SPINE links to IPN device in POD2 down

[danto]

Hi,
I have a setup for ACI Multipod, I've brought up POD1, and the IPN devices, but there is no link between the IPN device in POD 2 and the spines. I am using 40Gbps twinax cables,an 40Gbps SFP with fiber, either are not comming up.

Any ideas?

Thank you

Georgian

[Jason Williams]

Danto, 

Could you access both devices (IPN-2 and pod-2 spine) and provide the show interface eth x/y output of the interfaces used for the connectivity? 

Have you tried testing those same cables between the pod-1 spine and pod-1 IPN to isolate the issue from the cable? 

-JW

[danto]

Yes, for sure.

From IPN-2:

IPN-2# show interface e1/51
Ethernet1/51 is down (Link not connected)
admin state is up, Dedicated Interface
Hardware: 1000/10000/25000/40000/50000/100000 Ethernet, address: 006b.f137.343d (bia 006b.f137.3476)
Description: POD2-SPINE1-P23
MTU 9150 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
auto-duplex, 40 Gb/s, media type is 40G
Beacon is turned off
Auto-Negotiation is turned off, FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters never
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

BUC-IPN-1# show interface e1/51 transceiver details
Ethernet1/51
transceiver is present
type is QSFP-40G-CR4
name is CISCO-JPC
part number is P3007U103000-1
revision is A0
serial number is JPC20330172-B
nominal bitrate is 10300 MBit/sec per channel
Link length supported for copper is 3 m
cisco id is 13
cisco extended id number is 16
cisco product id is QSFP-H40G-CUXM

DOM is not supported

From SPINE switch:

(none)# show interface ethernet 1/23 transceiver detail

Ethernet1/23

    transceiver is present

    type is QSFP-40G-CR4

    name is CISCO-JPC

    part number is P3007U103000-1

    revision is A0

    serial number is JPC20330172-A

    nominal bitrate is 10300 MBit/sec per channel

    Link length supported for cable assembly is 3 m

    cisco id is -- 13

    cisco extended id number is 16

DOM is Disabled

(none)# show interface ethernet 1/23

Ethernet1/23 is down (notconnect)

admin state is up, Dedicated Interface

  Hardware: 1000/10000/100000/40000 Ethernet, address: 0000.0000.0000 (bia 006b.f13c.10b3)

  MTU 9150 bytes, BW 0 Kbit, DLY 1 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  Port mode is routed

  full-duplex, 40 Gb/s, media type is 40G

  FEC (forward-error-correction) : disable-fec

  Beacon is turned off

  Auto-Negotiation is turned on

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned off

  Rate mode is dedicated

  Switchport monitor is off

  EtherType is 0x8100

  EEE (efficient-ethernet) : n/a

  Last link flapped never

  Last clearing of "show interface" counters never

  0 interface resets

  30 seconds input rate 0 bits/sec, 0 packets/sec

  30 seconds output rate 0 bits/sec, 0 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 0 bps, 0 pps; output rate 0 bps, 0 pps

  L3 in Switched:

    ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes

  L3 out Switched:

    ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes

  RX

    0 unicast packets  0 multicast packets  0 broadcast packets

    0 input packets  0 bytes

    0 jumbo packets  0 storm suppression bytes

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    0 unicast packets  0 multicast packets  0 broadcast packets

    0 output packets  0 bytes

    0 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  0 output discard

    0 Tx pause

The other SPINE switch in POD 2 is connected to the same IPN device but with fiber SFP, but the same result. I've changed the port/sfp/cable, the ports between the spines and IPN device are not comming up.

[Rick1776]

Also what type of devices are you using as the IPN switches.

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-737855.html#_Toc463058878

[danto]

The IPN Devices in POD 1 and in POD2 are Nexus 93180YC-EX running NX-OS

[Rick1776]

I don't see the SFP QSFP-40G-CR4 as supported on the 93180YC-EX in NX-OS or ACI mode.

I only see the following....
QSFP-40G-SR4
QSFP-40G-CSR4
QSFP-40G-SR4-S
QSFP-40GE-LR4
QSFP-40G-LR4
QSFP-40G-LR4-S
QSFP-40G-ER4
WSP-Q40GLR4L

https://www.cisco.com/c/en/us/td/docs/interfaces_modules/transceiver_modules/compatibility/matrix/40GE_Tx_Matrix.html

[danto]

Hi,

The same equipments, and the same cables/SPF are used in POD1 and is working like a charm:

POD1-IPN-1# show lldp neighbors interface Eth1/51 detail
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID

Chassis id: 006b.f13c.18f3
Port id: Eth1/23
Local Port id: Eth1/51
Port Description: topology/pod-1/paths-211/pathep-[eth1/23]
System Name: POD1-SPINE-1
System Description: topology/pod-1/node-211
Time remaining: 111 seconds
System Capabilities: B, R
Enabled Capabilities: B, R
Management Address: 172.20.0.41
Management Address IPV6: not advertised
Vlan ID: not advertised

Total entries displayed: 1
POD1-IPN-1# show interface ethernet 1/51 transceiver details
Ethernet1/51
transceiver is present
type is QSFP-40G-CR4
name is CISCO-TYCO
part number is 2821248-5
revision is D
serial number is TED2035K1CY-A
nominal bitrate is 10300 MBit/sec per channel
Link length supported for copper is 3 m
cisco id is 13
cisco extended id number is 16
cisco product id is QSFP-H40G-CUXM

DOM is not supported

But in POD 1 I've configured the spine interface policies, but in POD2 I cannot do that, because in order to do that I have to adopt the devices in POD2, but in order to adopt the devices in POD2 i hava to have the links up, so it's a chicken and the egg problem

[Rick1776]

Hmm, you've got me stumped. Can you send a picture of your layout. Maybe that will shed some light onto the situation.

[danto]

Yes, sure

[Rick1776]

Are the connections form the IPN to the SPINE running OSPF, same MTU 9K, and DHCP-Relay, and PIM Bidir.
SO the first POD you brought up (Seed POD) is up and running no problem correct? You have active leafs in POD 2 as well?

[Jason Williams · Accepted solution]

Although these configurations are important (DHCP, PIM, MTU, etc), they have no impact on link layer connectivity; therefor, it is unneccessary to go down this path to troubleshoot the link being down. 

Could you try the following? 

- Plug both ends of the cable into the same IPN. Example: Connect Pod2-IPN Eth1/1 to Pod2-IPN Eth1/2

- Run the same test using the same cable on the spine. Connect the linecard to itself. 

In either case do you see the links come up? 

Repeat this test on multiple ports and multiple cables if needed. This would help further isolate the issue to a device or a cable. 

On the IPN side, you could try setting the link speed to either auto or 40000. 

-JW 

[Rick1776]

So do you think this could be just a layer 1 connectivity issue?

[danto]

Hmm, you might be right, I will test it tomorrow and let you know. I think it could be a problem with the IPN device as the links from the spines to the leafs in POD 2 are up.

[danto]

You'te right Jason,

Only interfaces Eth1/49 and Eth1/50 are working, interfaces Eth1/50-54 are not.

I'm gonna RMA the box.

Thanks for your help.

[Rick1776]

That’s so strange. Let us know if the new switch works.