cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2385
Views
10
Helpful
20
Replies
Highlighted
Beginner

SPINE links to IPN device in POD2 down

Hi,
I have a setup for ACI Multipod, I've brought up POD1, and the IPN devices, but there is no link between the IPN device in POD 2 and the spines. I am using 40Gbps twinax cables,an 40Gbps SFP with fiber, either are not comming up.

Any ideas?

 

Thank you

Georgian

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

Although these configurations are important (DHCP, PIM, MTU, etc), they have no impact on link layer connectivity; therefor, it is unneccessary to go down this path to troubleshoot the link being down. 

Could you try the following? 

- Plug both ends of the cable into the same IPN. Example: Connect Pod2-IPN Eth1/1 to Pod2-IPN Eth1/2

- Run the same test using the same cable on the spine. Connect the linecard to itself. 

In either case do you see the links come up? 

Repeat this test on multiple ports and multiple cables if needed. This would help further isolate the issue to a device or a cable. 

On the IPN side, you could try setting the link speed to either auto or 40000. 

-JW 

 

View solution in original post

20 REPLIES 20
Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

Danto, 

Could you access both devices (IPN-2 and pod-2 spine) and provide the show interface eth x/y output of the interfaces used for the connectivity? 

Have you tried testing those same cables between the pod-1 spine and pod-1 IPN to isolate the issue from the cable? 

-JW

Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

Yes, for sure.

From IPN-2:

IPN-2# show interface e1/51
Ethernet1/51 is down (Link not connected)
admin state is up, Dedicated Interface
Hardware: 1000/10000/25000/40000/50000/100000 Ethernet, address: 006b.f137.343d (bia 006b.f137.3476)
Description: POD2-SPINE1-P23
MTU 9150 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
auto-duplex, 40 Gb/s, media type is 40G
Beacon is turned off
Auto-Negotiation is turned off, FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters never
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

 

BUC-IPN-1# show interface e1/51 transceiver details
Ethernet1/51
transceiver is present
type is QSFP-40G-CR4
name is CISCO-JPC
part number is P3007U103000-1
revision is A0
serial number is JPC20330172-B
nominal bitrate is 10300 MBit/sec per channel
Link length supported for copper is 3 m
cisco id is 13
cisco extended id number is 16
cisco product id is QSFP-H40G-CUXM

DOM is not supported

 

From SPINE switch:

(none)# show interface ethernet 1/23 transceiver detail

 

Ethernet1/23

    transceiver is present

    type is QSFP-40G-CR4

    name is CISCO-JPC

    part number is P3007U103000-1

    revision is A0

 

    serial number is JPC20330172-A

    nominal bitrate is 10300 MBit/sec per channel

    Link length supported for cable assembly is 3 m

    cisco id is -- 13

    cisco extended id number is 16

 

DOM is Disabled

 

(none)# show interface ethernet 1/23

Ethernet1/23 is down (notconnect)

admin state is up, Dedicated Interface

  Hardware: 1000/10000/100000/40000 Ethernet, address: 0000.0000.0000 (bia 006b.f13c.10b3)

  MTU 9150 bytes, BW 0 Kbit, DLY 1 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  Port mode is routed

  full-duplex, 40 Gb/s, media type is 40G

  FEC (forward-error-correction) : disable-fec

  Beacon is turned off

  Auto-Negotiation is turned on

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned off

  Rate mode is dedicated

  Switchport monitor is off

  EtherType is 0x8100

  EEE (efficient-ethernet) : n/a

  Last link flapped never

  Last clearing of "show interface" counters never

  0 interface resets

  30 seconds input rate 0 bits/sec, 0 packets/sec

  30 seconds output rate 0 bits/sec, 0 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 0 bps, 0 pps; output rate 0 bps, 0 pps

  L3 in Switched:

    ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes

  L3 out Switched:

    ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes

  RX

    0 unicast packets  0 multicast packets  0 broadcast packets

    0 input packets  0 bytes

    0 jumbo packets  0 storm suppression bytes

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    0 unicast packets  0 multicast packets  0 broadcast packets

    0 output packets  0 bytes

    0 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  0 output discard

    0 Tx pause

 

 

The other SPINE switch in POD 2 is connected to the same IPN device but with fiber SFP, but the same result. I've changed the port/sfp/cable, the ports between the spines and IPN device are not comming up.

 

Highlighted
Contributor

Re: SPINE links to IPN device in POD2 down

Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

The IPN Devices in POD 1 and in POD2 are Nexus 93180YC-EX running NX-OS

Highlighted
Contributor

Re: SPINE links to IPN device in POD2 down

I don't see the SFP QSFP-40G-CR4 as supported on the 93180YC-EX in NX-OS or ACI mode.

I only see the following....
QSFP-40G-SR4
QSFP-40G-CSR4
QSFP-40G-SR4-S
QSFP-40GE-LR4
QSFP-40G-LR4
QSFP-40G-LR4-S
QSFP-40G-ER4
WSP-Q40GLR4L

https://www.cisco.com/c/en/us/td/docs/interfaces_modules/transceiver_modules/compatibility/matrix/40GE_Tx_Matrix.html
Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

Hi,

The same equipments, and the same cables/SPF are used in POD1 and is working like a charm:

 

 

POD1-IPN-1# show lldp neighbors interface Eth1/51 detail
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID

Chassis id: 006b.f13c.18f3
Port id: Eth1/23
Local Port id: Eth1/51
Port Description: topology/pod-1/paths-211/pathep-[eth1/23]
System Name: POD1-SPINE-1
System Description: topology/pod-1/node-211
Time remaining: 111 seconds
System Capabilities: B, R
Enabled Capabilities: B, R
Management Address: 172.20.0.41
Management Address IPV6: not advertised
Vlan ID: not advertised

Total entries displayed: 1
POD1-IPN-1# show interface ethernet 1/51 transceiver details
Ethernet1/51
transceiver is present
type is QSFP-40G-CR4
name is CISCO-TYCO
part number is 2821248-5
revision is D
serial number is TED2035K1CY-A
nominal bitrate is 10300 MBit/sec per channel
Link length supported for copper is 3 m
cisco id is 13
cisco extended id number is 16
cisco product id is QSFP-H40G-CUXM

DOM is not supported

 

But in POD 1 I've configured the spine interface policies, but in POD2 I cannot do that, because in order to do that I have to adopt the devices in POD2, but in order to adopt the devices in POD2 i hava to have the links up, so it's a chicken and the egg problem

Highlighted
Contributor

Re: SPINE links to IPN device in POD2 down

Hmm, you've got me stumped. Can you send a picture of your layout. Maybe that will shed some light onto the situation.
Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

Yes, suremultipod-problem.png

Highlighted
Contributor

Re: SPINE links to IPN device in POD2 down

Are the connections form the IPN to the SPINE running OSPF, same MTU 9K, and DHCP-Relay, and PIM Bidir.
SO the first POD you brought up (Seed POD) is up and running no problem correct? You have active leafs in POD 2 as well?
Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

Although these configurations are important (DHCP, PIM, MTU, etc), they have no impact on link layer connectivity; therefor, it is unneccessary to go down this path to troubleshoot the link being down. 

Could you try the following? 

- Plug both ends of the cable into the same IPN. Example: Connect Pod2-IPN Eth1/1 to Pod2-IPN Eth1/2

- Run the same test using the same cable on the spine. Connect the linecard to itself. 

In either case do you see the links come up? 

Repeat this test on multiple ports and multiple cables if needed. This would help further isolate the issue to a device or a cable. 

On the IPN side, you could try setting the link speed to either auto or 40000. 

-JW 

 

View solution in original post

Highlighted
Contributor

Re: SPINE links to IPN device in POD2 down

So do you think this could be just a layer 1 connectivity issue?
Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

Hmm, you might be right, I will test it tomorrow and let you know. I think it could be a problem with the IPN device as the links from the spines to the leafs in POD 2 are up.

 

Highlighted
Beginner

Re: SPINE links to IPN device in POD2 down

You'te right Jason,

Only interfaces Eth1/49 and Eth1/50 are working, interfaces Eth1/50-54 are not.

I'm gonna RMA the box.

Thanks for your help.

Highlighted
Contributor

Re: SPINE links to IPN device in POD2 down

That’s so strange. Let us know if the new switch works.