cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4612
Views
0
Helpful
1
Replies

N3K QSFP Link not connected issue

InsaneDevice
Level 1
Level 1

Hello everyone. 

We have a pair of nexus 3164q in vpc domain(900), also we have a pair of 3064 in other vpc domain (200).
3164 is the 2ru 64х 40G ethernet switch with two rows of ports 32 + 32.
3064 is the 1ru  48x10G+4x40G ethernet switch. 

We have connected two 40G ports of each 3064 to the one 3164 (switch A) and last 2x40G to second 3164 (switch B). 
In 3164 we are taking 1 port from top row and 1 port from bottom row.

The problem is that any bottom port from switch A (3164) won't work with any 3064 40G uplink port of any 3064 switch.
Switch B works fine with any uplink 40G port of any 3064 switch.
Any port of switch A (3164) are fine with any port of switch B (3164).

The problem ports detect a transceiver, and have normal signal (rx\tx), they are admin up, but link just not connected.
If we plug this MPO cable to upper port - the link is coming up.

We can do breakout (not a beautiful workaround) from two sides and then all 4 10G pairs of 40G port are up.
We have tried to replace QSFP - no matters.

 

If anybody have any ideas - please, go on, cause we are frustrated.

 

3064:

interface Ethernet1/50
description Eth1/34.cnx1.spb
lacp rate fast
switchport mode trunk
channel-group 201 mode active

x10p48dnx4.spb# sh run int po 201
interface port-channel201
description Po901.cnx1-2.spb
spanning-tree port type network
vpc 201
inherit port-profile VPC_CORE_PO

x10p48dnx4.spb# sh int status | i 1/50
Eth1/50 Eth1/34.cnx1.spb notconnec trunk full 40G QSFP-40G-SR4

x10p48dnx4.spb# sh int e1/50 transceiver
Ethernet1/50
transceiver is present
type is QSFP-40G-SR4
name is OEM
part number is QSFP-AOC-3
revision is 01
serial number is QEB6740011
nominal bitrate is 10300 MBit/sec per channel
Link length supported for 50/125um OM2 fiber is 3 m
cisco id is 13
cisco extended id number is 16

x10p48dnx4.spb# sh int e1/50
Ethernet1/50 is down (Link not connected)
admin state is up, Dedicated Interface
Belongs to Po201
Hardware: 40000 Ethernet, address: 6c20.56e6.feb9 (bia 6c20.56e6.feb9)
Description: Eth1/34.cnx1.spb
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
auto-duplex, 40 Gb/s, media type is 40G
Beacon is turned off
Auto-Negotiation is turned on, FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters 1w2d
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

x10p48dnx4.spb# sh int e1/50 transceiver details | i 'Rx|Tx'
Tx Power 0.15 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.18 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.06 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.60 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.45 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.37 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.15 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.55 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm

3164:

interface Ethernet1/34
lacp rate fast
switchport
switchport mode trunk
channel-group 901 mode active
no shutdown
x40p64cnx1.spb# sh run int po 901
!Command: show running-config interface port-channel901
!Time: Fri Mar 15 12:05:43 2019
version 7.0(3)I4
interface port-channel901
switchport
spanning-tree port type network
vpc 901
switchport trunk allowed vlan 1-4094
inherit port-profile VPC_CORE_PO
x40p64cnx1.spb# sh int e1/34 transceiver
Ethernet1/34
transceiver is present
type is QSFP-40G-SR4
name is OEM
part number is QSFP-AOC-3
revision is 01
serial number is QEB6740011
nominal bitrate is 10300 MBit/sec per channel
Link length supported for 50/125um OM2 fiber is 3 m
cisco id is 13
cisco extended id number is 16

x40p64cnx1.spb# sh int e1/34
Ethernet1/34 is down (Link not connected)
admin state is up, Dedicated Interface
Belongs to Po901
Hardware: 100/1000/10000/40000 Ethernet, address: 00f6.63d0.6414 (bia 00f6.63d0.6414)
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
auto-duplex, auto-speed, media type is 40G
Beacon is turned off
Auto-Negotiation is turned on, FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters never
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard 0 Rx pause TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

x40p64cnx1.spb# sh interface e1/34 transceiver details | i 'Rx|Tx'
Tx Power 0.18 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power -0.15 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.08 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power -0.04 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.48 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.44 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.18 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power -0.06 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
 

1 Accepted Solution

Accepted Solutions

InsaneDevice
Level 1
Level 1

After some investigation and tests it is clear that this issue about hardware / software bug, because breakout mode works fine and all diag tests are in pass state. 

 

For any other who are faced this issue we can recommend:

1. check if the link comes up with different switch

2. check with other ports of the switch.

3. check in breakout mode.

4. run diag tests.

 

If all of this is fine, may be you are also dealing with hardware bug and need to replace the switch.

View solution in original post

1 Reply 1

InsaneDevice
Level 1
Level 1

After some investigation and tests it is clear that this issue about hardware / software bug, because breakout mode works fine and all diag tests are in pass state. 

 

For any other who are faced this issue we can recommend:

1. check if the link comes up with different switch

2. check with other ports of the switch.

3. check in breakout mode.

4. run diag tests.

 

If all of this is fine, may be you are also dealing with hardware bug and need to replace the switch.

Review Cisco Networking for a $25 gift card