Hello everyone.
We have a pair of nexus 3164q in vpc domain(900), also we have a pair of 3064 in other vpc domain (200).
3164 is the 2ru 64х 40G ethernet switch with two rows of ports 32 + 32.
3064 is the 1ru 48x10G+4x40G ethernet switch.
We have connected two 40G ports of each 3064 to the one 3164 (switch A) and last 2x40G to second 3164 (switch B).
In 3164 we are taking 1 port from top row and 1 port from bottom row.
The problem is that any bottom port from switch A (3164) won't work with any 3064 40G uplink port of any 3064 switch.
Switch B works fine with any uplink 40G port of any 3064 switch.
Any port of switch A (3164) are fine with any port of switch B (3164).
The problem ports detect a transceiver, and have normal signal (rx\tx), they are admin up, but link just not connected.
If we plug this MPO cable to upper port - the link is coming up.
We can do breakout (not a beautiful workaround) from two sides and then all 4 10G pairs of 40G port are up.
We have tried to replace QSFP - no matters.
If anybody have any ideas - please, go on, cause we are frustrated.
3064:
interface Ethernet1/50
description Eth1/34.cnx1.spb
lacp rate fast
switchport mode trunk
channel-group 201 mode active
x10p48dnx4.spb# sh run int po 201
interface port-channel201
description Po901.cnx1-2.spb
spanning-tree port type network
vpc 201
inherit port-profile VPC_CORE_PO
x10p48dnx4.spb# sh int status | i 1/50
Eth1/50 Eth1/34.cnx1.spb notconnec trunk full 40G QSFP-40G-SR4
x10p48dnx4.spb# sh int e1/50 transceiver
Ethernet1/50
transceiver is present
type is QSFP-40G-SR4
name is OEM
part number is QSFP-AOC-3
revision is 01
serial number is QEB6740011
nominal bitrate is 10300 MBit/sec per channel
Link length supported for 50/125um OM2 fiber is 3 m
cisco id is 13
cisco extended id number is 16
x10p48dnx4.spb# sh int e1/50
Ethernet1/50 is down (Link not connected)
admin state is up, Dedicated Interface
Belongs to Po201
Hardware: 40000 Ethernet, address: 6c20.56e6.feb9 (bia 6c20.56e6.feb9)
Description: Eth1/34.cnx1.spb
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
auto-duplex, 40 Gb/s, media type is 40G
Beacon is turned off
Auto-Negotiation is turned on, FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters 1w2d
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
x10p48dnx4.spb# sh int e1/50 transceiver details | i 'Rx|Tx'
Tx Power 0.15 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.18 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.06 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.60 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.45 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.37 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.15 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.55 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
3164:
interface Ethernet1/34
lacp rate fast
switchport
switchport mode trunk
channel-group 901 mode active
no shutdown
x40p64cnx1.spb# sh run int po 901
!Command: show running-config interface port-channel901
!Time: Fri Mar 15 12:05:43 2019
version 7.0(3)I4
interface port-channel901
switchport
spanning-tree port type network
vpc 901
switchport trunk allowed vlan 1-4094
inherit port-profile VPC_CORE_PO
x40p64cnx1.spb# sh int e1/34 transceiver
Ethernet1/34
transceiver is present
type is QSFP-40G-SR4
name is OEM
part number is QSFP-AOC-3
revision is 01
serial number is QEB6740011
nominal bitrate is 10300 MBit/sec per channel
Link length supported for 50/125um OM2 fiber is 3 m
cisco id is 13
cisco extended id number is 16
x40p64cnx1.spb# sh int e1/34
Ethernet1/34 is down (Link not connected)
admin state is up, Dedicated Interface
Belongs to Po901
Hardware: 100/1000/10000/40000 Ethernet, address: 00f6.63d0.6414 (bia 00f6.63d0.6414)
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
auto-duplex, auto-speed, media type is 40G
Beacon is turned off
Auto-Negotiation is turned on, FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters never
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard 0 Rx pause TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
x40p64cnx1.spb# sh interface e1/34 transceiver details | i 'Rx|Tx'
Tx Power 0.18 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power -0.15 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.08 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power -0.04 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.48 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power 0.44 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
Tx Power 0.18 dBm 3.39 dBm -11.61 dBm 2.39 dBm -7.61 dBm
Rx Power -0.06 dBm 3.39 dBm -13.56 dBm 2.39 dBm -9.50 dBm
