QSFP-40G-SR4 and Nexus 93180LC-EX

[Geir Sand-Strand]

Hi,

I have 2 Nexus 93180LC-EX switches, connected to each other via Eth1/11.  The config is just for test:
interface Ethernet1/11
  description Test QSFP
  switchport
  switchport mode trunk
  no shutdown
This is on both interfaces connected to each other.
What I need to do, is to test a 40G Netscout TAP for IDS.  But first I need to get link between the interfaces before I plugg in the TAP.  But I cannot get it connected.  I'm running the latest NXOS, 7.0(3)I7(2) on both switches.  And as I understand the QSFP-40G-SR4 is supported.
Anyone have any idea why I don't get link?

Eth1/11  Test_QSFP  notconnec trunk     auto    auto    QSFP-40G-SR4

Geir

[Reza Sharifi]

Hi,

So, you are connecting these 2 switches back-to-back using 40Gig interfaces? 

Are the interfaces in up and up mode?

Can you provide the output of "sh int e1/11" from both switches?

HTH

[Geir Sand-Strand]

Yes, back to back.

The interfaces are down.

Sh int eth1/11 from the first n9k:
Ethernet1/11 is down (Link not connected)
admin state is up, Dedicated Interface
  Hardware: 1000/10000/40000 Ethernet, address: 6cb2.ae7b.128a (bia 6cb2.ae7b.128a)
  Description: Test QSFP
  MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is trunk
  auto-duplex, auto-speed, media type is 40G
  Beacon is turned off
  Auto-Negotiation is turned on  FEC mode is Auto
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  EEE (efficient-ethernet) : n/a
  Last link flapped never
  Last clearing of "show interface" counters 00:42:50
  0 interface resets
  30 seconds input rate 0 bits/sec, 0 packets/sec
  30 seconds output rate 0 bits/sec, 0 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
  RX
    0 unicast packets  0 multicast packets  0 broadcast packets
    0 input packets  0 bytes
    0 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    0 unicast packets  0 multicast packets  0 broadcast packets
    0 output packets  0 bytes
    0 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause

sh int eth1/11 from the other n9k:

Ethernet1/11 is down (Link not connected)
admin state is up, Dedicated Interface
  Hardware: 1000/10000/40000 Ethernet, address: 6cb2.aeb4.eb02 (bia 6cb2.aeb4.eb02)
  Description: Test QSFP
  MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is trunk
  auto-duplex, auto-speed, media type is 40G
  Beacon is turned off
  Auto-Negotiation is turned on  FEC mode is Auto
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  EEE (efficient-ethernet) : n/a
  Last link flapped never
  Last clearing of "show interface" counters 00:42:48
  0 interface resets
  30 seconds input rate 0 bits/sec, 0 packets/sec
  30 seconds output rate 0 bits/sec, 0 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
  RX
    0 unicast packets  0 multicast packets  0 broadcast packets
    0 input packets  0 bytes
    0 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    0 unicast packets  0 multicast packets  0 broadcast packets
    0 output packets  0 bytes
    0 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause

I have tried different QSFP's and also different MPO-fibers.

[Geir Sand-Strand]

Ok, found the problem.  There are different MPO cables.  Method A and Method B.  On type A adapters, the keys are inverted to ensure that the fiber at position 1 is connected to position 1 in the MTP/MPO cable connector at the opposing end. On type B adapters, both keys are oriented facing up in order that both MTP/MPO cable connectors are mated “key up” wise. The fiber at position 1 is connected to position 12 in the MTP/MPO connector at the opposing end.

I have tried several method A MPO cables, but when I tried a method B cable I got link.

[Reza Sharifi]

Glad you got it resolved. You can also use a couple of BI-DI optics and use LC connectors to connect the switches together. Much cheaper and easier to use with no need for MTP/MPO.  See link for more info.

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729493.html

HTH

[DavidSenior82442]

The only downside to BIDI is you cannot break one 40G in four lanes and connect to four other ports. So you have to go 40G to 40G.

At the moment its far more common to go from one 40G switch to four 10G ports in either switches or servers.