Cisco C9500-48Y4c switch: Uplink interfaces are not connected

[jshayne]

Problem:  The uplink interface (40G/100G) on Cisco C9500-48Y4c switch is not connected. There is no light when connecting QSFP+ 40GBASE-LR4 with the uplink port on the switch.

Our setup: We set up a connection between a computer equipped with Intel E810-C NIC and the Cisco C9500-48Y4c switch (uplink port). The computer and the switch both use QSFP+ 40GBASE-LR4. We use a single mode optic cable (SMF) to connect the computer with the switch. 

What happens: The LED on the Intel E810-C NIC (the computer side) is green, but there is no light on the connected uplink port (the switch side). Using the show command on the switch, I saw that the uplink interface could recognize the QSFP port, but the status is still not connected.

#show interfaces HundredGigE 1/0/49
HundredGigE1/0/49 is down, line protocol is down (notconnect)
Hardware is Hundred Gigabit Ethernet, address is 8024.8f2f.451f (bia 8024.8f2f.451f)
MTU 1500 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback set
Keepalive set (10 sec)
Full-duplex, 40Gb/s, link type is force-up, media type is QSFP 40G LR4 SFP
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 1d20h, output 19:27:09, output hang never
Last clearing of "show interface" counters 2d00h
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
44025253 packets input, 3172175099 bytes, 0 no buffer
Received 41529995 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 40901940 multicast, 0 pause input
0 input packets with dribble condition detected
55007359 packets output, 3920340020 bytes, 0 underruns
Output 630868 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 11 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Interestingly, when we plugged the QSFP ports into two uplink ports on the switch, the LEDs on the uplink ports were green. The log on the switch said that the uplink interfaces are connected. In addition, when we plugged the QSFP ports into two different computers equipped with Intel E810-C NIC, the LEDs on the NIC were green, and we could ping them with each other.

We even used 100GBASE-LR4 QSFP28 and still didn't make it work.

We have been getting stuck on this issue for more than 2 weeks. What could be the reason for this issue, and how can we fix it?

[andresfr]

Hello J,

What software version are you running on that C9500-48Y4C switch?

Please check the current recommended release using the link below:

Recommended Releases for Catalyst 9200/9300/9400/9500/9600 and Catalyst 3650/3850 Platforms

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-switches/214814-recommended-releases-for-catalyst-9200-9.html

Also, it's always a good idea to check the Cisco Optics-to-Device Compatibility Matrix:

https://tmgmatrix.cisco.com

Please try disabling FEC on this 100GE interface when using a 40GE transceiver:

configure terminal
 interface HundredGigE1/0/49
  fec off
  shut
  no shut
  end
!

You could also test by connecting that interface to another on the same switch (loop test) using the same set of SFPs and after properly configuring those as layer 3 interfaces or mapping to an unused VLAN.

Check the following Field Notice:

https://www.cisco.com/c/en/us/support/docs/field-notices/721/fn72192.html

Please let me know if this helps.

[jshayne]

Hi andresfr,

Thanks for your help. My software version is 17.3.4. I tried to disable fec on the switch, but unfortunately, it still did not work.

As mentioned, I tried the loop test on the switch. The LEDs on the two uplink ports were green, and the log told me that the interfaces were successfully connected. Therefore, I think the QSFP ports are working well on the switch. This made me really curious

Please let me know if you need more input. 

[andresfr]

Hi J,

I would like to ask the following.

What is the exact model of those Intel NICs? E810-CQDA1?

When you were testing between the two computers, were you using 40G-LR4 or 100G-LR4 QSFPs?

Regards,

[jshayne]

Hello andresfr,

The Intel NICs we use are E810-CAM2. Please see the link below.
https://www.intel.com/content/www/us/en/products/sku/187410/intel-ethernet-controller-e810cam2/specifications.html

For the test between 2 computers, we used 40G-LR4. As mentioned, we can ping two computers with each other.

100G-LR4 QSFPs do not seem to work for the connection between two computers.

Best regards,

[andresfr]

Hey J,

The product brief for this Intel controller shows the following:

Supports multiple port speeds
with a single architecture:
100/50/25/10/1GbE  <<---- Note 40GbE is missing

Then, the controller datasheet states the following:

For example, the E810
does not support 2.5G/40G/200G/400G speeds, and therefore those options should not be
selected for this device.

Could you please double-check with Intel support that this is the case?

If the card does not support or comply with the 40GbE standard then I won't expect it to work when connected to the C9500-48Y4C Cisco Catalyst Switch.

Best regards,

[jshayne]

Thanks, andresfr. I also tried with 100G QSFP. There is still no connection, as can be seen below.

I am still wondering what the issue could be. What I am thinking is that maybe there is a mismatch in the setting between the NIC and the switch. The thing is that it works well while doing the loop test on the switch. What do you think about this possibility? And what can I do in this case?

show int HundredGigE1/0/50
HundredGigE1/0/50 is down, line protocol is down (notconnect)
Hardware is Hundred Gigabit Ethernet, address is 8024.8f2f.44b2 (bia 8024.8f2f.44b2)
MTU 1500 bytes, BW 100000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Gb/s, link type is force-up, media type is QSFP 100G LR4
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 2d07h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
Output 0 broadcasts (0 multicasts)
0 output errors, 0 collisions, 6 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Best regards,

[andresfr]

Hi J,

I'm suspecting that the E810 Controller may not be compatible with the Cisco QSFP-100G-LR4 transceiver (Third party) from the Intel point of view.

Is there any equivalent QSFP on the Intel side that you can use for testing?

Otherwise, I would advice checking with Intel support. Cisco TAC can also help you get some trace logs on the C9500-48Y4C for further debugging to see if those provide additional details that you can use to better identify the root cause of this issue.

[jshayne]

Hi @andresfr ,
While we are waiting for new QSFPs and cables to try out, I would be grateful if you could help me with this question. As the LED on the Intel NIC lights up, the uplink interface on the switch is in notconnect status. Interestingly, when the uplink interface was administratively down, the LED on the Intel NIC was immediately off. Could you please help explain this? 

[MHM Cisco World]

https://bst.cisco.com/bugsearch/bug/CSCvy49060

check this bug

[Leo Laohoo]

Cold reboot the 9500 and the link should come up.

[jshayne]

Hi all,

Thanks for the help!
Cold restart does not help. Unfortunately, we do not have the Intel-compatible QSFPs to test. I think the compatibility could be a reason. I will check it and keep you updated about this issue. If it still does not work, then it might be a bug.

Best regards,

[jshayne · Accepted solution]

The problem is solved. I have to change QSFP cable. The one I used exceeds the power supply that the Intel NIC can provide.