Solved: Re: UCS 240 M4 with VIC 1387 works with Nexus 3064, but not with 3850 - Page 2

Elliot Dierksen · ‎01-02-2023

I have an absolutely maddening issue. I have a Nexus 3064 that has been the switch for my storage and vMotion networks for several years, and it has worked great. I am trying to transition some of the 40G ports to a 3850 so I don't have to run the Nexus all the time. Its fans are much louder than the 3850's. On the 3850, I get CDP in both directions when connected to either switch, but IP doesn't work. I can't ping, and no ARP entry for the UCS. When connected to the Nexus, everything works properly. It is a C240-M4SX with a 1387 MLOM card, and it is running ESXi 7.0. The port configurations are simple, and I have stared at them quite a bit, but still no joy. Here is the Nexus config fragment.

interface Ethernet1/51
  description vm-ucs1 - vmnic3
  switchport access vlan 252
  spanning-tree link-type point-to-point

Here is the 3850 config fragment.

interface FortyGigabitEthernet1/1/2
 description vm-ucs1 - NFS
 switchport access vlan 252
 switchport mode access
 switchport nonegotiate
 spanning-tree portfast

I see input and output packets on the 3850 side and the ESXi side, but IP fails. I am stumped at the moment. It is the exact same cable and QSFP. I moved it from the Nexus to the 3850. I plug in a different C240 M4 running TrueNAS (FreeBSD) with a Chelsio T580 card and it works fine.

Elliot Dierksen · ‎01-10-2023

I got an answer in the server networking forum.https://community.cisco.com/t5/server-networking/vic-1387-40g-connection-to-3850-switch/td-p/4752420

The problem is that the VIC 1387 always sends an 802.1p tag. If it didn't get a VLAN tag from the OS, it leaves the VLAN part of the 802.1p tag blank. The Nexus doesn't mind this, but the 3850 is not happy with that. When I force the VIC to use a VLAN tag, it all works fine.