VIC 1457 In Stand Alone C220 M5 with only 1 NIC showing connected ESXi

[electricd7]

Hello-

We have 3 c220 M5 servers each with a VIC 1457 installed in it.  I have it cabled like the following:

PORT 1 - > Nexus N9k-9372PX  Switch 1, 10g Port 1 (configured as standard trunk port)

PORT 2 ->  Nexus N9k-9372PX  Switch 2, 10g Port 1 (configured as standard trunk port)

I have port channel turned off in CIMC and it rebuilt 4 vNICs and 4 HBAs on the card.  I can confirm that port 1 and 2 are connected and show up that way in physical interfaces in CIMC.  I have the uplink set to 0 for the eth0 and the uplink set to 1 for eth1.  When I go in the OS (ESXi 7.0) I see the MAC addresses for 2 ports which match eth0 and eth1 in CIMC, however only eth0 shows connected in OS.  Nothing I do seems to make the second NIC show connected in the OS.  What am I doing wrong?  I expected to see 4 NICs in the OS.  We are not using fiber channel at all in this installation and I had hoped to just use the VIC 1457 as 4 standalone 10/25gb ethernet NICs.

[Wes Austin]

What cables/optics are you using to connect the devices? If you swap the cables between Port 1 and 2, does the issue remain with the port or the cable? Do both N9K switch ports work using port 1? Does it work if you use port 3 or 4 instead of port 2 ?

[electricd7]

I haven't went through and swapped the cabling all around yet as we don't have physical access to the datacenter at the moment.  We are using 10g-lrm optics and 10m single-mode fiber.  I am pretty sure it will just stick with PORT1 as it is like that on all 3 servers.  Here are some screenshots of the CIMC config as well as the OS showing MAC addresses which match but no connection. 

[Wes Austin]

I would get supported optics installed and see if the problem persist. Maybe try a 10G-LR or LR-S if using Single Mode fiber is required? I do not see 10G-LRM as a supported option.

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/unified-computing-system-adapters/datasheet-c78-741130.html 

[electricd7]

Actually the SFP+ module shows "Supported" in the matrix returned from the controller see below:

We just moved the SFP from the second port on the card to the third (port-2 in image) on the card and the OS instantly saw that adapter as "Connected."  So my overall problem is solved as I now have 2 connected interfaces.  I still don't understand why this card doesn't present 4 vNICs to ESXi, however, as I would expect based on the vNICs shown in CIMC.  I also don't understand why it took me moving the cabling from the second port to the third to get a connection in the OS when the uplink for vNIC2 is "Uplink 1" (0 based, so 0,1,2,3)?

[Wes Austin]

While the GUI may recognize that LRM optic, I am not sure it was tested/validated so your results may vary.

Can you please confirm you have disabled the "port-channel" settings from the VIC in the CIMC GUI? It seems like this is not disabled which is contributing to the behavior.

[electricd7]

I get it.  I can confirm that it is at least "working" with the LRM and that the GUI thinks it supported.  This is a temporary setup as we are awaiting some new Nexus switches and 25gb twinax for these.  

As far as the port channel, I agree that is how it acts for sure, but it is disabled and has been rebooted (which then rebuilt the vNICs as 4 instead of the original 2).  See screenshot:

[Wes Austin]

You could try to flash the VIC firmware to the latest and see if this has any impact. Otherwise, are you able to bring the second port up connected to any other devices? What is the configuration on the upstream switchports?

Can you move cable back to the old non-working port and confirm it goes back down?

[electricd7]

Yes.  I can confirm that moving the cable back to the second port makes it go back down.  The upstream switchports are very simple and look like the following:

interface Ethernet 1/11
description esxi01 vmnic1
switchport mode trunk
switchport trunk allowed vlan 140,141,127

interface Ethernet 1/11
description esxi01 vmnic2
switchport mode trunk
switchport trunk allowed vlan 140,141,127

[Wes Austin]

From ESXi you mention it only sees two ports, instead of four? I would expect ESXi in Port-channel mode to only see two interfaces, but four in non Port-channel. Did you potentially make these changes after the OS was installed?

Anyway to re-install ESXi as a test?

[electricd7]

Yes.  I made the changes after the initial install.  I haven't went back to try to re-install after putting it to non port-channel mode.  Perhaps that is the issue.