Host Interface Placement with two VICs

[mdrudge]

Hi!

We are working on setting up a new vSphere environment, and would like to use Host Interface Policies to order the six vNICs (in redundancy pairs) and two vHBAs assigned to each host. We have it working on blades that have one VIC, but now we want it to work on blades with two VICs (B200 M4), and are not sure what we are doing wrong since we can't seem to control the numbering. I've created a Host Interface Policy with round robin, and vCON1 (VIC 1340) assigned the even numbered interfaces and vCON2 (VIC 1380) assigned the odd numbered interfaces in the service profile Host Interface Placement. I would then expect the following:

vCON1:
UCS vNIC0 - ESXi vmnic0
vNIC2 - vmnic1
vNIC4 - vmnic2

vCON2:
vNIC1 - vmnic3
vNIC3 - vmnic4
vNIC5 - vmnic5

Instead I get:

vCON1:
vNIC0 - vmnic0
vNIC2 - vmnic1
vNIC4 - vmnic5

vCON2:
vNIC1 - vmnic3
vNIC3 - vmnic4
vNIC5 - vmnic2

I then took the same service profile, and the same interface placement, and applied a new HIP with linear ordered, and then got the following:

vCON1:
vNIC0 - vmnic0
vNIC1 - vmnic5
vNIC2 - vmnic1
vNIC3 - vmnic3
vNIC4 - vmnic2
vNIC5 - vmnic4

vCON2: Nothing

With a linear ordered HIP, does it just assign interfaces until it reaches the limit the VIC can handle? And what am I missing here to get the ordering that we need? I got the impression from my research that we could control the numbering in UCS, but do we need to manually configure something in ESXi as well?  My head hurts with all this lol.  Thanks!

[Steven Tardy]

The docs on this are available:

  https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-2/b_UCSM_GUI_Configuration_Guide_2_2/configuring_server_related_policies.html#d79727e13282a1635

The vCon order really has nothing to do with the OS order but vNIC <> physical VIC order.

To complicate things the VIC 1300 series has 2x PCIe bus connections and the vNICs are distributed across the connections.

VMware ESXi changed a while back (maybe ESXi 5.5) and instead of enumerating PCI devices depth first started doing breadth first (or vice versa).

Cisco believes this to be a "VMware issue". VMware believes this to be a "Cisco issue". Neither side will resolve this leading to end customers suffering.

The only way I've found to resolve the OS ordering is by modifying esx.conf following (now removed, but that article exists for other languages) (the internet archive / way back machine may work once the AWS outage is resolved.) doc: https://kb.vmware.com/s/article/2019871

But VMware will claim this method is "unsupported".

Looking at your vNIC order (if that is even still accurate) I use a command like (paste all of this into the shell) (backup esx.conf first to be sure):

vi /etc/vmware/esx.conf
:%s/vmnic5/vmnicTEMP/g
:%s/vmnic1/vmnic5/g
:%s/vmnic2/vmnic1/g
:%s/vmnic4/vmnic2/g
:%s/vmnicTEMP/vmnic4/g
:wq
reboot

Cross fingers and say a blessing that UCS vNIC order matches ESXi vmnic order.

If that does not work, put the original esx.conf back then reboot and send the output from:

esxcfg-nics -l

Hope that helps.

[mdrudge]

Thanks for your reply, Steven.  This is very helpful information, even if it's a little discouraging.  I think with newer versions of ESXi (I'm setting up 7.0.2, and also looking for our current 6.5 prod cluster for another project), the host config file doesn't quite work like with older versions.  I found this document which changed NIC arrangement in ESXi, but not in the way I expected: https://www.vxav.fr/2020-04-22-change-vmnic-order-on-vsphere-host/.  We are continuing with our investigation, to see what will work best for what we want to do.  I'll keep you posted, but wanted to put in a reply today.  Thanks!