Troubleshooting vNIC/vHBA Placement

[aaronrombaut]

-- Cisco UCS B200 M4 blade with Cisco UCS VIC 1340 (Adapter 1 / UCSB-MLOM-40G-03) and Cisco UCS VIC 1380 (Adapter 2 / UCSB-VIC-M83-8P) adapters.
-- Firmware is UCS Manager 4.0(1b) but was an issue in 3.x firmware as well.
-- Hosts run VMware ESXi, 6.5.0, 10175896, but issue existed in lesser builds as well.

-- MAC Pool configured, size 256 and assigned sequential

-- vNIC Templates consist of eight vNICs:
   -- two for management, one on FI-A and the other on FI-B (eth0 and eth1)
   -- two for vMotion, one on FI-A and the other on FI-B (eth2 and eth3)
   -- four for various VLANs, split round-robin between FI-A and FI-B (eth4; eth5; eth6; eth7)
 
-- LAN Connectivity Policy is configured with all eight vNICs

-- vNIC/vHBA Placement Policies have been created, one for Linear and another for Round Robin
 
In the Service Profile Template, I have played around with the vNIC/vHBA Placement trying to only use the Cisco VIC 1380 adapter, which is Adapter 2. My placement policy is set to Linear so according to Cisco documentation, vCon 1 and vCon 2 should be for Adapter 1 and vCon 3 and vCon 4 should be for Adapter 2 in a blade with two adapters.

Under vCon 3, I have the following:

Name	Order
vHBA fc0	1
vHBA fc1	2
vNIC eth0	3
vNIC eth1	4
vNIC eth2	5
vNIC eth3	6
vNIC eth4	7
vNIC eth5	8
vNIC eth6	9
vNIC eth7	10

So...in UCSM, when I go to Equipment > Chassis > Chassis 1 > Servers > Server# > Adapters > Adpater 2 > HBAs or NICs, they are all on Adapter 2! This seems exactly like what I am looking for.

However...in VMWare, click on Configure Management Network > Network Adapters, my network adapters are out of order (by MAC Address) and spanned across Mezzanine Slot 1 and Chassis slot 6! How? Why?

Ok, so this calls for drastic measures. I consulted the VMware article https://kb.vmware.com/s/article/2091560 which describes how VMware ESXi determines the order in which names are assigned to devices. So, rather than manually reorder the file and reboot, I decided to "nuke it" by using the 'Reset System Configuration'. The system came back on-line with the same problem of half the Network Adapaters on the Chassis and the other half on the Mezzanine.

Can anyone smarter than me figure this out?

[mojafri]

Hi, 

Please see answer inline: 

in VMWare, click on Configure Management Network > Network Adapters, my network adapters are out of order (by MAC Address) and spanned across Mezzanine Slot 1 and Chassis slot 6! How? Why?

Its a naming convention from vmware - you will find more in the link which you shared  https://kb.vmware.com/s/article/2091560

You can't change PCI ordering using 'Reset System Configuration' - https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuv34051/?reffering_site=dumpcr 

Question:

1. Have you added extra vnic/vhba in a service-profile after OS install? 

2. Was ESXi installed after creating SP with linear placement? 

3. What is the actual/desire order?  

4. Have you moved the SP from one blade to another where OS was already installed on target blade with diff SP? 

Since you haven't mentioned, 3rd Gen VIC card have diff architecture, having 2 PCIe lanes internally (Call 'host port'). So you have to consider them assigning manually to each vnic's/vhba's  

As an example, let's say we have configured 4 vNICs and 2 vHBAs in the service profile. With the automatic placement, the vNICs and vHBAs will be split between the host-ports:

host-port 1 : vnic0, vnic1, vhba0

host-port 2: vnic2, vnic3, vhba1

This load balancing will be seen in the Actual Order column of the vNIC/vHBAs.

Here we can see an obvious split between the vNICs and vHBA in terms of their "Actual Order". Using this method, we can confirm that vnic0, vnic1 and vhba1 are on host-port 1, and vnic2, vnic3 and vhba2 on host-port 2. 

Unless you manually set the Admin Host Ports, load balancing may occur on every vnic add/delete which will change the host-port of existing or old vnic/vhba.

I found a blog which has a brief description: https://jeremywaldrop.wordpress.com/2016/02/15/ucs-vic-13401380-pci-placement/

 Please rate if you find it helpful.

Regards,

MJ

[aaronrombaut]

MJ,

Thank you for replying. Just wanted to follow-up and say that I ended up just doing a manual re-ordering on one of my hosts using localcli. I am going to create a host profile and see if I can "fix" the ordering on other hosts as was suggested in a VMware KB article. I had updated UCSM to 4.0(1b) and also the Host Firmware. This whole thing was more or less an experiment to see if I could actually get the "desired order" to be the "actual order". I also needed to clean-up and rebuild most of the Service Profiles so I built new Service Profile Templates. The only thing I preserved was the WWNN Pool since we boot from SAN and I did not want to have to reconfigure zoning.

A few answers to your questions, though. This particular host is a B200 M4 with a Cisco VIC 1340 MLOM and Cisco VIC 1380. The reason I tried a 'Reset System Configuration' was that I wanted the reset VMware OS to enumerate the devices again and see what order they would be in. The reset did re-order the vNIC and vHBA, but still in a wonky order.

-- No extra vNIC or vHBA were added, at least this was not the root of the issue. I played around with changing the number of vNIC from eight to one and then back to eight, but then had non-contiguous MAC addresses. At one point, I didn't even have MAC addresses near the same range as one another. They were all over the place. I altered my MAC Pool and created two pools, one for each Fabric Interconnect and reflecting such in the MAC Pool. I also set the assignment order from default to sequential.

-- ESXi was originally not installed after creating Service Profile with linear placement, but doing the 'Reset System Configuration' essentially did make this so.

-- The Service Profile was not moved from one blade to another and these blades have only ever had VMware ESXi installed.

-- The desired order is basically to have 1:1:1 with the lowest to highest vmnic match the vNIC and the MAC address increase from low to high as well. See below for an example. This is now my "reference host".

Cisco UCS Name	VMware ESXi Name	MAC Address
vNIC eth0	vmnic0	00:25:B5:11:A0:00
vNIC eth1	vmnic1	00:25:B5:11:B0:00
vNIC eth2	vmnic2	00:25:B5:11:A0:01
vNIC eth3	vmnic3	00:25:B5:11:B0:01
vNIC eth4	vmnic4	00:25:B5:11:A0:02
vNIC eth5	vmnic5	00:25:B5:11:B0:02
vNIC eth6	vmnic6	00:25:B5:11:A0:03
vNIC eth7	vmnic7	00:25:B5:11:B0:03