Service Profile Portability between B200 M3 and B200 M4

[chrisrost]

Hi folks,

i have a question regaring on the portability of a service profile, actually running un an B200 M3 with UCSB-MLOM-40G-01, moving to an B200-M4 with UCSB-MLOM-40G-03. There are no failures while changing the association, but when the SP is newly started there are some new "vmNICs" created because different MAC Address association.

Problem is, that vmNIC behavior in VMWare 5.1.0 is then looking at the MAC adresses and my vSwitches are not configured correctly then.

When  switch back the service profile to any B200 M4, the vNIC s are "genereated" with the right MAC adresses an vSwitch confg will be correct.

so there are two question:

1- is it generally possible to move a SP between different Hardware Types like B200M3 and B200M4

2- is there a workaround for my problem.

If, and i think so, no.1 is defined, does anybody have the offficial documentation from cisco, showing this behavior?

Thanks guys,

Night

CRost

[Walter Dey]

Hi Chris

I don't understand your problem at all; when the original and destination blade have the same amount  of I/O adaptors, and the number of vnics in the SP are the same, it should work.

Generally speaking, the PCI architecture of the blades are different; this could lead to rearrangement of vnics, and therefore a mac address change.

see eg

https://clnv.s3.amazonaws.com/2015/usa/pdf/BRKCOM-2016.pdf

PCI Reenumeration

With Cisco VIC card every vNIC presented as PCI device to the OS.

PCI reenumeration possible if
    -Most OS (Windows/Linux)
    -Number of VIC cards changes (for example from Palo to VIC1240/VIC1280)
    -Add/remove number of vNICS

Always take backup of network config
to capture mac address to IP address mapping before service profile migration

Walter.

[chrisrost]

Hi Walter,

seems, that i interpreted the failure wrong by myself. Let me describe it new:

When i associate  SP to one of the M3 Servers, any of the 17 vNICs are generated and VMWare will show them as vmNIC0-vmNIC16.

When i associate the SP to the M4 Server, there are 17 vNICs generated too, the MAC addresses are also ascending in the right order, but there are 4 vmNICs "missing", vmNIC9-vmNIC12 and there are four "new" vmNIC 17-20.

i will post 2 Screenshots of the vmNICs in M3 and M4.

Thanks fpr any idea.

Chris

[Walter Dey]

Hi Chris

So in the SP, your 17 vnics are visible and with proper IP addresses.

The problem are the vmnics on ESXi. Correct ?

Isn't this a ESXi problem ? I'm convinced, that a new install of ESXi would solve the issue.

btw. do you boot the OS over the SAN ?

Did you see https://www.virten.net/2013/11/vmnicvmhba-sequence-changed-in-esxi-5-5/ ?

Walter.

[chrisrost]

Hi Walter,

the customer will have the ability, to move the SP between M3 and M4 Servers for hardware redundancy.

We use ESXi 5.1.0 U3 (2323236).

I dont think, that it is a VMWare problem, i think, the SP is generating any kind of new PCI association and so, VMWare is skipping some and generating some new adapters.

The association of vmNIC-vSwitch, i think, is basing on the vmNIC ids.

When you look at the screenshot, you see, that the mac adresses are associated to the vNICs in the same order (only skipping 4 adapters).

M3             ->          M4

vmNIC0                  vmNIC0

...                             ....

vmNIC8                   vmNIC8

vmNIC9                   vmNIC13

...                             ....

vmNIC16                 vmNIC20

Thanks

Chris

[Walter Dey]

Chris

Check this KB article:

ESXi/ESX host loses network connectivity after adding new NICs or an upgrade (2019871)

The PCI ID to VMNIC numbering relationship is determined at boot time and is automatically entered into the esx.conf file located at /etc/vmware/ for persistence. The ESX/ESXi host first scans the seg number, then the bus number, the slot number, and finally the function number. This order ensures that ports on the same multi-port NIC are numbered sequentially.

In ESXi 5.5, 6.0, the ordering algorithm for VMNICs is changed.

How VMware ESXi 5.5 and 6.0 determine the order in which names are assigned to devices (2091560)

Walter.

[ssumichrast]

Hi Chris. Sorry for late response. 

Specifically the reason this happens on M4 with VIC 1340 is that the VIC 1340 has two "admin host ports". These admin host ports act almost like having two VIC cards installed.  If you search here on the discussion board I made a lengthy post about this a few more than back when it bit us during a production upgrade. 

TAC worked my case because I wasn't sure whwas going on (had a hunch but wanted it confirmed). Tac and I agreed the only solution is whenever you add vNICs to a blade with the 1340 VIC (or any VIC supporting multiple host ports) is to have ESXi renumerate the PCI bus. Walter posted a link later in this thread with instructions on how to do it. 

If you want to be able to just move profile association without reenumerating you can go into the vNIC/vHBA placement setting on the blade. Place all vHBA and vNICs on VCON1 and set their admin host port to 1. This will make your PCI enumeration work like it did on a M3 with 1240 VIC. You won't get the benefit of having two PCI connections to the motherboard but you also won't have to renumerate in ESXi. You could then, at a later maintenance window (in my case, waiting til all blades are converted to VIC 1340) set the vHBA/vNIV placement back to let the system decide and then do your ESXi reenumerating. 

Hole that helps. This is an annoying caveat that I wish was better published by Cisco. Its not specifically their fault but the change with admin host port really screwed me during a routine procedure a few months ago.