UCS unexpected paths for vNICs.

[joeharb]

We have deployed a new UCS and ACI fabric and have noticed some odd behaviors in respect the presence of the MAC address that is assigned to FIA showing up on the port-channel designated for FIB.  The connection is simple, VPC between 2 leafs to each FI, we have verified by shutting down the interfaces that all links are mapped correctly, we have evacuated the fabric and traffic fails appropriately but the mac address of the actual vnic still reflects the A Pool.

Example:

MAC address POOL For A side:  00:25:B5:11:AC:00-00:25:B5:11:AC:FF

MAC address POOL For B side:  00:25:B5:11:BC:00-00:25:B5:11:BC:FF

From the APIC:

00:25:B5:11:AA:00 --- learned 101 102 vpc DC_OPS_CORE_FI_B vlan-45 not-applicable ---
00:25:B5:11:AA:03 --- learned 101 102 vpc DC_OPS_CORE_FI_B vlan-45 not-applicable ---
00:25:B5:11:AA:04 --- learned 101 102 vpc DC_OPS_CORE_FI_B vlan-45 not-applicable ---
00:25:B5:11:AA:06 --- learned 101 102 vpc DC_OPS_CORE_FI_B vlan-45 not-applicable ---
00:25:B5:11:AA:07 --- learned 101 102 vpc DC_OPS_CORE_FI_B vlan-45 not-applicable ---
00:25:B5:11:BA:02 --- learned 101 102 vpc DC_OPS_CORE_FI_B vlan-45 not-applicable ---

I would not expect to see any MAC's from the A pool through that VPC as it is for the port-channel to FIB.

Any suggestions on where to investigate further?

Thanks,

Joe

[Robert Burns]

ACI can't force MACs to be learned on an interface, that would be the fault of the sending device.  Do you have UCS vNICs configured for Fabric Failover?  If yes, this would explain things.  UCS vNICs enabled with FF will fail the MACs from FI-A over to the FI-B path (if/when all FI-A uplinks fail or are shutdown).   Confirm this first, then we'll look further. 

You can check the MACs on your FI's CLI:

connect nxos b

show mac address-table | inc 00:25:B5:11:AA

Robert

[joeharb]

Failover is not enabled:

The MAC address viewed from the FI directly matches what we are seeing on within ACI.

paducs01-B(nx-os)# sh mac add
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 35 0050.56a9.1c43 dynamic 0 F F Veth905
* 35 0050.56a9.4bed dynamic 0 F F Veth905
* 35 0050.56a9.8cea dynamic 0 F F Veth905
* 35 0050.56a9.f872 dynamic 0 F F Veth905
* 45 0025.b511.aa00 dynamic 0 F F Veth901
* 45 0025.b511.aa03 dynamic 0 F F Veth909
* 45 0025.b511.aa04 dynamic 0 F F Veth917
* 45 0025.b511.aa06 dynamic 0 F F Veth933
* 45 0025.b511.aa07 dynamic 0 F F Veth941

We did a Fabric Evac last night and all NICS on the ESXi associated with the A side showed down, but I never saw a change in the MAC address from the host itself.

Thanks,

Joe

[Robert Burns]

0025.b5 MACs are UCS-assigned MACs

0050:56 MACs are VMware assigned

So based on your FI output, ACI is doing nothing wrong.  UCS FIs are hosting those MACs on FI-B which is why you're seeing them traverse the B-side VPC.  With the FI's running in End-host mode, they will never learn MACs from the upstream.  They'll always learn MACs from connected servers/IOMs directly.

I'm a little rusty on UCSM debugging, so you might be better served posting this in the UCS forum, but I'll take a shot. 

For one of your affected servers, issue this from the UCSM CLI:
show service-profile circuit server x/y

connect nxos b
show pinning server-interfaces

[joeharb]

45 0025.b511.aa00 dynamic 0 F F Veth901

Veth901 is nic 2 on the ESXi server and reflects that B Pool within the UCS GUI, see attach screenshot but show up on Fabric B with an A Pool MAC address.

I don't mind moving this to a different forum if you think that would be valuable.

I appreciate all you help

[Robert Burns]

Need the two outputs request above to assist further.

Robert

[joeharb]

Please see attached output.

Thanks,

Joe

[Robert Burns]

Going to move this thread to the UCS forum.  Hopefully someone has seen this before.

Robert

[Wes Austin]

It almost sounds related to this VMware KB if its consistently vmk0 mac address showing on both fabric. This is because VMware inherits the mac address from UCS vNIC for vmk0 by default. Possibly when you failover, vmk0 is being moved/learned on FI-B up to the network on B side ACI fabric?

https://kb.vmware.com/s/article/1031111

You can test by taking a host and recreate management vmk0 from the KVM to get a unique mac address and check the behavior.

If a duplicate MAC address is confirmed, the MAC address must be changed by deleting and recreating the vmkernel interface.

To delete a vmknic from a port group, use this command:

# esxcfg-vmknic -d -p pgName

or

# esxcfg-vmknic -d pgName

To add a vmknic to a port group, run the command:

# esxcfg-vmknic -a -i DHCP -p pgName

or

# esxcfg-vmknic -a -i x.x.x.x -n 255.255.255.0 pgName