cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
890
Views
10
Helpful
4
Replies

MAC addresses moving from FI to FI

goapps.us
Level 1
Level 1

We had a consultant setup our Hyper-V SCVMM infrastructure and he apparently did not set out HV switch correctly. Long story short, the consultant setup the NIC teaming of our Tenant vSwitch to Switch Independent and with the LB mode of Dynamic

Based on case with TAC, this mode is not recommended/supported in a UCS system. The SE recommended to change the LB algorithm to Hyper-V port. 

After making the changes to our HV switch last night, I was disappointed to observe the same behaviors. If you look at the table below, you can see the MAC addresses "moving" from across the FI fabric. 

Does anyone have any thoughts? 

Thanks 

-A(nxos)# show platform fwm info mac 0051.5600.02D4 8
mac vlan 1.8 mac 0051.5600.02d4: vlan 1.8
mac vlan 1.8 mac 0051.5600.02d4: learned-on vif141 age 30 ref_map = 'vlan if'
mac vlan 1.8 mac 0051.5600.02d4: nohit_count 0 hw_programmed 1 mac_clone 0
mac vlan 1.8 mac 0051.5600.02d4: old_if_index 'null'
mac vlan 1.8 mac 0051.5600.02d4: pss_flags 0
mac vlan 1.8 mac 0051.5600.02d4 cfg attrs - not-cli-cfg not-static movable no-drop no-regmac non-netstack-learnt not-secure not-src-drop
mac vlan 1.8 mac 0051.5600.02d4: mcec_flags 0x1, mac_info_flags 0, rem_if 0, sync_count 0 rcv_count 0
mac vlan 1.8 mac 0051.5600.02d4: CDCE Address b:10:0:bc:a:22


Mac history (Last 35 operations):
Total operations: 3418:
  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Remote mcec down del local 1 (69)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:40 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:40 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:40 2016

  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016
FIC-A-B(nxos)# show platform fwm info mac 0051.5600.02D4 8
mac vlan 1.8: could not find mac address
THIS IS A DELETED MAC ADDRESS
mac vlan 1.8 mac 0051.5600.02d4: vlan 1.8
mac vlan 1.8 mac 0051.5600.02d4: learned-on vif141 age 10 ref_map = 'vlan if'
mac vlan 1.8 mac 0051.5600.02d4: nohit_count 0 hw_programmed 1 mac_clone 0
mac vlan 1.8 mac 0051.5600.02d4: old_if_index 'vif141'
mac vlan 1.8 mac 0051.5600.02d4: pss_flags 0
mac vlan 1.8 mac 0051.5600.02d4 cfg attrs - not-cli-cfg not-static movable no-drop no-regmac non-netstack-learnt not-secure not-src-drop
mac vlan 1.8 mac 0051.5600.02d4: mcec_flags 0x1, mac_info_flags 0, rem_if 0, sync_count 0 rcv_count 0
mac vlan 1.8 mac 0051.5600.02d4: CDCE Address b:10:0:bc:a:1a


Mac history (Last 35 operations):
Total operations: 3413:
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:39 2016

  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:40 2016

  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:35:41 2016

  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:41 2016

  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016

  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Thu Mar 24 03:42:48 2016
1 Accepted Solution

Accepted Solutions

Walter Dey
VIP Alumni
VIP Alumni

Hi

Switch independent is correct ! Hyper-V port I think is ok as well.

To me, this mac flapping looks like "IP hash" which is quite common as a problem.

How is your service profile ? 2 vnic's, one attached to fabric A, resp. B (no hardware failover)

Walter.

View solution in original post

4 Replies 4

Walter Dey
VIP Alumni
VIP Alumni

Hi

Switch independent is correct ! Hyper-V port I think is ok as well.

To me, this mac flapping looks like "IP hash" which is quite common as a problem.

How is your service profile ? 2 vnic's, one attached to fabric A, resp. B (no hardware failover)

Walter.

Hello Walter, thank for your reply. 

After posting yesterday, I found that one of my hosts had not been changed to the recommended LB method. I updated the host last night and performed more tests and I found that the MAC flapping is no longer an issue! My guess is that the MS virtual switch really did not implement the LB method change until all hosts were set to the same. See table below 

As for you questions, yes one nic bound to A and one nic bound to B w/o failover. (see https://supportforums.cisco.com/discussion/12937886/service-profile-2012r2-hv-cluster) 

Thanks 

results of tests after the LB method change

FIC-A-A(nxos)# show platform fwm info mac 0051.56f0.01f0 8
mac vlan 1.8: could not find mac address
THIS IS A DELETED MAC ADDRESS
mac vlan 1.8 mac 0051.56f0.01f0: vlan 1.8
mac vlan 1.8 mac 0051.56f0.01f0: learned-on vif183 age 20 ref_map = 'vlan if'
mac vlan 1.8 mac 0051.56f0.01f0: nohit_count 0 hw_programmed 1 mac_clone 0
mac vlan 1.8 mac 0051.56f0.01f0: old_if_index 'vif183'
mac vlan 1.8 mac 0051.56f0.01f0: pss_flags 0
mac vlan 1.8 mac 0051.56f0.01f0 cfg attrs - not-cli-cfg not-static movable no-drop no-regmac non-netstack-learnt not-secure not-src-drop 
mac vlan 1.8 mac 0051.56f0.01f0: mcec_flags 0x1, mac_info_flags 0, rem_if 0, sync_count 0 rcv_count 0
mac vlan 1.8 mac 0051.56f0.01f0: CDCE Address b:10:0:bc:a:5e
Mac history (Last 35 operations):
Total operations: 944:
  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 07:38:58 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 07:38:58 2016
  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 08:17:35 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 08:17:35 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 08:17:35 2016
  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
--More--        
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Thu Feb 25 10:41:52 2016
  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 14:25:58 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 14:25:58 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 14:25:58 2016
  Operation: Mac update due by interface (8)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 17:13:39 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 17:13:39 2016
  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 19:01:26 2016
  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 19:01:26 2016
  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 19:01:26 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Sun Feb 28 19:01:26 2016
  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 08:24:49 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 08:24:49 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 08:24:49 2016
  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
--More--   Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
  Operation: Mac subjected to HW delete in new learn (50)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b7 hint: 0)
        at Mon Feb 29 09:05:03 2016
trace mgr stats:
    deleted_mac_tree_full: 0
    deleted_mac_tree_not_found: 0
    deleted_mac_tree_add_err: 0
    deleted_mac_tree_del_err: 0
    mac_trace_new_err: 0
    update_history_err: 4
FIC-A-B(nxos)# show platform fwm info mac 0051/[J.56f0.01f0 8
mac vlan 1.8 mac 0051.56f0.01f0: vlan 1.8
mac vlan 1.8 mac 0051.56f0.01f0: learned-on vif141 age 0 ref_map = 'vlan if'
mac vlan 1.8 mac 0051.56f0.01f0: nohit_count 0 hw_programmed 1 mac_clone 0
mac vlan 1.8 mac 0051.56f0.01f0: old_if_index 'null'
mac vlan 1.8 mac 0051.56f0.01f0: pss_flags 0
mac vlan 1.8 mac 0051.56f0.01f0 cfg attrs - not-cli-cfg not-static movable no-drop no-regmac non-netstack-learnt not-secure not-src-drop 
mac vlan 1.8 mac 0051.56f0.01f0: mcec_flags 0x1, mac_info_flags 0, rem_if 0, sync_count 0 rcv_count 0
mac vlan 1.8 mac 0051.56f0.01f0: CDCE Address b:10:0:bc:a:1a
Mac history (Last 35 operations):
Total operations: 990:
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
--More--         (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Thu Mar 10 21:03:27 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
--More--         at Thu Mar 10 21:03:27 2016
  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Fri Mar 11 01:53:57 2016
  Operation: Mac local aged (12)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Fri Mar 11 01:53:57 2016
  Operation: Mac delete (10)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Fri Mar 11 01:53:57 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d0000b0 hint: 0)
        at Fri Mar 11 01:53:57 2016
  Operation: Mac create (9)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Fri Mar 25 10:39:04 2016
  Operation: Mac learned from hw (40)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Fri Mar 25 10:39:04 2016
  Operation: Mac hardware update successful (24)
        (flags: Loc  (0x1) mac_info_flags (0x0) if: 0x1d00008d hint: 0)
        at Fri Mar 25 10:39:04 2016
trace mgr stats:
    deleted_mac_tree_full: 0
    deleted_mac_tree_not_found: 0
    deleted_mac_tree_add_err: 0
    deleted_mac_tree_del_err: 0
    mac_trace_new_err: 0
    update_history_err: 6

Well Cisco community, it turns out that my issue is still happening. Since my last post, I have updated my firmware to 2.2.6e and also upgrade the eNic drivers to 3.4.0.14

I've been working with TAC and the engineer asked me to include Microsoft Support. WOW what a clown show!!! Their last recommendation would be change the load balancing mode from "Hyper-V Port" to "Address Hash". Based on my limited testing, this does work, but based on this article it is not suited for heavier workloads. Anyone out there with experience with this?

Questions :  

  • would using the 1000V mitigate this behavior?
  • At the moment, the 2 vNics (VM-A and VM-B) of my tenant vSwitch are associated to a specific FI and I have NOT enabled failover. To mitigate the behavior, I could change the design and associate both vNics to a primary FI and enable failover. This would pin the VM's to one FI and traffic would survive a link FI failure? 

I'm completely stuck! I invested 100K in hardware and I cannot boot a VM without affecting other VM's on the host. This is completely ridiculous!!! Does anyone have any recommendations? 

Best regards 

 

Walter, I wanted to report that we were finally able to address this issue. 

See https://support.microsoft.com/en-us/kb/3137691 

Thanks