Switch-independent load-balancing NIC teaming on server-side and MAC/ARP flapping on L2/L3 switches

dukenuk96 · ‎09-11-2014

Since active deployment of Windows Server 2012, our servers support team began to utilize new feature - switch-independent load-balancing NIC teaming. At first look it seems great - no additional network configuration is required and load balancing is performed by server itself by sending frames in round-robin or some hash algorithm out from different NICs (say two for simplicity) but with same MAC address. Theoretical bandwith is now grown up to 2Gbps (if we have two 1G NICs per server) against failover NIC teaming configuration, when one of two adapters is always down.

But how does this affect (if does) switching and routing performance of network equipment? From point of view of L2 switch - it has to rewrite its CAM table each time a server sends frame from different NIC. Isn't it expensive operation? Won't it affect switching in a bad way? We see in our logs that same server make switches to change mac-to-port associations several times per second.

Well, and how does it affect routing, if the switch to which server is connected is L3 switch an performs routing for the subnet server connected to? Will CEF operate well if ARP entry chages several times per second?

Thank you.

dukenuk96 · ‎10-09-2014

Since nobody answered here, we created service request and got the following answer (in short):

L2 MAC flapping between ports is very bad and you must avoid such configurations as much as possible. There is one possible variant that can be considered in your situation - use port-channel (either L2 or L3), in this configuration port-channel will be treted as single port and there won't be flapping.

Conversation example is here: https://ramazancan.wordpress.com/tag/best-practice/

fnature · ‎08-07-2019

I'm used to ESXi where you have load balancing based on virtual port id. This configuration is independent from the switch: it doesn't require extra configuration on the switch.

Similarly I think that independent mode and load balancing based on hyper-v port in windows server is the same: you do not run into the L2 MAC flapping.

fnature · ‎08-07-2019

ok, I digged further.

There should be no MAC flapping with independent mode at all.

There is a mechanism of MAC replacement for outbound frames as explained in https://docs.microsoft.com/en-us/windows-server/networking/technologies/nic-teaming/nic-teaming-mac-address-use-and-management

Following are lists that describe NIC Teaming MAC address replacement behavior, based on how the team is configured:

In Switch Independent mode with Address Hash distribution

All ARP and NS packets are sent on the primary team member

All traffic sent on NICs other than the primary team member are sent with the source MAC address modified to match the NIC on which they are sent

All traffic sent on the primary team member is sent with the original source MAC address (which may be the team's source MAC address)

In Switch Independent mode with Hyper-V Port distribution

Every vmSwitch port is affinitized to a team member

Every packet is sent on the team member to which the port is affinitized

No source MAC replacement is done

In Switch Independent mode with Dynamic distribution

Every vmSwitch port is affinitized to a team member

All ARP/NS packets are sent on the team member to which the port is affinitized

Packets sent on the team member that is the affinitized team member have no source MAC address replacement done

Packets sent on a team member other than the affinitized team member will have source MAC address replacement done

In Switch Dependent mode (all distributions)

No source MAC address replacement is performed

mmmiller60 · ‎05-11-2020

Adding some real world experience in this... 1. If this is a cluster you hyper "live migrate" to another host you will see 1 flap as the vm traffic is changed for the new host. 2. If you are seeing lots of flapping (this happened to me) and the switch is ignoring new Mac's because of it, you have a problem with the team on the windows side, you must delete the team and recreate it. Once I deleted the team and recreated it the problematic flapping was fixed, there was no other way to fix it, rebooting or removing NICs from the team did not help.