Problem Description
Cisco Catalyst 4500 Series experiences High CPU when VMware servers are connected with redundant links.
High CPU on Catalyst 4500 due to mac address flapping
When a Ethernet frame enters the catalyst 4500 Series switch, the switch will learn the source mac address if that mac address is not there on the mac address table. In order to learn the mac address, the first frame will be sent to switch CPU. Once the mac address is learned, the remaining frames from that source mac address will be forwarded in the hardware.
If a host has multiple links to the Catalyst 4500 series switch, but these links on the switch are not bundled using port channel and the host load balances the traffic using both links with the same source mac address, the switch has to learn the mac address continuously. If the host does a per packet load balancing, the switch will learn the mac address for every frame it receives alternatively from these two ports. This will result in sending all the packets to the switch CPU. You will see High CPU on the switch because of the frames sent to the Switch CPU.
How to verify the High CPU due to mac address flapping?
The below URL will help you to identify whether the High CPU is due to mac address flapping
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#host
Also you can enable the mac address table mac-move notification feature to easily identify the mac address flapping.
Switch#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#mac address-table notification mac-move
Switch#show mac address-table notification mac-move
MAC Move Notification: enabled
Switch# show log
%C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:XX:XX:XX in vlan 28 is flapping between port Gi 2/3 and port Gi 3/3
VMware Networking
You can connect a single virtual switch to multiple physical Ethernet adapters using the VMware Infrastructure feature called NIC teaming.
VMware’s NIC teaming allows a single virtual switch to connect to multiple physical Ethernet adapters. VMware provides the following load balancing options on the vSwitch.
Route based on the originating virtual switch port ID
Route based on source MAC hash
Route based on IP hash
Route based on the originating virtual switch port ID is the default load balancing method. When you use this setting, traffic from a given virtual Ethernet adapter is consistently sent to the same physical adapter. If the Virtual servers were using one Virtual Ethernet interface to connect to the network, you will not see the mac address flapping on the Catalyst switches. But the problem arises when the Virtual servers teaming multiple Virtual Ethernet adapters. This scenario is depicted as below
Resolution
Option 1:
Select “Route based on IP hash” on the vSwitch. Configure Port channel on the Catalyst switches to bundle the links to the physical adapters.
Option 2:
Select “Route based on source MAC hash” as the load balancing method on the vSwitch. Do not configure port channel on the Cisco Catalyst Switches.
Option 3:
If “Route based on the originating virtual switch port ID” is used on the vSwitch, do not team the Virtual Ethernet Adapters as Active/Active on the virtual servers. You may use as Active/Standby on the Virtual Servers. Do not configure port channel on the Cisco Catalyst Switches.
References:
http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048