Vlan Flapping between port-channels

ccietrainee1 · ‎03-11-2012

Dear Experts,

Need your help in fixing one major issue that i am currently facing. We have a pair Cisco 6509 switch in which 2 * 48 Port 1G line cards and 1 * 16 Port 10G line Card, FWSM and Sup 720 are installed.

We have Cisco UCS and HP Blade servers.Cisco UCS servers are connected to Cisco 6509 switch using Fabric Interconnect, and HP Servers are directly connected to core switches.

Recently the team made many changes in the network. Upgraded the IOS in Cisco 6509 switch, Configured Port profiling , MAC Pinning , HBA Cards to UCS / Nexus 1000V Infrastructure. After this change they lost the connectivity to UCS and HP Serers. Every tower is checking at their end.

The Network Team has reverted back the core switch with old IOS , but still the problem persisit.

I could only see the following error log in the core switch. There are two port-channels one between core 1 and core 2. The other is between core switch and FWSM module. Can anybody explain the root cause of the following error message.

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 8 is flapping between port Po308 and port Po256

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 78 is flapping between port Po308 and port Po256

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 100 is flapping between port Po256 and port Po308

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 6 is flapping between port Po256 and port Po308

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 75 is flapping between port Po256 and port Po308

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 100 is flapping between port Po256 and port Po308

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 75 is flapping between port Po256 and port Po308

Mar 11 11:43:02 IST: %MAC_MOVE-SP-4-NOTIF: Host 001f.6c67.6f80 in vlan 35 is flapping between port Po256 and port Po308

Thanks & Regards,

B.Senthilkumar

Kevin Dorrell · ‎03-11-2012

How are those port channels configured? I am guessing the mode is "on" rather than LACP. The problem with "on" mode is that if the switch has a different opinion to the servers about which links are bundled, then you get precisely this kind of flapping. The host thinks it is sending to two links of a single port-channel, while the switch thinks the two links are from different port channels.

This sort of thing happens if there is a cabling error, or, as in your case, there is some misconfigured network switch virtualisation in the server. In the worst case, when you have another redundant path, it can inadvertently cause a loop and subsequent meltdown.

For my own part, I always insist the server guys use LACP to form their port-channels. That way I can be sure there is no misunderstanding. If the connectivity is wrong, the switch treats the links as individual rather than as a port-channel.

Kevin Dorrell

Luxembourg

ccietrainee1 · ‎03-11-2012

Thanks Kevin for your qucik response.

Port-channel 256 is configured between two cisco 6509 and it is configured in "desirable" mode, which means they use PAGP. I will get the details of Port-channel 308 soon. It should be in FWSM module.

One more thing i noticed is that only one mac address is causing the issue, i will try to find out how the teaming or bonding is done at the server end.

Thanks again . I will come up with more data that may be helpful for further troubleshooting.