Cisco UCS and 6509 in VSS Configuration

[Jonathan Gill]

I am hoping experts on this forum can help me understand how the following actions led to a situation where no traffic was being forwarded on a pair of 6509's configured in a VSS topology.   I have a UCS domain, 6296UP interconnects, cross connected to a pair of 6509's.   Fabric A has two 10Gb interfaces in a port channel, same for fabric B (separate port channel).  I have been tracking a problem where I was observing a lot of MAC flapping between these port channels.  When troubleshooting the issue, I had our network engineering team remove the interfaces from the port channel on their end.   This was done prior to me removing the northbound interfaces on the FIs from the corresponding port channel on each fabric.  When our network team unbundled the interfaces, all traffic stopped forwarding on the VSS pair and I was unable to get back to the management interface to break down the port channel.   Our network team disabled the interfaces and traffic began forwarding again (this strikes me as particularly odd).   I am running the FI's in EHM, so I am not sure how this action could have stopped all traffic from forwarding on the 6500.   Does anyone have any ideas as to how this could of happened?   The UCS domain in question has 8 blades, all of which were powered down.  I am doing in-band management, the CIMC's are accessible via the northbound interfaces on the FIs.   I was observing the MAC addresses of the CIMC flapping between the port channels upstream.   UCS was not reporting anything via syslog that indicated a problem during the MAC flap.  There is something going on and I lack the expertise on Catalyst in a VSS topology with UCS.     

[ssumichrast]

VSS will look the same as a VPC to the UCS -- there's nothing much there. It just makes the two switches look like one to the downstream devices, that's all. 

Your network team did build the port channels correctly? (FI-A has a link on each 6509, and those are in one port channel and the FI-B links are in their own? I realize you said separate just making sure they had the right links in the right channels.)

Did they set the port channels to mode active so LACP fires up properly?

There is honestly nothing the UCS will know about the 6509 port channels than it would from a nexus VPC -- as long as the channels are built right on both sides the bundles should come up. The flapping makes me wonder if the port channels on the 6509 are messed up as far as membership. 

[Jonathan Gill]

While it may look the same, are there things that happen when connected to a pair of switches configured for VPC vs. VSS?   Along the lines of LACP vs. PAgP?  Similar constructs, looks the same, but if memory serves me there are a few additional things that happen with PAgP.  

They did build the port channel correct, it was a text book LACP bundle (mode active).  A dedicated bundle for each fabric interconnect.  Its been communicated that the down fabrics are each their own data plane, the cross connect between FIs is for heart beating and management... perhaps we could call it a stretched control plane in some respects.  

I though the same when I saw the flapping, perhaps they bundled both fabrics, but that is not the case.   I have verified this configuration. 

What I don't know, which is a big problem, is why the 6500's stopped forwarding traffic when the links were unbundled on their side.  I wanted to minimize variables by going down to single links and test EHM by having multiple unbundled links to see if there is flapping between the individual interfaces (for example, flapping between two interfaces on the same FI).   I realize I should have broke the port channel on my side (UCS) before our network team broke it on their side... however, I wouldn't have expected the large scale impact.   Could a loop have been created, a multicast storm?   I need to understand why this happened before I "play" with the uplink configuration more.  

[Walter Dey]

Which OS do you have, and which kind of algorithm do you use for NIC teaming / multipathing ?

see also

http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Data_Center/VMDC/1-1/vmdcDg11/design.html

[Jonathan Gill]

I have the servers powered down, and still experience MAC flapping and had the issue where the port channel was broken on 6509 which caused what was likely a loop and stopped all traffic from being forwarded on the 6500s until the links to UCS were brought down.