UCS mini with Catalyst 3850 stack northbound

[tomcriley]

I have a UCS mini with 2 uplinks from each 6324 connected to a 3850 stack.  I have cabled it so 6324-A is connected to port ten1/0/1 and ten2/0/1 (po1 on the 3850's), and 6324-B is connected to port ten1/0/2 and ten2/0/2 (po2 on the 3850's).  I have configured the corresponding port channels on the uplink ports in UCSM.  The only reachable IPs on the chassis (that traverse the uplinks) currently are the CIMC in-band mgmt IPs of the blades, so these are what I am testing connectivity with.  The FI's are in end-host mode.  When the ports are enabled and the port channels negotiate, the 3850 throws MAC flapping errors stating that the CIMC MACs are flapping between po1 and po2.  Should I be cabling and/or configuring this a different way?

[James Clifford]

Hi. What mode do you have the port-channel set to on the 3850's? I would make sure it's set to LACP and not PAGP.


I haven't checked if these are available on 3850 but they are on 3750 but try something like this.....
Interface ten1/1
channel-group 1 mode active
channel-protocol LACP 



Thanks 
James 

[christopher.lister]

I have not done this with 3850's specifically, but I have used 3850's extensively and I use a LOT of UCS, with both 1st, 2nd, & 3rd gen fabric interconnects.  I use all of these with Nexus switches upstream which is ideal but I also work for a large company that can afford to drop millions on this stuff.

That said, I am confident the issue you are running into is that, because a switch stack is managed as a single switch, despite having multiple physical switches, only one management plane is active and functional at a time.  This means you have one ARP table for the entire stack, rather than having individual ARP tables as you would have with two switches upstream.  Bear in mind that you have two completely separate paths configured from your blade vNICs all the way to your fabric interconnects.  The UCS mini card in IOM slot A should always connect to one and only one upstream switch, while all of the connections coming from the UCS mini in IOM slot B should be connected to one and only one upstream switch.  Of course, it follows that these two switches must be unique.

The MAC flapping you are seeing is expected behavior in your configuration because the upstream switch stack is seeing the same device report in from two different UCS minis, each configured with two different MAC addresses.  So it is reported first as MAC #1 then it is switched to MAC #2 by the other FI.  If you separate the stack into two switches, the problem will resolve.  The downside of this configuration is that you cannot use an upstream VPC (virtual port channel) unless you have a Nexus pair. 

Hope this helps...

[christopher.lister]

I have not done tis with 3850's specifically, but I have used 3850's extensively and I use a LOT of UCS, with both 1st, 2nd, & 3rd gen fabric interconnects.  I use all of these with Nexus switches upstream which is ideal but I also work for a large company that can afford to drop millions on this stuff.

That said, I am confident the issue you are running into is that, because a switch stack is managed as a single switch, despite having multiple physical switches, only one management plane is active and functional at a time.  This means you have one ARP table for the entire stack, rather than having individual ARP tables as you would have with two switches upstream.  Bearing in mind that you have two completely separate paths configured from your blade vNICs all the way to your fabric interconnects.  The UCS mini card in IOM slot A should always connect to one and only one upstream switch, while all of the connections coming from the UCS mini in IOM slot B should be connected to one and only one upstream switch.  Of course, it follows that these two switches must be unique.

The MAC flapping you are seeing is expected behavior in your configuration because the upstream switch stack is seeing the same device report in from two different UCS minis, each configured with two different MAC addresses.  So it is reported first as MAC #1 then it is switched to MAC #2 by the other FI.  If you separate the stack into two switches, the problem will resolve.  The downside of this configuration is that you cannot use an upstream VPC (virtual port channel) unless you have a Nexus pair. 

Hope this helps...