Nexus 5ks connected to 6513s

[Phillip Wilson]

I have a few questions on how the Nexus 5500s work and also what design would be the best in our situation.

Currently we have two 6513 Core switches  6513-1 and 6513-2 running HSRP and RSTP.  6513-1 is currently the Root Bridge, 6513-2 is setup as our secondary root.  We also have two 5548UPs setup with a peerlink between them.    Picture attached. "Current Setup.jpg"

     There is a 20gig port channel between the 6513s and also a 20gig port channel(peer link) between the 5548s. 5548-1 has a 10g fiber running back to 6513-1 and 5548-2 has a 10g fiber running back to 6513-2.  Currently now Spanning tree is blocking the link from 6513-2 going to 5548-2 which is what we expect.  We were working on moving some things to different racks the other day and moved switch 5548-2 to another rack and brought it back online without the peerlink fiber connected. We started to have issues and tried to plug the peer link back up, but still continued to have issues.  We started to troubleshoot and noticed that both 5548s were acting like the master of the VPC domain and was not letting traffic pass accross the Peer Link.  We then rebooted the 5548-2 with the thought that it would come back up as the secondary in the VPC domain.  When it finally finished booting back up it then caused a huge loop in our network accross both uplinks to the 6513s and the 5548s peer link which in turn took our network down.  Spanning tree did not work like it was suppose to and block the port going from 6513-2 to 5548-2 in time.

     My thought was this didn't seem like a good setup.  I went to the drawing board and decided we needed to have an uplink from both 5548-1 and 5548-2 going to 6513-1 setup as a port channel/VPC and also a uplink from 5548-1 and 5548-2 going to 6513-2 setup as a port channel/VPC.  Picture Attached. "NewSetup.jpg"  We are also planning on buying a "2K-C2224TP-1GE" to hang off the 5548s to use for ethernet.

     I'm pretty new to how the Nexus 5ks work and haven't done a lot of research on them. Would this not be a better setup since both switches will have an active link to the root bridge?  The links from the 5548s going to 6513-2 should always be in a block state until we loose both uplinks to 6513-1 or the entire switch itself correct?  Also how will the 5548s reactive if I was to loose the Peer link fiber?  Will they continue to work as normal?  When it looses this Peer Link does it suspend all the VPCs for the servers on the secondary switch in the Domain so the network doesn't get confused on which link to send traffic up since there isn't a peer link active or is this not the case?  Also with the peer link down will the 6513-1 know which 5548 to send the traffic to if there is only 1 device (not setup in a VPC) on one of the 5548s?  What are your recommendations/best practice on the setup for the 2k?

Thanks,

Phil

[matthew-frederick]

what are you using the nexus 5k's for? are they going to be just data center switches?

we have  2 nexus 5k's for our virtual cluster. the only difference is we are working with OLD 4507's with 1GB ports

we have two 4507's and two nexus 5k's

so we set it up kind of like this

all of the connections are layer 2 trunks only allowing the vlans that we need to route through our network. the vlans we do have to route - we create svi's on the sore and add the subnet to pur eigrp table.

i know this doesnt answer a lot of your questions - but i figured i would share how we've done it since it seems to be working thus far...

i think if our nexus's were to lose their peer link - it wouldnt susped the port channels, but they would not be able to route internally between eachother.

and if you set the links up between the 6's and 5's a different port channels - non of the should go into a blocking state

[Phillip Wilson]

The Nexus 5k's we are using because we are pretty much at our capacity on ten gig ports on our 6513s.  We have a plan later down the road to replace our 6513s with 7k's.    We are using the 5k's to attach our UCS Chassis via 10gig and also some ESX hosts via 10Gig.  We have ours setup like you do with a Layer 2 link and pruning out the vlans going to them.  We are only using them as L2 switches.  When we were moving one of the 5k's I did notice stuff in the log about VPC going in to a suspended mode when we removed the Peer Links.  This is why I asked the questions about how the 5k's worked.  I do think we need to have ours setup more like you have yours so we don't have to worry so much about spanning tree playing a part with the secondary switch.  As for my 2k I am just planning on making a VPC and attaching it to both 5k's. 

In your situation it is really nice even though you have a 1g uplink that you now have the capability to do 10gig now!  Which will come in very nice later on down the road.  Are you planning on eventually replacing your 4507s with the 5k's as your core?

[matthew-frederick]

on your port configurations connecting the 6's and the 5's - do you have you channel group configured as "on" or "active"?

[Phillip Wilson]

They are configured as "on"

[matthew-frederick]

Phillip Wilson wrote:
They are configured as "on" 

changing them to "active" on both sides will most likely solve your port suspending issues...

[matthew-frederick]

Phillip Wilson wrote:
In your situation it is really nice even though you have a 1g uplink that you now have the capability to do 10gig now!  Which will come in very nice later on down the road.  Are you planning on eventually replacing your 4507s with the 5k's as your core?

nope. our nexus swiches are solely for our virtual cluster - nothing more. we are actually going to replace the 4507's with 4500 x-series switches. then we will use 3750x stacked 24pt fiber switches running ipservices to do our distro and stacked 3750x to do ethernet. so our 4500x's will be doing nothing but core routing. 10GB links between all of them. it is going to be a HUGE step up for us.