Nexus 5k VPC/Spanning Tree

[kirkulees007]

We've depolyed a pair of Nexus 5020 on our Network and attached them to our existing 6509's using VPC.  Our server team is reporting an issue where their servers become unreachable and only a reboot brings them back online.  Diagram of the architcture below.  This happens to random servers at random times, 1 or twice a week.

I was able to troublshoot the issue last week.  This is what I found:

1.  6509-1 did not have an ARP entry for the server.  (not able to ping server)

2.  6509-2 did have an ARP entry.  (not able to ping server).  If I cleared the entry it came back.

Layer 2 path to the problem server:

6509-2 to 6509-1

6509-1 to NX5020-1

NX5020-1 to NX5020-2

Both Nexus 5020's are VPC to both 6509's with no spanning tree blocked ports.

Shutting the switch port the Active server NIC is connected to had no effect (server did not fail to passive NIC).

Shutting server side Active port had no effect, server did  not fail to passive and seemed to hang after shutting the port.

The big issue I see is the 6509-2 spanning tree blocking to Nexus5020-2.  What is the recomended design for a scenerio like ours?  Has anyone else ran into this kind of issue? 

[Jerry Ye · Accepted solution]

There are couple problems with your design.

1. If the N2K is single homed to 1 N5K, traffic would be not optimal

2. I am assuming the vPC is configured between the N5Ks, and your diagram is not correct

To be honest, I would change to the design to the following:

HTH,

jerry

[kirkulees007]

We are running a VPC peer link between 5k's.  The 5k's support fewer 2k's when you duel home them so we left them single homed and will rely on server side redundancy for failover.

[Jerry Ye]

N5010/N5020 supports 12 FEX. So in your case, I don't see any real problem if you are have 12 FEX, 1 vPC peer-link, and 1 vPC uplink to your to each 6500.

The 12 FEX limitation is really the hardware port-channal limitation in the N5000, you can only have 16 HW port-channels. This limitation has been lifted in the N55xx.

How many FEX are you having? Do you have any other PO single homed to a N5Ks?

HTH,

jerry

[kirkulees007]

12 FEX per 5k in our case, I think it's 12 total across both when duel homed right?  It's purely a money issue.  We aren't taking full

advantage of VPC and I've be told by our VAR that we need to add another physical connection between the 5k's for VLAN's that

aren't participating fully in VPC.  In this case, that would be the active/passive server in the diagram.

I believe he is going to recomend a 2nd link for spanning tree between 5ks, and to put all non active/active VLAN's on that connection.  Which for us is going to wind up being pretty much all VLAN's

I haven't seen the full proposal, so I honestly can't comment intellgentily on it at this time.  I have seen whispers of this kind of configuration but nothing well documented on google/Cisco.  I'll update this when I have a better understanding of the proposal.

Thansk,

[Jerry Ye]

You can do 2nd link for non-vPC VLAN if that is what you want. I still think the design in my previous post will do what you want and provide redundancy (that design will not work for active-active servers.

HTH,

jerry

[kirkulees007]

I'm really confused with the active/active vs. active/passive client part of this.  Our VAR has pretty much said that our design will not work with an active/passive server.  Again, I could well be misunderstanding him.  I have a TAC case open on this as well, and TAC is telling me the design is fine as is, but we are running on a buggy version of code (5.0(2)N1(1)), and it could be potentially causing problems.  His "recomendation" was to move to the most recent version.  TAC can't recomend code updates, so recomendation is a loose term here.

[Jerry Ye]

TAC doesn't generally provide any design to customer, they do break-fix. Cisco Advanced Services would provide design and recommendation including software version.

To your question, here is the picture of active-standby server design:

And here is a picture of active-active server design:

There is a reason behind it which is Cisco do not double-layer vPC now.

Regards,

jerry

[kirkulees007]

That helps!  The original design was for active/active but it never materialized for various reasons.

Thanks a lot for the input, I'll update this when I get more back from the VAR, I am curious what you might think of their solution.

[DAVE GENTON]

Are you using hsrp ? Try using the vpc peer-gateway command and that should fix the issue provided you are setup correctly. I cannot see your config.

Dave

Sent from Cisco Technical Support iPhone App

[Jerry Ye]

The N5K is L2 (N5020), peer-gateway has no effect in this setup.

HTH,

jerry

[kirkulees007]

Thanks Jerry, you are spot on.  We'll be working on correcting our various issues over the next month or 2