Configuring vPC to support highly-available hosts

[stuartkendrick]

Is there a design pattern document for implementing back-to-back vPC in a Data Center, in support of highly-available hosts?

i.e. a design in which the core / Layer 3 switches use vPC to speak LACP down to the top-of-rack access / Layer 2 switches. And the access / Layer 2 switches then use vPC to speak LACP down to servers equipped with dual NICs

Currently, I am ending up with STP blocking VLANs on an access switch / Layer 2 uplink, and I figure there is a way to do better. Does one end up with a single vPC domain crossing both the core switches and the access-layer switches? Or does one deploy separate vPC domains for each pair of access-layer switches providing LACP to hosts?

See diagram for detail

--sk

Stuart Kendrick

[MHM Cisco World]

No network topology attach ?

[stuartkendrick]

Ah, OK, I had missed Figure 9, thank you for pointing that out.  The text there clearly notes that vPC Domain IDs must be distinct.

In this scenario, vPC domain identifiers must be different on both layers because this
information is used as part of the LACP protocol.

And the text explains why

Ahh, I see, yes, I will try making the Aggregate Layer vPC number consistent between the (4) downlinks to the Access Layer (dc-200-esx and dc-201-esx) and then report back

--sk

[Sergiu.Daniluk]

Hi @stuartkendrick 

You mean like the picture below, where vPC domain 10 is the core/dist and vPC domain 20 is access?

Cheers,

Sergiu

[SystemAdministrator7710]

Yes, the diagram above fits my design, minus the (2) boxes to the left of the Host, at the access layer (we are only connecting Hosts at the access-layer)

From what document did you pull that diagram?  I would like to read it; perhaps it contains a discussion of the issue I am seeing

--sk

[Sergiu.Daniluk]

The one and only vPC design guide - https://www.cisco.com/c/login/index.html?referer=/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

Written for Nexus 7000, applicable for all nexus series.

Stay safe,

Sergiu

[SystemAdministrator7710]

Hi Sergiu,

I don't see that diagram in https://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

Your diagram seems like a cleaner version of Figure 31 and Figure 32 in the above vpc_best_practices_design_guide.pdf URL (without the complexity of (8) or (32) Aggregation to Access Layer links)

I notice that in the configuration example, both vPC Domain IDs are '1'

Although, in the Required Recommendation section on p. 49, the text suggests that one can use distinct vPC Domain IDs

Required Recommendation:
In a double-sided vPC topology, all interconnect links between the 2 vPC domains MUST belong to the same vPC.
All links form a unique vPC (on both sides of the 2 vPC domains). VPC id can be different across the 2 vPC
domains. However, vPC id must be the same across the 2 peer devices of the same domain

At the moment, I am using distinct vPC Domain IDs (vPC Domain 1 at the Aggregation Layer and vPC Domain 211 at the Access Layer)

Dang, I can see that my attempt to attach my diagram failed originally; see the PDF Data-Center-Back-to-Back-vPC.pdf for the illustration

--sk

[Sergiu.Daniluk]

The image from my post is Figure 9. Using a Different vPC Domain Identifier in Double-Sided vPC Topology  from the whitepaper.

vPC domains needs to be unique in the layer 2 domain because this id is used for different protocols such as STP or LACP.

Stay safe,

Sergiu

[SystemAdministrator7710]

OK, I am puzzled then -- I am using distinct vPC Domain IDs ... but I am still seeing STP blocking one of the uplinks

--sk

[MHM Cisco World]

Can i see stp for both, upper and lower peers.

NO NEED YOU ALREADY PAST IN YOUR TOPOLOGY.

[MHM Cisco World]

I Make Check, 
try the following 
config the both port in each Upper NSK  to be include the same PO, so no PO211 and PO 212 but one PO x.
config the lower NSK to be in same PO.
Using different Domain ID for both Upper & Lower Peers NSK.

[stuartkendrick]

I have made a physical change to the network, notably, making the uplinks off both switches 40G, rather than the previous case, in which one switch was uplinked at 10G.  I attach the updated diagram, Data-Center-Back-to-Back-vPC-Post.pdf, along with updated 'show span' output:  I see no change

And then, at the dc-x-rtr layer (the DC aggregate layer), changing vPC 201 --> vPC 200, i.e. joining the downlinks to both Access Layer switches into the same port-channel & vPC.  This fixed the STP blocking issue, at the cost of Suspending the downlinks to one of the switches -- I speculate because LACP system-identifier is different between the (2) Access Layer switches, and dc-x-rtr realize that the device on the other end is not in fact a single switch.  I attach a diagram illustrating this design, see Data-Center-Back-to-Back-vPC-Post-All-vPC-200.pdf

Puzzling isn't it?  If you have insights into what I am missing, do let me know.

--sk

[MHM Cisco World]

I don't get the issue here but I see suspend member of PO, if this case 
remove vPC from the PO and wait then add it again see if this way solve the issue.

[sp2720401]

Depending on how you configured your domain, you might have duplicate vPC system macs.

Check your vPC system mac (show vpc role).

If they are duplicate between the two pairs, you will see this issue show up in STP and LACP since the IDs will be the same for all 4 switches. You will need to manually set the system mac on one vPC pair or change up your domain settings. Only the vPC pair should share a common vPC system mac.

[SystemAdministrator7710]

I started from scratch, and I no longer see the suspended ports.  But STP continues to block the Port-Channel1 uplink from the left-hand switch

I see different 'vPC system mac', so I propose that this is good

I attach an updated diagram, with additional 'show' output

--sk