vPC Best Practices for Nexus 7000 and 5000

[Sandeep Singh]

Introduction

The biggest limitation to a classic port-channel communication is that port-channel operates only between two devices. To overcome this limitation, NX-OS has a technology called virtual Port Channel (vPC). A pair of switches acting as a vPC peer endpoint look like a single logical entity to port-channel attached devices; the two devices that act as the logical port-channel endpoint are actually two separate devices. This setup has the benefits of hardware redundancy combined with the benefits offered by port-channel, e.g loop management.

vPC Overview

A virtual port channel (vPC) allows links that are physically connected to two different Cisco Nexus 7000 Series devices to appear as a single port channel by a third device. The third device can be a switch, server, or any other networking device that supports port channels. You can use only Layer 2 port channels in the vPC. A vPC domain is associated to a single VDC, so all vPC interfaces belonging to a given vPC domain must be defined in the same VDC. You must have a separate vPC peer-link and peer-keepalive link infrastructure for each VDC deployed. Consolidating a vPC pair (two vPC peer devices of the same domain) in two VDCs of the same physical device is not supported. The vPC peer link must use 10-Gigabit Ethernet ports for both ends of the link or the link will not form.

vPC Control Plane Recommendations

• For the Peer Link, it is good to use two 10GbE ports on separate line cards for resilent connectivity.
• Also, for the Peer Link, make sure that the ports are in dedicated mode (not Shared).
• If using the Management Interface for Peer-Keepalive traffic, the management interface should be connected to a Layer 2 management switch, not back to back.
• The Peer Keepalive traffic should be over a separate keepalive link and not over the Peer Link.

vPC Member Connectivity Recommendations

• Make sure to dual attach all devices into the vPC Domain.
• It is good to use LACP for the Port Channels of vPC Member ports.

vPC Spanning Tree Recommendations

• Check that the Spanning Tree parameters match across vPC peers in accordance with vPC requirements.
• Do not use Bridge Assurance feature on vPC Member ports.
• Use Bridge Assurance feature on vPC Peer Link.
• Ensure all switches in the Layer 2 domain are using Rapid-PVST to avoid slow STP convergence time.
• Configure Portfast on Edge ports, to avoid slow STP convergence.

vPC Layer 3 Connectivity Recommendations

• Use separate layer 3 links to connect routers to the vPC Domain.
• Enable Layer 3 routing between vPC peers over a separate layer 3 link.
• Do not enable routing over vPC for vPC members. In the DCI environment, use HSRP tracking to ensure Layer 3 failover between Data Center.
• Do not use Link tracking for HSRP, use extended Object tracking.

Related Information

• Nexus - vPC Recommendations
• Nexus - Troubleshooting vPC Consistency Issue
• Nexus 5000 vPC Design Best Practices
• Best Practices for Virtual Port Channels (vPC) on Cisco Nexus 7000 Series Switches

[Andras Dosztal]

Nice work!

These should also go to the related information:

• Nexus 5000 vPC Design Best Practices on DocWiki
• Design and Configuration Guide: Best Practices for Virtual Port Channels (vPC) on Cisco Nexus 7000 Series Switches

[Sandeep Singh]

Thanks Andras.

I have added the links.

[Shahab Khan]

Hi,

What is the best practise related to STP while running vPC? Since all the ports in vPC should be in forwarding state, then should we enable STP or not? My topology is having Cisco Nexus 7K which is connected with Nexus 5K & which in turn connected to Nexus 2K. My gateway is configured on 7K. Almost all my servers are single-homed.

Regards,

Shahab Khan

[Steve Fuller]

Hi Shahab,

It is recommended that STP always remain enabled. Even though vPC removes loops from the topology, STP is seen as a backup in the event that vPC fails or is misconfigured. The following is from page 58 of the Cisco Design and Configuration Guide: Best Practices for Virtual Port Channels (vPC):

Strong Recommendations:

• Spanning Tree Protocol must remain enabled for all VLAN (even if all access devices are vPC-attached to vPC domain). Do not disable spanning-tree protocol!

There are additional recommendations and guidelines within the aforementioned document and it would certainly be a good investment in your time to review if you're new to vPC.

Regards

[ronbuchalski]

I have a question regarding vPC and multiple pairs of Nexus 5000s connected to a pair of Nexus 7000s.  This is for the example where each pair of Nexus 5000s have four uplinks, two connected to each Nexus 7000, and both sides of this four-link connection are configured as a vPC, so it looks like one port-channel connection between the N5K pair and the N7K pair, consisting of four links.

Now, I want to add a second pair of Nexus 5000s, and a third pair of Nexus 5Ks, and so on, with each pair connected with four links, similar to the first pair.

Does each Nexus 5000 pair need to have a unique vPC Domain ID, or can each Nexus 5000 pair use the same vPC Domain ID, as long as it is different than the vPC Domain ID for the Nexus 7000s?

After reading the documents, I would say that each Nexus 5000s pair needs to have a unique vPC Domain ID.  But I've seen several Data Centers that have multiple pairs of Nexus 5000s connected to a pair of Nexus 7000s.  The Nexus 7000 vPC Domain ID is 101.  Each Nexus 5000 pair is configured with the same vPC Domain ID of 1.

The Nexus 5000s have been in service for quite awhile, and vPC appears to work fine with this configuration, so I'm not sure if it's just a Best Practice to give each Nexus 5000 pair a unique vPC Domain ID, or if it's a POTENTIAL source of problems if they all use the same vPC Domain ID.

Thank you,

Ron Buchalski

[eduardopozo56]

Best practice and Cisco recommendation is to use a unique VPC domain id on contiguous layer 2 domains. VPC Domain id is used to form the VPC system mac address, so if you use the same domain id within the same Layer2 domain you might encounter spanning tree issues in some topologies.

"The vPC peer devices use the vPC domain ID that you configure to automatically assign a unique vPC system MAC address. Each vPC domain has a unique MAC address that is used as a unique identifier for the specific vPC-related operations, although the devices use the vPC system MAC addresses only for link-scope operations, such as LACP. We recommend that you create each vPC domain within the contiguous Layer 2 network with a unique domain ID. You can also configure a specific MAC address for the vPC domain, rather than having the Cisco NX-OS software assign the address."

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/interfaces/configuration/guide/if_vPC.html#wp1803808

[pheavens85]

Hi Ron

Regarding this statement:

"If using the Management Interface for Peer-Keepalive traffic, the management interface should be connected to a Layer 2 management switch, not back to back"

What is the disadvantage of a directly connected peer-keepalive on the Management port?

[Alcides Miguel]

I'm creating an infrastructure with N7K and my question is regarding L3, what I'm trying to do is to use the N7K as the core of the network and SVI should be created on these switches? and shoul I create one SVI per switch and assign different IP for each?

and the access switches(3560) should connect to N7K as vtp client?

Best regards,
AM  

[ronbuchalski]

You should really post this question in another topic, as your question does not relate to vPC Best Practices.

But yes, you would create SVIs on each Nexus 7K, and use a FHRP (i.e., HSRP) for redundancy.

Cisco does not advise using VTP in data center environments.  It is no longer a best practice (was it ever?)

Bringing this back to vPC, you would configure each of your access switches with a port-channel uplink, and connect one link to each Nexus 7K.  Then, you would configure the port-channel on the Nexus side across the two switches using vPC.

Note that the vPC connection can only be Layer 2, not Layer 3.  If you want to establish Layer 3 connectivity between your access switches and the Nexus switches, you cannot use vPC.  You would configure physical Layer 3 interfaces on the Nexus side and have point-to-point Layer 3 connections to your access switches, and your routing protocol will use equal cost multipath (ECMP) to balance the load across the paths.

-rb

[jefferybjohnson]

With NX-OS 7.2 out this is kind of a game changer in regards to dynamic routing over vPC's

[ahmad82pkn]

Hi Jeff, i am also interested in 7.2 dynamic routing explanation.

What i understand is now you can build routing neighbor ship across both VPC Peers for 2x router connected in L2 with nexus.

although not finding any design proof for that :)

[noel.boulene]

Hi,

In this page you recommend to :

• Enable Layer 3 routing between vPC peers over a separate layer 3 link

In this page : http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_L3_w_vpc_5500platform.html#wp1014719

It is not recommended for Nexus 5K :

In some circumstances, you might consider having a separate link between the two vPC switches, either to carry non-vPC VLAN traffic or to form Layer 3 routing protocol peering. While this design is supported on the Cisco Nexus 7000 Series switch, it does not work on the Cisco Nexus 5000 Series switch. With the Cisco Nexus 5000 Series switch, we recommend that you use a vPC peer link for Layer 3 peering to carry both vPC and non-vPC VLAN traffic.

So what is the recommended design for 5K ?

Thanks