- Introduction
- Requirements
- Recommendations
- Role and system priority
- Configuration parameters which must match across vPC peers
- Disabling vPC peer link compatibility check
- Related information
Introduction
This document describes general recommendations to optimize convergence and ensure stable and deterministic operation of a Virtual Port Channel topology.
Requirements
There are no specific requirements for this document.
Recommendations
vPC peer-link recommendations
The vPC peer link is a standard 802.1Q layer 2 trunk which potentially carries all VLANs. It also carries regular control plane traffic such as HSRP hellos, STP BPDUs, etc. The vPC Peer Link also carries the critical CFS (Cisco Fabric Services) traffic, which is marked with a CoS value of 6.
The vPC Peer Link is considered critical for vPC operation; therefore steps should be taken to ensure that this link operates in the most reliable and resilient way possible. As such, the general Peer Link recommendations are as follows:
- The vPC Peer Link should be formed of a minimum of 2 x 10GE links on separate linecards
- The 10GE ports should be configured to operate in dedicated bandwidth mode
- Configure vPC Peer Link ports as STP 'Network' ports to enable Spanning Tree Bridge Assurance on these links (assuming BA is enabled globally).
- Enable UDLD on vPC Peer Links
- Running a dedicated VLAN / SVI across the peer link to cater for uplink failure (e.g. to the core) may also be desirable.
The recommendation to use a minimum of 2 x 10GE links for the vPC peer link ensures that a single 10GE linecard failure will not result in a vPC failover situation, whereby reliance on backup features such as the Peer-Keepalive link or Spanning Tree may come into play.
The recommendation on dedicated rate mode is made to ensure that oversubscription should not become an issue on the vPC Peer Link. The Nexus 7000 32 port 10GE linecard (N7K-M132XP-12) has the ability to disable three out of four 10GE links within a given port group, resulting in full 10Gbps line rate capability on the remaining port in the group. The rate mode on a port can be configured using the following CLI commands:
interface Ethernet
rate-mode dedicated
Full details of rate mode and port group information are available at the following URL:
Bridge Assurance is an enhancement to the Spanning Tree Protocol that enables the sending of BPDUs on all operational network ports, including alternate and backup ports. If a port does not receive a BPDU within a specified time period, the port moves into the blocking state. In order to provide additional protection in the event of unexpected reliance upon Spanning Tree, the Bridge Assurance and UDLD features should be enabled on all vPC Peer Link connections.
vPC Peer-keepalive link recommendations
The vPC Peer-keepalive Link (PK-link) is used to provide protection against dual active scenarios in the event of the primary Peer Link being lost. If loss of the Peer Link takes place, the PK-link is used to determine the status of the opposite peer (in other words, to determine whether the loss of connectivity is due to link failure or node failure).
The PK-Link uses a simple heartbeat between vPC peers these messages are sent out every 2 seconds, and the link uses a 3 second hold timeout upon loss of the primary Peer Link.
In order of preference, the following types of interface should be used for the vPC PK-link:
1. Dedicated Link (1Gbps is sufficient) using dedicated VRF
2. mgmt0 interface (shared link with management traffic)
3. Routed over L3 infrastructure (least preferred)
In the event that the chassis is not equipped with any 1GE linecards (i.e. the chassis supports 10GE only), then option 1 becomes less desirable - it is considered inefficient and expensive to dedicate a 10GE connection solely for the PK-link. In this case, option 2 (mgmt0 interface) should be considered.
NOTE: If the mgmt0 interface is used for vPC peer-keepalive functionality, these interfaces must be connected together using an intermediate switch. This is because in a dual Supervisor system, only one management port is active at any one time (e.g. either slot 5 or slot 6). If the mgmt0 interfaces are connected without a switch, i.e. back-to-back, this will cause issues with vPC.
One topology which should not be considered is where the vPC PK-link is routed across the vPC Peer Link. This defeats the object of having a dedicated PK-link - the PK-link is designed to act as a backup for the primary Peer Link, therefore it does not make sense to route the PK-link across the Peer Link.
vPC member link recommendations
vPC member links are the interfaces taking part in the port channel itself; in other words, the connections to the downstream switch. The following recommendations apply to the vPC member links:
- The configuration of the vPC member links should be identical across the two vPC peer switches in particular, the vPC ID must match across the two peers, while the Port Channel number should match across the two peers for ease of management. Figure below shows an example of this:
Role and system priority
In order for vPC to function, the system must elect a primary and secondary peer device. By default, these roles are elected automatically by NX-OS, however it is beneficial to manually control which devices assume the primary and secondary roles.
The recommended practice for configuring role priorities is to ensure that the primary vPC peer is the same device used as the Spanning Tree root bridge and HSRP active peer. This can be achieved by lowering the role priority (default value is 32667), using the following commands:
vpc domain
role priority
The vPC System Priority parameter corresponds to the LACP System Priority value, and is used to guarantee that a peer partner cannot make a decision in terms of aggregation capability and advertisement. In the case of LACP, it is possible to have up to 16 Ethernet ports in a channel, however only eight of these ports can be active at any one time, with the remaining eight operating in standby mode. If there are more than eight ports in the link, the switch at the controlling end of the link will make a decision on which ports are placed into the channel.
It is recommended to manually configure the vPC System Priority to ensure that the vPC peer devices are the 'primary' devices in terms of LACP. It is also important that the vPC System Priority is identical on each pair. vPC System Priority is configured using the following commands:
vpc domain
system-priority
Configuration parameters which must match across vPC peers
There are a number of configuration parameters which must be identical across the two vPC Peer Switches. Failure to comply with this requirement will result in the vPC moving into suspend mode. The configuration parameters which must be identical are as follows:
- Port-channel mode: on, off, or active
- Link speed per channel
- Duplex mode per channel
- Trunk mode per channel:
- Native VLAN
- VLANs allowed on trunk
- Tagging of native VLAN traffic
- Spanning Tree Protocol (STP) mode
- STP region configuration for Multiple Spanning Tree
- Enable/disable state per VLAN
- STP global settings:
- Bridge Assurance setting
- Port type setting-We recommend that you set all vPC interfaces as network ports.
- Loop Guard settings
- STP interface settings:
- Port type setting
- Loop Guard
- Root Guard
- Maximum Transmission Unit (MTU)
- VLAN interface-Each device on the end of the vPC peer link must have a VLAN interface configured for the same VLAN on both ends and they must be in the same administrative and operational mode.Those VLANs configured on only one device of the peer link do not pass traffic using the vPC or peer link.You must create all VLANs on both the primary and secondary vPC devices, or the VLAN will be suspended.
Disabling vPC peer link compatibility check
The default NX-OS behaviour is to not allow any changes to be made to vPCs while the vPC peer link is in a failed state.The intention is to preserve the consistency of the configuration, but in some cases it may be desirable to make changes while the peer link is down, for example provisioning new vPC-attached servers when one of a pair of Nexus 5000s is out of service.
On the Nexus 5000 you can modify the default vPC behaviour when a peer link is down on the primary vPC switch.The default behaviour on the primary vPC switch, after a peer-link goes down, is to keep the vPCs down once they get flapped and to not bring up newly configured vPCs. This command allows newly configured vPCs and existing vPCs that are flapped to be brought up when a peer link is down.
vpc domain
switch(config-vpc-domain)# peer-config-check-bypass
NOTE: The command comes into effect when a peer link is down and the vPC role has been determined to be primary.This command has no effect on the behaviour of the secondary switch, which disables its vPCs in the event of peer-link failure.
