ā08-15-2014 10:23 AM - edited ā03-07-2019 08:24 PM
Welcome to the CiscoĀ® Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about CiscoĀ® NX-OS.
The biggest limitation to a classic port channel communication is that the port channel operates only between two devices. To overcome this limitation, Cisco NX-OS has a technology called virtual port channel (vPC). A pair of switches acting as a vPC peer endpoint looks like a single logical entity to port channel attached devices. The two devices that act as the logical port channel endpoint are actually two separate devices. This setup has the benefits of hardware redundancy combined with the benefits offered by a port channel, for example, loop management.
vPC technology is the main factor for success of Cisco NexusĀ® data center switches such as the Cisco Nexus 5000 Series, Nexus 7000 Series, and Nexus 2000 Series Switches.
This event is focused on discussing all possible types of vPC along-with best practices, failure scenarios, Cisco Technical Assistance Center (TAC) recommendations and troubleshooting
Vishal Mehta is a customer support engineer for the Cisco Data Center Server Virtualization Technical Assistance Center (TAC) team based in San Jose, California. He has been working in TAC for the past 3 years with a primary focus on data center technologies, such as the Cisco Nexus 5000 Series Switches, Cisco Unified Computing Systemā¢ (Cisco UCSĀ®), Cisco Nexus 1000V Switch, and virtualization. He presented at Cisco Live in Orlando 2013 and will present at Cisco Live Milan 2014 (BRKCOM-3003, BRKDCT-3444, and LABDCT-2333). He holds a masterās degree from Rutgers University in electrical and computer engineering and has CCIEĀ® certification (number 37139) in routing and switching, and service provider.
Nimit Pathak is a customer support engineer for the Cisco Data Center Server Virtualization TAC team based in San Jose, California, with primary focus on data center technologies, such as Cisco UCS, the Cisco Nexus 1000v Switch, and virtualization. Nimit holds a master's degree in electrical engineering from Bridgeport University, has CCNAĀ® and CCNPĀ® Nimit is also working on a Cisco data center CCIEĀ® certification While also pursuing an MBA degree from Santa Clara University.
Remember to use the rating system to let Vishal and Nimit know if you have received an adequate response.
Because of the volume expected during this event, Vishal and Nimit might not be able to answer every question. Remember that you can continue the conversation in the Network Infrastructure Community, under the subcommunity LAN, Switching & Routing, shortly after the event. This event lasts through August 29, 2014. Visit this forum often to view responses to your questions and the questions of other Cisco Support Community members.
Solved! Go to Solution.
ā08-19-2014 02:35 PM
I know this is for the Nexus 5000 but information is easy to find for this. Are there any up to date design guides for the Nexus 7000 and 2000 for VPC along with supported server connectivity? I find this information a bit hard to find, if you can provide any I would appreciate it, thanks.
ā08-19-2014 07:13 AM
Good time-of-day.
We have a couple of Nexus 5000 switches in vPC, configured with peer-gateway, but without peer-switch; running as first-hop (with HSRP) for 50 end-user VLANs.
Could you please hint what should be the correct steps to minimize service disruption (in terms of routing and STP), if we need to reload holder of primary vPC role?!
Thanks.
ā08-19-2014 03:13 PM
Hi Vasilii,
to answer your question.
The āvpc peer-gatewayā allows HSRP routers to accept frames destined for their vPC peers. This feature extends the virtual MAC address functionality to the paired routerās MAC address.
In addition vPC the peer-switch feature reduces convergence time as a result of a spanning-tree failure from 3 seconds to sub-second. so if you dont have the peer-switch option then you 3 second spt convergence time.
-Nimit
ā08-19-2014 04:10 PM
Hello Vasilii
Is your Primary vPC N5k switch root of Spanning-Tree ?
If so then Peer-Switch would help to minimize the disruption to just sub-seconds as explained by Nimit.
Please see below link (search for Peer-Switch) for explanation on how Peer-Switch configuration can help
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/design_guide_c07-625857.html#_Toc271759470
Otherwise if these Nexus 5000 are not roots then rebooting Primary N5k will not cause service disruption as Secondary N5k will continue to forward the traffic.
To summarize below are the options:
1. if N5k are not root....then reboot of primary N5k should not have traffic loss
2. if they are the root then better to configure peer-switch
3. if peer-switch cannot be configured and N5k are the roots, then 3 second outage expected
Let me know if more clarification is needed.
Thanks
Vishal
ā08-19-2014 11:50 PM
Thank you for reply.
The vPC is a root bridge for all VLANs, but peer-switch is not configured (clear about this - STP convergance should happen).
Won't the reload of vPC primary device disrupt the service for any significant time? Running firmware is 5.3.1 and my question is about proofed behavior of vPC if we reload operational primary (like, does peer-keepalive and peer-links go down simultaneously?)
ā08-21-2014 12:32 AM
Upon reboot of Primary vPC switch there will not be any traffic loss on vPC links as secondary vPC switch will become primary and forward the traffic. i.e There is no impact on north/south data traffic but east-west traffic over Peer-Link will be lost (black-holed) during reboot time of primary vPC switch
During reboot yes both Peer-Link and Keepalive link will go down simultaneously.
The details on expectation during vPC peer switch reboot are explained at link below:
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/operations/n5k_vpc_ops.html#wp425292
Let me know if further clarification is needed.
Regards,
Vishal
ā08-19-2014 02:35 PM
I know this is for the Nexus 5000 but information is easy to find for this. Are there any up to date design guides for the Nexus 7000 and 2000 for VPC along with supported server connectivity? I find this information a bit hard to find, if you can provide any I would appreciate it, thanks.
ā08-19-2014 03:40 PM
Hi,
HostātoāCisco Nexus 2000 Series Connectivity Options
A host can connect to a Cisco Nexus 2000 using all the most common technologies
like network interface card (NIC) teaming or MAC pinning (active-active uplinks using
the Cisco Nexus 1000V Switch, for example) or port channels. On the top of those,
the Nexus 2000 supports the Virtual Port Channel (vPC) functionality, a unique Cisco
feature that allows hosts to attach to a pair of Cisco Nexus 2000 with active/active
redundancy based on standard port channeling.
please find the attached for your reference.
i hope this help !!
let me know if you have any additioanl questions.
ā08-19-2014 04:12 PM
Hello KWillacey,
Yes there is Server deisgn guide with Nexus 7000 but it is specific for UCS-B chassis.
The recommendations listed below will be helpful in connecting any other servers with Nexus 7000
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white_paper_c11-623265.html
Please refer above link and feel free to ask us any follow-up questions
Thanks,
Vishal
ā08-20-2014 06:36 AM
Thanks for the information guys. I will review and ask any follow up questions.
ā08-20-2014 11:55 AM
Hello,
My questions are regarding the behaviour in what concerns L3 modules in Nexus 55xx and their routing adjacencies with external devices via vPC (when compared to N7000).
The N5k L3 and vPC operations guide
(http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/operations/n5k_L3_w_vpc_5500platform.html#wp999181)
states that a topology of a router attached to a pair of Nexus 5500 w L3 module works in unicast (N5k1 would forward the packet towards N5k2 via the peer link) but is not supported in Multicast. Usually this a major NO in N7k - see Brad Hedlunds website.
My question is that the majority of the routing protocols need to establish adjacencies based in multicast, so I'm wondering if a router would manage to establish adjacencies (let's say OSPF) with both N5k's or not? Would the traffic flow work?
Is the vPC loop avoidance rule applicable when the traffic is coming from a N5K-2 vPC member, through the peer link and is directed to the L3 module of the N5K-1, or is the L3 module seen as an orphan port, bypassing then this rule? In the N7k, this type of traffic seems to be caught by the loop prevention rule.
Does the same principle apply to Nexus 5600/6000/7000 ? if not, why?
Excepting for the suboptimal traffic flow, is there any reason for this topology not being recommended?
Another question that is also related is: if on a DCI scenario where two pairs of N5500/5600/6000 are interconnected to each other via a back to back vPC (no fabricpath) do you still need to separate L2 traffic from L3?
if the N5500 is able to forward L3 packets towards the peer L3 module, not breaking the vPC rule, why would the L3 /L2 links separation be needed?
Thank you in advance for your answers
Gustavo
ā08-21-2014 12:58 AM
Hi Gustavo,
I will research on your query and will provide detailed answer upon my findings.
Thanks,
Vishal
ā08-21-2014 05:16 PM
Hello Gustavo
Please see my responses to your questions:
Control Plane: To establish Routing adjacency, the first packet (hello) is punted to CPU. So in the case of triangle routed VPC topology as specified on the Operations Guide Link, multicast for routing adjacencies will work. The hellos packets will be exchanged across all 3 routers and adjacency will be formed over VPC links
Now for Data Plane we have two types of traffic ā Unicast and Multicast.
The Unicast traffic will not have any forwarding issues, but because the Layer 3 ECMP and port channel run independent hash calculations there is a possibility that when the Layer 3 ECMP chooses N5k-1 as the Layer 3 next hop for a destination address while the port channel hashing chooses the physical link toward N5k-2. In this scenario,N5k-2 receives packets from R with the N5k-1 MAC as the destination MAC.
Sending traffic over the peer-link to the correct gateway is acceptable for data forwarding, but it is suboptimal because it makes traffic cross the peer link when the traffic could be routed directly.
For that topology, Multicast Traffic might have complete traffic loss due to the fact that when a PIM router is connected to Cisco Nexus 5500 Platform switches in a vPC topology, the PIM join messages are received only by one switch. The multicast data might be received by the other switch.
Similarity: For both products, loop avoidance is possible due to VSL bit
The VSL bit is set in the DBUS header internal to the Nexus.
It is not something that is set in the ethernet packet that can be identified. The VSL bit is set on the port asic for the port used for the vPC peer link, so if you have Nexus A and Nexus B configured for vPC and a packet leaves Nexus A towards Nexus B, Nexus B will set the VSL bit on the ingress port ASIC. This is not something that would traverse the peer link.
This mechanism is used for loop prevention within the chassis.
The idea being that if the port came in the peer link from the vPC peer, the system makes the assumption that the vPC peer would have forwarded this packet out the vPC-enabled port-channels towards the end device, so the egress vpc interface's port-asic will filter the packet on egress.
Differences: In Nexus 5000 when it has to do L3-to-L2 lookup for forwarding traffic, the VSL bit is cleared and so the traffic is not dropped as compared to Nexus 7000 and Nexus 3000.
It still does loop prevention but the L3-to-L2 lookup is different in Nexus 5000 and Nexus 7000.
For more details please see below presentation:
https://supportforums.cisco.com/sites/default/files/session_14-_nexus.pdf
But in most scenarios I have seen pair of Nexus 5000 with pair of Nexus 7000 over DCI or 2 pairs of Nexus 7000 over DCI. If Nexus 7000 are used then L3 and L2 links are required for sure as mentioned on above presentation link.
Let us know if you have further questions.
Thanks,
Vishal
ā08-21-2014 06:07 PM
Hello Vishal,
Excellent answers and presentation (even though it focus more on 7k). You managed to clarify something that was getting very fuzzy.
I noted the PIM Multicast issue.
Now my last questions are:
- does 5600 or 6000 have the same behaviour as the 5500 to clear the VSL bit when doing L3 to L2 lookup? Given that their L3 engines seem to be integrated within FIB I'd expect them to be more like 7k than 5500...
- Where can we find reference information that explains the per-platform behaviour?
- will the unicast attachment via port-channel to a pair of N5500 be supported by TAC (if we have a case and we need assistance)?
Thank you again.
Gustavo
ā08-22-2014 12:12 AM
Hello Gustavo,
Thank you for the acknowledgement.
I have experience with Nexus 5000/5500 and Nexus 7000 but not on 5600/6000 series.
Kindly follow up with your Cisco System Engineer for architectural difference on L3-to-L2 lookup. I am curious to know as well and will reach to Engineering Team, once i have answer will surely respond back.
I dont think we publish on cisco.com the per-platform behavior and again Cisco Account Team can give you those specifics.
I did not understand the topology on unicast attachment, are you referring to Router unicast attachment via vPC to N5000 ? please elaborate the network setup.
But either way, any case is welcomed by Cisco TAC, if we think the topology is not right for your network on basis of platform, code and end-goal then we will recommend you the correct workaround and reasoning for un-supported topology
Feel free to ask us more questions.
Thanks,
Vishal
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide