cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4963
Views
40
Helpful
16
Replies

Ask the Expert: Different Flavors and Design with vPC on Cisco Nexus 5000 Series Switches

ciscomoderator
Community Manager
Community Manager

Welcome to the Cisco® Support Community Ask the Expert conversation.  This is an opportunity to learn and ask questions about Cisco® NX-OS.

The biggest limitation to a classic port channel communication is that the port channel operates only between two devices. To overcome this limitation, Cisco NX-OS has a technology called virtual port channel (vPC). A pair of switches acting as a vPC peer endpoint looks like a single logical entity to port channel attached devices. The two devices that act as the logical port channel endpoint are actually two separate devices. This setup has the benefits of hardware redundancy combined with the benefits offered by a port channel, for example, loop management.

vPC technology is the main factor for success of Cisco Nexus® data center switches such as the Cisco Nexus 5000 Series, Nexus 7000 Series, and Nexus 2000 Series Switches.

This event is focused on discussing all possible types of vPC along-with best practices, failure scenarios, Cisco Technical Assistance Center (TAC) recommendations and troubleshooting

Cisco Experts

Vishal Mehta is a customer support engineer for the Cisco Data Center Server Virtualization Technical Assistance Center (TAC) team based in San Jose, California. He has been working in TAC for the past 3 years with a primary focus on data center technologies, such as the Cisco Nexus 5000 Series Switches, Cisco Unified Computing System™ (Cisco UCS®), Cisco Nexus 1000V Switch, and virtualization. He presented at Cisco Live in Orlando 2013 and will present at Cisco Live Milan 2014 (BRKCOM-3003, BRKDCT-3444, and LABDCT-2333). He holds a master’s degree from Rutgers University in electrical and computer engineering and has CCIE® certification (number 37139) in routing and switching, and service provider.

Nimit Pathak is a customer support engineer for the Cisco Data Center Server Virtualization TAC team based in San Jose, California, with primary focus on data center technologies, such as Cisco UCS, the Cisco Nexus 1000v Switch, and virtualization. Nimit holds a master's degree in electrical engineering from Bridgeport University, has CCNA® and CCNP® Nimit is also working on a Cisco data center CCIE® certification While also pursuing an MBA degree from Santa Clara University.

Remember to use the rating system to let Vishal and Nimit know if you have received an adequate response. 

Because of the volume expected during this event, Vishal and Nimit might not be able to answer every question. Remember that you can continue the conversation in the Network Infrastructure Community, under the subcommunity LAN, Switching & Routing, shortly after the event. This event lasts through August 29, 2014. Visit this forum often to view responses to your questions and the questions of other Cisco Support Community members.

1 Accepted Solution

Accepted Solutions

Kelvin Willacey
Level 4
Level 4

I know this is for the Nexus 5000 but information is easy to find for this. Are there any up to date design guides for the Nexus 7000 and 2000 for VPC along with supported server connectivity? I find this information a bit hard to find, if you can provide any I would appreciate it, thanks.

View solution in original post

16 Replies 16

Good time-of-day.

We have a couple of Nexus 5000 switches in vPC, configured with peer-gateway, but without peer-switch; running as first-hop (with HSRP) for 50 end-user VLANs.

Could you please hint what should be the correct steps to minimize service disruption (in terms of routing and STP), if we need to reload holder of primary vPC role?!

Thanks.

Hi Vasilii,

to answer your question.

The “vpc peer-gateway” allows HSRP routers to accept frames destined for their vPC peers.  This feature extends the virtual MAC address functionality to the paired router’s MAC address.  

In addition vPC  the peer-switch feature reduces convergence time as a result of a spanning-tree failure from 3 seconds to sub-second. so if you dont have the peer-switch option then you 3 second spt convergence time.

-Nimit

 

Hello Vasilii

Is your Primary vPC N5k switch root of Spanning-Tree ?

If so then Peer-Switch would help to minimize the disruption to just sub-seconds as explained by Nimit.

Please see below link (search for Peer-Switch) for explanation on how Peer-Switch configuration can help

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/design_guide_c07-625857.html#_Toc271759470

Otherwise if these Nexus 5000 are not roots then rebooting Primary N5k will not cause service disruption as Secondary N5k will continue to forward the traffic.

To summarize below are the options:

1. if N5k are not root....then reboot of primary N5k should not have traffic loss
2. if they are the root then better to configure peer-switch
3. if peer-switch cannot be configured and N5k are the roots, then 3 second outage expected

Let me know if more clarification is needed.

Thanks

Vishal

Thank you for reply.

The vPC is a root bridge for all VLANs, but peer-switch is not configured (clear about this - STP convergance should happen).

Won't the reload of vPC primary device disrupt the service for any significant time? Running firmware is 5.3.1 and my question is about proofed behavior of vPC if we reload operational primary (like, does peer-keepalive and peer-links go down simultaneously?)

Upon reboot of Primary vPC switch there will not be any traffic loss on vPC links as secondary vPC switch will become primary and forward the traffic. i.e There is no impact on north/south data traffic but east-west traffic over Peer-Link will be lost (black-holed) during reboot time of primary vPC switch

During reboot yes both Peer-Link and Keepalive link will go down simultaneously.

The details on expectation during vPC peer switch reboot are explained at link below:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/operations/n5k_vpc_ops.html#wp425292

Let me know if further clarification is needed.

Regards,

Vishal

Kelvin Willacey
Level 4
Level 4

I know this is for the Nexus 5000 but information is easy to find for this. Are there any up to date design guides for the Nexus 7000 and 2000 for VPC along with supported server connectivity? I find this information a bit hard to find, if you can provide any I would appreciate it, thanks.

Hi,

Host–to–Cisco Nexus 2000 Series Connectivity Options
A host can connect to a Cisco Nexus 2000 using all the most common technologies
like network interface card (NIC) teaming or MAC pinning (active-active uplinks using
the Cisco Nexus 1000V Switch, for example) or port channels. On the top of those,
the Nexus 2000 supports the Virtual Port Channel (vPC) functionality, a unique Cisco
feature that allows hosts to attach to a pair of Cisco Nexus 2000 with active/active
redundancy based on standard port channeling.

please find the attached for your reference.

i hope this help !!

 

let me know if you have any additioanl questions.

 

Hello KWillacey,

Yes there is Server deisgn guide with Nexus 7000 but it is specific for UCS-B chassis.

The recommendations listed below will be helpful in connecting any other servers with Nexus 7000

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white_paper_c11-623265.html

 

Please refer above link and feel free to ask us any follow-up questions

Thanks,

Vishal

Kelvin Willacey
Level 4
Level 4

Thanks for the information guys. I will review and ask any follow up questions.

Gustavo Novais
Level 1
Level 1

Hello,

 

My questions are regarding the  behaviour in what concerns L3 modules in Nexus 55xx and their routing adjacencies with external devices via vPC (when compared to N7000).

The N5k L3 and vPC operations guide

(http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/operations/n5k_L3_w_vpc_5500platform.html#wp999181)

states that a topology of a router attached to a pair of Nexus 5500 w L3 module works in unicast (N5k1 would forward the packet towards N5k2 via the peer link) but  is not supported in Multicast. Usually this a major NO in N7k - see Brad Hedlunds website.

My question is that the majority of the routing protocols need to establish adjacencies based in multicast, so I'm wondering if a router would manage to establish adjacencies (let's say OSPF) with both N5k's or not? Would the traffic flow work?

Is the vPC loop avoidance rule applicable when the traffic is coming from a N5K-2 vPC member, through the peer link and is directed to the L3 module of the N5K-1, or is the L3 module seen as an orphan port, bypassing then this rule? In the N7k, this type of traffic seems to be caught by the loop prevention rule.

 

Does the same principle apply to Nexus 5600/6000/7000 ? if not, why?

 

Excepting for the suboptimal traffic flow, is there any reason for this topology not being recommended?

 

Another question that is also related is:  if on a DCI scenario where two pairs of N5500/5600/6000 are interconnected to each other via a back to back vPC (no fabricpath) do you still need to separate L2 traffic from L3?

if the N5500 is able to forward L3 packets towards the peer L3 module, not breaking the vPC rule, why  would the L3 /L2 links separation be needed?

 

Thank you in advance for your answers

 

Gustavo

Hi Gustavo,

I will research on your query and will provide detailed answer upon my findings.

Thanks,

Vishal

Hello Gustavo

Please see my responses to your questions:

  1. Yes almost all routing protocols use Multicast to establish adjacencies. We are dealing with two different type of traffic –Control Plane and Data Plane.

Control Plane: To establish Routing adjacency, the first packet (hello) is punted to CPU. So in the case of triangle routed VPC topology as specified on the Operations Guide Link, multicast for routing adjacencies will work. The hellos packets will be exchanged across all 3 routers and adjacency will be formed over VPC links

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/operations/n5k_L3_w_vpc_5500platform.html#wp999181

Now for Data Plane we have two types of traffic – Unicast and Multicast.

The Unicast traffic will not have any forwarding issues, but because the Layer 3 ECMP and port channel run independent hash calculations there is a possibility that when the Layer 3 ECMP chooses N5k-1 as the Layer 3 next hop for a destination address while the port channel hashing chooses the physical link toward N5k-2. In this scenario,N5k-2 receives packets from R with the N5k-1 MAC as the destination MAC.

Sending traffic over the peer-link to the correct gateway is acceptable for data forwarding, but it is suboptimal because it makes traffic cross the peer link when the traffic could be routed directly.

For that topology, Multicast Traffic might have complete traffic loss due to the fact that when a PIM router is connected to Cisco Nexus 5500 Platform switches in a vPC topology, the PIM join messages are received only by one switch. The multicast data might be received by the other switch.

  1. The Loop avoidance works little different across Nexus 5000 and Nexus 7000.

Similarity: For both products, loop avoidance is possible due to VSL bit

The VSL bit is set in the DBUS header internal to the Nexus.

It is not something that is set in the ethernet packet that can be identified. The VSL bit is set on the port asic for the port used for the vPC peer link, so if you have Nexus A and Nexus B configured for vPC and a packet leaves Nexus A towards Nexus B, Nexus B will set the VSL bit on the ingress port ASIC. This is not something that would traverse the peer link.

This mechanism is used for loop prevention within the chassis.

The idea being that if the port came in the peer link from the vPC peer, the system makes the assumption that the vPC peer would have forwarded this packet out the vPC-enabled port-channels towards the end device, so the egress vpc interface's port-asic will filter the packet on egress.

Differences:  In Nexus 5000 when it has to do L3-to-L2 lookup for forwarding traffic, the VSL bit is cleared and so the traffic is not dropped as compared to Nexus 7000 and Nexus 3000.

It still does loop prevention but the L3-to-L2 lookup is different in Nexus 5000 and Nexus 7000.

For more details please see below presentation:

https://supportforums.cisco.com/sites/default/files/session_14-_nexus.pdf

  1. DCI Scenario:  If 2 pairs are of Nexus 5000 then separation of L3/L2 links is not needed.

But in most scenarios I have seen pair of Nexus 5000 with pair of Nexus 7000 over DCI or 2 pairs of Nexus 7000 over DCI. If Nexus 7000 are used then L3 and L2 links are required for sure as mentioned on above presentation link.

Let us know if you have further questions.

Thanks,

Vishal

Hello Vishal,

 

Excellent answers and presentation (even though it focus more on 7k). You managed to clarify something that was getting very fuzzy.

I noted the PIM Multicast issue.

Now my last questions are:

- does 5600 or 6000 have the same behaviour as the 5500 to clear the VSL bit when doing L3 to L2 lookup? Given that their L3 engines seem to be integrated within FIB I'd expect them to be more like 7k than 5500...

- Where can we find reference information that explains the per-platform behaviour?

- will the unicast attachment via port-channel to a pair of N5500 be supported by TAC (if we have a case and we need assistance)?

 

Thank you again.

 

Gustavo

Hello Gustavo,

Thank you for the acknowledgement.

I have experience with Nexus 5000/5500 and Nexus 7000 but not on 5600/6000 series.

Kindly follow up with your Cisco System Engineer for architectural difference on L3-to-L2 lookup. I am curious to know as well and will reach to Engineering Team, once i have answer will surely respond back.

I dont think we publish on cisco.com the per-platform behavior and again Cisco Account Team can give you those specifics.

I did not understand the topology on unicast attachment, are you referring to Router unicast attachment via vPC to N5000 ? please elaborate the network setup.
But either way, any case is welcomed by Cisco TAC, if we think the topology is not right for your network on basis of platform, code and end-goal then we will recommend you the correct workaround and reasoning for un-supported topology

Feel free to ask us more questions.

Thanks,
Vishal

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: