VPC and Routing Design Question

[TONY SMITH]

Hi,

I'm trying to understand the implications of mixing routing protocols with a VPC topology.   To simplify the scenario, we will have two Nexus switches with Catalyst switch stack dual-homed.  Nexus to Catalyst link will be L2 trunk with all VLAN to VLAN routing at the Nexuses via SVIs and HSRP.

If I understand correctly the problem would arise if we then connect a router to the Catalyst using a dynamic routing protocol between the router and the two Nexuses.  Is that correct?  From what I can read the underlying issue is that the router may be learning a route from Nexus A but the VPC will feel free to send the frame to Nexus B.

If my understanding is correct, what are the best alternative options?  From a routing perspective we want the two Nexuses and the WAN routers all to be neighbours with the Nexuses advertising out the DC subnets and learning the WAN subnets from the routers.

Any comments appreciated.

Tony S

[shaps]

It depends on the platform as to how this would look, I remember having issues with a similiar set up before, the solution was simply to create separate L2 link between the two Nexus devices and dedicate that to peering for the router to nexus, I think now with newer code this is possible without that and the use of the peer gateway and peer router command which allows the secondary device to accept packets on behalf of its peer and not decrement the ttl value when building adjacenies. Without this you will constantly see exstart within the ospf process,

Pretty sure the 9K will need a dedicated link.

[TONY SMITH]

Thanks.   They're 9Ks, running NX-OS 9.3(1),

There will be a separate L2 link between the two Nexuses, separate from the VPC Peer Link.  

When you refer to "dedicate that to peering for the router to nexus" does that refer to specific configuration for that link, or just that router traffic was the sole reason for needing it?

[shaps]

Hey, sorry to be clear it just to facilitate the peering between the router and the nexus, so one vlan used to transit the adjaceny on the dedicated link whilst ensuring that is pruned from the peer link.

[TONY SMITH]

Hi, thanks again.   To help me with the background could you confirm that my understanding of the underlying issue is correct.  Let's call the two Nexuses NexusA and NexusB, and the Catalyst can just be Catalyst.  So let's see if I have this correct ..

(1) Fundamental principle of a VPC, if a frame is received on a VPC member link on NexusA then it can only be forwarded on either a VPC member on NexusA or a non-VPC link.  

(2) For HSRP and hosts talking to their default gateway that's no problem as either Nexus responds on the HSRP virtual MAC

(3) With dynamic routing protocols the two Nexuses will behave as two separate routers, not as one router.  A downstream router, connected via VPC will see these two different routers and may for example pick NexusA as the next hop for a given path.  The router's packet (frame I suppose) will be addressed to NexusA's MAC address, but the VPC may in fact forward it to NexusB, the wrong one.

Am I correctly stating the issue?

[shaps]

1 and 2 are correct and dont really require any more adding to them, point 3 opens a few questions mainly becuase it has been a while since I have been neck deep in nexus, the crux of it is when any routing happens via a vpc member link and uses the additional link between the two peers the logic of VPC is not a factor as there is no traversal of the peer link so will naturally go via the new link.
I believe with newer code, although I havent tried there is no requirement for the extra link between the pair, the commands peer gateway and layer 3 peer router overcome these limitations.
Hopefully this helps.

[Andrea Testino]

Hey Tony,

Routing over vPC has been supported for quite some time (since 7.0(3)I5..). Since you're running 9.x, you will definitely be able to use the required "layer3 peer-router" feature within the vPC domain.

I encourage you go over our Routing over vPC doc which will have a list of supported topologies, etc. It is very unusual for us to see customers still using the separate L2 link between the two vPC peers nowadays.

Hope that helps! Let me know if you have any other questions.

[TONY SMITH]

Andrea,

Thanks.  I have looked at various design documents, but haven't so far found one that looks completely relevant to this installation.  In the document that you linked, our equivalent would be L2 switches in the locations shown as L3-B and L3-C, connected as Layer 2 only to the Nexuses.  The routers will connect conventionally to those switches.  So the routing protocol neighbours would be the two Nexuses and the routers connected to the switches.

The Nexus-Catalyst links need to be L2 as there are a number of VLANs that need to be presented on both Catalysts.  I did think of taking the routers right off the Catalysts and connecting either directly to the Nexuses or via separate switches on non-VPC links.  However that still seems sub-optimal as traffic from the hosts on the VPC connected switches could then end up on the "wrong" Nexus if the path to the routers is non-VPC.

Or am I missing something?  I guess one solution would be to forget VPC for the Catalyst uplinks.  That seems a bit naff, but in may in fact turn out that the customer's high-bandwidth loads aren't going to be on those switches in any case.

[jjfaure]

Hi guys,

I have similar doubts relating BGP routing over  VPC. I have 1 pair of Nexus 5600 switches configured in VPC, and i need to know if is supported to have this config:

-1 Ebgp session between NexusA and a third party router (RouterA), using Nexus SVI interface

-Another Ebgp session between NexusB and another router (RouterB), using Nexus SVI interface

-An IBGP session between NexusA and NexusB to share routes leart from peer Nexus

-NexusA and Nexus B will be in same vpc domain, configured with "peer-gateway" and "layer3 peer-router" options

I have not seen specific documents or samples about a configuration like this one, please anyone can inform if is it supported?

Regards and TIA

Juan

[Sergiu.Daniluk]

Hi @jjfaure 

You can refer to this document for all supported routing options over vPC, and the required config, if any.

https://www.cisco.com/c/en/us/support/docs/ip/ip-routing/118997-technote-nexus-00.html 

Stay safe,

Sergiu

[jjfaure]

Hi @Sergiu.Daniluk 

I understand i can have all those functions working togheter in same VPC Nexus, is not it? This is the reason i think i need IBGP between primary and secondary VPC  Nexus switches

Regards

Juan