Nexus 9300 Series - vPC over LAN Extension Circuit

[TONY SMITH]

Hi,

We're looking at a design with a data centre and separate site with the actual users.  DC will be using Nexus 9300 series, exact model not yet determined.  Possibly the same at the user site.   The idea with the Nexuses is to connect everything via vPC so that no normal traffic has to cross the peer link, and so we have redundancy. 

DC is linked to the user site by a 1 gig point to point Ethernet LAN extension circuit.  We are looking at the option of a second link for redundancy, however I wondered whether that could be configured as vPC either single ended between the two DC Nexuses and something like a 3850 switch stack.   Or maybe double ended if we have two vPC capable Nexuses in the user site as well.

Are there documented limits on the link characteristics supported for vPC, for example latency?  The existing circuit has around 1ms RTT, the second circuit would be diversely routed so may not be exactly the same.

Thanks, TonyS

[Hector Gustavo Serrano Gutierrez · Accepted solution]

Hi @TONY SMITH,

https://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

You could "stretch" the layer 2 domain between your Data Center and "user site" similar to the above picture, but do you really have such need? Probably you don't want to make your Layer 2 domain too large.

On the other hand, you can just make the links connect to Layer 3 ports and leverage usual Layer 3 Routing between your Nexus Switches and the "user  site".

Keep in mind that Nexus Switches in a vPC cluster are still individual Control Plane devices so Routing between them should also be set up in order to exchange routes. This in case you only connect a single cable to only one of the Nexus in vPC.

Regards.

[TONY SMITH]

Thanks for the response.  We won't be extending the Layer 2 domain between the sites, it will be a router Layer 3 connection.  The reason I was thinking VPC rather than just two load sharing L3 links was to do with each Nexus always being able to forward on the frame (packet) without ever passing to their peer.  I guess that could still be arranged with conventional L3 routing if each Nexus prefers its own route over that of its peer.

Can you create a L3 VPC, in the same way as a L3 Etherchannel?   Or does the fundamental Nexus configuration mean that all its VPCs are one or the other?

Thanks, Tony S

[Hector Gustavo Serrano Gutierrez]

In that case, you can just make them Layer 3 ports and let the Nexus' Routing Table take the decision to forward the traffic to the user site using the locally connected link.

You could bundle these links in a Layer 2 vPC (there is no L3 vPC) and use SVIs (interface vlans) to enable Routing between the Nexus switches and the user site but I don't suggest this approach. One of the reasons is that PIM Multicast over vPC is not fully supported.

As a side note, I'd suggest you to add a new local link to interconnect the both Nexus in parallel to the vPC peer-link and make it a Layer 3 link. With it, you can enable any Routing Protocol and in the scenario where let's say Nexus-A is the only one with a link to the user site, Nexus-B can have a Routing entry in its Routing Table using Nexus-A as an IP next-hop over this Layer 3 link.

Best Regards.

[TONY SMITH]

Thanks, could I clarify the point about a L3 link between the Nexuses, which I assumes refers to a link between the two members of one VPC domain, ie the pair at one site?  What's the benefit of this compared to just letting the two peer with each other at L2?   This will be quite a collapsed design, it's not big enough to justify separate core, distribution etc equipment.  So the idea will be for the Nexus to do all local VLAN to VLAN routing, with SVIs for each VLAN and HSRP.  I was thinking of designating one VLAN for routing, and allow them to peer on that VLAN, both with each other and with any separate routers.  A separate L3 connection between Nexus implies (I think) we would also use separate L3 links to any other routers.

---

@Hector Gustavo Serrano Gutierrez wrote:

...

As a side note, I'd suggest you to add a new local link to interconnect the both Nexus in parallel to the vPC peer-link and make it a Layer 3 link. With it, you can enable any Routing Protocol and in the scenario where let's say Nexus-A is the only one with a link to the user site, Nexus-B can have a Routing entry in its Routing Table using Nexus-A as an IP next-hop over this Layer 3 link.

---

[Hector Gustavo Serrano Gutierrez]

Hi @TONY SMITH,

That's right. I am referring to an extra link between the two members of one VPC domain, ie the pair at one site. This does not imply use separate L3 links to any other Routers.

With this L3 link, you can keep the vPC peer-link out of the picture in case the uplink goes down on the local Nexus and traffic needs to be Routed (not Switched) to the other vPC member. I prefer this as I feel is a "more elegant" approach. However, you can still use the vPC peer-link and create a pair of SVIs to make the Nexuses peer each other using any Routing protocol.

Back to the original question, you use the Layer 3 links between the Nexus and the user site and not necessarily bundle them in vPC. Normal Routing (Equal Cost Multi-Path = ECMP) would take care traffic traverses both links.

Regards.

[TONY SMITH]

Thanks again.  

---

@Hector Gustavo Serrano Gutierrez wrote:

That's right. I am referring to an extra link between the two members of one VPC domain, ie the pair at one site. This does not imply use separate L3 links to any other Routers.

---

To clarify what I was meaning about other routers, let's say we start with Vlan 10 10.10.10.0 /24 where the routers are connected.  Nexuses will have Vlan 10 SVIs and routing protocol enabled to peer with the routers on Vlan 10.  Once that's configured then in the absence of anything to prevent it, the Nexuses would also peer with each other over the same VLAN.  With that filtered out in favour of a separate L3 link between the Nexuses, I can't help feel there's some sort of contradiction.  Maybe I need to just model that for my own benefit, as it's a routing issue not Nexus specific.   We're intending to use EIGRP if that makes a difference.

[Hector Gustavo Serrano Gutierrez]

Hi @TONY SMITH,

You are right, in that case Vlan 10 SVI already covers (indirectly) routing from Nexus-A to Nexus-B (and viceversa) as they will see each other as EIGRP neighbors over the vPC peer-link.

Use of any Routing Protocol or Static Route with a Router or Layer 3 Switch over vPC (using any VLAN with SVI) from your pair of Nexus is indeed supported nowadays. It wasn't supported in the early days of Nexus switches but that is now ancient history =)

Still, I'm more in favor of using L3 links to peer with a Router or Layer 3 Switch. Main reason is if Multicast PIM is a requirement in the future.

Supported Topologies for Routing over Virtual Port Channel on Nexus Platforms

[TONY SMITH]

Thanks again.  This will be a fairly sweeping redesign so we may be able to go for L3 links between routers.  I haven't got that level of detail yet.  However my original question, vPC over LAN Extension, is answered.  Yes it's possible, but in this specific design it's not needed and in fact would add no benefit.