Re: EVPN Multisite and Classic Ethernet

mmelbourne · ‎11-25-2022

Hi,

Is it possible to leverage EVPN Multisite where a vPC Border Gateway is connected to a "Classic" L2-segment only (i.e. no site-internal VTEPs)? I configured this and actually see Multisite in a non-operational state because of Fabric isolation (there are no L3 connections to a Fabric as this is a Classical L2 connection, and "evpn multisite fabric-tracking" can only be applied to L3 interfaces and not to an L2 port-channel). Therefore when Type-2s are advertised to the remote site, they always have the vPC VIP (and not the Multisite VIP). I assume that the Multisite VIP only comes it to play when there are site-internal VTEPs (e.g. migrating a Classic site to a Fabric design) and it needs to perform a rewrite for the local site-internal VTEPs.

Does this mean that with with only an L2 connection from the vPC BGW to a Classic Ethernet segment, Multisite is not actually in operation (and therefore we do not gain the benefit of EVPN Multisite Storm Control (for example), and indeed, there is no benefit to configuring Multisite for this use-case, as is not operational?

Sergiu.Daniluk · ‎11-25-2022

So in site1 you have a full vxlan fabric, and in site2 you have only BGWs right? Similar to this:

Its supported and should work along with .

Reference: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-739942.html#Legacysiteintegration

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/whitepaper-c11-742114.html#AppendixVXLANEVPNMultiSitewithvPCBGWdesignanddeploymentconsiderations

Maybe you can share the cfg and we can have a look together on what its missing.

Cheers,

Sergiu

mmelbourne · ‎11-27-2022

Hi Sergiu,

That is pretty much the design as it stands; migrating from a Legacy site to a Fabric-enabled site, and I agree that the documentation suggests EVPN Multi-Site should work in this way. Here, were are migrating away from the Legacy site, so we are not introducing any site-internal VTEPs as a migration strategy at that site.

However, on the legacy site, I can configure all of the multisite configuration, but cannot configure "evpn multisite fabric-tracking" on any interfaces (there are no L3 interfaces to configure, only DCI interfaces, and this command is not accepted on L2 vPC port-channels). In this situation multisite is shown to be operationally 'down' because of the "Fabric" isolation. Therefore, in this scenario is Multi-site even in play, or is this just a basic vPC BGW? In my view the documentation does hint at this, stating that the vPC VIP is used for directly-attached L2 devices, but how then can this BGW leverage some of the advantages of Multi-Site (e.g. storm-control) if Multi-Site is not active?

This then leads me to question whether Multi-Site is possible here as it's not active (unless there are workarounds to enable Multi-Site in this specific configuration)?

Unfortunately, the configuration is now in production, and I can't reapply the Multisite configuration on the legacy site, but I will look to build it again in a lab. I'd be interested to see a vPC Multisite BGW configuration for a Legacy site (no site-internal fabric links), but I can't find an example.

Thanks,
Matt

mmelbourne · ‎11-27-2022

What's interesting, is in the documentation:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/vxlan/configuration/guide/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x_chapter_01001.html

There is the comment:

"The "evpn multisite fabric-tracking" is mandatory only for anycast BGWs. For vPC based BGWs, this command is not mandatory. The NVE Interface will be brought up with just the dci tracked link in the up state"

The NVE interface is up and vPC VIP is used; what's not clear is which Multi-Site functions are still in operation, given Multi-Site is still operationally 'down'.

mmelbourne · ‎11-30-2022

I've build this using 9.3(8) in a virtual lab; the A-side of one vPC pair is attached the other site and the other site are configured similarly. IPs are:

10.<site>.0.254 A-side RID (BGP)
10.<site>.0.253 B-side RID (BGP)
10.<site>.0.252 A-side PIP
10.<site>.0.251 B-side PIP
10.<site>.0.250 vPC VIP
10.<site>.0.249 Multisite VIP

site-a-1# sh nve interface nve1 detail 
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [notified]
Local Router MAC: 5001.0000.1b08
Host Learning Mode: Control-Plane
Source-Interface: loopback1 (primary: 10.1.0.252, secondary: 10.1.0.250)
Source Interface State: Up
Virtual RMAC Advertisement: No
NVE Flags: 
Interface Handle: 0x49000001
Source Interface hold-down-time: 180
Source Interface hold-up-time: 30
Remaining hold-down time: 0 seconds
Virtual Router MAC: 0200.0a01.00fa
Virtual Router MAC Re-origination: 0200.0a01.00f9
Interface state: nve-intf-add-complete
Multisite delay-restore time: 180 seconds
Multisite delay-restore time left: 0 seconds
Multisite dci-advertise-pip configured: False
Multisite bgw-if: loopback100 (ip: 10.1.0.249, admin: Up, oper: Up)
Multisite bgw-if oper down reason: FABRIC isolated.

site-a-1# sh nve peers 
Interface Peer-IP State LearnType Uptime Route
r-Mac 
--------- -------------------------------------- ----- --------- -------- -----
------------
nve1 10.2.0.250 Up CP 00:17:00 n/a

The NVE to reach site 2 from Site 1 is the vPC VIP (10.2.0.250), not the Multisite VIP (10.2.0.249).

If this is the expected behaviour for devices L2-attached to the vPC BGW (no site-internal VTEPs), then what is the advantage of the Multisite cnfiguration in this scenario; do we still have Multisite storm-control in operation?