Re: EVPN Multisite VRF communication across sites

j.a.m.e.s · ‎05-06-2020

Dear All,

I would like to get HostA-VRFBlue talking to HostB-VRFBlue and HostA-VRFRed talking to HostB-VRFRed without traffic passing though the firewall.

Will configuring EVPN Multisite help me? The two VRFs have multiple different VNIs and VLANs with AnyCast gateways at either site, but they share the same L3VNI number.

Kind regards

James.

f00z · ‎05-09-2020

If VRF blue has the same L3VNI at both sites, it should already be working, provided they are in the same EVPN domain.

Multisite is for larger scale by separating the domains (i.e. evpn domain site a, evpn domain 'shared for multisite', evpn domain site b). I'd say it's REALLY NICE TO USE for large setups especially with multiple domains (ASN) and to limit failure zones but to

do what you are asking it isn't necessary.

Have the route reflectors peer with each other and exchange information and it should just work in that case.

The firewall is only for inter-tenant communcation and since the vrf blue exists on both sides with the same l3vni that isn't considered intra-vrf, unless I'm not understanding your configuration.

If you could annotate VNI's and ASNs on your diagram that would help. Or include some configuration.

j.a.m.e.s · ‎05-11-2020

Hi f00z,

I have updated the diagram to provide VNIs/ASNs and to show better how to firewall is connected.

The firewall is peered with vf-red and vf-blue on the south side and vrf-common on the north side.
It announces a default south
It announces the vf-red/blue subnets north (into vf-common)
The border leafs talk EIGRP with the core using VRF-Lite and only the vf-common and the underlay have the neighbourship
So vf-common has the vf-red/blue subnets and is responsible for announcing them into the EIGRP core

I have updated the diagram to provide VNIs/ASNs and to show better how to firewall is connected.

This also shows the default route (the only path to the other site) from the leaf's perspective:

leaf01-A# sho ip route 0.0.0.0 vrf vf-common
[...]
0.0.0.0/0, ubest/mbest: 2/0
    *via 10.0.227.71%default, [20/0], 11w1d, bgp-65001, external, tag 65000 (evpn) segid: 103967 tunnelid: 0xaeee347 encap: VXLAN

    *via 10.0.227.72%default, [20/0], 11w1d, bgp-65001, external, tag 65000 (evpn) segid: 103967 tunnelid: 0xaeee348 encap: VXLAN

leaf01-A# sho ip route 0.0.0.0 vrf vf-red[...]
0.0.0.0/0, ubest/mbest: 2/0
    *via 10.0.227.71%default, [20/0], 11w1d, bgp-65001, external, tag 65000 (evpn) segid: 103966 tunnelid: 0xaeee347 encap: VXLAN

    *via 10.0.227.72%default, [20/0], 11w1d, bgp-65001, external, tag 65000 (evpn) segid: 103966 tunnelid: 0xaeee348 encap: VXLAN

.71 and .72 are the border leaf PIP, so in both cases the traffic is vxlan encapsulated and sent to the Bdr Leaf. Once on the border leaf, vf-red has to be decapsulated and sent to the firewall for filtering, then the packet hops out on vf-common. If the packet started out in vf-common it will just get decapsulated and sent to the core using the EIGRP route.

BdrLeaf01-A# sho ip route 0.0.0.0 vrf vf-common[...]
0.0.0.0/0, ubest/mbest: 2/0
    *via 10.0.129.85, Eth1/1.511, [170/281856], 11w1d, eigrp-2, external # Next hop Core
    *via 10.0.129.89, Eth1/1.511, [170/281856], 11w1d, eigrp-2, external # Next hop Core

BdrLeaf01-A# sho ip route 0.0.0.0 vrf vf-red[...]
0.0.0.0/0, ubest/mbest: 1/0
    *via 10.0.255.224, [20/0], 11w1d, bgp-65001, external, tag 65002 #Next hop is FW

I appreciate what you are saying about peering the Spines at either site, but will this work? Surely this would result in Site-A (for example) having the subnet routes of Site-B, but what would be the next-hop behaviour for these routes? Would the border leaf not just end up using the default, which continues to be the firewall?

Wouldn't MultiSite help me by building a VXLAN tunnel between the border leafs and bypassing the firewall for the subnet routes learned at the remote site?

f00z · ‎05-11-2020

Which devices are the EVPN route reflectors? Multi site would work of course here , but if it's only 2 locations with not a lot of scale you can also do it manually. The underlay network would have to be extended to both sites in that case and you peer the EVPN route reflectors (ebgp multihop peer) at each location so they share the information, and rewrite the asn or set manual route targets.

Multi site would help with this but isn't necessary. Multi site would make another evpn domain where all sites are connected to (border leaves) and lets you translate between the VNI's instead of doing something like vrf-lite or OTV.

There's a really good white paper here although it was written before multi-site, so it does not show that method:

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-738503.html#_Toc477465165

Again if you only have two sites, you can stretch (multipod) the fabric so both sites are in the same EVPN domain.

The main reason to set up multi-site or some type of DCI translation is to scale out past the 256 vtep limit. Most devices can only peer with 256 vteps, and the only way to get past that is the methods listed in this white paper or multi-site. All the multi site stuff really does is hide the vteps from the other domains, so you can have 256? domains with 256 vteps each all connected to a 'common' evpn domain , so the common domain only sees the border gateway vteps and nothing behind it. Common way of scaling any networking setup until the hardware eventually does way more scale.

j.a.m.e.s · ‎05-12-2020

Since it's an eBGP underlay, there are no route-reflectors but you can think of the 4x Spines as the route-reflectors because of:

address-family l2vpn evpn
 nexthop route-map RM-EVPN-LEAF-UNCHANGED_N_HOP
 retain route-target all
exit

What I'm concerned about is that from VF-A or VF-B, the only way out of the local fabric is via VF-Common and to hop into this VRF, the packet must follow a default route via the firewall.

Let's consider that I was able to get the routes of the Site B spines into Site A spines. What would the next hop on these routes be set to?

f00z · ‎05-12-2020

When you peer the sites together, the routes get imported into the vrf from evpn, since you are using the same VNI. If they are different ASNs you use the command to rewrite the route targets. This of course doesn't separate the vteps between the sites and so you have a 256 vtep (or whatever your hardware supports) limit for the entire network. The vrf blue routes for site b will automatically get imported into the vrf for site a (with auto route targets) and they will not use the firewall. This is the multi-pod design in the whitepaper, or also here: http://yves-louis.com/DCI/wp-content/uploads/2015/10/VXLAN-Multipod-geographically-dispersed-white-paper-final.pdf

No multi-site, so no need for special devices that can support it or licenses.

So the way out of the site, is through the border leaf underlay, (as on your core devices it says vrf common and underlay), I'm assuming the underlay extends to the other site so the border leaf/spines at site a can talk to border leaf/spines at site b via underlay. Peering will distribute vtep information , client leaf at site a will now see a vtep on client leaf at site b and encap traffic and send it there as if it were at site a, it won't know the difference.

This method, there is no way out of the fabric, you are extending the fabric between sites so it's all one fabric. Does that make more sense?