EVPN Vxlan Layer2, adding a gateway on a VTEP doesn't seem to work

f00z · ‎05-05-2020

I have a EVPN layer2 only lab setup and I'm trying to do something simple which is not working.

Example 2 spine 8 leaf, nexus 9300

Everything is layer 2 except on leaf#8 i add vlan SVI:

vlan 100

vn-segment 10100

int vlan 100

ip address 60.0.0.1/24

The issue is that the MAC address of this SVI does not get inserted into EVPN locally. This SVI can send out arp request and gets a reply and can send IP traffic to an ip address anywhere in the EVPN domain on this VNI, but the traffic can't come back to it because the MAC address never gets populated.

Reading lots of documents this seems like it would work without a hitch, but it's not working. The only mac address i see populated from this VTEP is 0200.0000.0a0a.

I see there is an option in BGP called 'advertise-system-mac' which seems like it would do EXACTLY what I want to fix the problem, however of course it's locked out:

Cannot configure advSysMac. 'vendor controller interop' is not enabled

And of course the vendor controller interop is locked by a license.. sigh

?? advertise-system-mac Advertise extra EVPN RT-2 with system MAC

I would like it to do this so it knows where to send the replies. This works fine on other vendors, not sure why it is such an issue here.

Anyone have any ideas on how to get this to work? This would be a border leaf or gateway to other networks but doesn't seem to work.

Documentation and presentations seems to suggest this *SHOULD* work (cisco live presentations and such) -- i.e. using centralized layer3 gateway in EVPN where most of the leaves can only do layer2 or have lower scale.

The only thing it would need to do to get it to work, is advertise the system mac (or svi mac).

Sergiu.Daniluk · ‎05-05-2020

Hi @f00z

Correct me if I am wrong, but to me this looks like a gateway for vlan 100/vn-segment 10100. Is this what it is?

If yes, then a simpler solution would be to configure distributed anycast gateway on all VTEPs.

Regards,

Sergiu

f00z · ‎05-06-2020

Yes it is a gateway for VN 10100. It should work and it doesn't.

Using anycast gateway is a MUCH MUCH more complicated solution, it's not a simple solution as you say. Not only does it require that I configure the gateway on every VTEP, it also requires every VTEP have VRF for that client. That is a huge configuration.

What if the leaves only support vxlan bridging?
What if the leaves don't have a huge routing table /scale for what I need?

What if I have hundreds of leaves? This is just a lab test.

Putting a gateway on one of the leaves as a border is a super-simple solution and fits my need and should work, but doesn't. The way we have it now a router is connected to a layer2 leaf, but the whole point of IRB is that the leaf should do both, and it isn't because the MAC isn't advertised. Simple fix, maybe a bug in the software? Or is there a tunable somewhere I can use to make it work?

Like I said this works on other vendors gear and I was hoping it would work. Any ideas? I know it will work if the route gets advertised for the SVI mac because that is what the other gear does with some tunables.

f00z · ‎05-06-2020

It's doing the weird/odd MAC thing for anycast gateway setup also..

2020 May 7 03:27:24 test-1 %BGP-2-EVPN_STATIC_MAC: bgp- [948] Static/learnt MAC address conflict for 0200.0000.0a0a in VNI 653200

Even when in anycast gateway mode on multiple leaves, they still advertise this weird mac address to each other instead of the actual SVI mac address. (nexus 9300 9.3.4 and 7.0.3.I7.8 ) ..

The system is inserting static mac address entries that you can't delete, or immediately come back. If i put in my own static mac address entry, it shows up in the table (show mac addr) but it disappears from the configuration. Definitely something buggy going on.

The right mac address is L3VNI mac table (along with the crazy 0200.0000.0a0a mac addr, where is this coming from??), but the right mac addr is not in the L2VNI and it's giving me duplicate mac log messages. Anycast gateway works fine, but it's not the use case I'm shooting for, and I'm seeing strange behaviour.

This is a very simplistic lab setup with ingress replication. Any input would be appreciated.

f00z · ‎05-09-2020

I believe these random mac addresses were the result of having the "feature fabric forwarding" command on, but i'm not 100% positive. I wiped out the lab devices and started over without using this feature and I don't see the weird 0a0a mac address. I think this feature is for fabricpath only and not for vxlan (as HMM is still enabled and vxlan is still working without this). There is no documentation supporting this. In some vxlan configuration guides online they have this feature enabled, and in some they don't. So I am curious as to what EXACTLY it does.

On another note, the mac address for the SVI STILL isn't being injected into the mac routes. Maybe it isn't supported on this platform (evpn centralized gateway). It is supported on other vendors platforms so I am curious as to why it isn't here (there are knobs to adjust it i.e. advertise-mac , advertise-system-mac, that type of thing). Also support for spine anycast gateway (or border anycast gw) also works on other vendors , there's a spine-anycast-gateway command under NVE but does not seem to do anything, and of course there's no documentation for it (maybe a new feature coming out?)

Has anyone tried to do spine or border anycast gw (where the leaves do layer2 vxlan gateway only) with EVPN?

f00z · ‎05-26-2020

Finally figured this out and it was what I thought originally, the nexus devices won't advertise the system mac or virtual mac to the evpn.

I tested on another vendor's hardware and they have tunables to redistribute the system mac and virtual mac into the evpn as a type 2, so this works. This is missing from nexus and probably all it needs for it to work properly.

If I fire up EXABGP and I advertise a type 2 route for the system mac of the nexus, it would work. I suppose that's a viable option but very silly when the software could do it itself.