Migration from HSRP to VXLAN

[neil_titchener]

Hi All,  Firstly thankyou in advance for taking the time to respond to my question.

I'm new to VXLAN and looking into how we can migrate from a HSRP default gateway to VXLAN using Anycast IP.  I have two questions but feel free to add anything you might help me.

1.  The documentation always uses multiple VRFs.  Is this required in an enterprise where there is one tenant?

2.  The diagram below shows what I'm proposing and I think there are two options how to migrate from HSRP to DAG.  First one is to set the MAC address for HSRP and use the same for DAG.  This option would see servers uses both HSRP and DAG as the default gateway for a short time until HSRP was shut down.  The second is to use a new IP for DAG and migrate servers, one at a time to the new Default gateway.  Are both these approaches valid?

The diagram shows only one local network with the current access and HSRP distribution switches with the new DAG distribution switches on the right.

Any suggestions would be very welcome.

Thanks in advance.

[m.kafka]

Hi Neil,

I'm afraid, this will not work smoothly. Overlay VLANs (VXLAN, VNI) are configured like this:

vlan 123
 name [choose your name]
 vn-segment 4567

I doubt, whether such a VLAN can be operated in both ways, VXLAN overlay and classic L2-VLAN, simultaneously.

Your approach is smart, promising a smooth transition. But I doubt, it will work this way.

Regards, MiKa

[f00z]

Yeah I mean, it's a hack, having the same mac address will cause a lot of flapping between ports but it would work provided layer3 access is the same.  For EVPN to use layer3 you want a layer3 vni, and to use layer2 a layer2 vni, to use EVPN with both you will have to make a layer2 and a layer3 VNI.   I know it sounds a little strange but it's just the way it works.  VXLAN is layer2 only so it creates one layer2 vrf  and one layer3 vrf  (so one MAC VRF and one L3 vrf) , there are assigned different VNI so the other vteps know what to do with it. It's more complicated than a traditional setup and i'd say the minimum you want to do is at least have two VRF/VNI one for l2 one for l3 .. It just makes setting it up a lot easier.. There are other ways to hack it up by using one vrf or trying to merge it together but honestly it's not worth the hassle and setting it up the way it was designed is the best way to do it. 

[m.kafka]

Hi f00z,

First your question from the original post:

1. The documentation always uses multiple VRFs. Is this required in an enterprise where there is one tenant?

No, you don't need VRFs always, only for multiple tenants with duplicate/overlapping L2 or L3 addressing or requiring traffic separation.

Cisco published a Design Guide about 2 months ago for the Nexus 9000 series "Migrating Classic Ethernet Environments to VXLAN BGP EVPN":

https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/migrating-classic-ethernet-to-vxlan-bgp-evpn-white-paper.html

If your platform doesn't support coexistence of HSRP and DAG, a hack could work if you don't have east-west traffic:

If your platform doesn't support coexistence of HSRP and Distributed Anycast Gateway, you might hack routing between core and distribution. This can only work, if your subnet has some unused IPs. Start with a new VLAN configured for VXLAN/DAG and use a high IP address for the DAG if the HSRP is a low address.

If DHCP is in use start with a small DHCP scope (e.g. a /27) for the VXLAN and dhcp exclude this on the classic VLAN, maybe you will need do dhcp release/renew on endpoints if a dhcp renew will not obey the newly excluded range. As soon as there are no active hosts in the excluded range you can continue:

For routing during the migration phase use more specific static routes for this /27 to the DAG based VLAN. Start moving endpoints to the new VLAN.

Repeat until all hosts are in the new VLAN. Increase the DHCP scope for VXLAN on each iteration while adding more dhcp excluded to the classic. When all endpoints are in the new VLAN revert to your preferred routing (e.g. OSPF) and remove static routes.

This will likely break east-west traffic...

Best I can think of....

(edit: using the same MAC address is not a good option: duplicate address detection)

[f00z]

I've tested these things in a lab setup and also done some real world migrations. In the lab I found a problem around every corner.  That is a good document by the way, it has a lot of good information but it doesn't really prepare you for the MANY unexpected issues that can happen.  

From all of my testing it's better to set up EVPN with the underlay and separate l2 (macvrf) and l3 (vrf) for each tenant even if there's only one tenant.  It's also better to convert it to layer2 first and not do ANY layer3 on it until everything is migrated , keep using your existing routers.   Then per vlan or per tenant depending on your setup, use a script(ansible, or whatever) to roll out the anycast gw all at once across everything (testing it in the lab first of course) for that particular vlan or tenant and take down the old routing svi.   I also ran into MASSIVE problems, and i mean like, seriously broken things don't work right at all problems trying to route single /32's with EVPN.. like say i wanted to move 1 ip address over to another vlan, it wasn't having it, minimum of /31 worked but /32 no. So many issues.  

Most of my implementations I didn't use anycast gateway at all, ended up being too much of a burden in most cases and made things 100 times more difficult to debug.  Best use case for it is if there's tons of e-w trafffic in different vlans/subnets/tenants talking, or if you lack ARP/adjacency scale on whatever device is doing the routing.  For most implementations I've done the only traffic going to the gateway is usually to the firewall or external, so it's almost always going to 1 node anyway , making DAG useless. 

Also not using vrf for evpn (i.e. mixing it in with the underlay) proved to be also very annoying to debug things.  Have to pick a standard and do it that way and keep it that way or it gets too hackish 

[m.kafka]

Hi f00z, Thank you for the reply!

Which lab do you use, CML, Hardware?

DAG is only necessary if you have IP mobility (VM migration) to keep the MAC of the default gateway and want it simple. EVPN will (should) announce the gateway MACs of VTEPs to all other VTEPs within a VNI. Each VTEP maintains a list of gateway MACs for each VNI and serve any gateway MAC address within this VNI locally (sh mac addr table shows a "G" in the first column). So DAG is not even necessary, it only makes thing nicer and easier. I remember having read this on one of the hundreds of pages, can't find it right away.

I'm still convinced that (L2 or L3) VRFs are not necessary but I believe it is difficult to subtract that feature from the configuration guides. Several configurations will move to a different place when using VRFs.

I have the feeling that either the feature (VXLAN) or the documentation is built around a special use case, spine/leaf with VRF and route reflector, and any other topology is difficult to implement.

Regards, MiKa

[00u1661g7uPHO4xrg5d7]

What is issue in just shutting HSRP in Ethernet and enabling Anycast in VXLAN ?

[NetworkRequests]

This is a live network with around 100 servers in each VLAN.  shutting down HSRP and enabling VXLAN will cause an outage. Changing the DG MAC address will also cause issues as some servers may not accept the new MAC address so we would have to check every server has updated it's ARP table and force an ARP update where needed.

The idea of introducing a new Default gateway removes those issues and allows servers to be migrated one at a time.  We will be purchasing Ciscos CLM shortly so I'll test out these solutions.

I'll post my results