Trouble with dual isp failover configuration

[thakissick0986]

Hello All,

I am having a little trouble fully figuring out the necessary configuration to make this work. 

Currently I have a Catalyst 9200 L3 switch which is physically connected to an SDWAN ISR router on port te1/1/1. Port te1/1/1 is on VLAN 800 and VLAN 800 is set with ip address 10.70.254.250. The LAN interface on the ISR router is 10.70.254.249. I also have a 5G/LTE router physically connected to interface gi1/0/48. Port gi1/0/48 is set to VLAN 801 and VLAN 801 is set to ip address dhcp. It was giving VLAN 801 a DHCP address of 192.168.1.37.

The connection to the Internet (Comcast circuit) via the ISR router should be considered "primary" as this is our SDWAN and inter-site connectivity. The 5g Router is "backup" and will only provide Internet (no inter-site connectivity).

The L3 switch (with ip routing turned on) initially had only the following default route setup in the config:

ip route 0.0.0.0 0.0.0.0 10.70.254.249

So all is working/well if the Comcast gateway (plugged physically into the ISR router) is up/has connectivity to the Internet.

The following is what I did to try and create a failover to the 5G router as a backup:

- Created an ip sla with icmp-echo to 8.8.8.8 with timeout 5000 frequency 5

- Created track 1 ip sla 1 reachablity

- Started the track with lifetime of forever

Initially I created it with a source-interface of te1/1/1 but show ip sla summary was showing a timeout every time - I had to change the source-interface of the ip sla to VLAN 800 and then it was showing me a successful return from 8.8.8.8

So I thought that part of it was good.

I then did:

ip route 0.0.0.0 0.0.0.0 10.70.254.249 track 1

and 

ip route 0.0.0.0 0.0.0.0 gigabitEthernet 1/0/48 dhcp 254

One thing I thought was odd here is that after these commands I did sh run | i ip route and it showed 3 lines:

ip route 0.0.0.0 0.0.0.0 10.70.254.249 track 1

ip route 0.0.0.0 0.0.0.0 10.70.254.249

ip route 0.0.0.0 0.0.0.0 gigabitEthernet 1/0/48 dhcp 254

I figured it would have just 'replaced' the entry without the track 1 at the end with the one with track 1 but it did not. I am not sure if this is part of the problem or not.

To test, I powered off the Comcast gateway connected to the ISR router. I waited a few moments and ran sh ip route and it did show

gateway of last resort via 192.168.1.1 via DHCP or something similar 

So that part of it did work - however, I could not get the route to switch back to the 'primary'

I plugged the Comcast gateway back in, and after a few minutes the link light on the ISR router came back on. I figured the SLA reachability would go back to 'Up' and that it would switch the default route back... but it never did. I waited a solid 10 minutes to make sure everything was up.

I ran ping 8.8.8.8 source vlan 800 and it was timing out... 

I know the Comcast Internet is up. I removed the IP SLA and track, removed all the ip route 0.0.0.0 0.0.0.0 entries, and put the original ip route 0.0.0.0 0.0.0.0 10.70.254.249 back in, and instantly was able to ping 8.8.8.8 as normal.

Why did the SLA reachability never return to Up and switch back to the other default route?

[Flavio Miranda]

@thakissick0986 

 I think you should not ping 8.8.8.8 because as long as you have internet working this condition is always true and  will not trigger IPSLA. 

 Try to ping the comcast IP address instead.  when you shut the comcast down, the IPSLA will be triggered as there will be no response. Then, it will be triggered again when the comcast start responding again. 

[thakissick0986]

Ok this makes sense partially... but I would expect to see the default route switching back and forth between the 2 if this was working correctly.

If track reachability fails, remove 0.0.0.0 0.0.0.0 10.70.254.249 from route table, add 0.0.0.0 0.0.0.0 gi1/0/48 dhcp 254 as default route. Once this occurs, 8.8.8.8 would become reachable again, causing the sla track to go from Down -> Up and the original route would be added back. This cycle would repeat itself until 8.8.8.8 was reachable from the original route again.

[Flavio Miranda]

 Well, need to test but as I said, as 8.8.8.8 can be reached from both links the trigger might not work. You need to use some IP which will stop responding and start responding after the link is back. 

Here an example config. Need to change the IP appropriated. 

track 8 ip sla 1 reachability

ip sla 1
icmp-echo <comcast> source-ip <some IP on switch with reachability to the target IP>
ip sla schedule 1 life forever start-time now

ip route 0.0.0.0 0.0.0.0 <comcast> track 8
ip route 0.0.0.0 0.0.0.0  <5G/LTE>

[balaji.bandi]

ping with source interface or IP to reachability check - check some example testing's done here :

https://www.balajibandi.com/?p=1643