Failover Using Script in BGP or static Routing

[TaimurKhan]

Hi,

our router is connected to two isp. ISP 1 and ISP2. ISP 1 is connected through microwave and ISP 2 is connected through Fiber Link(provided by some vendor)..

ISP1 gives us P2P Pingable IP.

ISP2 gives us P2P Private IP and can be only pingable from our router.

BGP is running with both ISP.

i am having default route to both ISP and there is track on these routes so in case any link goes down all traffic flows in other direction and in normal circumstances do Load balancing.

recently we faced service issue from isp2. our bgp was up with isp2 but there was upstream issue from isp2. but still my router was having bgp up with isp 2 and still it was sending traffic that way. which was causing issue for our services.

i configured track on P2P ip and at that moment as P2P ip was up so my router does not find any problem and was doing its normal work. so i asked my isp2 to give me some IP or way to monitor your services as in case something happens again so our router can track that but they refused to do so.

so i need some other way to tackle this problem if it happens in future.

note that when that issue happened i was having only bgp up with isp2 but not receiving any prefixes.

[MHM Cisco World]

this is simple and no need specific IP from SP to track, only you need track 8.8.8.8 and use specific interface for IP SLA (additional you can use peer IP for IP SLA icmp-echo track also)

ip sla 8
icmp-echo 8.8.8.8 source-interface GigabitEthernet0/1
threshold 800
timeout 1000
frequency 1
ip sla schedule 8 life forever start-time now
ip sla 9
icmp-echo 31.13.65.1 source-interface GigabitEthernet0/1
threshold 800
timeout 1000
frequency 1
ip sla schedule 9 life forever start-time now

track 5 list boolean or
object 8
object 9
delay down 20 up 50
!
track 8 ip sla 8 reachability
delay down 5
! track 9 ip sla 9 reachability
delay down 5

event manager applet BGP_NEIGHBOR_DIRTY
description SHUT DOWN BGP NEIGHBOR IF RTT OVER 1000 FOR 5 SECONDS
event syslog pattern "5 list boolean or Up -> Down"
action 1.0 cli command "enable"
action 1.1 cli command "configure term"
action 1.2 cli command "router bgp xxxxx"
action 1.3 cli command "neighbor xxxxxxx shutdown"
action 1.4 cli command "end"
event manager applet BGP_NEIGHBOR_Clean
description BRING UP BGP NEIGHBOR IF RTT UNDER 1000 FOR 50 SECONDS
event syslog pattern "5 list boolean or Down -> Up"
action 1.0 cli command "enable"
action 1.1 cli command "configure term"
action 1.2 cli command "router bgp xxxxx"
action 1.3 cli command "no neighbor xxxxxxx shutdown"

[TaimurKhan]

hi ,

thanks for the reply,

i have checked and when i make source this interface and ping 8.8.8.8 or any other public ip, i can not ping. i reaches to isp router but then drops beyond that.

[MHM Cisco World]

I will run lab with many cases and I will  share here 

[TaimurKhan]

i tested that script in gns3

I tried to Ping google using loopback ip which is served through isp2, it works when i stop path to google  the ping has no response and it shutdowns the neighborship with isp2. But the problem is how it will enables it? as the source IP is served through ISP2 and neighborship is down

Kindly suggest

[Georg Pauwen]

Hello,

this can be tricky. What (if any) prefixes are you receiving from the ISPs ?

[TaimurKhan]

hi,

Thanks for the reply,

actually i am getting some prefixes from that isp but they are  the best only and not valid currently.

[paul driver]

Hello
you can apply a conditional default static route by tracking a public ip address such as google but you need to make sure that tracked ip isnt then reachable via the secondary isp because the tracking and failover will not work correctly - review here (example:2)

However your op suggests it’s not the default that is the problem as it seems that your primary default was active and your are actively policy routing some clients via isp2  and it’s that policy routing traffic  that was being black holed  when the isp2 experienced outage 

Can you confirm this is the case?

[TaimurKhan]

Hi,

The thing is the public Ip might be pingable through the other provider and i can define source of ping in sla. But as i checked my isp is not allowing me to Ping ip like google or other major pingable Ip. i opened ticket with them.

actually it is sort of PBR sort of scanario i am routing some of my traffic back and forth from one isp and some from other.

so when issue happend still my router was sending traffic to the isp that had issue as i was tracking the P2P ip.

[paul driver]

Hello
In most cases you would not get away with just source ipsla as again your source interface address may not be down or it may also be reachable through either isp hence why I like using a route-map with ipsla and nulling out reachability.

As for a icmp node to track from the "secomdary" ISP are you saying you are not able or are being denied ping to anything being advertsied from that ISP even the networks your policy routing towards?

[sidshas03]

It sounds like you experienced an issue with ISP2's upstream network that caused your router to continue sending traffic to ISP2, even though there was a problem with their network. This is a common issue with BGP, as BGP only monitors the availability of prefixes advertised by the upstream ISP, not the overall health of their network.

One way to tackle this problem is to implement BGP route dampening. Route dampening is a technique that allows you to suppress BGP updates from a particular upstream ISP if they are flapping or unstable. By suppressing updates from an unstable upstream ISP, you can prevent your router from sending traffic to that ISP until the issue is resolved.

Another solution is to implement an external monitoring system that can monitor the overall health of both ISP1 and ISP2's networks. This monitoring system should be able to detect issues with either ISP's network and send alerts to your router to adjust its routing accordingly.

Finally, you could also consider implementing policy-based routing (PBR) on your router. PBR allows you to specify traffic routing rules based on various criteria such as source IP, destination IP, and application protocol. With PBR, you can manually specify which ISP to use for specific traffic flows, instead of relying solely on BGP. This can give you more granular control over your traffic routing and allow you to react more quickly to network issues.

[TaimurKhan]

Hi,

Thanks for the reply

BGP route dampening : actually the P2P is not flapping and we had bgp session up when the issue happened.

Monitoring : The IP provided by the isp is P2P and do not have any way to monitor isp end. and it will be useless for me as still it might be up all the time as issue happened in services from upstream.

PBR : i have PBR but when issue happened then that time BGP was up and my router does not feel any problem and sending traffic to that ISP.

[sidshas03]

Thank you for providing more information.

Regarding BGP route dampening, if the issue is not related to P2P flapping, then it's possible that the BGP route dampening settings are too aggressive and causing legitimate routes to be suppressed. In that case, you may want to adjust the settings or disable route dampening altogether.

As for monitoring, it's understandable that you may not have visibility into the ISP's network. However, it's still important to monitor your own network to identify any issues that may be affecting your services. This can include monitoring traffic patterns, device performance, and service availability. You can also consider setting up alerts to notify you when certain thresholds are reached or when specific events occur.

Regarding PBR, it sounds like it may not be helpful in this particular situation since BGP was up and your router was sending traffic to the ISP. However, PBR can still be a useful tool for routing traffic based on various criteria, such as source/destination IP address or application type.

If the issue is ongoing or persistent, it may be worth reaching out to the ISP to see if they can provide any insight into the issue or if there are any known problems on their end.

[MHM Cisco World]

the route damping is not meaning the BGP peer up/down 
but the route that BGP peer advertise is up/down 
even with PBR still you need away to check the BGP Peer status which make us return to first point

[sidshas03]

You are correct that route damping in BGP is not directly related to the BGP peer being up or down. Rather, route damping is a mechanism used to suppress the propagation of unstable or flapping routes in BGP.

However, you also make a valid point that even with policy-based routing (PBR), it is still necessary to monitor the status of BGP peers to ensure that the correct routes are being advertised and used for traffic forwarding.