HSRP + BGP - LAN interface failover occurs but BGP fails to work

[gary.fortner]

I have 2 ISP's with 2 separate routers. 1 - 3845 and 1 - 2801. The routers are connected to a 2900 Catalyst switch stack on the LAN side. My Goal is to have BGP "follow" HSRP. When HSRP fails over I would like BGP to failover as well. Currently if I test failure of the LAN interface on the master router, HSRP works correctly but BGP fails to route traffic.

I have attached modified examples of the configs from both routers.

[Joseph Nelson]

Hi Gary,

Suppose router_a connects to ISPA; router_b connects to ISPB. Is your goal to have router_a bring down its eBGP session with ISPA if router_a is no longer the HSRP active router?

If so, I don't think there's a native way to do this, however, you may be able to use EEM scripts to bring down the session. 

See this article: http://notalwaysthenetwork.com/2014/02/05/monitoring-hsrp-failover-with-eem/

The solution depends very much on your goals. Post a topology and the results of your testing...

[gary.fortner]

j0nelsonCCNA,

Thanks for the response.

You are correct with your assumption. I will look into EEM scripts.

I have also found other options from my research. Would it be possible to:

- On router_a create an IP SLA process which starts PINGing the eBGP peer every 5 seconds.

- Next create an object which tracks this process.

- Next create a /32 static route to the peer using the peer itself as the next hop which uses the object status to validate itself.

- Next create a prefix to match this route.

- Then create a route-map which matches the prefix-list.

- Add this command to BGP “neighbor x.x.x.x fall-over route-map PEER_REACHABLE”

- Add a tracking statement to track the same object created above for BGP failover

Or is it possible to create loopback addresses and use iBGP?

Thanks

[Joseph Nelson]

Gary,

Some commentary on your other solutions:

• I'm assuming you meant you are pinging the "iBGP peer" for the "fall-over" configuration
• You don't actually need the IP SLA or the static-- you just need to redistribute connected into BGP at router_b. Create a loopback address on router_b and redistribute this /32 into BGP. If the path to router_b goes down or router_b goes down, the BGP session goes down and router_a implicitly withdraws the /32, then router_a falls over as planed

The problem with this solution is that if router_b goes down hard, router_a kills the BGP session and the entire site goes down. In other words, you fate-share the ISP connections which may not be what you want.

The reason I said there's no "native" way to do it is because under every scenario where you track the liveness of router_b ( i.e. IP SLA, Route injection/withdrawal, etc.), when router_b goes down or the path to router_b goes down, you end up unnecessarily bringing down the site ( by killing router_a eBGP session). I doubt you can justify the site going down, when you have redundant hardware/ISPs.

The problem is that from router_a's perspective you can only safely kill the BGP connection under a narrow set of circumstances:

1. You are sure ISPB is operational ( i.e. router_b has a default route from ISPB)
2. You are sure router_b is operational (i.e. if I go HSRP UNKNOWN is that because router_b is down or because the path to router_b is down?)

My #2 occurs if the path to router_b could be through N intermediate switches. If the link between two of these intermediate switch goes down. In this case, from a LAN perspective you have split-brain operation of the HSRP VIP. This means that you have a MAC flap. It may mean degrade service from an end-user perspective but at least they don't go down hard.

Can you post your topology. Don't need specifics, I just need to know how things are connected. Also are you running eBGP for a L3VPN (WAN) or is this for your Internet presence?

[gary.fortner]

j0nelsonCCNA,

This is for internet presense. Our firewall is dealing with VPN.

See simple topology attached.

[gary.fortner]

j0nelsonCCNA,

Ok, what if I leave the IP SLA tracking of the external address. Then on router_b add a IP SLA to track the external interface of my firewall (VIP).

ip sla 2
 icmp-echo 5.5.5.5 source-interface GigabitEthernet0/1
 timeout 500
 threshold 40
 frequency 3
ip sla schedule 1 life forever start-time now

Then I add EEM script to shutdown the external interface.

event manager applet SHUTDOWNEXTERNAL

event track 2 state down

action 1.0 cli command "enable"

action 1.1 cli command “config t”

action 1.2 cli command “interface gigabitethernet 0/0”

action 1.3 cli command “shutdown”

action 1.4 cli command “exit”

Then when track 2 is available again I use another EEM script to bring the interface back up.

event manager applet NOSHUTDOWNEXTERNAL

event track 2 state up

action 1.0 cli command "enable"

action 1.1 cli command “config t”

action 1.2 cli command “interface gigabitethernet 0/0”

action 1.3 cli command “no shutdown”

action 1.4 cli command “exit”

If this works, do I need to do anything on router_a?

Thanks

[Joseph Nelson]

Gary,

Now that your topology is in, the solution is a bit more obvious, here's what I would do:

LAN Side Failover

• On both router_a and router_b, create an IP SLA+Track (say, track 200) for the Firewall VIP
• In HSRP config, set it to track this new track object ( i.e. track 200 under gi0/1)
• Instead of null-routing your aggregates, null-route them and tie them to the track ( i.e. ip route <production_net> <mask> null0 track 200 )

Internet Side Failover ( Conditional BGP Announcement ) 

• Originate all your site prefixes on both router_a and router_b ( you already do this in your config; just adding to organize my thoughts)
• Redistribute connected into bgp on router_a ( for /30 which is off of gi0/0)
• Create conditional BGP announcement on router_b; your non-exist-map should include a prefix list that tracks router_a's ISPB /30; this makes router_b passive for ingress traffic. Also change your AS-4xx3 route-map to lower local preference on ISPB learned prefixes ( you implicitly do this on router_b, but you should make it explicit, say 120--it will help you you ever have to replace/soft cutover router_b). This makes router_b passive from a layer 3 perspective ( ingress and egress your site)

This should solve your problem because of the way you have things physically connected setup:

1. Failure: router_a goes down; router_b takes hsrp active role; begins announcing site prefixes to ISPB
2. Failure: Path to router_a fails ( lose 2960S member connected to router_b); router_b track 200 fails, it goes standby HSRP init/standby (doesn't matter because its connection to the LAN has failed. More important to fail BGP). BGP conditional announcement kicks in, but router_b doesn't announce your site prefix(es) to ISPB. This is because track 200 fails and your null-routes are removed. Having no valid route in the routing table, router_b withdraws its announcement to ISPB. (In reality, router_b never announces your prefixes to ISPB). ISPB re-routes traffic through router_a which we assume is up
3. Failure: Path to router_b fails ( loose 2960S member connected to router_a); same things happens as in 2. The null-route is removed, router_a withdraws the announcement to ISPA; router_b starts to announce to ISPB
4. Failure: FW Active member fails; VIP moved to standby member, no one cares :)
5. Failure: FW Passive member fails; no one cares

That should just about do it. Let me know if you need a config example and I can provide you one based on the configs you gave.

Let me know your thoughts...

Joe

[gary.fortner]

j0nelsonCCNA,

I like the solution. A config example would be very helpful if you don't mind.

Thanks,

Gary

[Joseph Nelson]

@Gary,

See attached document. Some disclaimers:

• Its the Internet, failover will not be fast. It will not be pleasant. In most cases, ( even in L3VPN scenario and on the open Internet), I've seen BGP converge in ~2 minutes. This depends on how your ISPs are connected and how your users are connected to the internet. (Waves hand vigorously, its the Internet, such is beyond the scope of this example :)
• Because of point 1 above, real-time traffic will drop during failover. But the failover is automatic so no 5am surprise while your stuck in traffic on the way into the office. Failback is also automatic ( although some people prefer users not to have two hits if at all possible. If that's the case, this isn't for you. As soon as that other active router returns to service, BGP will failback
• You should test the crap out of this in a lab environment there may be idiosyncrasies in your network I did not account for ( goes without saying)
• I would remove your track of "4.4.4.4" in the original configuration. Pinging stuff on the Internet is notoriously unreliable. More important, if it causes HSRP to switch _AND_ there's only a problem with the path to "4.4.4.4", you will have asymmetric routing within your network (i.e. egress from your site, you'll take router_b path, ingress to your site will take router_a path). Not totally as bad thing but something to be aware of. Whats more, egress path from internal network will go to router_a, come back on the 2960S stack, and then hit router_b. Again not totally a bad thing but something to think about. You can either add a direct link between router_a and router_b or track additional stuff ( I personally track Google and Microsoft's anycast DNS servers). The former solves the "back through stack" problem; the latter solves the "no one cares about pings / transient internet pathing problem"

One last thing. It occurred to me: What happens in this scenario:

• router_a ebgp with ISPA is up. ( Or router_a bgp is down but link is up/up?)
• router_a is up as well
• From router_b, path to router_a is up

You get a blackhole for your site prefixes. This is because router_b still does not advertise your site prefixes because it still has the ISPA connected /30 in is BGP RIB. The workaround is to condition the announcement on the existence of prefixes originated from ISPA directly ( see doc).  If you don't have such routes, you don't have a direct connection to ISPA and that verifies that router_a is no longer a viable ingress/egress point.

I'll make one last commentary: All these mechanisms -- EEM scripts, BGP conditional announcement, IP SLA tracking, etc-- add complexity. Not strictly a bad thing, but I like to keep things simple and avoid turning knobs where possible. That said, you wouldn't have to go through these theatrics to cover the failure cases if you had a direct link between router_a and router_b, I would seriously consider making it so.

If you can't spring for an EHWIC, you could recreate your edge network by connecting both interfaces on your edge routers to the 2960S. Both ISPs would connect to either 2960S switch. Logically, you establish a full mesh eBGP sessions between your routers and your ISPs. But that's just trading one complexity for another. I'd spring for the EHWIC and add a link between your routers...

Mark if you find helpful. Come back and show me your results as well...

[Joseph Nelson]

Also, see the link on BGP cond advertisement:

http://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/16137-cond-adv.html

[Joseph Nelson]

Gary,

Did you find that configuration example helpful?

[545985bryant]

j0nelsonCCNA

I know it has been over a year, but I wanted to let you know that I found this post very helpful.

Thanks for the effort you put in.