BGP not dropping route

[christopher_hall]

Problem:

We have setup IPSLA on a core switch and are having problems with BGP not dropping Route_1.

Setup:

We've setup IPSLA (on Switch_01) in our network to determine if a route, Route_1, is available. If it is available, then traffic flows normally. All locations that need to use the services provided by Route_1 learn about this through BGP. If Route_1 is not available, then Route_1 is dropped and Route_2 is used, which points to an alternative location in another LAN across our WAN.

It appears that IPSLA is working correctly. We unplug the router that connects to the services requested through Route_1 and the route is dropped by Switch_01 and it uses Route_2 to direct traffic to WAN_01. The problem is that WAN_01 (connected directly to Switch_01) never drops Route_1 from it's routing table and I don't think that Route_2 gets updated by BGP, so all 12 of our locations continue to come to our primary location where IPSLA is setup. Configs are attached.

Equipment:

Switch_01 = Catalyst 3560

WAN_01 = 2811

BGP:

Switch_01 and WAN_01 are bgp neighbors.

So the million dollar question(s) is/are: how long does it take for bgp to converge? As mentioned above, the WAN_01 is directly connected (is BGP neighbors with) Switch_01. We've had the router unplugged for about 15 minutes. The behavior we're experiencing is that when Switch_01 chooses Route_02 and takes the next hope to WAN_01, WAN_01 sends that traffic back to Switch_01. I can't ID any static routes that would be sending this back, nor is the Default Gateway doing do.

Thanks,

Chris

[Jon Marshall]

Chris

A topology diagram would help. If you are using BGP to learn the route why are you using IP SLA ie surely it would be easier to either run IBGP on the switch or redistribute the routes learnt from BGP into an IGP which the switch can then use. If the link goes down the route is lost and the alternate path is used.

That said i may not be understanding your setup correctly so perhaps a quick diagram/explanation would help.

In answer to your question BGP should converge a lot quicker than 15 minutes, something is obviously wrong somewhere.

Jon

[christopher_hall]

Hi Jon,

Here's one diagram that might help. I've got another that I can pass along, but would need to be modified so it makes sense.

Honestly, I know enough of BGP to make me dangerous. If IBGP could provide a better option, I would be okay with exploring that.

A little explanation of the diagram below:

1. Traffic would originate at the local networks

2. Traffic goes through the aVPN cloud to the Ops Router.

3. Next hop is Ops Switch. If the route to 10.45.45.1 is available,

then it takes the next hop to 10.20.102.125.

4. If not, the route is dropped and a new route is used (see the ip route statements in the image below)

5. Traffic would then go to Ops Router and out through the cloud to W Columbia, leading to the JConnect2.

Instead of heading out to the cloud, Ops Router is sending it back to Ops Switch.

1.

[Jon Marshall]

edited

[abdelseaudi]

I think this is because you are defining the route 10.15.1.0 via 2 static routes on the switch, so even if the 1st route is withdrawn, there is still another route in the routing table, and the switch is still advertising the network 10.15.1.0 to the wan router, saying that the switch is still the next hop for this subnet.

If we remove the static route "ip route 10.15.1.0 255.255.255.0 10.20.128.16 10", and if the ip sla is down, the static route "ip route 10.15.1.0 255.255.255.0 10.20.102.125 track 1" will be withdrawn from the routing table, and the WAN route coming from the WAN Router via BGP should be in the routing table instead. This is assuming the WAN Router is advertising the route to 10.15.1.0 to the switch.

[Jon Marshall]

Chris

I think Abdelmuhaimen is right in that the secondary route is installed in the routing table on the Ops switch and then advertised to the Ops router.

You need to the Ops router to be learning the route for 10.15.1.0 via the AVPN cloud and not from the Ops switch if the link goes down.

If you look on the W Columbia router and so a "sh ip bgp neighbor x.x.x.x advertised-routes" where x.x.x.x is the Ops router peer do you see 10.15.1.0/24 being advertised.

But i may not be as simple as getting that advertised because it's not clear how the clients come in. When you say source LANs are they via the AVPN cloud ?

Are both the Ops router and the W Columbia router meant to advertise 10.15.1.0/24 into the AVPN cloud ?

If they are do you always want the Ops router to be the preferred path.

Need to understand the full setup before any changes can be made.

Jon

[christopher_hall]

When I do a 'sh ip bgp neighbors x.x.x.x advertised-routes, I get: No such neighbor or address family.

[Jon Marshall]

Christopher

Is the W Columbia router not running BGP ?

Regarding your issue, it is the extra static route on the Ops switch. But it is not obvious what you want to happen. If the link fails and there is no path via the Ops switch then why send traffic in to the Ops router only to be sent back to the W Columbia router.

Is the W Columbia router in the same site and does it have a direct connection to the Ops router ?

To me it makes sense that if the Ops router path has failed then traffic should come in from the AVPN cloud direct to the W Columbia router. For this to happen you need to advertise out the same 10.15.1.0/24 network into the AVPN cloud from the W Columbia router but prefer the route to the Ops router unless that link fails.

Is there a reason your run EBGP between the Ops switch and the Ops router ie. why not just run EBGP on the Ops router ?

Do any of the devices on the right hand side of your diagram  run BGP ie. are they setup the same way as the devices on the left hand side ?

One final question. Do you know why you are running IP SLA ie. to me it would make more sense to have the L3 device connected to 10.15.1.0/24 to run an IGP with the JConnect router and advertise that subnet. The JConnect router runs an IGP with the Ops switch and advertises that route. That way on the Ops switch you do not need static routes and IP SLA, if the link fails anywhere the routes is no longer received by the Ops switch and is then no longer advertised into BGP

I'm not saying this is the thing to do as there may be good reason why you are using IP SLA, i was just wondering ?

It's very difficult to suggest a solution without knowing what the policies are in terms of what can run what and whjich way traffic is meant to flow

Jon

[christopher_hall]

Thanks Abdelmuhaimen & Jon for your insights.

The bigger picture of our network is this: we have 14 total locations, one of which is our primary data center (Ops) and the other a secondary/DR site (W Columbia). The application/service we're running is ATMs and all locations besides Ops will have one. Each of these sites will come back through the AVPN cloud to Ops. When traffic hits the 10.20.102.125 address, an IPSEC tunnel is created to reach it's final destination.

The normal process for this traffic is to come to the Ops location. If the route is unavailable, traffic should go to the secondary location. One question I have is if the primary route and secondary route are both advertised at the same time, how does traffic know to go out the primary route?

Hope this makes sense.

Chris

[Jon Marshall]

Chris

One question I have is if the primary route and secondary route are both advertised at the same time, how does traffic know to go out the primary route?

You need to manipulate the BGP route with either AS prepending or using MED. MED should be used only if you want to influence traffic in the neighboring AS. Basically both the Ops router and the W Columbia router need to advertise out the 10.15.1.0/24 subnet but you use one of the above to make sure that the remote site routers running BGP choose the Ops route unless the Ops router stops advertising it.

If you do this then there is no need for traffic to go into the Ops router -> Ops switch and then back across to W Columbia if the link has failed, instead it can go straight to the W Columbia router.

Do you use the same AS at all sites ie. the AVPN cloud will have it's own AS but for each site are you using the same AS or do you have a different AS per site ?

Jon

[christopher_hall]

Jon,

There are two of us working on this, so the advertising of both static routes has changed. The W Columbia router is learning the route to 10.20.102.125 (Ops location) from the AVPN cloud.

Each location has a different AS, but yes, the cloud has it's own. And each of the locations peer with this AS.

Chris

[Jon Marshall]

Chris

There are two of us working on this, so the advertising of both static routes has changed.

Not sure i understand ?

The W Columbia router is learning the route to 10.20.102.125 (Ops location) from the AVPN cloud

But where does it learn the route to 10.15.1.0/24 from ? It is this network ie. 10.15.1.0/24 that you want to have failover for isn't it ?

It must have a path to 10.15.1.0/24 via it's own links or there would be no point in your sending the traffic back from the Ops side if the link has failed. I am not sure where the 10.20.102.125 route comes into it. Presumably the JConnect2 router is also configured for IPSEC as a backup ?

If the W Columbia side can be used a backup to get to 10.15.1.0/24 then the easiest solution, as discussed, is to remove the second default route on the Ops switch ie. the one that tries to send it back to the Ops router and have both the Ops router and the W Columbia router advertise out the 10.15.1.0/24 network to the AVPN cloud and make the Ops route the preferred route.

If the link goes down on the JConnect router then the Ops switch removes the tracked route. Because there is no longer a route to 10.15.1.0/24 in the IP routing table (remember you have removed the second static route from the config) BGP will not then advertise the 10.15.1.0/24 network to the AVPN cloud. So the remote sites will use the other route via the W Columbia router.

When the link comes back up the Ops switch will reinstall the tracked route, the Ops router will then advertise it via BGP to the AVPN cloud and because it is the preferred route the remote sites will then switch back to using the primary path.

Jon

[christopher_hall]

Jon,

My head is a bit 'cloudy' due to a cold. Let me try this again.........

The W Columbia router is learning the route to 10.15.1.0/24 from the AVPN cloud. And yes, this is the network we need failover for. Disregard the comments on 10.20.102.125.

The W Columbia switch does have static routes pointing to 10.15.1.0/24. And the JConnect2 is identically configured (and managed by our vendor) as the JConnect at Ops.

So the solution you think would include something like this (as far as static routes):

Ops Switch:

ip route 10.15.1.0 255.255.255.0 10.20.102.125

W Columbia Switch:

ip route 10.15.1.0 255.255.255.0 10.20.128.16 250

Since we use BGP to update routes, if the routing statement on the ops switch is no longer available, then everything will automagically go to the statement advertised by the W Columbia switch (which should already be in the routing tables). Sound about right?

Thanks,

Chris

[Jon Marshall]

Chris

No problem.

Couple of things -

1) The W Columbia router needs to learn the route from inside as well. If it is only learning it from the AVPN cloud if the Ops router stops advertising that route then how will the W Columbia router know how to get to 10.15.1.0/24 ?

In addition the W Columbia router needs to learn it internally so it can also advertise it out to the AVPN cloud. So i assumed you had a similiar setup on the W Columbia side as the Ops router side in terms of BGP etc. Is this not the case ?

I'm also assuming that any device that needs access to 10.15.1.0/24 within the W Columbia side of your diagram (and there might not be any devices) would go direct ie. it wouldn't go out across the AVPN cloud to the Ops router.

2)  You still need to track the route on the Ops switch so you can remove it if the link from the JConnect router goes down. If you don't track it the Ops router will continue to advertise it even if the link has failed.

Jon

[christopher_hall]

1) The W Columbia router would (also) learn the route to 10.15.1.0/24 from the W Columbia switch. W Columbia is setup similar to Ops in that there's a core switch that controls the flow and a wan router. And I do believe that devices needing to access 10.15.1.0/24 from W Columbia would go direct from that site, instead of to Ops.

2) So the config, including the track statement, would look like:

----------- Ops Switch -----------------

track 1 ip sla 1 reachability

ip route 10.15.1.0 255.255.255.0 10.20.102.125 track 1

ip sla 1

icmp-echo 10.45.45.1 source-ip 10.20.102.1

timeout 1000

threshold 500

frequency 3

ip sla schedule 1 life forever start-time now

ip sla enable reaction-alerts

---------- W Columbia Switch ---------------

ip route 10.15.1.0 255.255.255.0 10.20.128.16 250

The routers at Ops and W Columbia would get updated via their respective switches, such that, if the route to the 10.15.1.0/24 from the Ops switch should fail, that statement would drop out and the route from W Columbia (which was already being advertised) would be used througout the WAN.

Getting closer?

Chris