BGP not dropping route

[christopher_hall]

Problem:

We have setup IPSLA on a core switch and are having problems with BGP not dropping Route_1.

Setup:

We've setup IPSLA (on Switch_01) in our network to determine if a route, Route_1, is available. If it is available, then traffic flows normally. All locations that need to use the services provided by Route_1 learn about this through BGP. If Route_1 is not available, then Route_1 is dropped and Route_2 is used, which points to an alternative location in another LAN across our WAN.

It appears that IPSLA is working correctly. We unplug the router that connects to the services requested through Route_1 and the route is dropped by Switch_01 and it uses Route_2 to direct traffic to WAN_01. The problem is that WAN_01 (connected directly to Switch_01) never drops Route_1 from it's routing table and I don't think that Route_2 gets updated by BGP, so all 12 of our locations continue to come to our primary location where IPSLA is setup. Configs are attached.

Equipment:

Switch_01 = Catalyst 3560

WAN_01 = 2811

BGP:

Switch_01 and WAN_01 are bgp neighbors.

So the million dollar question(s) is/are: how long does it take for bgp to converge? As mentioned above, the WAN_01 is directly connected (is BGP neighbors with) Switch_01. We've had the router unplugged for about 15 minutes. The behavior we're experiencing is that when Switch_01 chooses Route_02 and takes the next hope to WAN_01, WAN_01 sends that traffic back to Switch_01. I can't ID any static routes that would be sending this back, nor is the Default Gateway doing do.

Thanks,

Chris

[Jon Marshall]

Chris

Yes, much closer

if the route to the 10.15.1.0/24 from the Ops switch should fail, that statement would drop out and the route from W Columbia (which was already being advertised) would be used througout the WAN.

This is the bit i am still not sure about ie. does the W Columbia switch run BGP between itself and the W Columbia router ?

You say it is already being advertised but if the W Columbia router was advertising 10.15.1.0/24 into the AVPN cloud what stops a remote site picking the preferred path via the W Columbia link ?

Jon

[christopher_hall]

The switch and router are peers. Below is a snippet of the bgp config from the w columbia router: The 10.20.128.1 is the data vlan for the w columbia switch. The other neighbor (obviously) is how it peers with the wan.

neighbor 10.20.128.1 remote-as 2386

neighbor 192.168.0.97 remote-as 13979

You say it is already being advertised but if the W Columbia router was advertising 10.15.1.0/24 into the AVPN cloud what stops a remote site picking the preferred path via the W Columbia link ?

Ah! The whole reason for the post!! The only thing I can say (assume) is that the others never see the W Columbia link as preferred.

Chris

[Jon Marshall]

Chris

Ah! The whole reason for the post!! The only thing I can say (assume) is that the others never see the W Columbia link as preferred.

If possible could you post the configs of the W Columbia router and switch ? That way we could see if it is being advertised.

The other way would be to check on the W Columbia router ie.

"sh ip bgp neighbors x.x.x.x advertised-routes" where x.x.x.x is the AVPN peer router.

It would be useful to know how this is setup.

Jon

[christopher_hall]

See attached.

[Jon Marshall]

Chris

Just one other question. Are all the remote site routers that use BGP to peer with the AVPN cloud Cisco routers ?

Jon

[christopher_hall]

All of our sites use Cisco routers.

[Jon Marshall]

Chris

Thanks for the configs. Again a couple of points -

1)  You are doing AS prepending on the W Columbia switch towards the W Columbia router. However it doesnt look from the configs as though you are including 10.15.1.0/24 in that.

In addition the networks that are being subjected to AS prepending don't seem to be advertised on the Ops switch as well which i would have expected so it's still not entirely clear how it is working.

2) Each switch (Ops and W Columbia) seem to be peering with their equivalent router but they also have another peering.

Is this to the respective JConnect routers ?  eg from the Ops switch -

router bgp 2388

bgp log-neighbor-changes

neighbor 10.20.200.2 remote-as 65200 <-- this is the Ops router

neighbor 208.61.216.1 remote-as 2386

neighbor 208.61.216.1 ebgp-multihop 3

neighbor 208.61.216.1 update-source Vlan1

not sure what the 208.61.216.1 neighbor is ?

from the W Columbia switch -

router bgp 2386

no bgp log-neighbor-changes

neighbor 10.20.128.110 remote-as 65213  <-- this is the W Columbia router

neighbor 12.96.115.65 remote-as 2386 router bgp 2386

not sure what the 12.96.115.65 neighbor is ?

What i am not clear on is why you are AS prepending for networks that only seem to be advertised from the W Columbia router into the AVPN cloud.

As for the specific 10.15.1.0/24 it would appear it is being advertised from both the Ops and W Columbia routers but i am not sure how the primary path is always used because as far as i can see they go out as equal paths.

I will have a more thorough look at the configs but i can't help feeling i am missing something because they are not doing what i expect them to be doing.

I'm not trying to confuse the issue, i just don't like offering solutions if i don't fully understand the setup.

Jon

[christopher_hall]

This is the internet router for Ops: neighbor 208.61.216.1 remote-as 2386.

This is the internet router for W Columbia: neighbor 12.96.115.65 remote-as 2386

At one time, we had split our internet traffice between the two locations. When we went from NetVPN to aVPN (AT&T), we tried to replicate how our traffic was routing. Our vendor could never get it together, so we dropped it and now have internet traffic coming to Ops (except for W Columbia).

Thanks for all your time and effort.

Chris

[Jon Marshall]

Chris

Okay that makes sense. Like i said in my last post i think you should just be able to change the static routing on the Ops switch.

The main thing i still can't understand is why the Ops path is always chosen for the 10.15.1.0/24 subnet because i cannot see anything that makes it preferred over the W Columbia one. I may have missed something in the configs so i'll look again.

Jon

[Jon Marshall]

Chris

I appreciate you just want an answer so the short answer is yes, you need to implement the routes as you suggested on the Ops and W Columbia switches. If the 10.15.1.0/24 is being advertised out already, and it looks like it is, and for some reason the Ops path is always chosen then i can't see you modifying the static routes actually changing that.

There may be unexpected consequences although i can't see them at the moment but as i can't fully match up the configs with what i expected to see there may be an element of risk to it.

Jon

[christopher_hall]

As I posted in my last message, I appreciate the time and effort you've put into this. I guess we could give it a shot and monitor the situation. If we have any odd behavior, then we'll reverse the changes.

Chris

[Jon Marshall]

Chris

As I posted in my last message, I appreciate the time and effort you've put into this.

No problem. Sometimes posters just want answers rather than keep answering further questions and i realise there may be some pressure to get this done so i wasnt sure if that was the case here.

So there was no offence intended.

Jon

[christopher_hall]

I understand. No offence taken. This has been lingering and we do want this resolved, but it's not necessarily production stopping. I'm always open to learn something new, so if you have further thoughts, I'm here. In the meantime, we will try to test and I'll post back results.

Chris

[Jon Marshall]

Chris

Okay, i looked at the configs again and now i understand why the 10.15.1.0/24 always goes to the Ops router.

You have a "network 10.15.1.0 mask 255.255.255.0" entry on the W Columbia switch but there is no static route for it.  So if the link from JConnect goes down and the Ops switch removes the tracked route then the W Columbia will not be advertising this route to the W Columbia router and so it will not advertise to the AVPN cloud.

This is because for a network defined by the "network ..." command under the BGP config to be advertised there must be a matching route in the IP routing table.

So you are going to need a static route for 10.15.1.0/24 on the W Columbia switch pointing to JConnect2. Then you need to add 10.15.1.0/24 to the route-map DefaultRoute so that the path is prepended with multiple instances of the AS PATH.

That way both the Ops router and the W Columbia router will both be advertising out 10.15.1.0/24 but if the link is up the Ops router path should be preferred. 

Jon

[christopher_hall]

Today, we added the following to the W Columbia switch:

ip route 10.15.1.0 255.255.255.0 10.20.128.110 track 1

ip route 10.15.1.0 255.255.255.0 10.20.128.16 10

I'm not real clear on why this was done, but based on your information above, we should:

1.) Add a static route on the w columbia switch pointing to the 10.15.1.0/24:

     ip route 10.15.0.0 255.255.255.0 10.20.128.16 250

2) If our Route-Map looks like this:

route-map DefaultRoute permit 10

match ip address 5 10 20 25 15 30

set as-path prepend 2386 2386 2386

How would I add the 10.15.1.0/24 to it?

Chris