BGP route failover and failback to local preference

[Joe Conklin]

I am using a bgp config similar to the below. Works perfect for specifying one route to be preferable for specific source/destination traffic. However, if the preferred route route detects a link failover it will failover, but it will not fail back automatically. I've tried "clear ip bgp * soft in", "clear ip bgp * soft out", but only "clear ip bgp *" did the trick. How can I ensure a route with a higher preference is used without intervention if the link is restore?

router bgp 65001

 neighbor 10.50.0.1 remote-as 65005

 neighbor 10.100.0.1 remote-as 65003

 neighbor 10.100.0.1 route-map PBR_Voice_prefer_MPLS in

 neighbor 10.250.0.1 remote-as 65002

!

access-list 10 permit 10.25.0.0 0.0.255.255

access-list 10 permit 10.15.0.0 0.0.255.255

!

route-map PBR_Voice_prefer_MPLS permit 10

 match ip address 10

 set local-preference 200

[Francesco Molino]

Hi

per default bgp timers are 180s (3 min).

how long did you wait before doing clear ip bgp?

[Joe Conklin]

There is a 60 second keep alive timer before the 180 second hold down timer starts. To be safe I waited at least 5 minutes, and in some cases as much as 10. We basically simulated a link state failure between one of the two routes, primarily the one with the set preference. When we re-established it the route did not come back unless we restarted the bgp altogether.

[Francesco Molino]

Could you issue the command below and send the output?

show ip bgp neighbors x.x.x.x | begin capabilities

Thanks

[Joe Conklin]

Per your request see the attached outputs. Below is my topology as well, which I am sim in GNS3 to troubleshoot this using 7200 series routers. Below is some additional information you may find useless but I wanted to throw it out there.

I also switched the preference over to weight as I was thinking preference being sent to peers was part of the problem. Weight is not sent to peers, but regardless it had the same effect. The effect being once we sim link failure BGP does not set weight or preference unless BGP is completely reloaded.

EDGE-1

router bgp 65001
neighbor 10.100.0.1 route-map PBR_Voice_prefer_MPLS in

route-map PBR_Voice_prefer_MPLS permit 10
match ip address 10
no set local-preference 200
set weight 65535

EDGE-2

router bgp 65002
neighbor 10.100.1.1 route-map PBR_Voice_prefer_MPLS in

route-map PBR_Voice_prefer_MPLS permit 10
match ip address 10
no set local-preference 200
set weight 65535

Troubleshooting steps..

For giggles.. if I kill the MPLS in my sim everything failover to the Private ME no problem. If I restore the MPLS link between ASN 65003 and 65004  and wait 5 minutes and then kill the Private ME 10.25.X.X and 10.15.X.X is routable, but nothing else is routable once we failover again from the Private ME to the MPLS. If I remove the route-map from BGP everything works fine, excluding 10.25.X.X and 10.15.X.X preferring the MPLS line. It would thereby appear that when the Private ME line goes down routes for networks other than 10.25.X.X and 10.15.X.X are not being advertised and the route map is the source of the problem.

[Joe Conklin]

I figured out the route issue, but the preferred route still isn't coming back once I failover and back. I everything is working except the fact I need to restart BGP for it to apply a preference or weight again after failover and failback.

Missing advertised routes

I wasn't getting routes through the path where the route map was applied. Per the below I needed to add another route map with no matching IPs set so it would accept the routes not defined in the access list. Below is a copy of my config and everythink works now that I have a second route map with no matching IPs where relevant.

router bgp 65002
synchronization
bgp log-neighbor-changes
redistribute connected
neighbor 10.51.0.1 remote-as 65006
neighbor 10.100.1.1 remote-as 65004
neighbor 10.100.1.1 route-map PBR_Voice_prefer_MPLS in
neighbor 10.250.0.2 remote-as 65001
!
access-list 10 permit 10.25.0.0 0.0.255.255
access-list 10 permit 10.15.0.0 0.0.255.255
!
route-map PBR_Voice_prefer_MPLS permit 10
match ip address 10
set weight 65535
!
route-map PBR_Voice_prefer_MPLS permit 20

http://www.cisco.com/c/en/us/td/docs/ios/12_2/ip/configuration/guide/fipr_c/1cfbgp.html

It is proper behavior to not accept any autonomous system path not matching the match clause of the route map. This behavior means that you will not set the metric and the Cisco IOS software will not accept the route. However, you can configure the software to accept autonomous system paths not matched in the match clause of the route-map router configuration command by using multiple maps of the same name, some without accompanying set commands.

[Francesco Molino]

Sorry, just to sum up because you lost me. I would like to catch correctly your issue in order to help.

You were talking about network advertisement and after came with routing issue. Actually your issue is network advertisement, correct?

Based on the design you've attached.

Let's concentrate for example on EDGE 1 and EDGE 2. If I understand correctly, your issue is:

- Everything's fine when Private ME link is UP

- If this link is going down and came up after, all networks from EDGE2 are not advertised to EDGE 1 until you do a clear bgp?

On your GNS3 lab, which IGP is running?

Could you explain, why you've configure BGP synchronization?

[Joe Conklin]

My issue is weight that I set works fine when everything is up. If I failover from the MPLS to the ME and reinstate the MPLS connection the preference does not take affect. I have to reload BGP on EDGE1 and 2 and I do not understand why. BGP should use weight first during path selection.

Technically all the routers are running a similar to the below config, except the ISP-1 and 2 are not using a PBR. I also removed the synchronization, I added it just for giggles in the sim to make sure that wasn't an issue, and it wasn't. I've since removed it.

router bgp 65002
bgp log-neighbor-changes
redistribute connected
neighbor 10.51.0.1 remote-as 65006
neighbor 10.100.1.1 remote-as 65004
neighbor 10.100.1.1 route-map PBR_Voice_prefer_MPLS in
neighbor 10.250.0.2 remote-as 65001
!
access-list 10 permit 10.25.0.0 0.0.255.255
access-list 10 permit 10.15.0.0 0.0.255.255
!
route-map PBR_Voice_prefer_MPLS permit 10
match ip address 10
set weight 65535
!
route-map PBR_Voice_prefer_MPLS permit 20

[Francesco Molino · Accepted solution]

Hi

Sorry for my late answer as I was off today.

Ok I have a better understanding of your issue. On your EDGE 1 and 2 you need to announce only your local networks to ISP routers. You don't want that ISP routes from ISP 1 is going to edge1 and then it will announce to edge2,....  Without filtering you will face some kind of loops/issues. This is out it's going to be on your lab. On real life, ISPs will configure their BGP in order to prevent such things in some ways. You can imagine that your have 2 same networks to announce on both sites to your ISP. In order to make 1 active and the other site passive for example, he can play with community tagging.

You can filter by using prefix-list by selecting networks (much work) or simply announce in outbound only locally generated routes by using as-path acl. 

Here is the config of as-path acl:

On EDGE 1 and EDGE 2:

ip as-path access-list 10 permit ^$

!

route-map localonly permit 10
  match as-path 10

Under BGP configuration from edge 1 and 2, you need to do for all neighbours:

On EDGE 1: 

router bgp 65001

 neighbor 10.100.0.1 route-map locally out

 neighbour 10.250.0.1 route-map locally out

On EDGE 2: 

router bgp 65002

 neighbor 10.100.1.1 route-map locally out

 neighbour 10.250.0.2 route-map locally out

As per your route-map PBR, you will prefer going through ISP than going directly to edge2. If you shut the mpls link between ISPs, then traffic is going through backdoor Private ME. When MPLS link is coming back up, traffic will go through ISP instead of Private ME.

Let me know if it works. I built same lab as yours in my environment and I'm showing some outputs: (IP may not be same): In my lab, I also changed bop timers (you don't need to do that)

EDGE 1 BGP Config:

router bgp 65001
  no synchronization
 bgp router-id 3.3.3.3
 bgp log-neighbor-changes
 network 10.25.0.0 mask 255.255.0.0
 timers bgp 2 3 3
 neighbor 10.101.0.1 remote-as 65003
 neighbor 10.101.0.1 route-map PBR_Voice in
 neighbor 10.101.0.1 route-map localonly out
 neighbor 10.250.0.2 remote-as 65002
 neighbor 10.250.0.2 route-map localonly out
 no auto-summary
!
ip as-path access-list 10 permit ^$
!
route-map localonly permit 10
 match as-path 10
!

EDGE 2 BGP config:

router bgp 65002
 no synchronization
 bgp router-id 4.4.4.4
 bgp log-neighbor-changes
 network 10.15.0.0 mask 255.255.0.0
 timers bgp 2 3 3
 neighbor 10.101.1.1 remote-as 65004
 neighbor 10.101.1.1 route-map PBR_Voice in
 neighbor 10.101.1.1 route-map localonly out
 neighbor 10.250.0.1 remote-as 65001
 neighbor 10.250.0.1 route-map localonly out
 no auto-summary
!
ip as-path access-list 10 permit ^$
!
route-map localonly permit 10
 match as-path 10
!

Default behaviour of my BGP table.

On EDGE 2:

Network Next Hop Metric LocPrf Weight Path
*> 10.15.0.0/16 0.0.0.0 0 32768 i
*> 10.25.0.0/16 10.101.1.1 65535 65004 65003 65001 i
* 10.250.0.1 0 0 65001 i

On EDGE 1

Network Next Hop Metric LocPrf Weight Path
*> 10.15.0.0/16 10.101.0.1 65535 65003 65004 65002 i
* 10.250.0.2 0 0 65002 i
*> 10.25.0.0/16 0.0.0.0 0 32768 i

When MPLS is shutdown:

On EDGE 1:

Network Next Hop Metric LocPrf Weight Path
*> 10.15.0.0/16 10.250.0.2 0 0 65002 i
*> 10.25.0.0/16 0.0.0.0 0 32768 i

On EDGE 2:

Network Next Hop Metric LocPrf Weight Path
*> 10.15.0.0/16 0.0.0.0 0 32768 i
*> 10.25.0.0/16 10.250.0.1 0 0 65001 i

When MPLS is coming back.

On EDGE 2:

Network Next Hop Metric LocPrf Weight Path
*> 10.15.0.0/16 0.0.0.0 0 32768 i
*> 10.25.0.0/16 10.101.1.1 65535 65004 65003 65001 i
* 10.250.0.1 0 0 65001 i

On EDGE 1

Network Next Hop Metric LocPrf Weight Path
*> 10.15.0.0/16 10.101.0.1 65535 65003 65004 65002 i
* 10.250.0.2 0 0 65002 i
*> 10.25.0.0/16 0.0.0.0 0 32768 i

Hope this will help.

Thanks

[Joe Conklin]

All good, and thanks! I ended up using the below. Your advice did help. I was trying to use a distribute list but couldn't get it working exactly as I needed. Instead I used the additional route map like you suggested, just that I used a deny. The below excerpt works 100%, just needed to tweak for each config/router as necessary. 

router bgp 65001
bgp log-neighbor-changes
redistribute connected
neighbor 10.50.0.1 remote-as 65005
neighbor 10.100.0.1 remote-as 65003
neighbor 10.100.0.1 route-map PBR_Voice_prefer_MPLS in
neighbor 10.100.0.1 route-map PBR_deny_voice_adv out
neighbor 10.250.0.1 remote-as 65002
!
access-list 10 permit 10.25.0.0 0.0.255.255
access-list 10 permit 10.15.0.0 0.0.255.255
access-list 11 permit 10.15.0.0 0.0.255.255
!
route-map PBR_deny_voice_adv deny 11
match ip address 11
!
route-map PBR_deny_voice_adv permit 20
!
route-map PBR_Voice_prefer_MPLS permit 10
match ip address 10
set local-preference 200
!
route-map PBR_Voice_prefer_MPLS permit 20

[Francesco Molino]

Hi

I'm happy that we solved your issue.

acl and/or prefix-list are ok to use them in a deny statement within route-maps.

however, with as-path acl you will not need to modify each time your deny acl that matching subnets.

thanks for rating.

cheers