Trouble with BGP Failover

dpoehls · ‎09-30-2010

Hello there,

We are multihoming using two 7600 routers, each to a separate ISP. The two 7600's then have an iBGP peering between them. I am preferring routes to the first ISP using local preference so if I do a "show ip bgp summary" I see the following (essentially) on each router:

Router 1:

x.x.x.x (Primary ISP Peering) Received routes 300000

y.y.y.z (Router 2) Recieved routes 100 (routes directly to Backup ISP)

Router 2:

z.z.z.z (Secondary ISP Peering) Received routes 300000

y.y.y.x (Router 1) Recieved routes 299900

The issue is, that if I have an issue with the Primary ISP (either the BGP session drops or I lose a lot of upstream routes), the routes shared in the peering between the 2 routers do not change. So even though most or all of the 300000 routes to the Primary ISP are lost, those changes are not getting passed on to Router 2. Thus, Router 1 has only 100 known routes to the internet, and Router 2 has 299900 routes to nowhere!

I can get things working again by clearing the session between the 2 routers, but I have to do this manually.

Question then: Why is this not happening automatically, and is there configuration I can change that will make it so?

Please let me know if more information is needed. I just don't see what I'm missing.

Thank you!

andrew.prince · ‎09-30-2010

Post your config for review, remove any sensitive information - but include your
route-maps, as-path lists etc

dpoehls · ‎09-30-2010

I sincerely hope this isn't the most confusing thing ever. There are some other BGP peerings on these routers, but this should be all config relevant to this issue.

Thanks for taking a look!

***Router 1***

interface GigabitEthernet4/12

description Connection Router 1 -> Router 2

ip address x.x.x.y 255.255.255.252

!

interface GigabitEthernet7/6

description Connection to ISP_1

bandwidth 1000000

ip address y.y.y.x 255.255.255.252

speed nonegotiate

no cdp enable

!

router bgp XXXXX

bgp log-neighbor-changes

neighbor y.y.y.y remote-as YYYYY

neighbor y.y.y.y version 4

neighbor y.y.y.y activate

neighbor y.y.y.y prefix-list DENY-LEARNED in

neighbor y.y.y.y route-map UPSTREAM_PROVIDER in

neighbor y.y.y.y route-map ANNOUNCE-ISP_1 out

neighbor x.x.x.z remote-as XXXXX

neighbor x.x.x.z activate

no auto-summary

no synchronization

network ???.???.???.??? mask ???.???.???.???

!

ip as-path access-list 1 permit ^(YYYYY)(_\1)*$

ip as-path access-list 2 permit ^(ZZZZZ)(_\1)*$

ip as-path access-list 3 permit ^YYYYY_

ip as-path access-list 4 permit ^ZZZZZ_

ip as-path access-list 5 permit ^ZZZZZ_SPECIFIC_CUSTOMER$

!

ip prefix-list DENY-LEARNED seq 1 deny ???.???.???.???/??

ip prefix-list DENY-LEARNED seq 2 deny ???.???.???.???/??

ip prefix-list DENY-LEARNED seq 9 permit 0.0.0.0/0 le 24

ip prefix-list DENY-LEARNED seq 10 permit 0.0.0.0/0 ge 1

!

route-map UPSTREAM_PROVIDER permit 5

description RESET WHOLESALE

match ip address prefix-list WHOLESALE-NETWORKS

set local-preference 70

!

route-map UPSTREAM_PROVIDER permit 10

description prefer ISP_1 1-hop routes

match as-path 1

set local-preference 500

!

route-map UPSTREAM_PROVIDER permit 20

description prefer ISP_2 1-hop routes

match as-path 2

set local-preference 400

!

route-map UPSTREAM_PROVIDER permit 25

description prefer SPECIFIC_CUSTOMER ROUTES to go out ISP_2

match as-path 5

set local-preference 350

!

route-map UPSTREAM_PROVIDER permit 30

description prefer ISP_1 to ISP_2

match as-path 3

set local-preference 300

!

route-map UPSTREAM_PROVIDER permit 40

description set ISP_2 as Backup

match as-path 4

set local-preference 200

!

route-map ANNOUNCE-ISP_1 permit 5

match ip address 40

!

route-map ANNOUNCE-ISP_1 permit 10

match ip address 42

!

route-map ANNOUNCE-ISP_1 permit 20

match ip address 45

!

route-map ANNOUNCE-ISP_1 permit 25

match ip address 46

!

route-map ANNOUNCE-ISP_1 permit 30

match ip address 48

!

route-map ANNOUNCE-ISP_1 permit 35

match ip address 49

!

***Router 2***

interface GigabitEthernet4/12

description Connection Router 1 -> Router 2

ip address x.x.x.z 255.255.255.252

!

interface GigabitEthernet7/3

description Connection to ISP_2

ip address z.z.z.x 255.255.255.252

no cdp enable

!

router bgp XXXXX

bgp log-neighbor-changes

neighbor z.z.z.z remote-as ZZZZZ

neighbor z.z.z.z version 4

neighbor z.z.z.z activate

neighbor z.z.z.z route-map UPSTREAM_PROVIDER in

neighbor z.z.z.z route-map ANNOUNCE-ISP_2 out

neighbor x.x.x.y activate

neighbor x.x.x.y remote-as XXXXX

no auto-summary

no synchronization

network ???.???.???.??? mask ???.???.???.???

!

ip as-path access-list 1 permit ^(YYYYY)(_\1)*$

ip as-path access-list 2 permit ^(ZZZZZ)(_\1)*$

ip as-path access-list 3 permit ^YYYYY_

ip as-path access-list 4 permit ^ZZZZZ_

ip as-path access-list 5 permit ^ZZZZZ_SPECIFIC_CUSTOMER$

!

ip prefix-list PREFER-ISP_2 description Route these specifically to ISP_2

ip prefix-list PREFER-ISP_2 seq 10 permit ???.???.???.???/??

!

route-map UPSTREAM_PROVIDER permit 5

description RESET WHOLESALE

match ip address prefix-list WHOLESALE-NETWORKS

set local-preference 70

!

route-map UPSTREAM_PROVIDER permit 10

description prefer ISP_1 1-hop routes

match as-path 1

set local-preference 500

!

route-map UPSTREAM_PROVIDER permit 20

description prefer ISP_2 1-hop routes

match as-path 2

set local-preference 400

!

route-map UPSTREAM_PROVIDER permit 25

description prefer SPECIFIC_CUSTOMER ROUTES to go out ISP_2

match as-path 5

set local-preference 350

!

route-map UPSTREAM_PROVIDER permit 27

description Specifically route to ISP_2

match ip address prefix-list PREFER-ISP_2

set local-preference 325

!

route-map UPSTREAM_PROVIDER permit 30

description prefer ISP_1 to ISP_2

match as-path 3

set local-preference 300

!

route-map UPSTREAM_PROVIDER permit 40

description set ISP_2 as Backup

match as-path 4

set local-preference 200

!

route-map UPSTREAM_PROVIDER permit 50

description EXPLICIT PERMIT

!

route-map ANNOUNCE-ISP_2 permit 5

match ip address 40

!

route-map ANNOUNCE-ISP_2 permit 20

match ip address 45

!

route-map ANNOUNCE-ISP_2 permit 25

match ip address 46

set as-path prepend XXXXX

!

route-map ANNOUNCE-ISP_2 permit 30

match ip address 48

!

route-map ANNOUNCE-ISP_2 permit 35

match ip address 49

!

shivlu jain · ‎09-30-2010

Hi

I did not see the prefix list WHOLESALE-NETWORKS and access list 40,42,45,46,48 and 49. Can you post the full configs?

regards

Shivlu Jain

http://www.mplsvpn.info

Dan-Ciprian Cicioiu · ‎09-30-2010

Hi ,

Could you post from Router2

sh ip bgp y.y.y.y (y.y.y.y the IP on the neighbor from the Router1 )

or

sh ip route y.y.y.y

This route should have the next hop to the segment between the router1 and router2 and vice versa.

The problem could be the reachability of the next hop advertised by BGP ( the next-hop of the external routes doesnt change between Router1 and Router2 ). You could configure a static route on both routes for the interconection segments.

Dan

Mahesh Gohil · ‎10-01-2010

Hi,

Please confirm,

1- Is there any device between two 7600 routers. If yes and if you have not enabled "ip path mtu discovery" then i would suggest to enable in all device in between. (To ensure your BGP packet will be >536 bytes and takes less time to send all the routes across bgp peer)

Also as a short solution you can give "ip mtu 1500" interface command for connecting interface between two routers.

and also check if there is log in router 1 for memory issue as it will have to hold aroung 6lacs routes

this is a my thought and exact issue can be known with few outputs.

1- sh ip bgp neighbor (ibgp) from both router

2- sh ip route summary from both router

3- sh ip bgp x.x.x.x (any internet route) from both router

4- sh ver

Regards

mahesh

andrew.prince · ‎10-01-2010

"next-hop-self" required between router 1 & 2 on the neighbor statements.

Giuseppe Larosa · ‎10-01-2010

Hello Dpoelhs,

may I ask you why you are trying to set local preference for routes with an AS path starting with ISP2 ASN in the route-map of routes received by ISP1?

this is not correct, you need to increase local preference for these routes on R2 only on eBGP session with ISP2.

also both route-maps applied on received routes should have a final action like that means accepts everything with standard attributes ( local pref 100)

route-map UPSTREAM_PROVIDER permit 50

description EXPLICIT PERMIT

!

I don't see this final clause in first route-map applied on R1 if it is really missing you have a trouble.

The route-maps must be different to provide different local preference to different routes. They shouldn't be equal as explained above

Local preference is AS wide so to prefer a route received on Rx you should set local preference for that route on Rx only not also on Ry on the session with the other provider

Hope to help

Giuseppe