Redundant WAN route selection with BGP and OSPF redistribution

jvanderson2010 · ‎11-14-2011

Hi,

I have a requirement to implement a redundant WAN solution with BGP and OSPF redistribution, and I've got to a point that is getting beyond my knowledge and am after some assistance / guidance.

To [hopefully] set the scenario, see the attached / included diagram.

For the remote site, I want the primary WAN connection back to the main site to be the Telco WAN cloud. Under normal network all traffic to and from the Remote Site and Main Site should traverse the Telco WAN cloud. In the event of a failure of the Telco WAN cloud, including a link failure at MLB-RTR or a link failure at B1H-RTR or a failure within the Telco WAN cloud, I want all traffic to fail-over to the backup link, which is the Microwave/radio link.

Across the Microwave backup link I only want a default route advertised from CEN-RTR, as the Main Site is where the default route should go to.

When the Telco WAN cloud connection is restored, I want all traffic to fail-back to the Telco WAN cloud connection from the Microwave/radio link.

My problem is that fail-over works fine, but fail-back does not work properly.

When I cause of failure of the Telco WAN connection by shutting down the WAN interface on MLB-RTR all the routing fails over to the Microwave/radio backup link. However, when I bring that interface back up the fail-back does not work properly. Both the MLB-RTR and TSB-RTR routers restore their original routing and send all traffic destined to the Main Site via the Telco WAN cloud. However, the CEN-RTR and B1H-RTR do not restore their routing, and they continue to route traffic via the Microwave/radio backup link. ie. CEN-RTR continues to advertise into OSPF a path via redistributed BGP routes, and B1H-RTR prefers the routes to the remote site seen via OSPF over any routes it sees via BGP. For some reason B1H is either not receiving the Remote Site routes via BGP upon restoration, or it is ignoring them in favour of the OSPF path.

To restore the routing back to "normal" via the Telco WAN cloud, I have to manually shutdown the CEN-RTR WAN interface, which causes the routes to disappear from OSPF and then B1H-RTR picks up the routes from the Telco WAN cloud. Redistribution from BGP into OSPF by B1H-RTR then occurs and everything returns to normal. After which I can bring back up the CEN-RTR WAN interface and everything stays "normal".

However, when I cause a failure of the Telco WAN connection by shutting down the WAN interface on B1H-RTR, all the routing fails over. Then when I restore the interface, the B1H-RTR brings up the BGP neighbour successfully, and receives the routes to the Remote Site, and redistributes them into OSPF, which then become the preferred path in the OSPF network, and subsequently preferred by the CEN-RTR. So routing restores back to "normal".

Below is copies of the relevant sections of configuration from the key four devices illustrated in the diagram and participating in the design. I'm not BGP expert so this configuration may not be ideal and I'm open to suggestions for improvement.

B1H-RTR

-------

interface Loopback0

ip address 172.27.127.247 255.255.255.255

!

interface GigabitEthernet1/0/1

description Telco WAN Cloud

no switchport

ip address 172.27.126.102 255.255.255.248

speed 100

duplex full

no mdix auto

!

interface Vlan582

description To Campus Network

ip address 172.27.126.114 255.255.255.248

no ip redirects

no ip unreachables

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 0225267F5F562B701C

ip ospf hello-interval 4

!

interface Vlan583

description To Campus Network

ip address 172.27.126.122 255.255.255.248

no ip redirects

no ip unreachables

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 0225267F5F562B701C

ip ospf hello-interval 4

!

router ospf 1

log-adjacency-changes detail

auto-cost reference-bandwidth 10000

area 0 authentication message-digest

redistribute static metric 100 metric-type 1 subnets

redistribute bgp 64614 metric 10 metric-type 1 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface Vlan582

no passive-interface Vlan583

network 172.27.0.0 0.0.255.255 area 0

!

router bgp 64614

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.97 remote-as 64514

!

address-family ipv4

redistribute connected

redistribute static

redistribute ospf 1 match internal external 1 external 2 nssa-external 1 nssa-external 2

neighbor 172.27.126.97 activate

neighbor 172.27.126.97 default-originate

neighbor 172.27.126.97 weight 40000

neighbor 172.27.126.97 soft-reconfiguration inbound

neighbor 172.27.126.97 prefix-list PL-BGP-DROP-DEFAULT-ROUTE in

neighbor 172.27.126.97 route-map RM-BGP-WEIGHT in

distance 20 0.0.0.0 255.255.255.255

distance 20 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

network 0.0.0.0

exit-address-family

!

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE description DO NOT ACCEPT THE DEFAULT ROUTE FROM THE GWIP CLOUD

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31

route-map RM-BGP-WEIGHT permit 10

set local-preference 40000

set weight 40000

!

route-map RM-BGP-TO-OSPF permit 10

set tag 666

!

CEN-RTR

-------

interface Loopback0

ip address 172.27.127.248 255.255.255.255

!

interface Vlan563

description Microwave backup link

ip address 172.27.126.69 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 013024200F5B225E71

arp timeout 300

!

router ospf 1

log-adjacency-changes detail

auto-cost reference-bandwidth 10000

area 0 authentication message-digest

redistribute static subnets route-map STATIC_TO_OSPF

redistribute bgp 64614 metric 1000 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface GigabitEthernet1/4

no passive-interface Vlan3

no passive-interface Vlan67

no passive-interface Vlan559

no passive-interface Vlan561

no passive-interface Vlan562

network 10.127.0.0 0.0.255.255 area 0

network 10.176.0.0 0.0.255.255 area 0

network 172.27.0.0 0.0.255.255 area 0

network 192.168.0.0 0.0.255.255 area 0

!

router bgp 64614

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.68 remote-as 64616

!

address-family ipv4

neighbor 172.27.126.68 activate

neighbor 172.27.126.68 default-originate

neighbor 172.27.126.68 prefix-list PL-BGP-DROP-DEFAULT-ROUTE in

neighbor 172.27.126.68 route-map RM-BGP-WEIGHT in

neighbor 172.27.126.68 route-map RM-BGP-ROUTES-OUT out

neighbor 172.27.126.68 filter-list 1 out

distance bgp 190 200 1

distance 190 0.0.0.0 255.255.255.255

distance 190 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

network 0.0.0.0

exit-address-family

!

ip as-path access-list 1 deny .*

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE description DO NOT ACCEPT THE DEFAULT ROUTE FROM THE BACKUP LINK

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31

route-map STATIC_TO_OSPF permit 20

set metric 140

set metric-type type-2

!

route-map RM-BGP-ROUTES-OUT permit 10

set metric 200

set as-path prepend 64614 64614 64614

!

route-map RM-BGP-WEIGHT permit 10

set weight 1

!

route-map RM-BGP-TO-OSPF permit 10

set weight 1

!

MLB-RTR

-------

interface FastEthernet2/0/24

description Telco WAN Cloud

no switchport

ip address 172.27.126.110 255.255.255.248

speed 100

duplex full

no mdix auto

spanning-tree portfast

spanning-tree bpdufilter enable

spanning-tree bpduguard enable

!

interface Vlan568

description Campus Network Routing

ip address 172.27.127.178 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 123A2733465B28557A

!

router ospf 1

log-adjacency-changes detail

area 0 authentication message-digest

redistribute static subnets route-map STATIC_TO_OSPF

redistribute bgp 64616 metric 100 metric-type 1 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface Vlan568

network 0.0.0.0 255.255.255.255 area 0

default-information originate

!

router bgp 64616

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.105 remote-as 64516

neighbor 172.27.127.179 remote-as 64616

!

address-family ipv4

redistribute connected

redistribute static

redistribute ospf 1 route-map RM-OSPF-TO-BGP

neighbor 172.27.126.105 activate

neighbor 172.27.126.105 weight 40000

neighbor 172.27.126.105 prefix-list PL-BGP-DROP-DEFAULT-ROUTE out

no neighbor 172.27.127.179 activate

distance bgp 20 20 1

distance 20 0.0.0.0 255.255.255.255

distance 20 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

network 0.0.0.0

exit-address-family

!

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31

route-map STATIC_TO_OSPF permit 20

set metric 130

set metric-type type-2

!

route-map RM-OSPF-TO-BGP deny 10

match tag 666

!

route-map RM-OSPF-TO-BGP permit 20

set weight 40000

!

route-map RM-BGP-TO-OSPF permit 10

set tag 666

!

TSB-RTR

interface Loopback0

ip address 172.27.127.243 255.255.255.255

!

interface Vlan563

description Microwave backup link

ip address 172.27.126.68 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 013024200F5B225E71

arp timeout 300

!

interface Vlan568

description Campus Network Routing

ip address 172.27.127.179 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 013024200F5B225E71

!

router ospf 1

log-adjacency-changes detail

area 0 authentication message-digest

redistribute static subnets route-map STATIC_TO_OSPF

redistribute bgp 64616 metric 1000 metric-type 1 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface Vlan568

network 0.0.0.0 255.255.255.255 area 0

default-information originate metric 1000

!

router bgp 64616

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.69 remote-as 64614

neighbor 172.27.127.178 remote-as 64616

!

address-family ipv4

redistribute connected

redistribute static

redistribute ospf 1 route-map RM-OSPF-TO-BGP

neighbor 172.27.126.69 activate

neighbor 172.27.126.69 prefix-list PL-BGP-DEFAULT-ROUTE-ONLY in

no neighbor 172.27.127.178 activate

distance bgp 190 200 1

distance 190 0.0.0.0 255.255.255.255

distance 190 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

exit-address-family

!

ip prefix-list PL-BGP-DEFAULT-ROUTE-ONLY description ONLY ACCEPT A DEFAULT ROUTE FROM RADIO LINK

ip prefix-list PL-BGP-DEFAULT-ROUTE-ONLY seq 10 permit 0.0.0.0/0

route-map STATIC_TO_OSPF permit 20

set metric 130

set metric-type type-2

!

route-map RM-OSPF-TO-BGP deny 10

match tag 666

!

route-map RM-OSPF-TO-BGP permit 20

set weight 1

!

route-map RM-BGP-TO-OSPF permit 10

set tag 666

!

vikz230884 · ‎11-14-2011

Hi John,

Looks like the traffic from B1H will choose to route to CEN since it have less as-path, because with telco router you run different AS.

I can think one of the solution is to prepend the as-path at CEN, so the route calculation will be based on weight & local preference (which you have already configured).

HTH,

Vikram

jvanderson2010 · ‎11-14-2011

Vikram,

Does the AS-PATH come into the equation in this situation because there is no BGP peer between CEN and B1H? It is only an OSPF cloud.

Thanks.

vikz230884 · ‎11-14-2011

John,

My mistake, I initially tought that there is iBGP between them.. you're right..then when you're bringing the link, you need to see if there is actually routes advertise by the peer.

Are the show ip bgp, showed the routes with *> ?

I think the reason it still using the OSPF routes is because the routes from eBGP is not valid or there is no routes advertise from that peer...

HTH,

Vikram

milan.kulik · ‎11-15-2011

Hi,

mutual BGP/OSPF redistribution is tricky sometimes.

You need to think about WHAT you are redistributing and WHEN you are redistributing.

IMHO, you problem on B1H-RTR might be following:

When the Telco connection fails, the Remote site eBGP prefixes are removed from the BGP table.

The same prefixes are received via OSPF then and redistributed to BGP.

Note: BGP is using a default weight attribute value of 32768 for the prefixes redistributed in this scenario.

When the Telco connection is restored again, the Remote site prefixes are received again from eBGP.

But as the same prefixes are present in the BGP table already (redistributed from OSPF) with weight=32768 and they are beating the eBGP prefixes at the moment!

So B1H keeps the redistributed prefixes as the bets ones within BGP and keeps routing via OSPF consequently.

You can check using sh ip route and sh ip bgp x.x.x.x for one of the Remote site prefix at that moment - I guess you will see the weight value 32768.

So if I'm correct, you need to add a route-map with set weight 0 command to the B1H config for OSPF to BGP redistribution.

I'm not sure this will fix all problems in your design as it's quite complex.

But as I said, you need to think "What happens when the prefix comes from this side first and what happens if the prefix comes from the other side first?".

HTH,

Milan

Anukalp S · ‎08-16-2015

Hello.. I have been facing same problem and i did try to set weight 0 while redistributing from ospf to BGP but still routes see from ospf. I had to clear ospf process to learn routes from BGP.

I want to know if some one had been facing similar issue and was able to fix this problem.

This is very tricky issue.

milan.kulik · ‎08-16-2015

Hi,

IMHO, you need to check why the prefixes redistributed from OSPF are still beating the same prefixes received from the BGP neighbor now.

Possibly another BGP attribute needs to be modified?

I remember similar case when I had to increase Local Preference of the prefixes received from the BGP neighbor to force them to win.

Best regards,

Milan

Anukalp S · ‎08-21-2015

I would try increasing LP in another downtime window. Would like to know whether in your case increasing value of LP resolved this problem.

Anukalp S · ‎09-25-2015

hi.. i tried with manipulating local preference, i confrigured LP value 50 while redistributing from ospf to BGP but still ospf routes is choosen.

What else can do to resolve this case as it seems a bit complex.

milan.kulik · ‎09-25-2015

Hi,

can you provide

sh ip bgp subnet/length

example of one of the prefixes which is still chosen redistributed from OSPF and also received via BGP at the same time?

The BGP attributes should be visible in such an output and we should be able to see why the redistributed prefix was chosen.

BR,

Milan

Anukalp S · ‎09-25-2015

Hi,

pls see below output.

============================================

Incase of Mpls down i receive routes from ospf(fine int this case) but when mpls up still i see this route. I only see this route from ospf only.

MPLS#sh ip bgp 192.168.242.254
BGP routing table entry for 192.168.242.0/24, version 27328
Paths: (1 available, best #1, table default)
Advertised to update-groups:
     4
Local
    192.168.80.17 from 0.0.0.0 (192.168.80.22)
      Origin incomplete, metric 2, localpref 50, weight 50, valid, sourced, best

MPLS#sh ip o ne

Neighbor ID Pri State Dead Time Address Interface
192.168.199.254 1 FULL/DR 00:00:31 192.168.80.17 GigabitEthernet0/1

=============================================================

But when i reset ospf i started seeing below route.

MPLS#sh ip bgp 192.168.242.254

BGP routing table entry for 192.168.242.0/24, version 27034

Advertised to update-groups:

4755 4755
    172.16.0.17 from 172.16.0.17 (192.168.231.212)
      Origin incomplete, localpref 100, weight 200, valid, external, best
4755 4755, (received-only)
    172.16.0.17 from 172.16.0.17 (192.168.231.212)
      Origin incomplete, localpref 100, valid, external

MPLS#sh ip bgp summ | b N

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

172.16.0.17 4 4755 8940 8967 27815 0 0 5d14h 63

milan.kulik · ‎09-25-2015

Hi,

so it seems to me:

You are advertising the OSPF prefix 192.168.242.0/24 to your BGP neigbor 172.16.0.17, aren't you?

And when MPLS is Up again, that neighbor is still not advertising 192.168.242.0/24 to you, i.e., it keeps the 192.168.242.0/24 prefix received from you as the best one even when MPLS is up?

In that case, you need to advertise the 192.168.242.0/24 prefix to the 172.16.0.17 neighbor with some worse BGP attributes to make the neighbor to prefer 192.168.242.0/24 prefix received from MPLS the preferred one!

You might try to prepend your AS number several times when advertising, e.g.

It might be also worth to discuss with the neighbor router admin why they prefer your prefix over the MPLS one.

BR,

Milan

Anukalp S · ‎09-25-2015

See below topology. 192.168.242.0/24 is Site B addess. So when Site B mpls down i get routes on site A over Backup but when site B mpls up, i still receive route from ospf and is advertise to BGP peer, though it should receive this route from BGP peer.

OSPF domain is backup as i am redisbuting b/w ospf processes and making E2 routes and while over BGP i am making E1 routes.

Apologize for poor drawing,

milan.kulik · ‎09-25-2015

Well, I'm getting almost lost.

I'd suggest to focus on the question:

Why when the MPLS is Up again, the BGP router on site B is NOT advertising the 192.168.242.0/24 subnet to site A via BGP?

Isn't it because it is receiving the same prefix from site A and takes it as the best BGP prefix?

Can you provide the

sh ip bgp 192.168.242.0/24

output from BGP router on site B at the moment when MPLS is Up again?

BR,

Milan

Anukalp S · ‎09-25-2015

Now you are correct.. when MPLS site B up again i dont see this route being advertised to BGP peer but what i see this route is being recevied from the peer.

Because site A is still advertising this route to its BGP peer rather being received.

I cannot make this route to stop advertising on site A. I tried to filter(deny) this route on site B to be received from peer but it did not make any change on site A. Site A was still advertising this route to peer(Although it should receive this route from peer)