Hi MilanI agree with the main - Page 2

jvanderson2010 · ‎11-14-2011

Hi,

I have a requirement to implement a redundant WAN solution with BGP and OSPF redistribution, and I've got to a point that is getting beyond my knowledge and am after some assistance / guidance.

To [hopefully] set the scenario, see the attached / included diagram.

For the remote site, I want the primary WAN connection back to the main site to be the Telco WAN cloud. Under normal network all traffic to and from the Remote Site and Main Site should traverse the Telco WAN cloud. In the event of a failure of the Telco WAN cloud, including a link failure at MLB-RTR or a link failure at B1H-RTR or a failure within the Telco WAN cloud, I want all traffic to fail-over to the backup link, which is the Microwave/radio link.

Across the Microwave backup link I only want a default route advertised from CEN-RTR, as the Main Site is where the default route should go to.

When the Telco WAN cloud connection is restored, I want all traffic to fail-back to the Telco WAN cloud connection from the Microwave/radio link.

My problem is that fail-over works fine, but fail-back does not work properly.

When I cause of failure of the Telco WAN connection by shutting down the WAN interface on MLB-RTR all the routing fails over to the Microwave/radio backup link. However, when I bring that interface back up the fail-back does not work properly. Both the MLB-RTR and TSB-RTR routers restore their original routing and send all traffic destined to the Main Site via the Telco WAN cloud. However, the CEN-RTR and B1H-RTR do not restore their routing, and they continue to route traffic via the Microwave/radio backup link. ie. CEN-RTR continues to advertise into OSPF a path via redistributed BGP routes, and B1H-RTR prefers the routes to the remote site seen via OSPF over any routes it sees via BGP. For some reason B1H is either not receiving the Remote Site routes via BGP upon restoration, or it is ignoring them in favour of the OSPF path.

To restore the routing back to "normal" via the Telco WAN cloud, I have to manually shutdown the CEN-RTR WAN interface, which causes the routes to disappear from OSPF and then B1H-RTR picks up the routes from the Telco WAN cloud. Redistribution from BGP into OSPF by B1H-RTR then occurs and everything returns to normal. After which I can bring back up the CEN-RTR WAN interface and everything stays "normal".

However, when I cause a failure of the Telco WAN connection by shutting down the WAN interface on B1H-RTR, all the routing fails over. Then when I restore the interface, the B1H-RTR brings up the BGP neighbour successfully, and receives the routes to the Remote Site, and redistributes them into OSPF, which then become the preferred path in the OSPF network, and subsequently preferred by the CEN-RTR. So routing restores back to "normal".

Below is copies of the relevant sections of configuration from the key four devices illustrated in the diagram and participating in the design. I'm not BGP expert so this configuration may not be ideal and I'm open to suggestions for improvement.

B1H-RTR

-------

interface Loopback0

ip address 172.27.127.247 255.255.255.255

!

interface GigabitEthernet1/0/1

description Telco WAN Cloud

no switchport

ip address 172.27.126.102 255.255.255.248

speed 100

duplex full

no mdix auto

!

interface Vlan582

description To Campus Network

ip address 172.27.126.114 255.255.255.248

no ip redirects

no ip unreachables

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 0225267F5F562B701C

ip ospf hello-interval 4

!

interface Vlan583

description To Campus Network

ip address 172.27.126.122 255.255.255.248

no ip redirects

no ip unreachables

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 0225267F5F562B701C

ip ospf hello-interval 4

!

router ospf 1

log-adjacency-changes detail

auto-cost reference-bandwidth 10000

area 0 authentication message-digest

redistribute static metric 100 metric-type 1 subnets

redistribute bgp 64614 metric 10 metric-type 1 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface Vlan582

no passive-interface Vlan583

network 172.27.0.0 0.0.255.255 area 0

!

router bgp 64614

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.97 remote-as 64514

!

address-family ipv4

redistribute connected

redistribute static

redistribute ospf 1 match internal external 1 external 2 nssa-external 1 nssa-external 2

neighbor 172.27.126.97 activate

neighbor 172.27.126.97 default-originate

neighbor 172.27.126.97 weight 40000

neighbor 172.27.126.97 soft-reconfiguration inbound

neighbor 172.27.126.97 prefix-list PL-BGP-DROP-DEFAULT-ROUTE in

neighbor 172.27.126.97 route-map RM-BGP-WEIGHT in

distance 20 0.0.0.0 255.255.255.255

distance 20 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

network 0.0.0.0

exit-address-family

!

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE description DO NOT ACCEPT THE DEFAULT ROUTE FROM THE GWIP CLOUD

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31

route-map RM-BGP-WEIGHT permit 10

set local-preference 40000

set weight 40000

!

route-map RM-BGP-TO-OSPF permit 10

set tag 666

!

CEN-RTR

-------

interface Loopback0

ip address 172.27.127.248 255.255.255.255

!

interface Vlan563

description Microwave backup link

ip address 172.27.126.69 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 013024200F5B225E71

arp timeout 300

!

router ospf 1

log-adjacency-changes detail

auto-cost reference-bandwidth 10000

area 0 authentication message-digest

redistribute static subnets route-map STATIC_TO_OSPF

redistribute bgp 64614 metric 1000 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface GigabitEthernet1/4

no passive-interface Vlan3

no passive-interface Vlan67

no passive-interface Vlan559

no passive-interface Vlan561

no passive-interface Vlan562

network 10.127.0.0 0.0.255.255 area 0

network 10.176.0.0 0.0.255.255 area 0

network 172.27.0.0 0.0.255.255 area 0

network 192.168.0.0 0.0.255.255 area 0

!

router bgp 64614

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.68 remote-as 64616

!

address-family ipv4

neighbor 172.27.126.68 activate

neighbor 172.27.126.68 default-originate

neighbor 172.27.126.68 prefix-list PL-BGP-DROP-DEFAULT-ROUTE in

neighbor 172.27.126.68 route-map RM-BGP-WEIGHT in

neighbor 172.27.126.68 route-map RM-BGP-ROUTES-OUT out

neighbor 172.27.126.68 filter-list 1 out

distance bgp 190 200 1

distance 190 0.0.0.0 255.255.255.255

distance 190 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

network 0.0.0.0

exit-address-family

!

ip as-path access-list 1 deny .*

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE description DO NOT ACCEPT THE DEFAULT ROUTE FROM THE BACKUP LINK

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31

route-map STATIC_TO_OSPF permit 20

set metric 140

set metric-type type-2

!

route-map RM-BGP-ROUTES-OUT permit 10

set metric 200

set as-path prepend 64614 64614 64614

!

route-map RM-BGP-WEIGHT permit 10

set weight 1

!

route-map RM-BGP-TO-OSPF permit 10

set weight 1

!

MLB-RTR

-------

interface FastEthernet2/0/24

description Telco WAN Cloud

no switchport

ip address 172.27.126.110 255.255.255.248

speed 100

duplex full

no mdix auto

spanning-tree portfast

spanning-tree bpdufilter enable

spanning-tree bpduguard enable

!

interface Vlan568

description Campus Network Routing

ip address 172.27.127.178 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 123A2733465B28557A

!

router ospf 1

log-adjacency-changes detail

area 0 authentication message-digest

redistribute static subnets route-map STATIC_TO_OSPF

redistribute bgp 64616 metric 100 metric-type 1 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface Vlan568

network 0.0.0.0 255.255.255.255 area 0

default-information originate

!

router bgp 64616

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.105 remote-as 64516

neighbor 172.27.127.179 remote-as 64616

!

address-family ipv4

redistribute connected

redistribute static

redistribute ospf 1 route-map RM-OSPF-TO-BGP

neighbor 172.27.126.105 activate

neighbor 172.27.126.105 weight 40000

neighbor 172.27.126.105 prefix-list PL-BGP-DROP-DEFAULT-ROUTE out

no neighbor 172.27.127.179 activate

distance bgp 20 20 1

distance 20 0.0.0.0 255.255.255.255

distance 20 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

network 0.0.0.0

exit-address-family

!

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0

ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31

route-map STATIC_TO_OSPF permit 20

set metric 130

set metric-type type-2

!

route-map RM-OSPF-TO-BGP deny 10

match tag 666

!

route-map RM-OSPF-TO-BGP permit 20

set weight 40000

!

route-map RM-BGP-TO-OSPF permit 10

set tag 666

!

TSB-RTR

interface Loopback0

ip address 172.27.127.243 255.255.255.255

!

interface Vlan563

description Microwave backup link

ip address 172.27.126.68 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 013024200F5B225E71

arp timeout 300

!

interface Vlan568

description Campus Network Routing

ip address 172.27.127.179 255.255.255.248

no ip proxy-arp

ip ospf message-digest-key 1 md5 7 013024200F5B225E71

!

router ospf 1

log-adjacency-changes detail

area 0 authentication message-digest

redistribute static subnets route-map STATIC_TO_OSPF

redistribute bgp 64616 metric 1000 metric-type 1 subnets route-map RM-BGP-TO-OSPF

passive-interface default

no passive-interface Vlan568

network 0.0.0.0 255.255.255.255 area 0

default-information originate metric 1000

!

router bgp 64616

bgp log-neighbor-changes

timers bgp 1 3

neighbor 172.27.126.69 remote-as 64614

neighbor 172.27.127.178 remote-as 64616

!

address-family ipv4

redistribute connected

redistribute static

redistribute ospf 1 route-map RM-OSPF-TO-BGP

neighbor 172.27.126.69 activate

neighbor 172.27.126.69 prefix-list PL-BGP-DEFAULT-ROUTE-ONLY in

no neighbor 172.27.127.178 activate

distance bgp 190 200 1

distance 190 0.0.0.0 255.255.255.255

distance 190 0.0.0.0 0.0.0.0

no auto-summary

no synchronization

bgp redistribute-internal

exit-address-family

!

ip prefix-list PL-BGP-DEFAULT-ROUTE-ONLY description ONLY ACCEPT A DEFAULT ROUTE FROM RADIO LINK

ip prefix-list PL-BGP-DEFAULT-ROUTE-ONLY seq 10 permit 0.0.0.0/0

route-map STATIC_TO_OSPF permit 20

set metric 130

set metric-type type-2

!

route-map RM-OSPF-TO-BGP deny 10

match tag 666

!

route-map RM-OSPF-TO-BGP permit 20

set weight 1

!

route-map RM-BGP-TO-OSPF permit 10

set tag 666

!

milan.kulik · ‎09-25-2015

Hi,

are you administering routers on both sites?

Why can't you stop the router on site A to advertise the prefix to site B?

To avoid routing loops, you should never advertise a prefix received from site B back to teh same site!

Generally, when configuring mutaul redistribution between two routing protocols, you need to think:

What happens when the prefix comes from this protocol first? And what happens when it comes from the other protocol first?

How can I configure both protocols to prefer the prefixes the way I want all the time? How to avoid routing loops (local prefixes received back from WAN)?

BR,

Milan

Anukalp S · ‎09-25-2015

Hello Milan,

Reason for why i cant stop advertising site B prefix from site A is i have others lovations over MPLS. And when Site B mpls down, site B should reach othe location through site A.

Cant we setup some kind of conditional advertisement like as long as site B mpls up, site A should not advertise site B prefix and when site B mpls down, site A start advertising site B prefixes.

I understand this is not a recommended setup but for now i have to do like so.

There should be some workaround. Please suggest.

milan.kulik · ‎09-25-2015

Hi,

in that case you should configure site B to ignore its own prefixes received from MPLS.

You said you have done that already?

Did you issue

clear ip bgp * soft

on BGP router on site B after that?

If yes, can you provide the BGP config part from router on site B and also the

show ip bpg 192.168.242.0/24

output from that router?

BR,

Milan

Anukalp S · ‎09-25-2015

Currently everything is fine as i had reseted ospf process, i can show you current output.

-----------------------------------------------------------------------------------------

MPLS-SiteB#sh ip rou 192.168.242.0
Routing entry for 192.168.242.0/24
Known via "ospf 1", distance 110, metric 2, type inter area
Redistributing via bgp 65354
Advertised by bgp 65354 match internal external 1 & 2 route-map OUT
Last update from 192.168.80.225 on GigabitEthernet0/0, 7w0d ago
Routing Descriptor Blocks:
* 192.168.80.225, from 192.168.247.254, 7w0d ago, via GigabitEthernet0/0
Route metric is 2, traffic share count is 1
MPLS-SiteB#
MPLS-SiteB#sh ip o ne

Neighbor ID Pri State Dead Time Address Interface
192.168.247.254 1 FULL/DR 00:00:32 192.168.80.225 GigabitEthernet0/0

MPLS-SiteB#sh ip bgp 192.168.242.0
BGP routing table entry for 192.168.242.0/24, version 7756
Paths: (1 available, best #1, table default)
Advertised to update-groups:
     4
Local
    192.168.80.225 from 0.0.0.0 (192.168.80.227)
      Origin incomplete, metric 2, localpref 100, weight 32768, valid, sourced, best

----------------------------------------------------------------------------------------------------------

router bgp 65354
bgp log-neighbor-changes
network 172.16.0.28 mask 255.255.255.252
redistribute ospf 1 match internal external 1 external 2 route-map OUT
neighbor 172.16.0.29 remote-as 4755
neighbor 172.16.0.29 soft-reconfiguration inbound

route-map OUT permit 10
match ip address Adv

ip access-list extended Adv

permit ip 192.168.242.0 0.0.0.255 any

milan.kulik · ‎09-25-2015

Hi,

well, what I'd be afraid of is using extended ACLs in redistribution route-maps.

As you can see here

http://blog.ine.com/2008/01/04/using-extended-access-lists-in-a-distribute-list/

the syntax is quite confusing.

So I'd recommend using ip prefix-lists instead.

Even when it seems to work in your case, I'd configure:

ip prefix-list site_B_prefixes seq 5 permit 192.168.242.0/24

!

route-map OUT permit 10
match ip address prefix-list site_B_prefixes

!

route-map IN deny 10

match ip address prefix-list site_B_prefixes

route-map IN permit 20

!

router bgp 65354

neighbor 172.16.0.29 route-map IN in

!

This way the router should never accept the 192.168.242.0/24 prefix from the MPLS neighbor.

And it should keep redistributing it from OSPF and advertisng to the BGP neighbor all the time.

BR,

Milan

Anukalp S · ‎09-25-2015

Thanks, but I used ACL and it was working too, i was not receiving 192.168.242.0 prefix from BGP peer at site B after applying inbound filter list and i was able to see at site B prefix 192.168.242.0 is advertising to peer properly.

But concern here is Site A, that time site A was still advertising 192.168.242.0 instead of receiving from BGP peer.

Cant we setup here kind of conditional advertising.

milan.kulik · ‎09-25-2015

Well,

with the info I've got available, I'd say:

The only way to recognize on site A the MPLS line is Up on site B is receiving the site B prefix from MPLS.

But if site B is not advertising it correctly all the time (and it seems not), it makes no sense to configure any conditional advertisement.

I guess there might had been another issue happening on site B:

Imagine the BGP router stopped receiving 192.168.242.0/24 via OSPF for any reason. So it stopped advertising it to MPLS and finally accepted the BGP prefix received from site A.

And after OSPF recovered finally, there was an eBGP prefix within the RIB (AD=20) and kept as the best route!

IMHO, blocking 192.168.242.0/24 received from MPLS on site B is the way!

BR,

Milan

Anukalp S · ‎09-25-2015

Milan,

Please understand, I have tried with blocking prefix 192.168.242.0/24 at site B but it didnt make site A to advertise this prefix.After blocking i saw i was not receving this prefix but still aite A was advertising.

So i am focusingg here on site A to stop advertising this prefix conditionally. When this prefix doesnt advertise at Site A then site B wont be receiving it.

milan.kulik · ‎09-25-2015

Hi,

I believe, you need to harden site B to advertise its prefixes under all conditions.

(EVEN after local OSPF flap.)

And then to configure site A to prefer site B prefixes recieved from MPLS/BGP over those received from OSPF.

BR,

Milan

Jon Marshall · ‎09-25-2015

Edited because it is a race condition at site B.

When the MPLS link comes back up at site B if it redistributes OSPF into BGP before it receives the route from MPLS it will advertise it.

If it receives the route before redistributing it won't.

So you need to -

1) modify the weight at site A for BGP routes > 32768

2) filter inbound BGP routes at site B to block it's own prefix.

Which I think is what Milan was saying all along :-)

Jon

milan.kulik · ‎09-25-2015

Hi Jon,

actually where I see the root problem cause is following scenario:

Imagine OSPF flaps on site B for any reason (LAN issue, e.g.).

I.e., the 192.168.242.0/24 prefix is removed from OSPF database on the router B.

As it had been redistributed to BGP there, it's also withdrawn from BGP database and withdrawn on the MPLS BGP neigbhor.

After a short time site A takes the same prefix received from its OSPF as the best one and starts to advertise via BGP.

Finally, site B receives this prefix via eBGP and accepts as the best one.

Then OSPF recovers on site B.

But there is the 192.168.242.0/24 prefix on router B in RIB received via eBGP already!

So even while received via OSPF now, the eBGP prefix wins and the OSPF prefix is not the best one and does NOT get redistributed to BGP!!

So the crutial thing is to

1) filter inbound BGP routes at site B to block it's own prefix.

2) regarding modifying the weight at site A for BGP routes > 32768:

The question is if the setup is not symmetrical possibly, i.e., site B should work as a backup connection for site A?

If yes, only site B prefixes should get better weight at site A.

And its own (site A's) prefixes should be also blocked on site A.

Quite complex, isn't it?

BR,

Milan

Jon Marshall · ‎09-25-2015

Hi Milan

I agree with the main issue although i was thinking about if from an MPLS failure scenario rather than a route flap although they could lead to the same thing in the end.

It really depends on what route is the RIB at any particular time, the key thing being there is no guarantee it will be the right one.

I agree also that you need the same filtering and weight manipulation at both sites and i would have thought that would sort the issue out.

It does depend on which sites are meant to be backing each other up ie. if the other sites (not A or B) do not back any other sites up then no need to redistribute OSPF into BGP except for the site's own subnets.

It is quite complex because of the backing up of sites with each other and how that affects what you advertise from each site.

Hope I haven't added to the confusion :-)

Jon

milan.kulik · ‎08-24-2015

Yes, changing the Weight of the redistributed prefixes to 0 and increasing the LP of

prefixes received via BGP fixed the problém that time.

best regrds,

Milan

Damian Cleveland · ‎11-14-2013

Hello,

Was Milan's solution effective?

Thank You

Redundant WAN route selection with BGP and OSPF redistribution