Hi,
I have a requirement to implement a redundant WAN solution with BGP and OSPF redistribution, and I've got to a point that is getting beyond my knowledge and am after some assistance / guidance.
To [hopefully] set the scenario, see the attached / included diagram.
For the remote site, I want the primary WAN connection back to the main site to be the Telco WAN cloud. Under normal network all traffic to and from the Remote Site and Main Site should traverse the Telco WAN cloud. In the event of a failure of the Telco WAN cloud, including a link failure at MLB-RTR or a link failure at B1H-RTR or a failure within the Telco WAN cloud, I want all traffic to fail-over to the backup link, which is the Microwave/radio link.
Across the Microwave backup link I only want a default route advertised from CEN-RTR, as the Main Site is where the default route should go to.
When the Telco WAN cloud connection is restored, I want all traffic to fail-back to the Telco WAN cloud connection from the Microwave/radio link.
My problem is that fail-over works fine, but fail-back does not work properly.
When I cause of failure of the Telco WAN connection by shutting down the WAN interface on MLB-RTR all the routing fails over to the Microwave/radio backup link. However, when I bring that interface back up the fail-back does not work properly. Both the MLB-RTR and TSB-RTR routers restore their original routing and send all traffic destined to the Main Site via the Telco WAN cloud. However, the CEN-RTR and B1H-RTR do not restore their routing, and they continue to route traffic via the Microwave/radio backup link. ie. CEN-RTR continues to advertise into OSPF a path via redistributed BGP routes, and B1H-RTR prefers the routes to the remote site seen via OSPF over any routes it sees via BGP. For some reason B1H is either not receiving the Remote Site routes via BGP upon restoration, or it is ignoring them in favour of the OSPF path.
To restore the routing back to "normal" via the Telco WAN cloud, I have to manually shutdown the CEN-RTR WAN interface, which causes the routes to disappear from OSPF and then B1H-RTR picks up the routes from the Telco WAN cloud. Redistribution from BGP into OSPF by B1H-RTR then occurs and everything returns to normal. After which I can bring back up the CEN-RTR WAN interface and everything stays "normal".
However, when I cause a failure of the Telco WAN connection by shutting down the WAN interface on B1H-RTR, all the routing fails over. Then when I restore the interface, the B1H-RTR brings up the BGP neighbour successfully, and receives the routes to the Remote Site, and redistributes them into OSPF, which then become the preferred path in the OSPF network, and subsequently preferred by the CEN-RTR. So routing restores back to "normal".
Below is copies of the relevant sections of configuration from the key four devices illustrated in the diagram and participating in the design. I'm not BGP expert so this configuration may not be ideal and I'm open to suggestions for improvement.
B1H-RTR
-------
interface Loopback0
ip address 172.27.127.247 255.255.255.255
!
interface GigabitEthernet1/0/1
description Telco WAN Cloud
no switchport
ip address 172.27.126.102 255.255.255.248
speed 100
duplex full
no mdix auto
!
interface Vlan582
description To Campus Network
ip address 172.27.126.114 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip ospf message-digest-key 1 md5 7 0225267F5F562B701C
ip ospf hello-interval 4
!
interface Vlan583
description To Campus Network
ip address 172.27.126.122 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip ospf message-digest-key 1 md5 7 0225267F5F562B701C
ip ospf hello-interval 4
!
router ospf 1
log-adjacency-changes detail
auto-cost reference-bandwidth 10000
area 0 authentication message-digest
redistribute static metric 100 metric-type 1 subnets
redistribute bgp 64614 metric 10 metric-type 1 subnets route-map RM-BGP-TO-OSPF
passive-interface default
no passive-interface Vlan582
no passive-interface Vlan583
network 172.27.0.0 0.0.255.255 area 0
!
router bgp 64614
bgp log-neighbor-changes
timers bgp 1 3
neighbor 172.27.126.97 remote-as 64514
!
address-family ipv4
redistribute connected
redistribute static
redistribute ospf 1 match internal external 1 external 2 nssa-external 1 nssa-external 2
neighbor 172.27.126.97 activate
neighbor 172.27.126.97 default-originate
neighbor 172.27.126.97 weight 40000
neighbor 172.27.126.97 soft-reconfiguration inbound
neighbor 172.27.126.97 prefix-list PL-BGP-DROP-DEFAULT-ROUTE in
neighbor 172.27.126.97 route-map RM-BGP-WEIGHT in
distance 20 0.0.0.0 255.255.255.255
distance 20 0.0.0.0 0.0.0.0
no auto-summary
no synchronization
bgp redistribute-internal
network 0.0.0.0
exit-address-family
!
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE description DO NOT ACCEPT THE DEFAULT ROUTE FROM THE GWIP CLOUD
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31
route-map RM-BGP-WEIGHT permit 10
set local-preference 40000
set weight 40000
!
route-map RM-BGP-TO-OSPF permit 10
set tag 666
!
!
CEN-RTR
-------
interface Loopback0
ip address 172.27.127.248 255.255.255.255
!
interface Vlan563
description Microwave backup link
ip address 172.27.126.69 255.255.255.248
no ip proxy-arp
ip ospf message-digest-key 1 md5 7 013024200F5B225E71
arp timeout 300
!
router ospf 1
log-adjacency-changes detail
auto-cost reference-bandwidth 10000
area 0 authentication message-digest
redistribute static subnets route-map STATIC_TO_OSPF
redistribute bgp 64614 metric 1000 subnets route-map RM-BGP-TO-OSPF
passive-interface default
no passive-interface GigabitEthernet1/4
no passive-interface Vlan3
no passive-interface Vlan67
no passive-interface Vlan559
no passive-interface Vlan561
no passive-interface Vlan562
network 10.127.0.0 0.0.255.255 area 0
network 10.176.0.0 0.0.255.255 area 0
network 172.27.0.0 0.0.255.255 area 0
network 192.168.0.0 0.0.255.255 area 0
!
router bgp 64614
bgp log-neighbor-changes
timers bgp 1 3
neighbor 172.27.126.68 remote-as 64616
!
address-family ipv4
neighbor 172.27.126.68 activate
neighbor 172.27.126.68 default-originate
neighbor 172.27.126.68 prefix-list PL-BGP-DROP-DEFAULT-ROUTE in
neighbor 172.27.126.68 route-map RM-BGP-WEIGHT in
neighbor 172.27.126.68 route-map RM-BGP-ROUTES-OUT out
neighbor 172.27.126.68 filter-list 1 out
distance bgp 190 200 1
distance 190 0.0.0.0 255.255.255.255
distance 190 0.0.0.0 0.0.0.0
no auto-summary
no synchronization
bgp redistribute-internal
network 0.0.0.0
exit-address-family
!
ip as-path access-list 1 deny .*
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE description DO NOT ACCEPT THE DEFAULT ROUTE FROM THE BACKUP LINK
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31
route-map STATIC_TO_OSPF permit 20
set metric 140
set metric-type type-2
!
route-map RM-BGP-ROUTES-OUT permit 10
set metric 200
set as-path prepend 64614 64614 64614
!
route-map RM-BGP-WEIGHT permit 10
set weight 1
!
route-map RM-BGP-TO-OSPF permit 10
set weight 1
!
MLB-RTR
-------
interface FastEthernet2/0/24
description Telco WAN Cloud
no switchport
ip address 172.27.126.110 255.255.255.248
speed 100
duplex full
no mdix auto
spanning-tree portfast
spanning-tree bpdufilter enable
spanning-tree bpduguard enable
!
interface Vlan568
description Campus Network Routing
ip address 172.27.127.178 255.255.255.248
no ip proxy-arp
ip ospf message-digest-key 1 md5 7 123A2733465B28557A
!
router ospf 1
log-adjacency-changes detail
area 0 authentication message-digest
redistribute static subnets route-map STATIC_TO_OSPF
redistribute bgp 64616 metric 100 metric-type 1 subnets route-map RM-BGP-TO-OSPF
passive-interface default
no passive-interface Vlan568
network 0.0.0.0 255.255.255.255 area 0
default-information originate
!
router bgp 64616
bgp log-neighbor-changes
timers bgp 1 3
neighbor 172.27.126.105 remote-as 64516
neighbor 172.27.127.179 remote-as 64616
!
address-family ipv4
redistribute connected
redistribute static
redistribute ospf 1 route-map RM-OSPF-TO-BGP
neighbor 172.27.126.105 activate
neighbor 172.27.126.105 weight 40000
neighbor 172.27.126.105 prefix-list PL-BGP-DROP-DEFAULT-ROUTE out
no neighbor 172.27.127.179 activate
distance bgp 20 20 1
distance 20 0.0.0.0 255.255.255.255
distance 20 0.0.0.0 0.0.0.0
no auto-summary
no synchronization
bgp redistribute-internal
network 0.0.0.0
exit-address-family
!
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 10 deny 0.0.0.0/0
ip prefix-list PL-BGP-DROP-DEFAULT-ROUTE seq 20 permit 0.0.0.0/0 ge 1 le 31
route-map STATIC_TO_OSPF permit 20
set metric 130
set metric-type type-2
!
route-map RM-OSPF-TO-BGP deny 10
match tag 666
!
route-map RM-OSPF-TO-BGP permit 20
set weight 40000
!
route-map RM-BGP-TO-OSPF permit 10
set tag 666
!
TSB-RTR
interface Loopback0
ip address 172.27.127.243 255.255.255.255
!
interface Vlan563
description Microwave backup link
ip address 172.27.126.68 255.255.255.248
no ip proxy-arp
ip ospf message-digest-key 1 md5 7 013024200F5B225E71
arp timeout 300
!
interface Vlan568
description Campus Network Routing
ip address 172.27.127.179 255.255.255.248
no ip proxy-arp
ip ospf message-digest-key 1 md5 7 013024200F5B225E71
!
router ospf 1
log-adjacency-changes detail
area 0 authentication message-digest
redistribute static subnets route-map STATIC_TO_OSPF
redistribute bgp 64616 metric 1000 metric-type 1 subnets route-map RM-BGP-TO-OSPF
passive-interface default
no passive-interface Vlan568
network 0.0.0.0 255.255.255.255 area 0
default-information originate metric 1000
!
router bgp 64616
bgp log-neighbor-changes
timers bgp 1 3
neighbor 172.27.126.69 remote-as 64614
neighbor 172.27.127.178 remote-as 64616
!
address-family ipv4
redistribute connected
redistribute static
redistribute ospf 1 route-map RM-OSPF-TO-BGP
neighbor 172.27.126.69 activate
neighbor 172.27.126.69 prefix-list PL-BGP-DEFAULT-ROUTE-ONLY in
no neighbor 172.27.127.178 activate
distance bgp 190 200 1
distance 190 0.0.0.0 255.255.255.255
distance 190 0.0.0.0 0.0.0.0
no auto-summary
no synchronization
bgp redistribute-internal
exit-address-family
!
ip prefix-list PL-BGP-DEFAULT-ROUTE-ONLY description ONLY ACCEPT A DEFAULT ROUTE FROM RADIO LINK
ip prefix-list PL-BGP-DEFAULT-ROUTE-ONLY seq 10 permit 0.0.0.0/0
route-map STATIC_TO_OSPF permit 20
set metric 130
set metric-type type-2
!
route-map RM-OSPF-TO-BGP deny 10
match tag 666
!
route-map RM-OSPF-TO-BGP permit 20
set weight 1
!
route-map RM-BGP-TO-OSPF permit 10
set tag 666
!
