Why did my OSPF config fail to failover

Drake22x · ‎06-26-2017

Hi everybody,

I have 3 ISPs connected to a site with 2 tunnels each going to separate physical locations (6 tunnels in total).

Tunnel1 - secondary to DC1 (ISP B)

Tunnel2 - primary to DC1 (ISP A)

Tunnel3 - tertiary to DC1 (ISP C)

Tunnel4 - primary to DC2 (ISP A)

Tunnel5 - secondary to DC2 (ISP B)

Tunnel6 - tertiary to DC2 (ISP C)

Unfortunately, few days ago I've experienced OSPF neighbor drop to DC1 and 2 (in Tunnels 2 and 4), but the OSPF did not failover on the backup tunnels and I cannot figure out why. I have contacted ISP A as it looked weird that both tunnels to separate locations would drop at the same time, but they've advised they experienced no outages and provided NMS graphs to prove so. I have the following tracking routes in place:

ip route vrf INTERNET 0.0.0.0 0.0.0.0 83.x.x.x5 10 track 2
ip route vrf INTERNET 0.0.0.0 0.0.0.0 167.x.x.x3 track 3
ip route vrf INTERNET 0.0.0.0 0.0.0.0 79.x.x.x1 20

The tunnel configs are as follows:

interface Tunnel1
description Secondary DC1 tunnel
ip vrf forwarding LOCAL
ip address 10.60.60.2 255.255.255.252
ip ospf network broadcast
ip ospf mtu-ignore
ip ospf cost 300
tunnel source 83.x.x.x9
tunnel mode ipsec ipv4
tunnel destination 85.x.x.x8
tunnel vrf INTERNET
tunnel protection ipsec profile CUST_Profile
end

MULLON-RT01#sh run int tu2
Building configuration...

Current configuration : 265 bytes
!
interface Tunnel2
description Primary DC1 tunnel
ip vrf forwarding LOCAL
ip address 10.60.60.10 255.255.255.252
ip ospf cost 100
tunnel source 167.x.x.x6
tunnel mode ipsec ipv4
tunnel destination 85.x.x.x8
tunnel vrf INTERNET
tunnel protection ipsec profile CUST_Profile
end

MULLON-RT01#sh run int tu3
Building configuration...

Current configuration : 266 bytes
!
interface Tunnel3
description Tertiary DC1 tunnel
ip vrf forwarding LOCAL
ip address 10.60.60.18 255.255.255.252
ip ospf cost 700
tunnel source 79.x.x.x5
tunnel mode ipsec ipv4
tunnel destination 85.x.x.x8
tunnel vrf INTERNET
tunnel protection ipsec profile CUST_Profile
end

MULLON-RT01#sh run int tu4
Building configuration...

Current configuration : 311 bytes
!
interface Tunnel4
description Primary DC2 tunnel
ip vrf forwarding LOCAL
ip address 10.60.60.25 255.255.255.252
ip ospf cost 3500
tunnel source 167.x.x.x6
tunnel mode ipsec ipv4
tunnel destination 85.y.y.y3
tunnel vrf INTERNET
tunnel protection ipsec profile CUST_Profile_R2
end

MULLON-RT01#sh run int tu5
Building configuration...

Current configuration : 315 bytes
!
interface Tunnel5
description Secondary DC2 tunnel
ip vrf forwarding LOCAL
ip address 10.60.60.29 255.255.255.252
ip ospf cost 3600
tunnel source 83.x.x.x9
tunnel mode ipsec ipv4
tunnel destination 85.y.y.y3
tunnel vrf INTERNET
tunnel protection ipsec profile CUST_Profile_R2
end

MULLON-RT01#sh run int tu6
Building configuration...

Current configuration : 313 bytes
!
interface Tunnel6
description Tertiary DC2 tunnel
ip vrf forwarding LOCAL
ip address 10.60.60.33 255.255.255.252
ip ospf cost 3700
tunnel source 79.x.x.x5
tunnel mode ipsec ipv4
tunnel destination 85.y.y.y3
tunnel vrf INTERNET
tunnel protection ipsec profile CUST_Profile_R2
end

and ospf config as follows:

router ospf 2 vrf LOCAL
capability vrf-lite
network 10.50.0.0 0.0.0.7 area 0
network 10.50.0.17 0.0.0.0 area 0
network 10.51.0.0 0.0.0.7 area 1
network 10.60.60.0 0.0.0.255 area 1

The outage lasted 10 minutes and it was just dead timer expired on Tu2 and Tu4, and then came back, however other tunnels never became active.

Hoping to shed some light onto this, many thanks for any help :)

Philip D'Ath · ‎06-26-2017

You don't need route tracking for the tunnels. You should probably use local policy routing to force a specific tunnel out a specific ISP, regardless of weather that ISP is up or down. At the moment it will be a bit indeterminate.

Use something like the below. You might have to play with the VRF. Not sure. I would also add the command "keepalive" under each tunnel for each site, so that the tunnels check the end to end path, to make sure the path is good and self shut themselves down on failure to help OSPF decide.

ip access-list extended toISPA
 permit ip host 167.x.x.x6 any
ip access-list extended toISPB
 permit ip host 79.x.x.x5 any

route-map chooseISP permit 10
 match ip address toISPA
 set ip next-hop 167.x.x.x3
route-map chooseISP permit 20
 match ip address toISPB
 set ip next-hop 79.x.x.x1

ip local policy route-map chooseISP

Drake22x · ‎06-27-2017

Philip, thanks for your input. I was wondering if using route map configuration that you've posted would potentially cause premature failovers if say, just one or two pings are dropped, or it would failover back and forth if the link is intermittent?

The tracking setup with ip sla I have is like this:

track 2 ip sla 2 reachability
delay down 30 up 180
track 3 ip sla 3 reachability
delay down 30 up 180
ip route vrf INTERNET 0.0.0.0 0.0.0.0 83.x.x.x5 10 track 2
ip route vrf INTERNET 0.0.0.0 0.0.0.0 167.x.x.x3 track 3

ip sla 2
icmp-echo 83.x.x.x5 source-interface GigabitEthernet0/0/1.201
vrf INTERNET
frequency 5
ip sla schedule 2 life forever start-time now

ip sla 3
icmp-echo 167.x.x.x3 source-interface GigabitEthernet0/0/1.202
vrf INTERNET
frequency 5
ip sla schedule 3 life forever start-time now

Wouldn't the frequency command be essentially the same as a keepalive? And do you have any idea as to reason why the OSPF would drop to both tunnels 2 and 4, as I am really curious to why it happened and why it didn't switch to the other tunnels

Many thanks

Philip D'Ath · ‎06-26-2017

I missed you were using three ISPs. Just extend the framework I have given you to support as many links as you want.