NX-OS 9000 eBGP Failover Help

[jobrien2]

Hello,

   I've been struggling getting eBGP failover between 2 Data Centers to work properly while deploying some new VPN hardware. I need routing to go through MAIN DC, and only go to BACKUP DC if the MAIN DC fails. I can get this to work just fine. However, when MAIN DC comes back online, routing does not fail back and will stay at BACKUP DC. I see the metric get pushed to MAIN DC from BACKUP DC, but the AD stays at the 110 for OSPF, even though it should(?) switch back to 20 for local eBGP. I've not found any articles or posts that relate to failing back to a MAIN DC using NX-OS BGP.

The VPN tunnels are confirmed built to both sites. MAIN DC and BACKUP DC are connected via MPLS.

I have attached the current configuration with everything that I've tried. I know it's a lot, but as you can see I've been trying everything I can find. I kept it in for now to try and find what works. Please help!

• MAIN DC

route-map OSPF->eBGP permit 10
route-map eBGP->OSPF permit 10
router ospf 1
   network (internals) area 0.0.0.0
   redistribute bgp 248 route-map eBGP->OSPF
router bgp 248
   address-family ipv4 unicast
      redistribute ospf 1 route-map OSPF->eBGP
   neighbor 10.1.1.1
      remote-as 250
      update-source Ethernet1/1
      address-family ipv4 unicast

• BACKUP DC

route-map OSPF->eBGP permit 10
route-map eBGP->OSPF permit 10
   set metric +23456
   set weight 65000
   set as-path prepend last-as 10
   set distance 251 251 251
router ospf 1
   network (internals) area 0.0.0.0
   redistribute bgp 248 route-map eBGP->OSPF
router bgp 248
   address-family ipv4 unicast
      redistribute ospf 1 route-map OSPF->eBGP
      distance 251 251 251
   neighbor 10.2.2.2
      remote-as 250
      address-family ipv4 unicast
         weight 65000

•  SHOW IP ROUTE 10.x.x.x

MAIN DC# show ip route 10.x.x.x

10.x.x.x/yy, ubest/mbest: 1/0
*via BACKUP DC LINK, [110/23457], 00:33:37, ospf-1, type-2, tag 250

[Georg Pauwen]

Hello,

post the full running configurations of both the main and the backup site routers.

[jobrien2]

Hi, thank you. I have attached as much of the config that I can post, as well as a high-level diagram of what we have. We're aware our 9ks have a lot of issues that need corrected in Q4/Q1 (in the middle of redoing the network architecture), but for now our priority is just the VPN with BGP failover. Unfortunately I'm the most experienced with NX-OS as a 1 year CCNA, so that's our knowledge base with NX-OS.

We have 3 sites: 1 Main Data Center, 1 Backup Data Center, and 1 building full of nothing but users. It is possible that this issue is on the SD-WAN hardware side, but I would like to exhaust all of my options so that my configuration is clean if I have to go back to their techs.

[Jon Marshall]

Do you have an alternate route between your DCs ie, not via MPLS ? 

Jon

[jobrien2]

Hi, thank you. Kind of, I attached the high level diagram to the post above. We have a triangle between 3 sites, but the DC to User site links are used only if the MPLS were to fail.

[Jon Marshall · Accepted solution]

It's still not clear to me what is backing up what and which way you want the routing to work so take the following as merely a suggestion. 

With mutual redistribution what can happen is if your main DC router loses it's connection with it's BGP peer it then uses the OSPF route learnt from the backup DC and because you are redistributing OSPF into BGP that route is then installed as a BGP route on the main DC router. 

This BGP route will have a weight of 32768 because it is a locally generated route. 

When the main DC relearns the BGP route from it's peer that route will have a weight of 0 but it already has a BGP route with a better weight (32768) so it does nothing ie. it does not install the newly learned BGP route into the routing table.

If it does not install the newly learned BGP route into the IP routing table it will stay with the OSPF route it learnt from the backup DC. 

The solution to this is to increase the weight of BGP routes learnt from it's peer to be > 32768. 

Jon

[jobrien2]

Hi Jon, thanks again. I think you may have solved this, and I'll test tonight. This is the only BGP that we have, so please bear with me as I walk through this.

I've changed DC names to help clarify. Each DC has its primary NX-9k form a BGP peer with the SD-WAN Headend. Each remote site SD-WAN box builds a tunnel to each DC, and the Headend box advertises the remote IPs to the NX-9k. We want all traffic to take the tunnel to the PRIMARY DC, and only take SECONDARY DC if the PRIMARY fails. Once the PRIMARY comes back online, we want traffic to fail back to the PRIMARY DC. This BGP routing will be redistributed through OSPF that runs through our network. Does that make sense? It seems to make sense to me after reading your post and doing more research in that direction.

I have seen the fail-over to SECONDARY work, but I could not get it to return to PRIMARY once it is back online. I think you're pointing out that it's as simple as me having the weight in the wrong DC? This would be great news.

Here is the planned configuration, removing all excess config and adding weight 65000 to the PRIMARY to SD-WAN peer:

• PRIMARY DC

route-map OSPF->eBGP permit 10
route-map eBGP->OSPF permit 10
router ospf 1
   network (internals) area 0.0.0.0
   redistribute bgp 248 route-map eBGP->OSPF
router bgp 248
   address-family ipv4 unicast
      redistribute ospf 1 route-map OSPF->eBGP
   neighbor 10.1.1.1
      remote-as 250
      update-source Ethernet1/1
      address-family ipv4 unicast

         weight 65000

• SECONDARY DC

route-map OSPF->eBGP permit 10
route-map eBGP->OSPF permit 10
router ospf 1
   network (internals) area 0.0.0.0
   redistribute bgp 248 route-map eBGP->OSPF
router bgp 248
   address-family ipv4 unicast
      redistribute ospf 1 route-map OSPF->eBGP
   neighbor 10.2.2.2
      remote-as 250
      address-family ipv4 unicast

[Jon Marshall]

 I haven't used SD-WAN but from your description of the setup and the actual problem it does sounds like that is the issue. 

I did wonder what the weight was doing on the backup router and yes the weight needs to be applied on the main router for the BGP routes learnt from the SD-WAN device to be used when the main connection comes back up. 

Let me know how it goes. 

Jon

[jobrien2]

Jon I can't thank you enough, that was my issue. My failover now works as intended. I added the weight on the Primary DC, and then added distance 200 to my Secondary DC to bump the AD over OSPF. Thanks again!

[Jon Marshall]

Glad you got it working, thanks for letting me know. 

Jon