BGP failover to IPSEC design - Configuration problems

[Marc Bouchard]

I am working on a design in GNS3 to failover our MPLS links to IPSEC. At the moment, all I'm trying to do is setup BGP properly and i'm having a heck of a hard time figuring this out (very rusty here...)

Based on the diagram below:

on the left is our datacenter. We have 2 MPLS links and an IPSEC router, as well as our core switches. Core switches run OSPF only. The others run both OSPF and BGP. The branch office on the right has a similar setup with a single MPLS link.

Objectives:

DataCenter: Load balance MPLS-A and MPLS-B, and failover if both are down to IPSEC tunnel (will be a DMVPN config with all branch offices).

BranchOffice: Failover to IPSEC if MPLS goes down.

- Fastest convergence possible

- Nice to have: failover if MPLS is up but connection has quality issues (packet loss for example).

- Nice to have: failback once all issues are resolved (but must avoid flapping)

BGP is using a single AS (iBGP only)

I created a generic configuration for all BGP routers. Since I don't want to fully mesh, every single router in the branch offices and the ones in the datacenter, I will need to use route reflectors. I think this is where my issue is.

I am peering all local routers in a site, and peering with the corresponding remote router.

BGP Peerings:

DBL-MPLS-A: with DBL-MPLS-B, DBL-IPSEC and MTL-MPLS

DBL-MPLS-B: with DBL-MPLS-A, DBL-IPSEC and MTL-MPLS

MTL-MPLS: with DBL-MPLS-A, DBL-MPLS-B and MTL-IPSEC

DBL-IPSEC: with DBL-MPLS-A, DBL-MPLS-B and MTL-IPSEC

MTL-IPSEC: with MTL-MPLS and DBL-IPSEC

My basic configuration is this (both remote and local have the same config right now, but I figured I MIGHT have to have different settings so I setup two peer-groups)

router bgp <as_number>
 bgp log-neighbor-changes
 bgp redistribute-internal
 bgp scan-time 20
 network <local_aggregated_subnet> mask 255.255.0.0
 aggregate-address <local_aggregated_subnet>  255.255.0.0 summary-only
 timers bgp 5 15
 redistribute ospf <instance> metric 1
 neighbor MFS-Local peer-group
 neighbor MFS-Local remote-as <as_number>
 neighbor MFS-Local update-source Loopback1
 neighbor MFS-Local route-reflector-client
 neighbor MFS-Local next-hop-self
 neighbor MFS-Local soft-reconfiguration inbound
 neighbor MFS-Remote peer-group
 neighbor MFS-Remote remote-as <as_number>
 neighbor MFS-Remote update-source Loopback1
 neighbor MFS-Remote route-reflector-client
 neighbor MFS-Remote next-hop-self
 neighbor MFS-Remote soft-reconfiguration inbound
 neighbor <remote_site_peer> peer-group MFS-Remote
 neighbor <local_site_peer> peer-group MFS-Local

My problem is that from DBL-Core, I can't ping MTL-Core (loopback addresses). Seems like the traffic dies somewhere (loops?) Any insight would be very much appreciated.

Thanks!

Marc

[jmattbullen]

an easy way to fix the routes coming back in on the remote site add the following on both MTL-MPLS and MTL-IPSEC

router bgp 200
 neighbor 192.168.1.5 filter-list 120 out
!
ip as-path access-list 120 permit ^$

This will cause the remote side to only advertise routes back to HQ if the are initiated at the remote side.

Good luck and have fun!

[Jon Marshall]

Marc

I don't know whether this will help but I built a quick lab for you with an MPLS network in between (see the attached .png and ,txt files).

I don't have GNS3 but I do have access to another emulator so you should be able to copy and paste the configurations as long as you edit the interface parts.

The routers don't have IPSEC so I just ran a BGP connection between the failover routers but that should work in terms of testing failover.

So in a nutshell it emulates pretty much what you are trying to do. All the OSPF, BGP has been configured and works and the MPLS part works as well although you only need to configure CE, failover router and switches.

I have used OSPF area 0 for all sites and within the MPLS cloud but note these are all separate areas as far as OSPF is concerned ie. they are not joined into one large area.

I have a loopback on sw1 and a loopback on sw2 and they are successfully being advertised over MPLS and you can ping between them as long as you use an extended ping using the loopback as the source IP.

Redistribution is being done on the MPLS routers and IBGP is running between the MPLS and failover routers in each site but I haven't configured redistribution on the failover routers so you would need to add that.

Finally I found an article suggesting using IBGP across DMVPN was not the best idea ie EBGP is better but it's up to you. I have used a different AS number per site so the failover connection is EBGP. If you want to use IBGP let me know and I can sort out the configs because there is a bit of change to be done in a few places and additional configuration needed for BGP.

See this link for details -

http://blog.ipspace.net/2014/03/scaling-bgp-based-dmvpn-networks.html

Hope it is enough to get you up and running so you can test it to make sure it all works.

Let me know if you have any questions etc.

PS - sorry about all the files, I don't have a winzip version at the moment :-)

Jon

[Marc Bouchard]

Wow. Thanks for taking the time to do this. I'll look into it tonight and see what I can come up with. I have also forwarded my GNS3 setup to the consultants to look into it...

One issue I just "fixed" was that the routes coming through the "MPLS simulation" i.e. the other AS, weren't coming back in as they originated from the same AS. I had to add the allowas-in parameter for it to work and now the routes are added to the table.

Another issue I have now is that the IPSEC router in DBL sees all 3 routes to the MTL office. Great. But the two MPLS routers only see each other as possible paths. This is more than likely due to iBGP route propagation restrictions so I think I need to add a route reflector at the IPSEC level. I tried that but that seems to have made things worse :) I configured the peering with the 2 MPLS with the route-reflector-client parameter, and I ended up losing those 2 extra paths on the IPSEC router. Not the result I was hoping for.

Thankfully I wont be setting this up (the service provider will) but I want to learn and understand how to make it work so I can give them a working model.

Your help is greatly appreciated, very rarely had someone try to help out like this. Find me on Twitter if you want (@SUBNET192) - Thanks!

[Jon Marshall]

Marc

That was one of the things I meant about extra configuration if you want to run IBGP ie. allowas-in.

Yes your issue is IBGP route advertisements.

Not entirely sure what you did but you would need to make the failover routers the route reflectors in each site and then each failover router is a client of the other failover router and the MPLS router is a client of it's local failover router.

I would look at the link because you may be able to run EBGP across the DMPVN links rather than IBGP but of course that would mean using a different AS number per site.

I really do have my doubts about this design ie. the consultants say it is to avoid complexity but with local preference, route reflectors etc. it seems to be to be getting quite complex as it is already and that is before testing.

There are simpler ways to do this but it really depends on how much flexibility in path choice you want using IBGP ie. send some traffic over MPLS some over the DMPVN

If the DMVPN really is just a backup it seems unnecessarily complex to me but like I say I don't have the full picture and the consultants presumably do.

I have saved that lab so I may try testing out some failover scenarios and if I get the chance I may try IBGP across the failover link as well.

As I said before, if you need help with anything just shout :-)

Jon