BGP- Failover between Dual Homed MPLS Connections

[awatson20]

We have a location where we have 2 MPLS connections coming in from 2 separate providers.  These MPLS networks connect back to our Central HQ.  The Primary connection is a 100MB MPLS with ISP1.  The 2nd connection is a 20MB MPLS connection with ISP2.  We are doing BGP with both.  For bandwidth requirements, we prefer the traffic to go across ISP1, the 100MB link.  What is happening is at times the connection to ISP1 will drop, or we will lose BGP adjacency, and the routing will automatically failover to the 2nd MPLS connection ISP2.  The automtic faiolver is good, but that link is only 20MB and cannot handle the load for a long time.  When the primary MPLS connection is re-established, and we learn routes again, and BGP adjacency is formed, the router will not fail back over to the primary.  It will continue to prefer ISP 2 until we manually take down that link, forcing the traffic back across ISP1.  Ideally, we want the traffic to route back across the primary when it comes back up.  Is there a way to modify the BGP preferences to make this happen.  Any suggestions or advice would be very much appreciated.

[Jon Marshall]

What sits behind those routers in the location ie. is it  L3 switch for example. If it is are you redistributing BGP into another routing protocol/using static routes/HSRP etc ?

Are you running IBGP between your 2 MPLS routers ?

Jon

[awatson20]

Yes, layer 3 swiiches.  On the WAN router which is a 3945 we are redistributing BGP into EIGRP.  Both MPLS connections terminate into this single router.

[Jon Marshall]

So it's one router for both EBGP connections. If this is the case then you can use the weight command to influence which routes to use when both are up. So at the moment when you do a "sh ip bgp" you should see 2 routes for each HQ prefix with a weight of 0 because 0 is the weight of an route learned from an EBGP peer.  Weight is the first thing used in the BGP best path selection process and is Cisco specific.

What you can do is under your bgp config is use the "neighbor x.x.x.x  weight

Note the above command applies the same weight to all routes learned from a neighbor which sounds like what you want but you can if needed filter on specific routes using a filter-list.

Jon

[awatson20]

Ok, thanks.  We were thinking that a weight modification would have to be made.  So by doing this, if the primary connection goes down, once it comes back up and those BGP routes are learned, because they have a higher weight it should prefer that path?

[Jon Marshall]

So by doing this, if the primary connection goes down, once it comes back up and those BGP routes are learned, because they have a higher weight it should prefer that path?

Yes.

Jon

[awatson20]

We are getting ready to make the weight changes so BGP will prefer routes from the one primary neighbor.  If there is a scenario where the routes are flapping, is there a way to specify a time period before BGP will converge and prefer that route to minimize the effect of flapping?

[Jon Marshall]

Yes, you can use BGP dampening although i have never personally used it -

http://www.cisco.com/en/US/docs/ios/iproute_bgp/command/reference/irg_bgp1.html#wp1113513

Jon

[awatson20]

We implemented the weight change which made BGP prefer the higher weight route from the primary neighbor. However, what we noticed was after doing some failover testing, traffic was going out towards HQ the correct way, but from HQ to this location it was going across the other connection, not the one we would prefer.

At HQ, we have 2 routers each with a connection to both MPLS providers. Will we also have to modify the BGP config there as well? Would we need to create a route filter for that specific subnet at the remote location and apply the weight to that neighbor using the filter? Thanks for your help.

Sent from Cisco Technical Support iPhone App

[awatson20]

Below is the route-map we are looking at applying.  Would this correctly apply a weight of 500 to 172.28.0.0 routes learned from the neighbor X.X.X.X?

access-list 2 permit 172.28.0.0 0.0.255.255

route map TEST permit 10
match ip address 2
set weight 500

router bgp XXXXX
neighbor X.X.X.X route-map TEST in

cleap ip bgp X.X.X.X

[Jon Marshall]

traffic was going out towards HQ the correct way, but from HQ to this location it was going across the other connection, not the one we would prefer.

At HQ, we have 2 routers each with a connection to both MPLS providers

traffic was going out towards HQ the correct way, but from HQ to this location it was going across the other connection, not the one we would prefer.

At HQ, we have 2 routers each with a connection to both MPLS providers

Weight is only locally significant on a single router. So the remote site has one WAN router with 2 MPLS connections. You applied a weight so that routes would be preferred over the primary link and this worked but it only affect the traffic on that router.  If you want the return traffic from HQ to come back via the same link then yes you need to modify the config at HQ.

You have 2 separate routers at HQ though so you cannot use weight because the routers don't communicate weight between them. So you either use -

1) local preference on the HQ routers if these routers have an IBGP peering between them

or

2) if they don't have an IBGP peering are you simply redistributing into EIGRP at HQ. If so when you redistribute you need to influence the metrics so the router you would to use at HQ for return traffic is sent back over the right link.

Can you explain how HQ is setup in terms of the routers/L3 switches, routing protocol(s) etc.

How does HQ choose which router to use ?

Note using weight at HQ will not work if the return packets from the remote site could go to either HQ WAN router from the internal network.

Jon

[awatson20]

At HQ, we have 2 edge routers both doing EBGP with 2 MPLS providers.  These routers are actually in 2 different Data Centers.  These routers are also doing IBGP between them.

On these routers, we are redistributing BGP into EIGRP.  We are doing EIGRP between the edge router and our core 6500 switches.

Typically though, even though there are 2 routers both with redundant links to both MPLS providers, the traffic will usually prefer 1 over the other which is the primary.  We do apply a metric on the redistribution into eigro.  See below.

router eigrp XXXX

network 172.31.0.0

redistribute bgp 65100 metric 170 1000 200 200 512

So on your last point, (Note using weight at HQ will not work if the return packets from the remote site could go to either HQ WAN router from the internal network.) they can go to either WAN router but currently only go through 1 unless there is a failure scenario.

This particular remote site happens to have a link to both MPLS providers.  Most of our remote locations are either on one or the other.  We want the return traffic to prefer the primary connection for this location, rather than the other MPLS connection.  I hope I have provided enough information.  Thank you.

[Jon Marshall]

Thanks for the explanation.

they can go to either WAN router but currently only go through 1 unless there is a failure scenario.

We want the return traffic to prefer the primary connection for this location, rather than the other MPLS connection.

I am a bit confused about the above statements which seem to contradict each other ie. all traffic from HQ goes via the primary but the second statement suggests that isn't happening.

So HQ has 2 DCs interconnected, each DC has an MPLS router. Each DC has a core pair of switches ?

Traffic from the remote site is sent down the primary link to HQ. Which link does it come in on ie. if it is the primary link at HQ and all traffic is returning using that primary link then why is traffic going via the other link ?

Jon

[awatson20]

I apologize for the confusion.  I will attempt to explain.

So HQ has 2 DCs interconnected, each DC has an MPLS router. Each DC has a core pair of switches ? YES

At the HQ(main DC, backup DC), there are 2 edge WAN routers.  Each has a connection to both MPLS providers.  One is AT&T and the other is Windstream.  So both routers are peering to both AT&T and Windstream at the Main Data Center and the backup Data Center at HQ.

At this remote site, there is a connection to both AT&T and Windstream.  The windstream connection has more bandwidth, so we want it to prefer Windstream, and back across Windstream for the return traffic.  Since we modified the weight, it is preferring that path outbound from the remote site.

From the HQ perspective, when I say primary, I am referring to the Main Data Center edge router.  Whats happening now is when there is a failure at the remote site with the Windstream connection, traffic will fail over and route across AT&T for both outbound, and inbound traffic from the HQ.  When Windstream comes back online, the weighted path is preferred on the router at the remote location, but its still coming back across the AT&T link at HQ.

For both AT&T and Windstream, traffic will typically route across the Primary edge router in our main data center, not the backup edge router.  It can route across the backup edge router, we just usually see it go out the primary.  The backup does peer with the same providers, and also is peering to the primary using IBGP, but unless the primary router, or those pipes fail, it will not traverse the backup.

I hope this makes ssome sense.

[Jon Marshall]

Yes it does make sense.

So basically when the Windstream connection fails at the remote site traffic is sent via AT&T link and comes into primary DC on AT&T. Because the remote site is no longer advertising routes via Windstream link then HQ only gets routes to the remote site via AT&T link so return traffic from HQ primary uses AT&T link.

When the Windstream link comes back online at the remote site routes are then advertised via that link to primary HQ site. The weight setting on the remote site means the Windstream link is used but when HQ comes to return the traffic it still uses the AT&T connection.

What i didn't understand was that the router at the primary HQ site has both links on the same router so yes, you can use weight. So if your route map config was for the primary DC router then yes it would apply that weight to the inbound route advertisement for 172.28.0.0 0.0.0.255 which is presumably in the remote site ?

I don't have anything to test on but you may need a second empty permit statement in your route-map to allow in all other routes. I can't remember whether the route map you configured would deny any routes not matching the acl or whether it would allow other routes but simply not set any weight. 

Finally you are doing BGP to EIGRP redistribution but i have assumed you are not doing EIGRP to BGP redistribution as well ?

Jon