MPLS failover between different ISP networks

[Chad Parish]

Question regarding MPLS WAN networks. If you have three separate sites communicating to one another via an ISP's MPLS network, all sites must be using that same network.  But if at one of these sites, the ISP drops the BGP established state between their PE router and your CE router then that site will be unable to send or receive with the rest of the sites over said MPLS network, but will then see the other network and send it's traffic over that one.

So in our case we use two different ISP's who each provide us with an MPLS network.  At any given time we are using only one of these networks between all sites by manipulating the "delay" metric in our EIGRP to BGP redistribution statement.  So in essence, both are always up but one is preferred by EIGRP.  

So with that said, if the ISP MPLS network with the preferred metrics should lose it's BGP neighbor state with one of our sites, shouldn't that site then proceed to route traffic over the less preferred ISP's MPLS network. And as the other sites still have the less preffered MPLS network established (but simply not using it), then the traffic from the one site out to the others should just failover with the other sites communicating with it over the less preffered MPLS network.

Last night, one of our local loop providers lost power to their equipment on our site, so we lost BGP connectivity with the preferred MPLS network.  However, we did not failover to the other less preffered MPLS network which still had connectivity.

Could this be due to the fact that we showed our edge router interface to the preferred ISP as UP DOWN?  In such a situation do we need to administratively shut down that interface so traffic fails over to the other MPLS network?  And if the Local Loop carrier's equipment loses power, shouldn't I see our interface out to the ISP as DOWN DOWN?

[Mark Malone · Accepted solution]

Yes in theory you should have automatically switched to the less preferred link but it depends on the BGP table was the less preferred paths in the BGP table , when you look at it now when everything is back up , do you have 2/3 routes to same destination but with different metrics like below , you shouldn't have to shut the link down if the network is unreachable and the BGP table is correct it should have flipped automatically to less preferred route once primary route failed and was removed from BGP as reachable

I would check the network in the BGP table make sure its paths are all there

example

2 valid routes for 10.136 network choses EBGP first but if there's a failover also knows it can reach it through less preferred IBGP

* i10.136.0.0/13    172.29.221.13            0    300      0 4449 65310 i
*>                          10.11.67.1                        300    0 4449 65310 i