BFD, OSPF, and Metro Ethernet

[mfarrenkopf]

We are having some issues with MPLS stability in our metro Ethernet environment (LDP fails; OSPF does not).  We're going to turn on BFD.  We have multiple remote sites using OSPF peered to 3 or 4 hub routers on a /24.  I've done a lot of reading, a little virtual labbing, but no production experience, so I just want some confirmation

It is my understanding that BFD must establish a BFD neighborship first.  By default, when OSPF registers with BFD, it has BFD establish neighborships with the DR and BDR (any router with which it is in FULL state).  Once those neighbors are established, a failure in the forwarding path causes the BFD neighborship to go down and OSPF gets signaled.

On this metro Ethernet environment, we have about 35 routers (31 sites plus 4 hubs).

If I turn on BFD on the hubs (two of which are the DR and BDR), then turn on BFD on five remote sites (the ones having the largest issues), the rest of the sites will be unaffected since a BFD neighborship was never established to them, correct?  Ultimately, I want to turn BFD on for all remote sites, but since this is my first foray into BFD in production, I just want to double check with the worst sites. 

[Evgeniy Prichinin]

You should move to p2p networks. After that you can tune hello/dead-intervals and enable BFD.

Using broadcast segments are very bad in mpls environment.

[mfarrenkopf]

I appreciate that, but that's not the situation I'm facing right now.  We have an opportunity to put several of our remote sites on their own fiber ring, and we're pursuing it, but it takes time.

The issue is that OSPF is staying up when LDP fails.  We recently migrated to OSPF from EIGRP.  Coincidentally, perhaps, EIGRP's hold timer and LDP's hold timer are both 15 seconds.  So the issue may have been masked by the fact that they both expired at close to the same time.

So we at least need an interim solution until we can get the fiber design in place.

[Giuseppe Larosa]

Hello mfarrenkopf,

>> We are having some issues with MPLS stability in our metro Ethernet environment (LDP fails; OSPF does not)

This is strange because by default LDP timers are longer then OSPF.

However, you have 35 devices on the same broadcast domain.

This is quite challenging for LDP because:

OSPF has the DR/BDR concept to avoid the full mesh N*(N-1)/2 problem that makes the number of adjacencies to follow a 2*N+2.

LDP uses UDP 646 for discovery and then has to build unicast TCP based on port 646 sessions with all of them

So each device needs to setup and mantain 34 LDP sessions on the link.

So I agree with the other colleague: you need to change the underlying L2 infrastructure in order to reduce the number of LDP sessions.

If your metro ethernet service supports 802.1Q Vlan tags you can build multiple smaller broadcast domains by using different Vlan-IDs.

I don't think that BFD can provide benefits here as the issues are on LDP scalability.

Probably buffer tuning and increasing the input buffer on the interface connecting to the metro ethernet service can mitigate the instability.

However, your network design requires a great review on the light of what I have explained above: LDP has no DR/BDR concept and so it requires a full mesh of LDP unicast TCP sessions between all devices.

This puts an excessive pressure on some devices.

In addition to this you need to think that for each prefix LDP is going to store 34 alternative labels from each of the LDP neighbors (all of them are received and stored in MPLS packet mode but only the ones coming from best paths according to OSPF metric are used this is called retention).

You can verify this with

show mpls ldp binding <prefix>.

This is again a waste of resources unless you are using a form of LDP label filtering to allocate and advertise labels only for loopbacks.

Hope to help

Giuseppe

[mfarrenkopf]

I just replied to the other responder that we recently changed to OSPF from EIGRP.  It could be the problem was masked because LDP's hold timer is 15 seconds (shorter than OSPF) and EIGRP's hold timer is also 15 seconds; OSPF's dead timer is 40 seconds.

Our recent problems do not appear to be an inherent LDP stability problem.  We had a 20 minute outage last week that our provider admitted was their fault, although they didn't tell us what happened.  But LDP failed while OSPF stayed up.  For 20 minutes.  Somehow OSPF HELLOs had to be getting through, at least often enough.

Our OSPF config uses LDP sync.  I believe that was available on all our platforms.  Most of our platforms have prefix suppression as well.  (We have a couple of devices not on the metro-E that are running a 12.2-series of code, which doesn't support prefix suppression.)  So that should reduce the number of labels.

For context, our two primary metro-E hubs are in the data centers and have 1 Gb links.  The remote sites have 50-100 Mb links and are not saturated, so I don't believe this to be a buffering issue.  This is why I'm looking at BFD, is to cause the OSPF sessions to fail as well.  If that happened, traffic would've shifted to redundant links at these sites and would've been a blip instead of an outage.

I also told the other respondent we do have an opportunity to connect several remote sites via a fiber ring, and we're working on that, but it's not a quick fix.

[mfarrenkopf]

I don't know if this reply got deleted for some reason.  I got no e-mail saying it violated any terms, and I got this copied to me as a reply I posted, so . . . ???

I just replied to the other responder that we recently changed to OSPF from EIGRP. It could be the problem was masked because LDP's hold timer is 15 seconds (shorter than OSPF) and EIGRP's hold timer is also 15 seconds; OSPF's dead timer is 40 seconds.
Our recent problems do not appear to be an inherent LDP stability problem. We had a 20 minute outage last week that our provider admitted was their fault, although they didn't tell us what happened. But LDP failed while OSPF stayed up. For 20 minutes. Somehow OSPF HELLOs had to be getting through, at least often enough.
Our OSPF config uses LDP sync. I believe that was available on all our platforms. Most of our platforms have prefix suppression as well. (We have a couple of devices not on the metro-E that are running a 12.2-series of code, which doesn't support prefix suppression.) So that should reduce the number of labels.
For context, our two primary metro-E hubs are in the data centers and have 1 Gb links. The remote sites have 50-100 Mb links and are not saturated, so I don't believe this to be a buffering issue. This is why I'm looking at BFD, is to cause the OSPF sessions to fail as well. If that happened, traffic would've shifted to redundant links at these sites and would've been a blip instead of an outage.
I also told the other respondent we do have an opportunity to connect several remote sites via a fiber ring, and we're working on that, but it's not a quick fix.

[Giuseppe Larosa]

Hello mfarrenkopf,

don't worry sometimes also Cisco forums have their issues.

>> But LDP failed while OSPF stayed up. For 20 minutes

This is a different matter where your ISP has some responsabilities.

>> Our OSPF config uses LDP sync. I believe that was available on all our platforms

With LDP sync OSPF metric should be maxed until LDP is up again that is each link in a router LSA should have maximum metric 65535 until LDP comes up again to make the OSPF path not used.

However, in a single broadcast domain with 35 routers I don't know if OSPF LDP sync is triggered only if all the LDP sessions are down at the same time.

It is good that the you are using label filtering.

As a short time fix I would look for using Vlan tags to divide the broadcast domain in multiple smaller domains.

Using BFD can be helpful to tear down OSPF if there are short connectivity interruptions.

>> I also told the other respondent we do have an opportunity to connect several remote sites via a fiber ring, and we're working on that, but it's not a quick fix.

Fiber rings are not the best topology for routing and MPLS at least for a service provider, but it can be better then current topology.

Hope to help

Giuseppe