Strange OSPF Behavior

[dennylester]

We have this strange OSPF behavior that seems to have suddenly cropped up.

Here is the setup

RouterA-2811

Attached to Sprint MPLS network (they are distributing OSPF into BGP to run across their backbone)

Running OSPF

Backups are ISDN lines using Demand Dial Routing with floating static routes (metric set at 200)

RouterB-1841

Attached to Sprint MPLS network

Running OSPF

BRI configured to accept calls

If RouterB is rebooted, when it comes up RouterA learns of RouterB subnets via OSPF and routes through the MPLS network. All appears to be well for a random number of days, weeks or months, then we'll get a call about a slow network. What we find is RouterA is dialed into RouterB and routing via the ISDN line. RouterB however, is still routing across the MPLS network.

Here's where things get confusing.

   RouterB has 3 subnets behind it.

   RouterA routes 2 of the subnets over ISDN and one of the subnets over MPLS

Sprint says they do not see RouterB advertising the other two subnets to them.

Is there a debug command I can run on RouterB to see the actual subnets being advertised? I'd like to clear the ospf processes and see what goes out.

I have over 30 other sites setup this way and saw this problem two other times over the last two months. Each time the remote 1841 was running 12.4(25b). Thinking it was an IOS bug, we upgraded to 12.4(25e) and the problem seemed to have cleared, until today.

Any suggests are welcome.

Denny

[Peter Paluch]

Hello Denny,

This is a strange problem.

A couple of questions:

1. What do you do to correct the problem once it appears? Is it sufficient to do the clear ip ospf process on the Router B?
2. When this problem appears, do both Router A and Router B report successful OSPF neighborship to Sprint's routers in the show ip ospf neighbor command?
3. Do you do any kind of redistribution on either Router A or Router B?

While it does not make much sense, it looks as if the Router B forgot to reoriginate some of its LSAs and send it towards Sprint, resulting in the Sprint aging-out that LSA and losing the connectivity in the process. It would be very helpful if the Sprint could provide you with the output of the show ip ospf database for your VRF so that you could compare it to your Router's B output - the topological databases MUST be identical.

There is sadly no debug to see which networks are being advertised to a particular neighbor, partially because OSPF as a link-state protocol does not really advertise networks but rather topological components, out of which the networks are computed afterwards.

Best regards,

Peter

[dennylester]

Hi Peter,

Thank you for the reply.

I tried clearing the ip ospf processes, removing the networks from the OSPF process and adding them back in and so far, it looks like the only thing that works is rebooting RouterB

Both routers do show their OSPF neighbors, both have their routing tables full of OSPF learned routes (minus the subnets in question missing from RouterA's routing table). I do want to mention that the 1 of 3 subnets that was learned via OSPF, we do not have a floating static route for it on RouterA. In the other two instances I mentioned seeing this problem, it was the same case, one of the subnets being learned didn't have a static route on RouterA. Perhaps this is where the hangup is occuring. As a test I tried forcing the Dialer interface down hoping the routes would kick it but they didn't. This is when I called Sprint to ask them what they saw coming from me.

We are slowly migrating from Sprint Reseller to Sprint direct and have two MPLS networks setup. On RouterA I do have two OSPF processes running, one for each MPLS network and am redistributing between the two. Since RouterA is the default gateway for everything anyway, I removed the redistribution between the processes and still didn't learn the route.

The next time it happens I will try to call Sprint and have them pull their database for my VRF so I can compare. I'm starting to think it's an IOS bug, but don't currently have maintenance contracts to open a ticket with Cisco.

Thank you for the suggestions.

Denny

[Peter Paluch]

Hello Denny,

Do you believe you could post a quick sketch of your network depicting the Router's A and B connection both to the internal networks and to the ISP? Also, would it be possible for you to post a sanitized version of the Router's A and B configuration? This may be a bug but considering the fact that you are running two OSPF processes compounded by floating static routes, there is some space for race conditions caused by subtle configuration glitches. I would like to have a look.

Best regards,

Peter