Intermittent Problem with Frame Relay Circuits

[incanus555]

Here's the problem: Every few hours or so, we stop receiving packets on either one of two Serial interfaces on our Cisco 2610 router. However, the interface line protocol stays "up" the whole time. Reloading the router fixes the problem temporarily, as does "shutdown" then "no shutdown" on the interface in question. Our ISP and Frame Relay providers (separate entities) said it was probably our router, so we bought a new one, and it's still happening. Now, both said entities are pointing fingers at each other instead of fixing the problem. Can anyone help? I can provide any information that I can get from our router, and some from those other two parties involved. I may be able to get one or both of them to join in this discussion. This is a serious issue for my company, and I'm out of ideas! Any help would be greatly appreciated.

-A. Bronson

[incanus555]

We just purchased this router from a third party reseller a few weeks ago. It was to replace our other of the same model, but this one has a newer (12.2 instead of 12.0 i think) IOS version.

Yes, we're using BVI because when the router was first configured, several years ago, our ISP didn't have software to do it any other way. However, they do have that capability now, and we are going to try eliminating the BVIs next.

As for the input queue idea, we've actually just investigated for the same thing, but it's not getting stuck at 76/75, and we can fix this problem without a reload, so it's not that, we don't think.

Our frame relay provider tried something new today, turning off what they called "policing" on the circuit. So far, the interfaces have both been running fine for over 15 hours (longer than it has ever worked continuously since this problem began). So for now, we're gonna keep our fingers crossed.

-A. Bronson

[Phillip Remaker]

I'd love to know what frame relay gear they have. I wouldn't be surprised if they have a bug in the policing code. Did they start using it recently?

Eliminating BVIs is a good idea anyway. Anytime you can reduce complexity, that is a good thing.

[incanus555]

I'm not sure what they're using. All I know is that it's a switch with several "cards" of some sort, ours of which has been replaced already to no avail.

Not sure about the policing. This is the first I've heard of such a thing, surprisingly. I hope it turns out to be the culprit, as it would at least prove a domain for the problem.

As for eliminating the BVIs, I agree. It's the next step.

[marikakis]

From the configuration posted one can see that the signaling protocol setting

of the serial interfaces is Cisco LMI ( frame-relay lmi-type cisco).

For signaling to work across the UNI, the user device and peering switch port

must use the same signaling protocol.

So, I would bet there exists at least one Cisco device in the FR provider's network,

and it is the one to which your router connects.

A bug is not impossible, even with Cisco.

"Policing" in a VC that crosses a FR/ATM WAN switching network is a set of parameters

that affect how the traffic that traverses your VC will be handled under congestion situations.

For your traffic to drop to zero congestion should be heavy, and the policing parameters

should be aggressive (for example, 1% for the util parameter of the VC).

A question here is if the congestion problem is frequent, or was a special incident

and resulted from another major problem inside their network.

In such cases, removing policing from your VC may make things better for you for a while,

but doesn't resolve the real problem, unless it is a bug indeed.

Even in the bug case, workarounds can only give short-term solutions.

A provider's network is a big mystery, unless you have access to the actual devices.

They are probably not obliged to tell you exactly what the problem was,

but maybe you deserve an explanation if you consider that they had you

have your router replaced, before they seriously examined your issue.

This is not only about deserving an explanation, but can have practical aspects too.

If you happen to experience a similar problem in the future,

you would be able to at least direct them to some possible cause,

before proceeding to replace your router in vain once again.

Anyway, if they finally resolved your problem, I suppose everyone is happy.

M.

[incanus555]

I hope this isn't a temporary fix. So far, it seems stable, though, as it's been up for several days now. They said they increased our CIR to over twice what it was, too, at no additional charge. That sort of compensates for things, which is nice.

But, the problems don't seem to end. Now, for the past couple days, we've been struggling with very slow performance. I've had the weekend off so I haven't looked into it much yet, but an all-day ping to our ISP's DNS server is averaging 700+ ms (usually <100). So I guess we'll see tomorrow.

I want to thank everyone who gave their ideas and help. This is the best experience I've had on an online forum.

A. Bronson

[marikakis]

If you already started experiencing delays,

this probably means that congestion started to build up.

Perhaps you are not the only customer who requested preferential treatment.

"Gossip troubleshooting methodology" can help you find out

if there exist many of their customers with the same issue ;-)

I smell a big issue here, and I hope it will be resolved quickly

for the sake of all the customers and the provider as well.

M.

[rrijk]

What I miss in the configuration is a "bandwith" statement and something like traffic shaping. I could not find the CIR rate which your SP is offering (maybe it is your T1 interface rate).

I assume this is NOT what is causing your problem (since I do not see any BECN in your captures), but I was just curious...

regards,

RdR

[marikakis]

Is this some DSL service ?

A problem inside the FR/ATM core could cause your PVCs

to somehow terminate in one DSLAM.

If both peering bridges have Spanning Tree Protocol enabled,

STP could be blocking one of your PVCs.

Such issues are inherent in some DSL deployment setups,

and I suppose this has to do with the DSL deployment strategy of the providers.

Turning STP off might solve your issue.

I cannot say for sure, because I cannot possibly know

exactly how they provide you this service.

If this doesn't solve your issue, maybe more features

(VLANs and MSTP) could do it.

I know this might sound like science fiction,

but what gets me very suspicious is the fact that when the problem occurs,

you always seem to have one PVC working.

That is a very generous problem indeed.

M.

p.s. The command " bridge protocol ieee " is the STP related command.

[incanus555]

Pretty sure it's not DSL. We have two T1's - one on each Serial/Frame Relay interface.

About always seeming to have one of the two PVCs working, here's why, I think: The problem happens to one interface or the other every couple hours. A script on a unix machine detects the problem within five minutes, and shuts down and restarts the interfaces. On the occasion that the script is disabled (when i'm messing with it) and the problem happens on one interface, eventually it'll happen on the other, too, and then both will be down until they are reset. They just don't go down at the same time.

[kapishmohole.cisco]

Hello,

I had this type of problem in my network, but the only difference is we have our own private WAN switching network.

We have PVC, connecting to another center, router to router. But this PVC goes through our own WAN switches, IGX; no service provider.

Same problem occurred, there was no flow of traffic on PVC but the interface status on both routers was ‘UP’.

Finally problem was with the WAN switching side, there was problem with firmware of UFM card on IGX WAN switch. We reset that IGX card and PVC started to work.

May be there is similar problem in your case, next time when you will face this thing try to concentrate on service providers side, let them test their UFM card on IGX/MGX (it wont be easy task!).

If this scenario is true about you then there is no problem with your router, that’s sure.

Another thing you can use is frame relay end-to-end keep-alive; this won’t solve your problem but will help in pointing the root cause.

Kapish