Router Crashing when ATM Interface under Heavier Load

[g_dunaway]

Hi all,

Looking for a little guidance before I engage TAC (AKA, waiting for maintenance contract purchase approval). I have a 3945 router running IOS 15.1.2T5 with an ATM interface card and adapter. It is currently configured with a 30Mbps PVC. As it reaches about 20 to 21Mbps and 14000 packets per second, I start to see output drops on the interface, which I had previously attributed to a combination of ATM overhead + volume of traffic being pushed through it. About half of the traffic that passes through is VoIP, so it is a lot of smaller packets, which hurts on ATM overhead, if I recall correctly. Starting yesterday, though, the circuit peaked at 23Mbps and 17000 packets per second. About 5 minutes, later, the router crashed and rebooted itself. Crash info reports the following:

PCIe Interrupt - Root Complex 0

Last PCIe Read CFG Data =  0x81000030
Last PCIe Write CFG Data = 0x81000034
Last PCIe CFG Op: Write
PCIe Interrupt: Error Detect Reg =  0x80100000
PCIe Err Enable = 0xBC8000 PCIe Err Cap Status = 0x5
PCIe Err Cap R0 = 0xC000040 PCIe Err Cap R1 = 0xFF000004
PCIe Err Cap R2 = 0x20000000 PCIe Err Cap R3 = 0xAAAA0300

Fatal PCIe error  - Multiple Errors Detected
Fatal PCIe error  - Inbound transaction was not mapped to any inbound window

%Software-forced reload

 12:24:26 CST Mon Dec 29 2014: Unexpected exception to CPU: vector 1500, PC = 0x4A3A114 , LR = 0x4A3A0AC

-Traceback= 0x4A3A114z 0x4A3A0ACz 0x4A3DF54z 0x4124D00z 0x4124D00z 0x4A3E010z 0x6644898z 0x664480Cz 0x6644B04z 0x6621B94z 0x8F86324z 0x8DBCF4Cz 0x8DD0420z 0x8DD0A60z 0x8DAE54Cz 0x8DAEE84z

Plus the various memory dumps.

This has happened 3 times now and in all cases, traffic levels have been the same just before the crash.

Any help in understanding why this is happening would be appreciated and thanks in advance.

-Gary

[Leo Laohoo]

%Software-forced reload

The router is running on a "T" train IOS.  Please investigate of the possibility of upgrading the IOS to the more stable "M" train.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Unless you really need your particular IOS version, like Leo, I too would recommend moving to a mainline recommended release that suites your needs.

PS:

It's likely you're correct that you're seeing drops before you hit your ATM "bandwidth" cap.  The ATM cell "tax" is a 5 byte header in each 53 byte cell and, if doing IP, there's cell padding (packets will sized up to the next multiple of 48).

For example, a 97 byte payload would consume 159 bytes (about 39% overhead).

[Nikhil Kapoor]

Hi Gary,

Can you please check crashinfo and look for the process. That may give a clue if everytime the crash has happened because of same process? 

You can find process as per below. Look for 

---- Partial decode of process block ----

Under this there will be Process=

Also you may want to upgrade the IOS and check if its stable?

HTH,
Nikhil 

[g_dunaway]

Thanks, all. I had suspected I would end up having to do this. I opened up a ticket with TAC just to see if this was related to a similar bug I saw that was reported as being fixed in the IOS version this router is on. It wasn't a 100% match though.

-Gary