2610 Router that hangs

[bbohan]

Network topology:

We have a 2600 router with two WIC1-T1-DSUs with two full T1s of Internet (3Meg Internet connection) that connect to a 12000 backbone router. We implement ip cef, ip load-sharing per-packet and ospf, and set the MTU size to 4470 to match that of the 12000.

Issue:

Every day or two the router hangs, in that it requires a reboot in order to restore optimum throughput. When the customer calls, we try to gain telnet access, however, we are unable to telnet to the router. We have dispatched a field tech to the site in order to console into the 2600 and he receives a memory error which does not allow console access as well.

Troubleshooting steps we have taken:

-We have replaced the 2600 with another one and the issue came back.

-We had the field tech, when on site, remove the Ethernet cable from the E0/0 interface and wait a few minutes to see if he was able to gain access via the console port, however, he was still getting the same error. Unfortunately, we had to have him reboot the router. Once he rebooted it, he was able to gain console access and we were able to gain telnet access. After about an hour, we lost telnet access again. We have checked our MRTG stats as well as the interface stats and there does not seem to be enormous amounts of traffic going through the interfaces. Another interesting issue is that once we gained telnet access, we checked the ip cef settings in the startup-config and it is in the config, however if we do a show ip cef, it says that ip cef is not running. We then did a "show running-config" and there was no output.

I have researched Cisco's site regarding "wedged interfaces" to look at the buffers and if there were any leaks. This is only a theory and we were unable to check the input queue since we needed to reboot.

Below I have attached the config and some show commands. Any recommendations would be greatly appreciated.

[thisisshanky]

What error message do you get when you try to console in ? Is it a Traceback ? Can you paste the message. Also can you paste a sh version output ?

[bbohan]

Fulfillment#show log history

Syslog History Table:1 maximum table entries,

saving level warnings or higher

36 messages ignored, 0 dropped, 0 recursion drops

7687 table entries flushed

SNMP notifications not enabled

entry number 7688 : SYS-2-MALLOCFAIL

Memory allocation of 20000 bytes failed from 0x802E1F88, alignment 0

Pool: Processor Free: 38872 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

timestamp: 8042637

[bbohan]

show mem free

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 814917F4 5695500 5599312 96188 0 29044

I/O 1A00000 6291456 1575172 4716284 4685360 4692796

Here are others symptoms we've experienced:

No output from some show commands

-"Low on memory" messages

-The console message "Unable to create EXEC - no memory or too many processes"

-Router hanging, no console response.

[phillipediab]

I think it is a memory problem.did you try to upgrade the DRAM ?

[bbohan]

The router has 32meg of ram.

[phillipediab]

I would add more DRAM. I had a similar problem on a 7200 series that was running bgp with too many serial and FE interfaces. I think you can add another 32M

regards,

[mcdonald.richards]

Try graphing your free memory as well via snmp and setting up syslog server logging to see if anything significant happens to trigger this. It sounds like a memory leak.

What version IOS are you running? It's likely a known bug.

[bbohan]

Here is our update. We have dispatched a field tech to swap out the 2610 with another one with two new wic1-T1-dsu's and upgraded the memory from 32meg to 64meg. This was done yesterday and thus far the connection has been good and we consistently gain telnet access to the router, which we were not getting before.

[bbohan]

sho ver

Cisco Internetwork Operating System Software

IOS (tm) C2600 Software (C2600-IS-M), Version 12.2(28), RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2005 by cisco Systems, Inc.

Compiled Thu 24-Feb-05 20:56 by kellmill

Image text-base: 0x8000808C, data-base: 0x8111D3F0

ROM: System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)

ROM: C2600 Software (C2600-IS-M), Version 12.2(28), RELEASE SOFTWARE (fc5)

Fulfillment uptime is 22 hours, 20 minutes

System returned to ROM by power-on

System image file is "flash:c2600-is-mz.122-28.bin"

cisco 2610 (MPC860) processor (revision 0x202) with 61440K/4096K bytes of memory

.

Processor board ID JAB03220014 (2794528994)

M860 processor: part number 0, mask 49

Bridging software.

X.25 software, Version 3.0.0.

1 Ethernet/IEEE 802.3 interface(s)

2 Serial network interface(s)

32K bytes of non-volatile configuration memory.

16384K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102