2951 router - sh run produces no output

[Keith Lawrence]

Hi,

We have a 2951 router which we are using for CME/CUE for about 20 phones. We use an ITSP sip trunk for incoming/outgoing calls.

It is on a recent IOS - Cisco IOS 15.4(3)M5 (UNIVERSALK9)

The issue we are seeing is after several days, we cannot use "sh run" from the console/vty connection. The command is accepted but nothing happens. e.g.:

RDX-ARD-2951-1#sh run
RDX-ARD-2951-1#sh run
RDX-ARD-2951-1#

We can still use "sh start" to view the config with no problems. Conf t also allows us to change the config as needed.

"wr mem" works but we get an odd vlan message:

RDX-ARD-2951-1#wr mem
Building configuration...

Error on initialize VLAN database 1: VTP feature not yet initialized[OK]
RDX-ARD-2951-1#

Power cycling the router resolves the issue for several more days but it starts happening again soon after.

I have checked the memory usage just now (while the sh run issue is happening) on the router and it doesn't look too excessive :

RDX-ARD-2951-1#sh mem
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   139B605C   226070436   218974508     7095928        1824       66256
           I/O   1CA00000    56623104    31445240    25177864    24107328    23155900

Functionally, the router is fine when this is happening - CME/CUE keep working as normal etc.

Any ideas as to what would be causing this problem?

Keith

[Carlos Villagran]

Hi!

This issue indeed is caused by low memory. The only way to fix this that I have found is to delete unnecessary VLANs and trunks (to reduce number of STP instances too) in the switch and if possible put it on VTP transparent mode.

Hope it helps, best regards!

JC

[Keith Lawrence]

Hi Carlos,

This is a router, not a switch and has no vlans apart from vlan1 configured. Here is some further information:

RDX-ARD-2951-1#sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 52
Number of existing VLANs        : 5
VTP Operating Mode              : Transparent
VTP Domain Name                 :
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : *removed*
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00

RDX-ARD-2951-1#sh vlans

No Virtual LANs configured.

RDX-ARD-2951-1#sh spanning-tree

VLAN1 is executing the ieee compatible Spanning Tree protocol
Bridge Identifier has priority 32768, address *removed*
Configured hello time 2, max age 20, forward delay 15
We are the root of the spanning tree
Topology change flag not set, detected flag not set
Number of topology changes 1 last change occurred 1w6d ago
from SM1/1
Times: hold 1, topology change 35, notification 2
hello 2, max age 20, forward delay 15
Timers: hello 0, topology change 0, notification 0, aging 300

Port 115 (SM1/1) of VLAN1 is forwarding
Port path cost 4, Port priority 128, Port Identifier 128.115.
Designated root has priority 32768, address *removed*
Designated bridge has priority 32768, address *removed*
Designated port id is 128.115, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
BPDU: sent 596034, received 0

Port 169 (SM2/1) of VLAN1 is forwarding
Port path cost 4, Port priority 128, Port Identifier 128.169.
Designated root has priority 32768, address *removed*
Designated bridge has priority 32768, address *removed*
Designated port id is 128.169, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
BPDU: sent 596034, received 0

Any other ideas?

[Mark Malone]

This sounds like a bug or else you may have a hardware issue between the memory nvram and ram etc somewhere , nvram must be working as the startup is there , instead of rebooting router did you try copy startup back into running config from flash

Have you tried to upgrade the image , a memory leak could cause something like this and a software upgrade should fix that , I see your on the most recommended image but you still may have triggered something

if your in conf t mode and you use a do show run do you get the same result or  do show run all

[Keith Lawrence]

Hi Mark,

In conf t, do sh run just has the same result.

Now even more wierdness happening. Just tried a wr mem and got this:

RDX-ARD-2951-1#wr mem
Processor Pool Total: 226070436 Used: 219080416 Free: 6990020
I/O Pool Total: 56623104 Used: 31499664 Free: 25123440

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 306240168 149675884 147682904 1528 12096183 *Init*
0 0 12876 2468084164 12876 0 0 *Sched*
0 0 1640029356 650642064 69676480 115478 115478 *Dead*
0 0 0 0 985200 0 0 *MallocLite*
1 0 1872332 70540 1809088 0 0 Chunk Manager
2 0 240 240 4296 0 0 Load Meter
3 0 29424 748 54636 0 0 LICENSE AGENT
4 0 65588 0 90884 0 0 EDDRI_MAIN................

It now seems every command opens a memory report similar to the above.

I think a power cycle will be needed soon.

Is there a way of testing for memory leaks or faulty RAM etc?

Keith

[Mark Malone · Accepted solution]

You can collect these below and run them through the Cisco analyser if the device is under support , if not output interpreter might tell you on Cisco website if that's still available its pretty useful tool for memory issues

usually when there's a leak you will see maybe alloc alerts in logs but to be honest if your getting memory outputs when you run a standard command you have a leak or bug from my experience anyway

show mem failures alloc

sh memory allocating-process totals

sh memory dead totals

https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl

[Keith Lawrence]

Hi Mark,

Thanks for the additional info.

I ran the sh mem failures alloc command and got the following:

RDX-ARD-2951-1#show mem failures alloc

Caller Pool Size Alignment When

0x61DF27C Processor 262136 0 15:52:59

0x61DF27C Processor 262136 0 15:46:30

0x61DF27C Processor 262136 0 15:46:30

0x4E54E5C Processor 259644 0 15:46:09

0x4E54E5C Processor 259644 0 01:31:23

0x4E54E5C Processor 259644 0 00:56:57

0x61DF27C Processor 262136 0 00:03:38

0x61DF27C Processor 262136 0 00:03:38

0x61DF27C Processor 262136 0 00:01:20

0x61DF27C Processor 262136 0 00:01:20

So it looks like the router is out of memory.

A quick check on our NMS shows there is only 215Mb RAM available on this unit and we are using 92% of that currently.  The specs for IOS 15 seem to indicate 512Mb RAM is recommended.

I have ordered up a 1GB stick for this router and will get that fitted to see if it improves matters.

Thanks for your help!

Keith

[Mark Malone]

Hi if your seeing the failures there you could have a leak too , before spending anything i would try a newer image , either way if your putting ram in the device will need to be rebooted anyway, you could be out of memory due to a leak in software

[Keith Lawrence]

So, after putting the order in for the RAM, I did another reload.

Lo and behold, everything is now stable - RAM usage at 60% and no allocation failures...

Ill probably go ahead and install the RAM anyway. We have CME licensing on this unit for 75 phones so the extra memory may come in useful once we start getting near that figure.

Keith

[Mark Malone]

Always the way :) hopefully it stays that way