ISR 2921, memory leak

[ngtransge]

Hello,

Got problem with 2921, it was restarting 2 times automatically(the second time it went down for couple of minutes, only power LED was active, all others down, it repairs  from that state without any interaction), after that it is operaiting normally about 5 day now, but I was restarting it manually, during that time. I was analising statistics, before load and after load with cisco output interpreter, and found that after load its '*Dead*'  process is holding more than 1 MB of ram. This is cisco output interpreter output:

WARNING: The process '*Dead*' is holding more than 1 MB of memory.
  This is considered to be high, and indicates a possible memory leak. The 'Dead'
  process is not a real process. It only accounts for the memory allocated under
  the context of another process which is terminated.
  TRY THIS: If a memory leak is detected, and the 'Dead' process seems to be consuming
  the memory, issue the "show memory dead" command from your device, and analyze
  which process consumes more memory. To do so, look at the "What" section of
  the output. This problem can also be caused due to a Cisco IOS Bug. Review the
  Bug Tool Kit for potential bugs and upgrade the device with the latest version
  of Cisco IOS software. If the problem persists, open a service request with Cisco
  at Service Request Tool.
 
  INFO: The top 3 processes that are holding less than 1 MB of memory are:
    'l4f mgt task' is holding 988708 bytes
    'Chunk Manager' is holding 715332 bytes
    'VLAN Manager' is holding 440520 bytes

I am interesting is it Normal ? or could it be the possible reason for router automatic restarting ?

In production network it was operating under small load.

It is running c2900-universalk9-mz.SPA.150-1.M5.bin. Also c2900-universalk9-mz.SPA.151-4.M.bin was installed for cheking but without any results.

[paolo bevilacqua]

It is not said (but certainly possible) that the crashes are related to the memory leak.

To know, you must engage the TAC, ask for escalation and intepretation of crash files.

[ngtransge]

router did not generate any crash file.

[paolo bevilacqua]

You can take "show stacks" or a console log when it crashes. That should be enough to diagnose the problem.

Since 15.0(1)M5 is known to be stable, one cannot exclude an hardware issue either.

[ngtransge]

This is logs, it generate after crash. SNMP is not running on this router.

%IFMGR-7-NO_IFINDEX_FILE: Unable to open nvram:/ifIndex-table No such file or directory
%ESWMRVL_FLTMG-5-NOTICE: Notice: FPGA Rev 0x71
%VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Initialized
%VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Enabled
%LINK-3-UPDOWN: Interface GigabitEthernet0/0,changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet0/1,changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet0/2,changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/2, changed state to down
%SYS-5-CONFIG_I: Configured from memory by console
%LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to administratively down
%LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to administratively down
%LINK-3-UPDOWN: Interface FastEthernet0/0/0, changed state to up
%LINK-3-UPDOWN: Interface FastEthernet0/0/1, changed state to up
%SYS-5-RESTART: System restarted --
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.0(1)M5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 23-Feb-11 15:41 by prod_rel_team
%SNMP-5-COLDSTART: SNMP agent on host CoreM is undergoing a cold start
%SSH-5-ENABLED: SSH 1.99 has been enabled
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
%CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
%CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON

[paolo bevilacqua]

I mean console log not buffer log.

Logs after crash are useless.

You can only keep monitoring the device at this point.

[ngtransge]

Thank you for help.

Is there any way to test router for any software or hardware problems without waiting and monitoring for a long time.

[paolo bevilacqua]

Try reproducing the conditions when the problem happened. E.g. it was at boot time, with a different configuration, etc.

Keep a console cable attached so you can see if any error message is generated.

Thanks for the nice rating and good luck!

[Timothy Stewart · Accepted solution]

It can be normal to see the Dead process holding memory.  The Dead process is simply a holding area for memory that was allocated by a process that was released and the memory is still being used.  A good example of Dead memory usage is when you telnet into a router, make a config change then kill the telnet session.  When you initually telnet into the router a Virtual Exec process is created to deal with the telnet session.  While telneted in, if you go into config mode and increase the logging buffered to say 10mb then 10mb of memory will be allocated for the log.  The allocation happened under the Virtual Exec process (a show mem proc would show Virtual Exec holding 10mb).  When the telnet session is killed, the memory allocated by Virtual Exec is still needed, therefore the 10mb is moved to the Dead process and the Virtual Exec process is removed.   Now, if the router is rebooted, the 10mb allocated for the Log will now show up under the Init process (Init allocates memory when the router is booting).

Seeing 1mb in Dead is nothing to be concerned about.  If you saw 30mb or more then it may warrent a closer look.  As far as the router rebooting, a show version will tell you when the last reboot occurred and the reason for it.  If it was a crash, you should see the reboot reason be something like "Bus error" or "Segv".  A crash should also result in a  crashinfo being produced and saved to the flash.

Tim

[ngtransge]

Thank you for help, and explanation.