How to recover memory taken up by *Dead* Process

[asaykao73]

Hi There,

We're hitting about 95% memory utilization and it appears most of it is being consume by *Dead* processes (see below).

Is there some way to recover the memory being used up by the *Dead* processes or will we have to reboot the router?

RTR-01#sh proc memory sorted

Processor Pool Total:  133318240  Used:  127600908 Free:    5717332

     I/O Pool Total:   40893952 Used:    13781152 Free:   27112800

PID TTY  Allocated       Freed    Holding    Getbufs    Retbufs Process

  0   0  151244900   52146784    99991348     554104          0 *Dead*

  0   0   63691228   22433984    36948272          0          0 *Init*

139   0  129059308  128081536      932496      24948          0 HTTP CORE

192   0     304768      34984      275612          0          0 IPSEC key engine

188   0     115776       3764      137384          0          0 Crypto ACL

133   0     136472        196      135296          0          0 DHCPD Receive

170   0     118884        472      127392          0          0 Crypto WUI

181   0      88248          0       95228          0          0 QOS_MODULE_MAIN

  4   0      65588          0       90568          0          0 EDDRI_MAIN

113   0      67384          0       77364          0          0 IP RIB Update

116   0      73340          0       73340          0          0 CEF process

179   0        940          0       61920          0          0 http client proc

157   0      44876        196       51660          0          0 CCVPM_HDSPRM

74 450     116092      82560       46124          0          0 SSH Process

195   0      24912          0       37892          0          0 Crypto Delete Ma

172   0      26180          0       36160          0          0 CCVPM_HTSP

36   0    3799060        780       35744       5388          0 Net Background

  6   0   32526052   46627708       35684   25629576   32805804 Pool Manager

60   0      27448        196       34232          0          0 VNM DSPRM MAIN

RTR-01#sh  ver

Cisco IOS Software, 3800 Software (C3845-ADVIPSERVICESK9-M), Version  12.4(3e), RELEASE SOFTWARE (fc2)

Technical Support:

http://www.cisco.com/techsupport

Copyright (c) 1986-2006  by Cisco Systems, Inc.

Compiled Wed 14-Jun-06 01:49 by  alnguyen

ROM: System Bootstrap,  Version 12.3(11r)T2, RELEASE SOFTWARE (fc1)

RTR-01  uptime is 20 weeks, 1 day, 18 hours, 26 minutes

System returned to ROM by  power-on

System restarted at 17:17:23 AEDT Mon Mar 26 2012

System image  file is "flash:c3845-advipservicesk9-mz.124-3e.bin"

Thanks.

Andy

[asaykao73]

It appears to be a series of dead SSH sessions.

RTR-01#show memory dead

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor   659DB9A0   133318240   127629396     5688844     5112068     5543780

      I/O   2D900000    40893952    13801756    27092196    26974320    27053116

          Processor memory

Address      Bytes     Prev     Next Ref     PrevF    NextF Alloc PC  what

659DB9A0 0000020004 00000000 659E07F4 001  -------- -------- 60D87F30  Acct Req chunk

659F47BC 0000004100 659F2CDC 659F57F0 001  -------- -------- 6173BB94  SSH Process

659F57F0 0000004100 659F47BC 659F6824 001  -------- -------- 6173BB94  SSH Process

659F6824 0000004100 659F57F0 659F7858 001  -------- -------- 6173BB94  SSH Process

659F7858 0000004100 659F6824 659F888C 001  -------- -------- 6173BB94  SSH Process

659F888C 0000004100 659F7858 659F98C0 001  -------- -------- 6173BB94  SSH Process

659F98C0 0000004100 659F888C 659FA8F4 001  -------- -------- 6173BB94  SSH Process

659FA8F4 0000004100 659F98C0 659FB928 001  -------- -------- 6173BB94  SSH Process

659FB928 0000004100 659FA8F4 659FC95C 001  -------- -------- 6173BB94  SSH Process

659FC95C 0000004100 659FB928 659FD990 001  -------- -------- 6173BB94  SSH Process

659FD990 0000004100 659FC95C 659FE9C4 001  -------- -------- 6173BB94  SSH Process

659FE9C4 0000004100 659FD990 659FF9F8 001  -------- -------- 6173BB94  SSH Process

659FF9F8 0000004100 659FE9C4 65A00A2C 001  -------- -------- 6173BB94  SSH Process

65A00A2C 0000004100 659FF9F8 65A01A60 001  -------- -------- 6173BB94  SSH Process

65A01A60 0000004100 65A00A2C 65A02A94 001  -------- -------- 6173BB94  SSH Process

65A037BC 0000004100 65A02A94 65A047F0 001  -------- -------- 6173BB94  SSH Process

65A047F0 0000004100 65A037BC 65A05824 001  -------- -------- 6173BB94  SSH Process

65A05824 0000004100 65A047F0 65A06858 001  -------- -------- 6173BB94  SSH Process

65A06BD4 0000004100 65A06858 65A07C08 001  -------- -------- 6173BB94  SSH Process

65A07C08 0000004100 65A06BD4 65A08C3C 001  -------- -------- 6173BB94  SSH Process

65A08C3C 0000004100 65A07C08 65A09C70 001  -------- -------- 6173BB94  SSH Process

65A09C70 0000004100 65A08C3C 65A0ACA4 001  -------- -------- 6173BB94  SSH Process

65A0ACA4 0000004100 65A09C70 65A0BCD8 001  -------- -------- 6173BB94  SSH Process

65A0C74C 0000004100 65A0BCD8 65A0D780 001  -------- -------- 6173BB94  SSH Process

65A0D780 0000004100 65A0C74C 65A0E7B4 001  -------- -------- 6173BB94  SSH Process

65A0E7B4 0000004100 65A0D780 65A0F7E8 001  -------- -------- 6173BB94  SSH Process

65A0F7E8 0000004100 65A0E7B4 65A1081C 001  -------- -------- 6173BB94  SSH Process

65A1081C 0000004100 65A0F7E8 65A11850 001  -------- -------- 6173BB94  SSH Process

65A11850 0000004100 65A1081C 65A12884 001  -------- -------- 6173BB94  SSH Process

65A12884 0000004100 65A11850 65A138B8 001  -------- -------- 6173BB94  SSH Process

65A138B8 0000004100 65A12884 65A148EC 001  -------- -------- 6173BB94  SSH Process

65A148EC 0000004100 65A138B8 65A15920 001  -------- -------- 6173BB94  SSH Process

65A15920 0000004100 65A148EC 65A16954 001  -------- -------- 6173BB94  SSH Process

65A16BF4 0000000172 65A16954 65A16CD0 001  -------- -------- 612F21FC  SSH Process

[paolo bevilacqua]

You're running very old software, update.

[Peter Paluch]

Hello Andy,

Paolo is completely correct, your IOS is strongly outdated. You should consider an upgrade. Regarding the taken memory, I am afraid the only way to reclaim it is to reload the device.

Best regards,

Peter

[asaykao73]

Thanks for the replies guy.

Will look at the IOS upgrade option.

[KAROLY KOHEGYI]

Hi Andy,

We have got same problem.

The device is also 3800.

We tried it iwith following IOS.

c3845-advipservicesk9-mz.124-24.T3.bin

c3845-advipservicesk9-mz.151-4.M4.bin

In both case the memory is decreasing continuously and there are many  SSH dead process.

Our questions is have you found any "good" IOS which does not consume the memory?

Thanks in advance!

[paolo bevilacqua]

There is always the possibility of bugs. With these, you have to contact the TAC.

Check if someone isn't attempting many ssh accesses without you knowing.

[KAROLY KOHEGYI]

Hi,

You right. It may be a bug but the question is where is the bug.

But i thought other CISCO costumer had bumbed into this problem also and had already opened TAC.

The SSH access is restricted by ACL.

Only the LMS manages the router via SSH periodically.

I think the LMS SSH process causes the DEAD memory problems.

I have not got any clue for this because the router does not show what kind of SSH source address were there.

I have same problem on ACE module. There the LMS SSH session are still exists and not disconnected.

Regards,

[cfbisdtac]

Could you be guarding (via ACL) against SSH from the Internet, but allowing access to a wider group of internal addresses than intended.  Could somebody be half-opening many SSH sessions programmatically, doing a sort of SSH DDOS from an internal host?

In general, could this be memory that IOS is not properly reclaiming when an SSH process times out, rather than when the user quits normally?

I see that Paolo had already posted a comment somewhat along that DDOS angle.