memory issue on ISR 2911 (cant login via SSH or TELNET)

[amralrazzaz]

dear all

i have :c2900-universalk9-mz.SPA.154-3.M1.bin" (Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)) with memory as below
Cisco CISCO2911/K9 (revision 1.0) with 483328K/40960K bytes of memory.
Processor board ID FCZ190368FM
3 Gigabit Ethernet interfaces
1 terminal line
1 Virtual Private Network (VPN) Module
8 Voice FXO interfaces
DRAM configuration is 64 bits wide with parity enabled.
255K bytes of non-volatile configuration memory.
255488K bytes of ATA System CompactFlash 0 (Read/Write)
-----------
Router#show telephony-service
CONFIG (Version=10.5)
=====================
Version 10.5
Max phoneload sccp version 17
Max dspfarm sccp version 18
Cisco Unified Communications Manager Express
---------

error massege appears to me suddenly and wont let me access to the router via telnet or ssh so i connected to the router via console port and showing me below error
then i reloead the router and then it fixed and i can connect via telnet and shh

so whats this issue and how to solve it and if there any thing to do to minimize the usage of used memory ??
also is it available to increase the ram usage to 1 or 1.5 GB ? is it available ?

May 31 12:10:21.225: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x3002F180, alignment 0
Pool: Processor Free: 1506336 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CCSIP_SPI_CONTROL", ipl= 0, pid= 474
-Traceback= 30010528z 300109FCz 3001F288z 30009BACz 3000F3C8z 370251A0z 37025940z 370259F8z 35943EC0z 3598FFDCz 3599ED4Cz 359BA444z 35CE7434z 35CE7838z 30032934z 30032918z
May 31 12:10:21.661: %SYS-2-CFORKMEM: Process creation of TFTP Server failed (no memory). -Process= "IP Input", ipl= 0, pid= 180
-Traceback= 3005A0D0z 32301070z 3273861Cz 3273A428z 301DC3F4z 301E2540z 301DA9Ccz 301DAC58z 301DAD44z 301DAF90z 30032934z 30032918z
May 31 12:10:25.621: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
May 31 12:10:26.077: %LICENSE-2-VLS_ERROR: 'VLSnotifyBirthAndExpiryEvents' failed with an error - rc = 25 - 'Error[25]: Could not understand the message received from license server on host "no-net".
Client-server version mismatch?
'
-Traceback= 30032934z 30032918z

User Access Verification

Password:
May 31 12:10:32.097: %SYS-6-RESERVE_MEMORY_CREATED: Reserve memory is being created
May 31 12:10:32.097: %SYS-6-USING_RESERVE_MEMORY: Reserve memory is being used
May 31 12:10:35.793: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
May 31 12:10:45.929: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
Password:
May 31 12:10:51.389: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x3002F180, alignment 0
Pool: Processor Free: 1499184 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CCSIP_SPI_CONTROL", ipl= 0, pid= 474
-Traceback= 30010528z 300109FCz 3001F288z 30009BACz 3000F3C8z 370251A0z 37025940z 370259F8z 35943EC0z 3598FFDCz 3599ED4Cz 359BA444z 35CE7434z 35CE7838z 30032934z 30032918z
May 31 12:10:51.897: %SYS-2-CFORKMEM: Process creation of TFTP Server failed (no memory). -Process= "IP Input", ipl= 0, pid= 180
-Traceback= 3005A0D0z 32301070z 3273861Cz 3273A428z 301DC3F4z 301E2540z 301DA9CCz 301DAC58z 301DAD44z 301DAF90z 30032934z 30032918z
May 31 12:10:56.121: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
May 31 12:11:06.213: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
Password:
May 31 12:11:16.329: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
May 31 12:11:22.105: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x3002F180, alignment 0
Pool: Processor Free: 1523552 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CCSIP_UDP_SOCKET", ipl= 0, pid= 476
-Traceback= 30010528z 300109FCz 3001F288z 30009BACz 3000F3C8z 35963A2Cz 359524A4z 3595DFB8z 35A1445Cz 35A14F6Cz 35CFF208z 35CFF3E0z 30032934z 30032918z
May 31 12:11:22.569: %SYS-2-CFORKMEM: Process creation of TFTP Server failed (no memory). -Process= "IP Input", ipl= 0, pid= 180
-Traceback= 3005A0D0z 32301070z 3273861Cz 3273A428z 301DC3F4z 301E2540z 301DA9CCz 301DAC58z 301DAD44z 301DAF90z 30032934z 30032918z
May 31 12:11:26.373: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
May 31 12:11:36.509: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z

User Access Verification

Password:
May 31 12:10:32.097: %SYS-6-RESERVE_MEMORY_CREATED: Reserve memory is being created
May 31 12:10:32.097: %SYS-6-USING_RESERVE_MEMORY: Reserve memory is being used
May 31 12:10:35.793: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
May 31 12:10:45.929: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
Password:
May 31 12:10:51.389: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x3002F180, alignment 0
Pool: Processor Free: 1499184 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CCSIP_SPI_CONTROL", ipl= 0, pid= 474
-Traceback= 30010528z 300109FCz 3001F288z 30009BACz 3000F3C8z 370251A0z 37025940z 370259F8z 35943EC0z 3598FFDCz 3599ED4Cz 359BA444z 35CE7434z 35CE7838z 30032934z 30032918z
May 31 12:10:51.897: %SYS-2-CFORKMEM: Process creation of TFTP Server failed (no memory). -Process= "IP Input", ipl= 0, pid= 180
-Traceback= 3005A0D0z 32301070z 3273861Cz 3273A428z 301DC3F4z 301E2540z 301DA9CCz 301DAC58z 301DAD44z 301DAF90z 30032934z 30032918z
May 31 12:10:56.121: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z
May 31 12:11:06.213: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCSIP CCB Pool . No memory available -Process= "Chunk Manager", ipl= 4, pid= 1
-Traceback= 30032934z 30032918z

CISCO2911-EGCAI01#show memory free
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 3D5321E0 279014844 184796540 94218304 44138236 88403120
I/O D800000 41943040 18570408 23372632 23318576 23317244

CISCO2911-EGCAI01#sh processes memory sorted
Processor Pool Total: 279014844 Used: 184599452 Free: 94415392
I/O Pool Total: 41943040 Used: 18570336 Free: 23372704

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 277346512 186302628 82482364 0 0 *Init*
474 0 354937868 295725024 37638312 0 0 CCSIP_SPI_CONTRO
1 0 29422800 859536 28570528 0 0 Chunk Manager
0 0 0 0 8790232 0 0 *MallocLite*
452 0 9628828 1075520 7595768 0 0 CC-API_VCM
483 0 18519144 12005420 6519860 0 0 STILE PERIODIC T
332 0 5851708 2856 5442672 0 0 FLEX DSPRM MAIN
0 0 211850612 210468120 4972236 1757683 0 *Dead*
476 0 19457076 31329988 1819996 0 0 CCSIP_UDP_SOCKET
472 0 1374888 240 1435912 0 0 CCH323_CT
362 0 3264936 1894620 1373924 0 0 Crypto IKEv2
408 0 1384776 20596 1371444 0 0 EEM Server
164 0 1379780 46352 1242320 0 0 CCSIP-REGISTER
481 0 737448 0 744712 0 0 l4f mgt task
479 0 507180 0 532444 0 0 RTPSPI
163 0 1290520 528832 479044 0 0 CDP Protocol
459 0 412644 240 437668 0 0 swmtp_msp
478 0 314432 240 339456 0 0 CCSIP_TLS_SOCKET
477 0 312332 240 337356 0 0 CCSIP_TCP_SOCKET
496 0 7243348 6927588 317780 0 0 Skinny Socket Se
391 0 287068 4988 292344 100548 0 EEM ED Syslog
47 0 272704 1664 278096 0 0 RF SCTPthread
357 0 266424 0 276688 0 0 QOS_MODULE_MAIN
467 0 229772 240 242796 0 0 VOICE_HA_ACTIVE
334 0 213212 0 226476 0 0 CCVPM_HTSP
PID TTY Allocated Freed Holding Getbufs Retbufs Process
392 0 214056 4988 219332 67468 0 EEM ED Generic
294 0 218620 608 216012 0 0 Crypto HW Proc
361 0 201748 0 215012 0 0 VOIP_FMD_MAIN
456 0 179788 240 204812 0 0 DSMP
124 0 202604 316 184288 0 0 USB Startup
116 0 160460 240 167484 0 0 VNM DSPRM MAIN
132 0 374812 212080 156204 0 0 mDNS
66 0 135784 0 143048 0 0 IF-MGR control p
133 0 272560 210704 133292 0 0 AFW_application_
409 0 130004 106752 118044 0 0 Call Home proces
453 0 51352 876 111740 0 0 VOIP_RTCP
457 0 71336 240 96360 0 0 VTSP
4 0 65588 1368 90852 0 0 EDDRI_MAIN
371 0 64216 2648 86832 0 0 Crypto ACL
3 0 328248 246300 81904 0 0 Skinny Msg Serve
310 0 71168 240 80192 0 0 Crypto WUI
369 0 351936 20236 78668 0 0 IPSEC key engine
237 0 75420 18988 77660 0 0 IP Background
246 0 109284 31536 76900 0 0 HTTP CORE
367 0 64596 3828 74728 0 0 Crypto IKMP
7 0 6700400 6700400 74268 6569100 6569100 Pool Manager
291 0 66436 240 73460 0 0 HTTP Process
403 0 292424 292876 70080 0 0 AFW_application_
236 0 57720 0 67592 0 0 IP RIB Update
458 0 41192 0 66456 0 0 TSP
176 0 49392 0 62656 0 0 IPAM Manager
355 0 944 0 62208 0 0 http client proc
253 0 944 0 62208 0 0 COPS
136 0 54556 416 61404 0 0 VRRS Main thread
PID TTY Allocated Freed Holding Getbufs Retbufs Process
189 0 131660 484 61164 0 0 AFW_application_
62 0 1988 1988 61128 0 0 AFW_application_
284 0 66260 672 61128 0 0 AFW_application_
354 0 748 748 61128 0 0 AFW_application_
298 0 860 860 61128 0 0 AFW_application_
356 0 66084 496 61128 0 0 AFW_application_
256 0 964 964 61128 0 0 AFW_application_
417 0 89860 28068 60788 0 0 DHCPD Receive
324 0 308176 262260 59936 0 0 Crypto CA
311 0 51488 104 58584 0 0 MMA DB TIMER
437 388 1106840 1074004 58036 0 0 Virtual Exec
317 0 50364 552 57176 0 0 Timer Library
161 0 49420 0 56684 0 0 ACCT Periodic Pr
179 0 49384 0 56648 0 0 IP ARP Retry Age
312 0 30400 240 55424 0 0 STUN_APP
405 0 29244 772 54372 0 0 LICENSE AGENT
114 0 48968 320432 53828 0 0 EEM ED ND
238 0 42872 0 53136 0 0 IP Connected Rou
469 0 39324 240 52348 0 0 Voip HA Session
494 0 35752 240 48776 0 0 MLD
120 0 41012 0 48140 0 0 Skinny MOH Loopb
455 0 265540 608 46084 0 0 Session Applicat
373 0 35876 14188 44844 0 0 IPSec background
241 0 35080 5236 37108 0 0 SR CapMan Proces
178 0 23668 76 36932 0 0 IP ARP Adjacency
498 0 55320 23760 34028 0 0 NTP
360 0 25684 240 32708 0 0 Dialog Manager
34 0 110072 85412 31924 3900 0 ARP Background
151 0 5432 0 30696 0 0 SMI Backup Proce
PID TTY Allocated Freed Holding Getbufs Retbufs Process
289 0 16824 240 29848 0 0 IPS Auto Update
149 0 3168 0 28432 0 0 SMI Director DB
58 0 68548 868 27032 2268 0 Net Background
200 0 1840 240 26864 0 0 Ethernet CFM
423 0 21568 4988 26844 0 0 EEM ED OIR
243 0 85324 240 26520 0 0 SCTP Main Proces
281 0 1196 240 26220 0 0 gk process
316 0 944 0 26208 0 0 FW_TEST_TRP
313 0 944 0 26208 0 0 STUN_TEST
280 0 1100 240 26124 0 0 Border Element p
41 0 18744 80 25928 0 0 Entity MIB API
345 0 864 240 25888 0 0 CTI Front End Pr
358 0 284 0 25548 0 0 RPMS_PROC_MAIN
444 0 240 0 25504 0 0 tHUB
29 0 0 0 25264 0 0 Crash writer
125 0 0 0 25264 0 0 CF_INTERDEV_SCTP
112 0 0 0 25264 0 0 RF_INTERDEV_SCTP
359 0 0 0 25264 0 0 VoIP AAA
203 0 240 240 25264 0 0 Ethernet CFM PAL
33 0 20208 3424 24048 0 0 ARP Input
349 0 324272 314004 23980 0 0 VOICE REG BG Pro
159 0 16832 240 23856 0 0 AAA Server
65 0 23536 16164 22968 0 0 RF Slave Main Th
389 0 20504 5728 22120 0 0 DCM Core Thread
416 0 13052 1208 22108 0 0 EEM Policy Direc
433 0 16816 4988 22092 0 0 EEM ED CRASH
432 0 16816 4988 22092 0 0 EEM ED DS
435 0 16672 4988 21948 0 0 EEM ED Nf
353 0 7608 240 20632 0 0 Sip MPA
PID TTY Allocated Freed Holding Getbufs Retbufs Process
413 0 13588 264 20508 0 0 snmp dcm ma shim
293 0 12580 240 19604 0 0 CIFS Proxy Proce
220 0 15500 3044 19264 0 0 CEF: IPv4 proces
470 0 12180 724 19008 0 0 EEM Helper Threa
180 0 886132 45016 18796 0 0 IP Input
395 0 13412 4988 18688 0 0 EEM ED Resource
233 0 4968 240 17992 0 0 RG Media Timer
350 0 4968 240 17992 0 0 CME Extension Mo
272 0 10708 0 17836 0 0 SSH Event handle
118 0 10292 240 17316 0 0 UCSE Monitor
420 0 12000 4988 17276 0 0 EEM ED Interface
424 0 12280 4988 17100 0 0 EEM ED RF
428 0 11684 4988 16960 0 0 EEM ED Timer
404 0 11404 1552 16956 0 0 VLAN Manager
390 0 11660 4988 16936 0 0 EEM ED Routing
394 0 11656 4988 16932 0 0 EEM ED Track
427 0 11620 4988 16896 0 0 EEM ED SNMP Noti
393 0 11620 4988 16896 0 0 EEM ED NHRP
113 0 11596 5020 16840 0 0 EEM ED Identity
438 0 11560 4988 16836 0 0 EEM ED RPC
431 0 11516 4988 16792 0 0 EEM ED Env
421 0 11516 4988 16792 0 0 EEM ED IOSWD
430 0 11516 4988 16792 0 0 EEM ED Config
425 0 11516 4988 16792 0 0 EEM ED SNMP
439 0 11516 4988 16792 0 0 EEM ED Ipsla
429 0 11516 4988 16792 0 0 EEM ED Test
426 0 11516 4988 16792 0 0 EEM ED SNMP Obje
419 0 11516 4988 16792 0 0 EEM ED Counter
422 0 11516 4988 16792 0 0 EEM ED None
PID TTY Allocated Freed Holding Getbufs Retbufs Process
418 0 11516 4988 16792 0 0 EEM ED CLI
451 0 3644 240 16668 0 0 TRPAPP
489 0 10209128 10206948 15676 0 0 SNMP ENGINE
460 0 2600 240 15624 0 0 RECMSPAPP
343 0 8244 0 15508 0 0 Skinny HW confer
406 0 2468 480 15252 0 0 RBM CORE
412 0 9948 1864 15136 0 0 DCM snmp dp Thre
275 0 4728 0 14992 0 0 Ethernet PM Proc
436 0 14436 12916 14784 0 0 Syslog
446 0 1728 240 14752 0 0 MRIB Process
366 0 1600 1408 14624 0 0 Crypto IKE Dispa
490 0 1260 240 14324 0 0 IP SNMPV6
63 0 66872 0 14308 0 0 CTS CORE
487 0 397432 396388 14308 0 0 IP SNMP
327 0 944 0 14208 0 0 PKI OCSP
461 0 828 0 14092 0 0 lib_off_app
64 0 1044 1476 14068 0 0 SXP CORE
307 0 1044 240 14068 0 0 TPLUS
414 0 10100 14776 14060 0 0 Bulkstat-Client
309 0 1004 240 14028 0 0 LDAP process
488 0 992 0 14024 0 0 PDU DISPATCHER
210 0 704 88 13968 0 0 CEF background p
411 0 6432 0 13696 0 0 Call Home DSfile
31 0 2417604 2409996 13652 0 0 Skinny MOH Serve
195 0 480 240 13504 0 0 Spanning Tree
255 0 56464 56240 13488 0 0 ADJ resolve proc
169 0 224 0 13488 0 0 CEF switching ba
258 0 224 0 13488 0 0 MFIB Master back
222 0 224 0 13488 0 0 Collection proce
PID TTY Allocated Freed Holding Getbufs Retbufs Process
211 0 224 0 13488 0 0 fib_fib_bfd_sb e
221 0 224 0 13488 0 0 ADJ background
445 0 3269224 3269116 13372 0 0 mdns Timer Proce
484 0 0 0 13336 0 0 CFT Timer Proces
167 0 240 240 13264 0 0 Ethernet LMI
152 0 240 240 13264 0 0 RG Faults Timer
153 0 240 240 13264 0 0 RG Protocol Time
174 0 240 240 13264 0 0 ATM OAM TIMER
329 0 0 0 13264 0 0 encrypt proc
154 0 240 240 13264 0 0 RG Transport Tim
177 0 240 240 13264 0 0 IPAM Events
39 0 0 0 13264 0 0 Policy Manager
83 0 0 0 13264 0 0 Crypto PKI-HA
323 0 0 0 13264 0 0 RSA background p
326 0 0 0 13264 0 0 Crypto PKI-Rev-P
325 0 240 240 13264 0 0 Crypto PKI-CRL
173 0 240 240 13264 0 0 ATM OAM Input
303 0 0 0 13264 0 0 Opaque Database
60 0 240 240 13264 0 0 Logger
365 0 0 0 13264 0 0 Crypto INT
187 0 240 240 13264 0 0 PPP IP Route
59 0 760 760 13264 0 0 IDB Work
344 0 0 0 13264 0 0 CCSIP_EVENT_TRAC
454 0 240 240 13264 0 0 confmsp
191 0 0 0 13264 0 0 SSS Manager
192 0 0 0 13264 0 0 SSS Policy Manag
288 0 0 0 13264 0 0 IPS Process
53 0 240 240 13264 0 0 XML Proxy Client
138 0 240 240 13264 0 0 Auth Manager
PID TTY Allocated Freed Holding Getbufs Retbufs Process
196 0 240 240 13264 0 0 SSM connection m
197 0 0 0 13264 0 0 WAAS Process
386 0 0 0 13264 0 0 Test AAA Client
51 0 240 240 13264 0 0 Dialer event
306 0 1073612 1073612 13264 0 0 LOCAL AAA
201 0 240 240 13264 0 0 Ethernet Timer C
202 0 240 240 13264 0 0 Ethernet Msec Ti
401 0 0 0 13264 0 0 IKEv2 FlexVPN Pr
277 0 0 0 13264 0 0 Ethernet PM Moni
491 0 0 0 13264 0 0 SNMP ConfCopyPro
102 0 0 0 13264 0 0 sal_dpc_process
271 0 0 0 13264 0 0 VPDN call manage
208 0 240 240 13264 0 0 KRB5 AAA
269 0 240 240 13264 0 0 X.25 Background
328 0 0 0 13264 0 0 PKI Revocation
497 0 240 240 13264 0 0 Web Write Housek
212 0 240 372 13264 0 0 Flow Exporter Ti
213 0 240 240 13264 0 0 Flow Exporter Pa
234 0 0 0 13264 0 0 RG If-Mgr Timer
257 0 240 240 13264 0 0 Tunnel
465 0 240 240 13264 0 0 vxcmsp
235 0 0 0 13264 0 0 Routing Topology
468 0 240 240 13264 0 0 VOICE_HA_STANDBY
492 0 240 240 13264 0 0 SNMP Traps
99 0 240 240 13264 0 0 SM Installer
475 0 0 0 13264 0 0 CCSIP_DNS
44 0 0 0 13264 0 0 License IPC stat
45 0 0 0 13264 0 0 License IPC serv
245 0 91868 532916 13264 0 0 TCP Protocols
PID TTY Allocated Freed Holding Getbufs Retbufs Process
244 0 0 9220 13264 0 0 TCP Timer
330 0 5564 104 12828 0 0 MMA DP TIMER
0 0 12716 1130412 12716 0 0 *Sched*
388 0 5592 316 12668 0 0 dcm_cli_provider
331 0 5368 0 12632 0 0 PERFMON ASYNC TI
318 0 4968 240 11992 0 0 Crypto Support
442 0 4660 0 11924 0 0 trunk conditioni
87 0 18044 10640 11028 0 0 RF Master Main T
259 0 996 240 11020 0 0 MVPN Mgr Process
415 0 3780 264 10696 0 0 dcm_expression_p
260 0 480 240 10504 0 0 Multicast Offloa
214 0 224 0 10488 0 0 Tunnel FIB
6 0 3376 240 10400 0 0 Check heaps
92 0 440 304 10400 0 0 Per-Second Jobs
376 0 0 0 10264 0 0 Crypto Device Up
368 0 0 0 10264 0 0 IPSEC manual key
273 0 0 0 10264 0 0 IPv6 Static Hand
158 0 240 240 10264 0 0 IP Host Track Pr
380 0 0 0 10264 0 0 Licensing MIB pr
168 0 240 240 10264 0 0 Ethernet OAM Pro
188 0 0 0 10264 0 0 HQF TARGET DYNAM
270 0 60652 60652 10264 0 0 QoS stats proces
274 0 0 0 10264 0 0 Ethernet Datapla
265 0 0 0 10264 0 0 IPv6 RIB Event H
207 0 240 240 10264 0 0 EAPoUDP Process
264 0 0 0 10264 0 0 IPv6 RIB Cleanup
266 0 0 0 10264 0 0 IP IRDP
263 0 240 240 10264 0 0 App Route Proces
262 0 240 4020 10264 0 0 static
PID TTY Allocated Freed Holding Getbufs Retbufs Process
276 0 240 240 10264 0 0 Ethernet PM Soft
148 0 3208 240 10232 0 0 call_home_les_oi
407 0 3104 480 9888 0 0 CTS credentials
387 0 2532 0 9796 0 0 dcm_cli_engine
23 0 2104 0 9368 0 0 IPC Seat Manager
447 0 0 0 9264 0 0 Key Proc
134 0 1960 0 9224 0 0 FHRP Main thread
135 0 1940 0 9204 0 0 VRRP Main thread
171 0 1892 0 9156 0 0 TRACK Main threa
172 0 1864 0 9128 0 0 TRACK Client thr
292 0 2068 240 9092 0 0 CIFS API Process
165 0 58680 0 9016 0 0 Call Home Timer
48 0 1688 0 8952 0 0 CHKPT RG SCTPthr
88 0 1688 0 8952 0 0 RF Master Status
410 0 9996 260 8952 0 0 Call Home DS
308 0 60448 58844 8868 0 0 NIST rng proc
297 0 1376 240 8400 0 0 CRM_CALL_UPDATE_
400 0 944 0 8208 0 0 IKEv2 Cluster Lo
205 0 944 0 8208 0 0 Auth-proxy HTTP
287 0 1184 240 8208 0 0 URL filter proc
337 0 932 0 8196 0 0 VPM_CONN_CHK_BAC
144 0 1152 240 8176 0 0 DTP Protocol
228 0 796 160 8060 0 0 L2MM
54 0 137548 148032 7804 0 0 crypto sw pk pro
338 0 436 0 7700 0 0 CCVPM_R2
30 0 380 0 7644 0 0 Exception contro
336 0 584 240 7608 0 0 VPM_CABLE_STAT_P
240 0 312 0 7576 0 0 Service Routing
449 0 544 240 7568 0 0 Resource Monitor
PID TTY Allocated Freed Holding Getbufs Retbufs Process
227 0 480 240 7504 0 0 Dot1x Supplicant
225 0 480 240 7504 0 0 Dot1x Supplicant
226 0 480 240 7504 0 0 Dot1x Supplicant
375 0 228 0 7492 0 0 GKM KS PROCESS
282 0 396 240 7420 0 0 SDEE Management
500 0 224 0 7352 0 0 DHCPD Database
379 0 0 0 7320 0 0 crypto raw delet
13 0 1058994296 1058995124 7308 0 0 Licensing Auto U
224 0 0 0 7264 0 0 Critical Auth
249 0 0 0 7264 0 0 bsm_timers
186 0 240 240 7264 0 0 PPP Bind
32 0 0 0 7264 0 0 IFS Agent Manage
295 0 0 0 7264 0 0 IPv6 Inspect Tim
296 0 0 0 7264 0 0 IPSECv6 PS Proc
206 0 0 0 7264 0 0 IP Admin SM Proc
185 0 240 240 7264 0 0 PPP SIP
299 0 240 240 7264 0 0 AAA Cached Serve
300 0 240 240 7264 0 0 ENABLE AAA
301 0 0 0 7264 0 0 EM Background Pr
302 0 0 0 7264 0 0 IDMGR CORE
184 0 240 240 7264 0 0 MOP Protocols
250 0 0 0 7264 0 0 bsm_xmt_proc
305 0 240 240 7264 0 0 LINE AAA
290 0 0 0 7264 0 0 Select Timers
190 0 0 0 7264 0 0 RIB LM VALIDATE
193 0 0 0 7264 0 0 SSS Feature Mana
209 0 0 0 7264 0 0 FRR Background P
223 0 0 0 7264 0 0 Socket Timers
268 0 0 0 7264 0 0 PAD InCall
PID TTY Allocated Freed Holding Getbufs Retbufs Process
267 0 0 0 7264 0 0 LAPB Process
286 0 240 240 7264 0 0 CCE DP URLF cach
314 0 240 240 7264 0 0 VSP_MGR
315 0 0 0 7264 0 0 EAP Process
285 0 240 240 7264 0 0 FW DP Inspect pr
252 0 240 240 7264 0 0 PPP Compress Res
219 0 0 0 7264 0 0 IPC UDP Input
194 0 0 0 7264 0 0 SSS Feature Time
9 0 240 240 7264 0 0 Timers
283 0 240 240 7264 0 0 Inspect process
322 0 0 3192 7264 0 0 crypto engine pr
15 0 240 240 7264 0 0 Environmental mo
254 0 240 240 7264 0 0 Dialer Forwarder
198 0 0 0 7264 0 0 X.25 Encaps Mana
242 0 0 0 7264 0 0 SNMP Timers
199 0 0 0 7264 0 0 IPv6 ping proces
156 0 240 240 7264 0 0 RG AR
279 0 240 240 7264 0 0 PPP NBF
150 0 0 0 7264 0 0 SMI CDP Update H
278 0 240 240 7264 0 0 RLM groups Proce
8 0 0 0 7264 0 0 DiscardQ Backgro
218 0 240 240 7264 0 0 IPMUXv6
204 0 0 0 7264 0 0 Auth-proxy AAA B
335 0 240 240 7264 0 0 VPM_MWI_BACKGROU
251 0 240 240 7264 0 0 PPP Compress Inp
217 0 240 240 7264 0 0 IPMUXv4
175 0 240 240 7264 0 0 IPAM/ODAP Events
339 0 0 0 7264 0 0 IP TRUST Registe
261 0 0 0 7264 0 0 RARP Input
PID TTY Allocated Freed Holding Getbufs Retbufs Process
216 0 0 0 7264 0 0 HQF Input Shaper
342 0 240 240 7264 0 0 EPHONE MWI BG Pr
147 0 240 240 7264 0 0 EtherChnl
146 0 240 240 7264 0 0 PI MATM Aging Pr
247 0 0 0 7264 0 0 IUA Main Process
346 0 240 240 7264 0 0 CTI Call Control
347 0 240 240 7264 0 0 CTI Line Monitor
348 0 0 0 7264 0 0 CME CTI Device M
304 0 0 0 7264 0 0 Key chain liveke
183 0 240 240 7264 0 0 NAT64 Bgnd Proce
351 0 0 0 7264 0 0 Presence Process
352 0 0 0 7264 0 0 DS0 PCM Capture
182 0 240 240 7264 0 0 PIM register asy
141 0 0 0 7264 0 0 802.1x Webauth F
215 0 0 0 7264 0 0 HQF Output Shape
139 0 0 0 7264 0 0 AUTH POLICY Fram
137 0 240 240 7264 0 0 IP NAT HA BGND P
181 0 0 0 7264 0 0 ICMP event handl
166 0 240 240 7264 0 0 AAA Dictionary R
35 0 240 240 7264 0 0 ATM Idle Timer
28 0 0 0 7264 0 0 IPC Loadometer
126 0 240 240 7264 0 0 BPSM stat Proces
363 0 0 0 7264 0 0 IKEv2 AAA handle
364 0 240 572 7264 0 0 CRYPTO MAP FREE
36 0 0 0 7264 0 0 ATM ASYNC PROC
123 0 240 240 7264 0 0 Call Management
122 0 0 0 7264 0 0 IGMP Snooping Re
239 0 0 0 7264 0 0 IP Traceroute
27 0 0 0 7264 0 0 IPC Keep Alive M
PID TTY Allocated Freed Holding Getbufs Retbufs Process
370 0 0 0 7264 0 0 CRYPTO QoS proce
26 0 0 0 7264 0 0 IPC Seat TX Cont
372 0 0 0 7264 0 0 Crypto PAS Proc
121 0 0 0 7264 0 0 IGMP Snooping Pr
374 0 240 240 7264 0 0 GKM GM Process
319 0 0 0 7264 0 0 EPM MAIN PROCESS
229 0 0 0 7264 0 0 MRD
377 0 0 0 7264 0 0 Multi-ISA Event
378 0 0 0 7264 0 0 Multi-ISA Cleanu
155 0 0 0 7264 0 0 RG VP
11 0 0 0 7264 0 0 License Client N
381 0 240 240 7264 0 0 Control-plane ho
382 0 0 0 7264 0 0 PM Callback
383 0 240 240 7264 0 0 dhcp snooping sw
385 0 240 2560 7264 0 0 AAA SEND STOP EV
119 0 240 240 7264 0 0 Ether-Switch RBC
230 0 0 0 7264 0 0 IGMPSN
117 0 7896 240 7264 0 0 FLEX DNLD MAIN
37 0 0 0 7264 0 0 CEF MIB API
115 0 0 0 7264 0 0 VIDEO DSPRM MAIN
25 0 0 0 7264 0 0 IPC Seat RX Cont
24 0 0 0 7264 0 0 IPC Check Queue
111 0 0 0 7264 0 0 RF_INTERDEV_DELA
110 0 240 240 7264 0 0 ehwic_eswge Proc
109 0 0 0 7264 0 0 esw_mrvl mat que
396 0 240 240 7264 0 0 RMON Recycle Pro
397 0 240 240 7264 0 0 RMON Deferred Se
398 0 0 0 7264 0 0 Syslog Traps
399 0 0 0 7264 0 0 WAN Optimization
PID TTY Allocated Freed Holding Getbufs Retbufs Process
333 0 0 0 7264 0 0 FLEX DSP KEEPALI
108 0 240 240 7264 0 0 DXMRVL
402 0 0 0 7264 0 0 Crypto cTCP proc
106 0 240 240 7264 0 0 ESWILPPM
105 0 240 240 7264 0 0 Eswilp Storm Con
104 0 240 240 7264 0 0 ESWPPM
103 0 0 0 7264 0 0 ARL Table Manage
12 0 0 0 7264 0 0 Image License br
232 0 0 0 7264 0 0 MAB Framework
321 0 240 240 7264 0 0 CSDB Timer proce
162 0 0 0 7264 0 0 AAA System Acct
98 0 240 240 7264 0 0 Bryce I2C CMD Qu
97 0 240 240 7264 0 0 SM Monitor
96 0 0 0 7264 0 0 HC Counter Timer
95 0 0 196764 7264 0 0 Transport Port A
160 0 0 0 7264 0 0 AAA ACCT Proc
94 0 0 0 7264 0 0 Token Daemon
93 0 0 0 7264 0 0 AggMgr Process
157 0 0 0 7264 0 0 DHCP Snooping
91 0 0 3488 7264 0 0 Per-minute Jobs
90 0 240 240 7264 0 0 Compute load avg
89 0 0 0 7264 0 0 Net Input
340 0 0 0 7264 0 0 EPHONE MWI Refre
86 0 240 240 7264 0 0 TDM Management
85 0 0 0 7264 0 0 IPSEC HA Mgr
84 0 0 0 7264 0 0 IKE HA Mgr
82 0 0 0 7264 0 0 CHKPT Test clien
81 0 0 0 7264 0 0 CHKPT Test clien
80 0 0 0 7264 0 0 CHKPT Test clien
PID TTY Allocated Freed Holding Getbufs Retbufs Process
79 0 0 0 7264 0 0 CHKPT Test clien
78 0 0 0 7264 0 0 CHKPT Test clien
77 0 0 0 7264 0 0 CHKPT Test clien
76 0 0 0 7264 0 0 CHKPT Test clien
75 0 0 0 7264 0 0 CHKPT Test clien
434 0 4480 4908 7264 0 0 IP NAT Ager
74 0 0 0 7264 0 0 CHKPT Test clien
73 0 0 0 7264 0 0 CHKPT Test clien
72 0 0 0 7264 0 0 CHKPT Test clien
71 0 0 0 7264 0 0 cpf_process_tpQ
70 0 0 0 7264 0 0 cpf_msg_rcvq_pro
440 0 0 0 7264 0 0 IP SLAs Ethernet
441 0 0 0 7264 0 0 trunk conditioni
69 0 0 0 7264 0 0 cpf_msg_holdq_pr
443 0 0 0 7264 0 0 VDC process
38 0 0 0 7264 0 0 AAA_SERVER_DEADT
67 0 0 0 7264 0 0 IF-MGR event pro
61 0 77036 240 7264 0 0 TTY Background
341 0 0 0 7264 0 0 FB/KS Log HouseK
448 0 240 240 7264 0 0 TRP_PSEUDOTIMER
145 0 0 0 7264 0 0 SFF8472
450 0 240 240 7264 0 0 Resource Availab
57 0 0 0 7264 0 0 Critical Bkgnd
5 0 0 0 7264 0 0 RO Notify Timers
22 0 0 0 7264 0 0 IPC Process leve
56 0 0 0 7264 0 0 ISM-VPN Cleanup
55 0 240 240 7264 0 0 ISM-VPN main
21 0 0 0 7264 0 0 IPC Deferred Por
20 0 0 0 7264 0 0 IPC Periodic Tim
PID TTY Allocated Freed Holding Getbufs Retbufs Process
143 0 240 240 7264 0 0 CMD HANDLER
19 0 0 0 7264 0 0 IPC Zone Manager
52 0 0 0 7264 0 0 SERIAL A'detect
50 0 240 240 7264 0 0 GraphIt
462 0 0 0 7264 0 0 Media Record
463 0 240 240 7264 0 0 Voice Player
464 0 0 0 7264 0 0 Resource Measure
49 0 0 0 7264 0 0 ARP Snoop
466 0 240 240 7264 0 0 Synthetic Video
18 0 0 0 7264 0 0 IPC Session Serv
142 0 0 0 7264 0 0 EAP Framework
40 0 240 240 7264 0 0 DDR Timers
46 0 8736 8812 7264 0 0 SMART
471 0 0 0 7264 0 0 IP NAT WLAN
17 0 0 0 7264 0 0 IPC Dynamic Cach
473 0 0 0 7264 0 0 CCH323_DNS
140 0 312 312 7264 0 0 Dot1x Mgr Proces
43 0 240 240 7264 0 0 Serial Backgroun
42 0 996 996 7264 0 0 PrstVbl
16 0 0 0 7264 0 0 IPC Event Notifi
248 0 240 240 7264 0 0 RUDPV1 Main Proc
482 0 0 0 7264 0 0 L4F OOB Process
480 0 0 0 7264 0 0 IP VFR proc
495 0 240 240 7264 0 0 Skinny MOH Event
486 0 240 240 7264 0 0 IP Flow Top Talk
493 0 240 240 7264 0 0 BUSYOUT SCAN
499 0 0 0 7172 0 0 DHCPD Timer
501 0 0 0 7128 0 0 NBAR LLT timer s
130 0 240 0 4504 0 0 Async write proc
PID TTY Allocated Freed Holding Getbufs Retbufs Process
129 0 240 0 4504 0 0 Async write proc
127 0 240 0 4504 0 0 Async write proc
128 0 240 0 4504 0 0 Async write proc
170 0 224 0 4488 0 0 ADJ NSF process
131 0 0 0 4264 0 0 DFS flush period
101 0 0 0 4264 0 0 dev_device_remov
100 0 0 0 4264 0 0 dev_device_inser
231 0 240 240 4264 0 0 L2X Switching Ev
68 0 0 0 4264 0 0 Inode Table Dest
14 0 0 0 4264 0 0 RMI RM Notify Wa
10 0 0 0 4264 0 0 WATCH_AFS
320 0 240 240 4264 0 0 RSCMSM VOLUME MO
2 0 240 240 4264 0 0 Load Meter
107 0 240 240 4264 0 0 Eswilp Storm Con
202996124 Total
CISCO2911-EGCAI01#

[marce1000]

 - Such issues may sometimes be resolved (but not guaranteed) in later software releases for your device, if applicable , and or feasible - upgrade and check if the problem persists, 

 M.

[Richard Burts]

The symptoms suggest that the problem may be a memory leak. The traceback messages certainly point to some type of software problem (rather than to a hardware problem). I agree that if you are able to upgrade to a different version of code it might very well solve your problem. 

Reboot provides a solution but it is likely be only a temporary solution. The long term solution is a different version of code (if you are able to do this)

[amralrazzaz]

what do u mean bu version of code ? u mean ios ?

and what if i go with upgrading memory if available on market ? is it okay ?

also if i disable the VOIP option fore sometimes it will help ?

is it available to upgrade the memory of this device ? how much ?

[amralrazzaz]

upgrading what ? IOS or memory ?

how many slots free in this router ? and is it available in markets to buy and install memory ? 

also with reload it sometimes solving issue but maybe it come back ?

how much is the memory and shall i buy 1 gb so total will be 1,5 GB of ram ?

[Richard Burts]

amr alrazzaz

Upgrading memory is not going to solve this issue. The only long term solution is to upgrade the version of code. 

[amralrazzaz]

what is version of code mean ? do u mean IOS version to the latest ?

[Georg Pauwen]

Hello,

post the full running configuration of the router, just to be sure that the configuration does not somehow cause this...

[amralrazzaz]

ill send it to u sir .. and i  have another question for the ios 

can i download the ios.bin from different site rather than cisco cite because they ask for contract service account and i have normal account and its just ios 

actually i downloaded this version from different site and dont know if its okay to install it on my router to upgrade what i have ?? is it fine and work normally ?

c2900-universalk9-mz.SPA.154-3.M1.bin -  this is current ios im using 

c2900-universalk9-mz.SPA.157-3.M5.bin - this is what i installed from internet (not cisco site!) 

so is **bleep** fine to use this for upgrade?

[Richard Burts]

amr alrazzaz

You ask "what is version of code mean ? do u mean IOS version to the latest ?"  Yes we are talking about IOS version. The Cisco site is the only site to get authentic official versions of IOS. You might use the code you downloaded from a different site. It might work fine. It might have issues. We can not know about it and you use it at your own risk.

[amralrazzaz]

actually i need to ask if usually the ios file came with code to use when install on router ? 

the ios costing how much approximately  if u know ? 

another question ? if i upgrade with new verison all configurations and flash memory and licenses will be removed all or nothing will happen ?

also the newer ios will be replaced on current old one ? and no way to get back to old one ?

thanks

[Richard Burts]

amr alrazzaz

Here are responses to your questions:

- For the 2911 the code file is typically a bin file that you load to the 2911 and it boots it. I think you are asking if there is a "loader" program that installs the code. While some other Cisco platforms use this approach the 2911 does not. It simply loads the bin file.

- you do not purchase the IOS code file. You purchase a service contract for your 2911 and the code file is part of what you get. There are options about what is included in the service contract (hours of coverage, guaranteed response time, etc) which impact the cost of the service contract. I could not guess what a service contract would cost you (but assume that it is pretty expensive).

- when you upgrade the code the flash memory and licenses should stay the same. For the most part the configuration should stay the same. There might be some changes in syntax for some commands, or there might be some change in defaults, which might cause some change in the configuration. Sometimes a new feature is added which causes some additions to the config to support that feature. Very occasionally support for a feature is dropped (especially if the update is to a new major release version) which might remove some commands from the configuration. But in general your configuration should stay stable.

- depending on the size of flash in your 2911 and depending on the size of the code file it might be possible to keep the old version in flash and also have the new version in flash. When both versions of code are in flash you would use boot system commands to control which version of code the router will boot (frequently specify to try the new code first and to use the old code if there is some problem in booting the new code).

If there is not room for both code files in flash you will need to remove the old version. Before you remove the old code file you should copy the old version of code (tftp and ftp are the typical ways to copy the code but there are other options for how to copy the code file) to your PC or to a server which can store the old code in case you need to revert to it. It is always better when you maintain a copy of the old code file. If there is a problem with the new code you can remove it and copy the old version back into flash.

[FullTimeWebHosting]

as far i can see, the issue is related to dead process taking memory  and not releasing it

check the memory usage with 

show processes memory sorted Allocated

and copy the first 20 lines.