ISR with a very high percentage of the DRAM

[BorislavPenchev0962]

Hi,

we have Cisco ISR 4k with a very high percentage of the DRAM

what could be the reason and how to resolve this ?

IOS 16.12.04

show processes memory sorted
Processor Pool Total: 1736323776 Used: 290005768 Free: 1446318008
reserve P Pool Total: 102404 Used: 88 Free: 102316
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 317237864 109669544 179217896 0 0 *Init*
729 0 34427568 51768 34751800 0 0 SBC main process
89 0 8520500392 7115579280 11003304 0 0 IOSD ipc task
213 32132 106115216 93460008 7848944 0 0 iosp_vty_100001_
0 0 227317216 221907496 5349296 8381679 0 *Dead*
201 0 8564912 1545400 5004848 0 0 CWAN OIR Handler
584 0 73193336 69161456 4073824 849828 0 EEM ED Syslog
158 0 236663832 251141400 3803536 0 0 SDWAN PB rx
1 0 3228512 31248 3227208 0 0 Chunk Manager
616 0 1802000 83136 1749376 0 0 EEM Server
546 0 696632 0 738576 0 0 QOS_MODULE_MAIN
749 0 573008 2976 567680 0 0 OSPF-101 Router
232 0 735480 268360 521136 0 0 IP ARP Adjacency
6 0 562728 52944 477088 0 0 RF Slave Main Th
0 0 0 0 461016 0 0 *MallocLite*
236 0 802512 334008 444008 0 0 mDNS
327 0 834384 426472 440600 0 0 CEF background p
585 0 388264 5704 424504 72316 0 EEM ED Generic
355 0 461664 44710680 404392 0 0 IP RIB Update
564 0 308320 1072 361192 0 0 Crypto IKEv2
190 0 1719920 1857032 324152 0 0 IPSec background
549 0 1976 0 247920 0 0 http client proc
188 0 218400 448 247896 0 0 VNM DSPRM MAIN
358 0 1824 0 247768 0 0 COPS
356 0 255728 78680 240752 0 0 IP Background
617 0 276344 160000 239272 0 0 Call Home proces
730 0 167720 448 221216 0 0 MRIB Process
10 0 38289107736 38289149448 216168 36573756336 36573786256 Pool Manager
PID TTY Allocated Freed Holding Getbufs Retbufs Process
745 0 205456 1424 215784 0 0 ONEP Network Ele
104 0 139584048 139430888 211488 0 0 DBAL EVENTS
522 0 1548392 1344176 205912 0 0 Crypto CA
357 0 179536 16608 204872 0 0 IP Connected Rou
221 0 400488 256288 194616 0 0 VRRS Main thread
77 0 415984 74952 191104 0 0 Net Background
404 0 156160 448 185656 0 0 RUDPV1 Main Proc
591 866 1936672 1820560 181688 0 0 SSH Process
15 0 33217561384 33217395344 178832 0 0 ARP Input
485 0 73336 448 174832 0 0 AN
410 0 140776 328 170152 0 0 L2FIB Event Disp
747 0 174208 58344 169976 0 0 CEF: IPv6 proces
500 0 132304 976 169272 0 0 Crypto WUI
293 0 65632 0 167576 0 0 st_pw_oam
7 0 65632 0 167576 0 0 EDDRI_MAIN
194 0 52080 2760 154160 0 0 LICENSE AGENT
230 0 98736 0 152680 0 0 IPAM Manager
510 0 43664 448 145160 0 0 STUN_APP
770 0 207352 14504 135256 0 0 OSPF-109 Router
663 0 84680 4104 134232 0 0 VLAN Manager
507 0 102904 0 132848 0 0 MMA DB TIMER
268 0 100816 0 130760 0 0 radius radsec cl
622 0 547864 476120 128856 0 0 EEM ED CLI
513 0 99192 744 128816 0 0 Timer Library
458 0 98744 0 128688 0 0 AAA HA cleanup
210 0 98736 0 128680 0 0 ACCT Periodic Pr
751 0 1415904 1284400 125560 0 0 DHCPD Receive
411 0 65576 0 119520 0 0 ATM HA
342 0 14824 0 116416 0 0 HTTP CORE
547 0 8576 0 110632 0 0 VoIP AAA
640 0 0 0 105624 0 0 PnP-Monitor
316 0 3312 160 105256 0 0 SKA IPC process
653 0 11480 8544 104880 0 0 WSMAN Process

Regards

[balaji.bandi]

check some troubleshoot steps : and we need to find what process is taking RAM : (to find any bugs reported same ?)

https://www.cisco.com/c/en/us/td/docs/routers/access/4400/troubleshooting/memorytroubleshooting/isr4000_mem.html

[Leo Laohoo]

---

@BorislavPenchev0962 wrote:

IOS 16.12.04

---

Do not use 16.12.4 and above.  

Downgrade to 16.12.3 or 16.12.3a or use 17.3.4.

Post the complete output to the following command "sh platform software status con brief".

[BorislavPenchev0962]

here the output of the following command

sh platform software status con brie
Load Average
Slot Status 1-Min 5-Min 15-Min
RP0 Healthy 0.24 0.40 0.35

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
RP0 Healthy 4039284 3452476 (85%) 586808 (15%) 4634252 (115%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
RP0 0 2.00 1.40 0.00 96.59 0.00 0.00 0.00
1 0.30 0.30 0.00 99.40 0.00 0.00 0.00
2 2.20 1.60 0.00 96.10 0.00 0.10 0.00
3 1.90 1.10 0.00 96.99 0.00 0.00 0.00
4 2.10 1.90 0.00 96.00 0.00 0.00 0.00
5 1.10 0.20 0.00 98.70 0.00 0.00 0.00
6 1.40 1.70 0.00 96.89 0.00 0.00 0.00
7 0.10 0.10 0.00 99.79 0.00 0.00 0.00

[Leo Laohoo]

---

@BorislavPenchev0962 wrote:

Memory (kB)
Slot Status   Total     Used (Pct)  Free  (Pct) Committed (Pct)
RP0  Healthy 4039284 3452476 (85%) 586808 (15%) 4634252 (115%)

---

Ok, I have some bad news and I have some f**king bad news.  

Bad News:  The router has a memory leak and will probably crash in the next 5 days.  

F**king Bad News:  The router is running a f**cking buggy code, 16.12.4, which is infamously known for a multitude of memory leaks, CPU hogs, SNMP-caused crashes, and the list goes on and on.  

I still have a command to try to find out what is causing the memory leak: 

sh process memory platform sort location r0 

[BorislavPenchev0962]

for now there is no crash , here the output of

#sh process memory platform sort location r0
System memory: 4039284K total, 3419656K used, 619628K free,
Lowest: 445264K
Pid Text Data Stack Dynamic RSS Name
----------------------------------------------------------------------
18548 360499 627260 136 224 627260 linux_iosd-imag
9808 1891 196216 136 171260 196216 confd
31355 95 173716 136 84 173716 cpp_cp_svr
30193 13816 129632 136 100 129632 fman_fp_image
9745 599 73288 136 280 73288 sysmgrd
30419 69 66192 136 12 66192 cpp_sp_svr
10159 3448 65232 224 0 65232 ftmd
30721 370 53764 136 12 53764 cpp_ha_top_leve
31019 151 50468 156 12 50468 cpp_driver
25123 276 49984 136 44 49984 bsm
19622 676 46148 136 48 46148 smand
10176 1707 44688 1956 8 44688 vdaemon
22529 7883 38400 136 88 38400 fman_rp
10164 1483 34260 136 0 34260 ompd
19366 193 33616 136 68 33616 smd
4145 240 32980 136 72 32980 ndbmand
2346 71 30212 136 72 30212 pubd
23444 164 29100 136 60 29100 dbm
23729 1292 28952 136 52 28952 cmand
9248 538 19344 136 76 19344 iomd
24043 59 15056 136 88 15056 cli_agent
32088 320 14880 136 32 14880 cman_fp
31754 66 14128 136 8 14128 cpp_cdm_svr
10154 1180 11224 136 0 11224 fpmd
20336 376 11000 136 48 11000 repm
12536 2544 10732 132 1060 10732 dhcpd
3225 202 10728 136 44 10728 syncfd
22084 387 10480 488 44 10480 hman
5756 720 9552 136 48 9552 cmcc
19087 900 9148 136 44 9148 vman
7859 720 9092 136 48 9092 cmcc
9048 720 9084 136 48 9084 cmcc
18103 78 8796 136 44 8796 psd
5133 156 8704 136 44 8704 dmiauthd
13762 99 8516 136 44 8516 ngiolite
10151 1630 8300 136 0 8300 cfgmgr
10152 565 7868 136 0 7868 cxpd
3570 206 7804 136 44 7804 nesd
10170 437 7552 136 264 7552 ttmd
24689 191 7492 136 168 7492 btman
7057 387 7348 488 44 7348 hman
8108 387 7344 484 44 7344 hman
13891 1270 7340 136 44 7340 nginx
4701 387 7188 484 44 7188 hman
29662 387 7144 488 44 7144 hman
10153 309 7108 136 140 7108 dbgd
16933 933 6768 248 4820 6768 psvp.sh
16747 933 6716 132 3892 6716 rollback_timer.
32293 191 6712 136 168 6712 btman
11226 933 6564 252 3336 6564 reflector.sh
10600 933 6544 244 3336 6544 droputil.sh
8789 191 6488 136 168 6488 btman
7075 191 6488 136 168 6488 btman
9728 191 6480 136 168 6480 btman
1 1506 6208 132 1580 6208 systemd
21651 115 6072 136 160 6072 keyman
21435 161 6040 136 44 6040 lman
229 366 5932 132 132 5932 systemd-journal
10626 498 5780 132 132 5780 libvirtd
17013 933 5524 244 3604 5524 pvp.sh
1257 933 5244 252 3388 5244 pvp.sh
28385 933 5240 248 3380 5240 pvp.sh
4068 933 5232 252 3388 5232 pvp.sh
2654 933 5228 248 3388 5228 pvp.sh
11519 933 5224 248 2080 5224 iptbl.sh
20720 88 5132 136 44 5132 plogd
13690 1270 4688 136 44 4688 nginx
19923 90 4560 136 40 4560 sdavc_proxy
6020 712 4404 136 40 4404 ncsshd_bp
21031 56 4364 136 48 4364 octeon_logger
286 472 3848 132 528 3848 systemd-udevd
426 188 3844 132 132 3844 dbus-daemon
25912 71 3712 136 40 3712 tams_proc
10599 933 3628 132 660 3628 oom.sh
24428 933 3584 252 2204 3584 cedge-tanman.sh
14047 933 3448 252 2204 3448 cedge-tanman.sh
16852 9 3412 132 40 3412 rotee
20785 554 3344 136 132 3344 journalctl
11792 9 3332 132 40 3332 rotee
26097 85 3328 136 40 3328 tamd_proc
11596 241 3320 136 52 3320 ncsshd
11287 9 3280 132 40 3280 rotee
10758 9 3236 132 40 3236 rotee
11141 9 3132 132 40 3132 rotee
6975 241 3124 136 52 3124 ncsshd
11632 9 3120 132 40 3120 rotee
10587 574 2976 132 132 2976 virtlogd
10594 933 2828 132 132 2828 libvirtd.sh
16752 933 2824 132 132 2824 auxinit.sh
11602 933 2708 252 1624 2708 netconf-subsys.
408 100 2664 132 132 2664 auditd
10687 68 2520 132 132 2520 rpc.statd
11601 241 2512 136 52 2512 ncsshd
23006 933 2476 136 1380 2476 flash_check.sh
27055 60 2380 136 40 2380 tam_svcs_esg_cf
10614 45 2152 132 132 2152 rpcbind
26200 933 2100 252 960 2100 pman.sh
25867 933 2100 252 964 2100 pman.sh
24811 933 2100 252 964 2100 pman.sh
23840 933 2100 256 960 2100 pman.sh
23241 933 2100 256 960 2100 pman.sh
22426 933 2100 256 964 2100 pman.sh
19210 933 2100 256 960 2100 pman.sh
18689 933 2100 256 956 2100 pman.sh
18078 933 2100 252 964 2100 pman.sh
5601 933 2100 256 964 2100 pman.sh
25505 933 2096 252 960 2096 pman.sh
23530 933 2096 248 964 2096 pman.sh
22974 933 2096 252 960 2096 pman.sh
20716 933 2096 252 960 2096 pman.sh
20167 933 2096 252 960 2096 pman.sh
19528 933 2096 252 956 2096 pman.sh
12898 933 2096 252 960 2096 pman.sh
10698 96 2096 132 132 2096 rpc.mountd
4840 933 2096 252 960 2096 pman.sh
2922 933 2096 252 960 2096 pman.sh
2580 933 2096 252 964 2096 pman.sh
1481 933 2096 252 960 2096 pman.sh
25266 933 2092 248 960 2092 pman.sh
24193 933 2092 248 960 2092 pman.sh
22745 933 2092 248 960 2092 pman.sh
22036 933 2092 248 960 2092 pman.sh
21691 933 2092 248 960 2092 pman.sh
21065 933 2092 248 960 2092 pman.sh
19848 933 2092 248 960 2092 pman.sh
18919 933 2092 248 964 2092 pman.sh
18497 933 2092 248 960 2092 pman.sh
17493 933 2092 248 960 2092 pman.sh
4205 933 2092 248 964 2092 pman.sh
2287 933 2092 248 964 2092 pman.sh
1196 933 2092 248 960 2092 pman.sh
7223 933 2088 256 960 2088 pman.sh
31701 933 2084 252 960 2084 pman.sh
31395 933 2084 252 960 2084 pman.sh
30432 933 2084 252 960 2084 pman.sh
8637 933 2084 252 960 2084 pman.sh
8283 933 2084 248 960 2084 pman.sh
7580 933 2084 252 960 2084 pman.sh
4909 933 2084 252 960 2084 pman.sh
31054 933 2080 248 960 2080 pman.sh
30744 933 2080 252 960 2080 pman.sh
30113 933 2080 252 960 2080 pman.sh
29824 933 2080 252 960 2080 pman.sh
29596 933 2080 248 960 2080 pman.sh
29116 933 2080 248 956 2080 pman.sh
25491 933 2080 252 964 2080 pman.sh
13438 933 2080 252 960 2080 pman.sh
9091 933 2080 252 960 2080 pman.sh
7950 933 2080 252 960 2080 pman.sh
6078 933 2080 248 960 2080 pman.sh
5980 933 2080 248 960 2080 pman.sh
3971 933 2080 252 960 2080 pman.sh
16760 150 2032 132 132 2032 xinetd
25255 9 2016 136 40 2016 rotee
1820 65 1992 136 40 1992 pttcd
16749 150 1920 132 132 1920 xinetd
6323 9 1920 136 40 1920 rotee
13628 9 1908 132 40 1908 rotee
26190 9 1904 136 40 1904 rotee
14846 150 1904 132 132 1904 xinetd
25858 9 1900 136 40 1900 rotee
24796 9 1900 136 40 1900 rotee
24766 9 1896 136 40 1896 rotee
23406 9 1896 136 40 1896 rotee
23099 9 1896 136 40 1896 rotee
21148 9 1896 136 40 1896 rotee
20326 9 1896 136 40 1896 rotee
19037 9 1896 136 40 1896 rotee
18786 9 1896 136 40 1896 rotee
17839 9 1896 136 40 1896 rotee
5520 9 1896 136 40 1896 rotee
2738 9 1896 136 40 1896 rotee
31947 9 1892 132 40 1892 rotee
31008 9 1892 132 40 1892 rotee
30381 9 1892 132 40 1892 rotee
30048 9 1892 132 40 1892 rotee
29844 9 1892 132 40 1892 rotee
29319 9 1892 132 40 1892 rotee
23708 9 1892 136 40 1892 rotee
21432 9 1892 136 40 1892 rotee
20664 9 1892 136 40 1892 rotee
19916 9 1892 136 40 1892 rotee
18393 9 1892 136 40 1892 rotee
9323 9 1892 132 40 1892 rotee
8963 9 1892 132 40 1892 rotee
8645 9 1892 132 40 1892 rotee
8253 9 1892 132 40 1892 rotee
7797 9 1892 132 40 1892 rotee
7559 9 1892 132 40 1892 rotee
6392 9 1892 132 40 1892 rotee
5330 9 1892 132 40 1892 rotee
4349 9 1892 132 40 1892 rotee
3117 9 1892 136 40 1892 rotee
1932 9 1892 136 40 1892 rotee
418 34 1892 132 132 1892 audispd
31733 9 1888 132 40 1888 rotee
30681 9 1888 132 40 1888 rotee
24255 9 1888 136 40 1888 rotee
22339 9 1888 136 40 1888 rotee
21935 9 1888 136 40 1888 rotee
19338 9 1888 136 40 1888 rotee
6444 9 1888 132 40 1888 rotee
4680 9 1888 136 40 1888 rotee
3666 9 1888 136 40 1888 rotee
17131 9 1884 132 40 1884 rotee
13294 9 1868 136 40 1868 rotee
1511 9 1868 132 40 1868 rotee
28581 9 1860 132 40 1860 rotee
4488 9 1860 132 40 1860 rotee
3170 9 1860 132 40 1860 rotee
23971 9 1852 136 40 1852 rotee
26646 9 1840 136 40 1840 rotee
23186 9 1840 136 40 1840 rotee
22721 9 1840 136 40 1840 rotee
17002 9 1840 132 40 1840 rotee
31323 9 1836 132 40 1836 rotee
25619 9 1836 136 40 1836 rotee
19589 9 1836 136 40 1836 rotee
1465 9 1836 136 40 1836 rotee
14841 150 1796 132 132 1796 xinetd
10659 150 1772 132 132 1772 xinetd
10593 48 1656 132 132 1656 sntp
10369 933 1652 132 132 1652 sysmgr_stats_st
10362 933 1644 132 132 1644 sysmgr_stats_st
10357 933 1640 132 132 1640 sysmgr_stats_st
10332 933 1640 132 132 1640 sysmgr_stats_st
10373 933 1636 132 132 1636 sysmgr_stats_st
10330 933 1636 132 132 1636 sysmgr_stats_st
10371 933 1632 132 132 1632 sysmgr_stats_st
11794 23 1620 132 132 1620 inotifywait
11306 23 1568 132 132 1568 inotifywait
16858 23 1556 132 132 1556 inotifywait
9744 933 1536 132 132 1536 viptela-logrota
11158 23 1528 132 132 1528 inotifywait
10785 35 1208 132 132 1208 sleep
26412 933 1180 136 132 1180 sort_files_by_i
11665 68 1052 132 36 1052 netconf-subsys
1794 23 656 132 132 656 inotifywait
28749 23 652 132 132 652 inotifywait
17213 23 652 132 132 652 inotifywait
17008 23 652 132 132 652 inotifywait
4757 23 652 132 132 652 inotifywait
3480 23 652 132 132 652 inotifywait
20082 35 576 136 132 576 sleep
21080 35 572 136 132 572 sleep
21295 35 568 132 132 568 sleep
21289 35 568 132 132 568 sleep
21286 35 568 132 132 568 sleep
21293 35 564 132 132 564 sleep
21288 35 564 132 132 564 sleep
21283 35 564 132 132 564 sleep
21282 35 564 132 132 564 sleep
21290 35 560 132 132 560 sleep
14046 23 384 136 132 384 inotifywait
11031 32 220 136 132 220 inet_gethost
11030 32 184 136 132 184 inet_gethost

[Leo Laohoo · Accepted solution]

---

@BorislavPenchev0962 wrote:

#sh process memory platform sort location r0
System memory: 4039284K total, 3419656K used, 619628K free,
Lowest: 445264K
Pid    Text  Data  Stack  Dynamic   RSS Name
----------------------------------------------------------------------
18548 360499 627260 136    224   627260 linux_iosd-imag
 9808   1891 196216 136 171260   196216 confd
31355     95 173716 136     84   173716 cpp_cp_svr

---

Contact Cisco TAC and ask them to see which one this is: CSCvg26073, CSCvf52877, CSCuw66262 or CSCuv52648

Recommendation:  Do not use 16.12.X.   Use the latest 16.6.X or 16.9.X but avoid 16.12.X.