Hello.
i use 3850-12XS-S switch for work. and it has problem for a week ago.
this switch running for BGP routing.
there are so many logs about memory like on below. and it seems occurred on a week ago.
Apr 28 15:06:49.805: %PLATFORM-4-ELEMENT_WARNING:Switch 1 R0/0: smand: 1/RP/0: Used Memory value 91% exceeds warning level 90%
and my switch's memory usage is
Processor Pool Total: 886455648 Used: 329807384 Free: 556648264
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832
my switch's uptime has over 1 year, 33 weeks, 5 day running time and it happened suddenly.
so i found https://www.cisco.com/c/en/us/support/docs/field-notices/701/fn70110.html , https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd45973/?rfs=iqvred
this field note and bug note said it will be fix after os upgrade over 16.3.3 and my switch os version is 16.3.5b now. but it didn't fixed yet.
it had happened when my switch's version was 16.3.3 but it happened again a few years later.
If anybody know how to fix a.k.a memory leak bug like this. please give me solutions.
Thank you.
and my switch R0 's memory usage list on below
show platform software process list switch active R0 sort memory
$rm software process list switch active R0 sort memory
Name Pid PPid Group Id Status Priority Size
------------------------------------------------------------------------------
wcm 3833 1426 3833 S 20 3784200192
fed main event 18173 16715 18173 S 20 2540150784
linux_iosd-imag 31144 29676 31144 S 20 1844625408
tams_proc 15426 14377 15426 S 20 1798717440
sif_mgr 15324 13805 15324 S 20 1399877632
platform_mgr 14136 13095 14136 S 20 1118101504
cli_agent 28518 27271 28518 S 20 872423424
smand 662 31808 662 R 20 641703936
fman_fp_image 27052 25933 27052 S 20 633511936
dbm 29199 28257 29199 S 20 596742144
smd 1354 32104 1354 S 20 595251200
nginx 11317 10359 10359 S 20 589058048
nginx 10359 9588 10359 S 20 584011776
fman_rp 29517 28761 29517 S 20 540725248
tms 1836 968 1836 S 20 520183808
bt_logger 9798 8461 9798 S 20 514048000
cmm 28732 27544 28732 S 20 513323008
lman 12176 10926 12176 S 20 510853120
nif_mgr 13355 12399 13355 S 20 510398464
hman 10994 9599 10994 R 20 509689856
psd 31663 30739 31663 S 20 506712064
stack_mgr 16330 14849 16330 S 20 506494976
plogd 31666 30596 31666 S 20 503742464
epc_ws_liaison 17657 16258 17657 S 20 503554048
swift_sampled 16832 15257 16832 S 20 501649408
tamd_proc 15238 14108 15238 S 20 497856512
keyman 11851 10509 11851 S 20 494370816
torchd 18313 16933 18313 S 20 492240896
repm 571 31245 571 S 20 259956736
libvirtd 1110 1061 1061 S 20 173350912
rotee 32183 1 32177 S 20 13553664
rotee 29920 1 29916 S 20 13553664
rotee 10745 1 10743 S 20 13553664
rotee 9891 1 9890 S 20 13553664
rotee 32555 1 32554 S 20 13549568
rotee 31916 1 31915 S 20 13549568
rotee 29136 1 29131 S 20 13549568
rotee 28714 1 28712 S 20 13549568
rotee 28103 1 28101 S 20 13549568
rotee 27705 1 27700 S 20 13549568
rotee 26431 1 26430 S 20 13549568
rotee 18054 1 18050 S 20 13549568
rotee 17500 1 17498 S 20 13549568
rotee 16507 1 16500 S 20 13549568
rotee 15410 1 15407 S 20 13549568
rotee 15023 1 15017 S 20 13549568
rotee 14300 1 14298 S 20 13549568
rotee 13512 1 13509 S 20 13549568
rotee 13031 1 13026 S 20 13549568
rotee 12281 1 12276 S 20 13549568
rotee 11819 1 11816 S 20 13549568
rotee 11460 1 11459 S 20 13549568
rotee 10679 1 10676 S 20 13549568
rotee 10091 1 10087 S 20 13549568
rotee 9412 1 9411 S 20 13549568
rotee 7804 1 7803 S 20 13549568
rotee 7662 1 7661 S 20 13549568
rotee 1788 1 1786 S 20 13549568
rotee 1272 1 1271 S 20 13549568
rotee 14902 1 14901 S 20 13430784
rotee 14552 1 14551 S 20 13430784
rotee 10251 1 10243 S 20 13430784
rotee 31090 1 31089 S 20 13426688
rotee 31005 1 31004 S 20 13426688
rotee 17737 1 17732 S 20 13426688
rotee 16939 1 16935 S 20 13426688
rotee 15902 1 15894 S 20 13426688
rotee 14711 1 14708 S 20 13426688
rotee 14263 1 14258 S 20 13426688
rotee 13182 1 13180 S 20 13426688
rotee 12689 1 12686 S 20 13426688
rotee 11076 1 11073 S 20 13426688
rotee 9010 1 9008 S 20 13426688
rotee 8655 1 8654 S 20 13426688
rotee 8343 1 8341 S 20 13426688
rotee 7539 1 7538 S 20 13426688
rotee 1992 1 1991 S 20 13426688
rotee 1988 1 1987 S 20 13426688
rotee 1908 1 1907 S 20 13426688
rotee 1437 1 1436 S 20 13426688
auto_upgrade_se 9470 8205 9470 S 20 13365248
systemd-journal 105 1 105 S 20 12734464
ncd.sh 12346 11286 12346 S 20 9793536
issu_stack.sh 11345 10060 11345 S 20 9662464
auto_upgrade_cl 9009 8040 9009 S 20 9539584
issu_stack.sh 23030 11345 23030 S 20 9527296
periodic.sh 13748 12712 13748 S 20 8540160
systemd 1 0 1 S 20 8519680
pvp.sh 7682 1 7681 S 20 6877184
btrace_rotate.s 17119 15841 17119 S 20 6631424
btrace_rotate.s 10032 8772 10032 S 20 6594560
psvp.sh 7513 1 7513 S 20 6565888
reflector.sh 1047 1 1047 S 20 6549504
droputil.sh 1046 1 1046 S 20 6549504
pvp.sh 14320 1 14316 S 20 6500352
pman.sh 28257 7682 7681 S 14 6139904
pman.sh 27544 7682 7681 S 14 6139904
pman.sh 14108 7682 7681 S 14 6139904
pman.sh 13095 7682 7681 S 14 6139904
pman.sh 12399 7682 7681 S 14 6139904
pman.sh 9588 7682 7681 S 14 6139904
pman.sh 8461 7682 7681 S 14 6139904
pman.sh 8205 7682 7681 S 14 6139904
pman.sh 968 7682 7681 S 14 6139904
pman.sh 10926 7682 7681 S 14 6135808
pman.sh 9599 7682 7681 S 14 6135808
pman.sh 31245 7682 7681 S 14 6025216
pman.sh 1426 7682 7681 S 14 6025216
pman.sh 31808 7682 7681 S 14 6021120
pman.sh 30596 7682 7681 S 14 6021120
pman.sh 29676 7682 7681 S 14 6021120
pman.sh 16933 7682 7681 S 14 6021120
pman.sh 14377 7682 7681 S 14 6021120
pman.sh 10509 7682 7681 S 14 6021120
pman.sh 8040 7682 7681 S 14 6021120
pman.sh 32104 7682 7681 S 14 6017024
pman.sh 28761 7682 7681 S 14 6017024
pman.sh 27271 7682 7681 S 14 6017024
pman.sh 15257 7682 7681 S 14 6017024
pman.sh 14849 7682 7681 S 14 6017024
pman.sh 13805 7682 7681 S 14 6017024
pman.sh 12712 7682 7681 S 14 6017024
pman.sh 11286 7682 7681 S 14 6017024
pman.sh 10060 7682 7681 S 14 6017024
pman.sh 30739 7682 7681 S 14 6012928
pman.sh 8772 7682 7681 S 14 6012928
pman.sh 14133 7682 7681 S 14 6008832
pman.sh 25933 14320 14316 S 14 5885952
pman.sh 16715 14320 14316 S 14 5885952
pman.sh 16258 14320 14316 S 14 5885952
pman.sh 15841 14320 14316 S 14 5885952
iptbl.sh 1476 1 1476 S 20 5419008
systemd-udevd 114 1 114 S 20 5369856
sort_files_by_i 15522 14133 15522 S 20 4796416
rollback_timer. 7263 1 7263 S 20 4739072
oom.sh 1051 1 1051 S 20 4739072
boothelper_evt. 1120 1 1057 S 20 4333568
stack_sntp.sh 26366 12346 26366 S 20 4268032
mcp_chvrf.sh 24417 1 24417 S 20 4268032
mcp_chvrf.sh 7259 1 7259 S 20 4268032
libvirtd.sh 1061 1 1061 S 20 4268032
sntp 26377 26366 26366 S 20 4038656
rpc.mountd 1215 1 1215 S 20 3407872
rpcbind 1074 1 1074 S 20 3207168
xinetd 24423 24417 24417 S 20 3141632
xinetd 24410 1 24410 S 20 3141632
xinetd 7269 7259 7259 S 20 3141632
xinetd 1055 1 1055 S 20 3141632
rpc.statd 1106 1 1106 S 20 2789376
inotifywait 7164 9009 7164 S 20 2629632
inotifywait 23480 12346 23480 S 20 2625536
inotifywait 23148 1047 23148 S 20 2625536
inotifywait 23039 11345 23039 S 20 2625536
inotifywait 15436 14320 14316 S 20 2625536
inotifywait 7944 7682 7681 S 20 2625536
inotifywait 7678 7513 7513 S 20 2625536
inotifywait 2073 1046 2073 S 20 2625536
inotifywait 22425 10032 22425 S 20 2564096
inotifywait 23028 11345 23028 S 20 2560000
inotifywait 21894 17119 21894 S 20 2560000
inotifywait 7583 7263 7583 S 20 2560000
inotifywait 1969 1476 1969 S 20 2560000
inotifywait 1130 1120 1057 S 20 2560000
sleep 22719 15522 15522 S 20 2392064
sleep 22672 13748 13748 S 20 2392064
sleep 1452 1051 1452 S 20 2392064
kworker/u12:2 24287 2 0 S 20 0
kworker/5:1 17578 2 0 S 20 0
lsmpi-rx 2300 2 0 S 15 0
lsmpi-xmit 2299 2 0 S 15 0
lsmpi-refill 2298 2 0 S 15 0
kworker/u12:1 1910 2 0 S 20 0
nfsd 1141 2 0 S 20 0
nfsd 1140 2 0 S 20 0
nfsd 1139 2 0 S 20 0
nfsd 1138 2 0 S 20 0
nfsd 1137 2 0 S 20 0
nfsd 1136 2 0 S 20 0
nfsd 1135 2 0 S 20 0
nfsd 1134 2 0 S 20 0
lockd 1133 2 0 S 20 0
loop10 897 2 0 S 0 0
loop9 773 2 0 S 0 0
loop8 753 2 0 S 0 0
loop7 703 2 0 S 0 0
loop6 680 2 0 S 0 0
loop5 660 2 0 S 0 0
loop4 619 2 0 S 0 0
loop3 599 2 0 S 0 0
loop2 534 2 0 S 0 0
loop1 514 2 0 S 0 0
loop0 477 2 0 S 0 0
kworker/1:2 228 2 0 S 20 0
kworker/3:1H 227 2 0 S 0 0
kworker/2:1H 226 2 0 S 0 0
kworker/0:1H 225 2 0 S 0 0
kworker/1:1H 224 2 0 S 0 0
kworker/4:1H 223 2 0 S 0 0
kworker/5:1H 217 2 0 S 0 0
kworker/4:2 200 2 0 S 20 0
kworker/3:2 199 2 0 S 20 0
kworker/0:2 195 2 0 S 20 0
usb-storage 193 2 0 D 20 0
scsi_eh_1 192 2 0 S 20 0
kworker/3:1 91 2 0 S 20 0
deferwq 90 2 0 S 0 0
irq/97-octeon-h 79 2 0 S 4294967245 0
octeon-ethernet 78 2 0 S 0 0
scsi_eh_0 73 2 0 S 20 0
nfsiod 66 2 0 S 0 0
fsnotify_mark 65 2 0 S 20 0
ksmd 64 2 0 S 25 0
kswapd0 63 2 0 S 20 0
kworker/0:1 62 2 0 S 20 0
kworker/2:1 61 2 0 S 20 0
rpciod 59 2 0 S 0 0
edac-poller 58 2 0 S 0 0
md 57 2 0 S 0 0
khubd 56 2 0 S 20 0
ata_sff 55 2 0 S 0 0
kblockd 54 2 0 S 0 0
crypto 53 2 0 S 0 0
bioset 52 2 0 S 0 0
writeback 51 2 0 S 0 0
irq/99-octeon-h 50 2 0 S 4294967245 0
irq/98-octeon-h 49 2 0 S 4294967245 0
irq/96-octeon-h 48 2 0 S 4294967245 0
irq/127-octeon- 47 2 0 S 4294967245 0
irq/95-octeon-h 46 2 0 S 4294967245 0
irq/93-octeon-h 45 2 0 S 4294967245 0
irq/92-octeon-h 44 2 0 S 4294967245 0
irq/91-octeon-h 43 2 0 S 4294967245 0
irq/94-octeon-h 42 2 0 S 4294967245 0
irq/90-octeon-h 41 2 0 S 4294967245 0
irq/89-octeon-h 40 2 0 S 4294967245 0
irq/60-octeon-h 39 2 0 S 4294967245 0
netns 38 2 0 S 0 0
kdevtmpfs 37 2 0 S 20 0
khelper 36 2 0 S 0 0
kworker/5:0H 35 2 0 S 0 0
kworker/5:0 34 2 0 S 20 0
ksoftirqd/5 33 2 0 S 20 0
migration/5 32 2 0 S 4294967196 0
rcuc/5 31 2 0 S 20 0
kworker/4:0H 30 2 0 S 0 0
kworker/4:0 29 2 0 S 20 0
ksoftirqd/4 28 2 0 S 20 0
migration/4 27 2 0 S 4294967196 0
rcuc/4 26 2 0 S 20 0
kworker/3:0H 25 2 0 S 0 0
ksoftirqd/3 23 2 0 S 20 0
migration/3 22 2 0 S 4294967196 0
rcuc/3 21 2 0 S 20 0
kworker/2:0H 20 2 0 S 0 0
kworker/2:0 19 2 0 S 20 0
ksoftirqd/2 18 2 0 S 20 0
migration/2 17 2 0 S 4294967196 0
rcuc/2 16 2 0 S 20 0
kworker/1:0H 15 2 0 S 0 0
kworker/1:0 14 2 0 S 20 0
ksoftirqd/1 13 2 0 S 20 0
migration/1 12 2 0 S 4294967196 0
rcuc/1 11 2 0 S 20 0
rcuc/0 10 2 0 S 20 0
rcu_sched 9 2 0 S 20 0
rcu_bh 8 2 0 S 20 0
migration/0 7 2 0 S 4294967196 0
kworker/0:0H 5 2 0 S 0 0
ksoftirqd/0 3 2 0 S 20 0
kthreadd 2 0 0 S 20 0
