CSCvn30230 - Catalyst 3k/9k: Slow memory leak in linux_iosd-imag

[noriwocchi]

Does this bug also affect Cisco Catalyst 3650 Series Switches?

[Leo Laohoo]

---

@noriwocchi wrote:

Does this bug also affect Cisco Catalyst 3650 Series Switches?

---

Yes, of course.

[noriwocchi]

Are you planning to add "Cisco Catalyst 3650 Series Switches" to Products in Details of https://bst.cisco.com/bugsearch/bug/CSCvn30230?

[Leo Laohoo · Accepted solution]

I do not work for Cisco.  

Raise a TAC case and get the TAC agent to include 3650. 

Good luck!

[fgabo87]

Hi All!

I did not want to open a new topic, but I am also suspecting a possible slow memory leak issue in relation to linux_iosd-imag, however, condition 2 is not valid in my case., as RSS value did not increase in the last 3 weeks. (Symptom 2: In the output of "show proc mem platform detailed name iosd smaps" go to the last heap section [IOSd heap 3] and check if the RSS counter increases over time.). Our switch is a 9407R + SUP-1XL-Y with IOS-X 17.9.4 . 

I have been monitoring the case for almost 3 weeks, and the linux_iosd seems to be fishy for me, however, I have not found any exact bug for this particular version of sw and platform.  Maybe this one? https://bst.cisco.com/bugsearch/bug/CSCwd07516 

27.November:

Switch_XYZ#show proc memory platform sorted
System memory: 15993664K total, 7284452K used, 8709212K free,
Lowest: 8634300K
Pid Text Data Stack Dynamic RSS Name
----------------------------------------------------------------------
5916 283193 2232956 136 464 2232956 linux_iosd-imag
26115 209 743880 136 138284 743880 fed main event
2139 1183 269284 1312 53136 269284 iomd
31113 1183 267828 1308 53052 267828 iomd
25844 1183 266680 1312 53076 266680 iomd
22132 473 262904 136 5268 262904 dbm
30373 3389 257624 136 1308 257624 confd.smp
27043 91 242540 136 8516 242540 pubd
27695 346 211928 136 368 211928 ndbmand
20792 1183 210156 1312 3580 210156 iomd

08. November:

Switch_XYZ#show proc memory platform sorted
System memory: 15993664K total, 6930716K used, 9062948K free,
Lowest: 8994668K
Pid Text Data Stack Dynamic RSS Name
----------------------------------------------------------------------
5916 283193 1929504 136 464 1929504 linux_iosd-imag
26115 209 740864 136 138284 740864 fed main event
2139 1183 267868 1312 53136 267868 iomd
31113 1183 266244 1308 53052 266244 iomd
25844 1183 265280 1312 53076 265280 iomd
22132 473 262576 136 5268 262576 dbm
30373 3389 257956 136 1308 257956 confd.smp
27043 91 241192 136 8516 241192 pubd
27695 346 211576 136 368 211576 ndbmand
20792 1183 208680 1312 2792 208680 iomd

So, it seems it eats up ~15Mb of memory a day. 

Did anyone else experienced the same/similar issue?

Thanks a million for your thoughts on this issue.

Regards,

Gábor

[Leo Laohoo]

---

@fgabo87 wrote:
 as RSS value did not increase in the last 3 weeks.

---

Wut? 

The RSS value has increased, from 1929504 (08 November 2023), to 2232956 (27 November 2023).  

Post the complete output to the command "sh platform resources" and post the "first page" of the command "sh process memory platform sort location r0".

[fgabo87]

Hi @Leo Laohoo ,

Than kyou for your thoughts on this issue. With the not increasing RSS value I referred to the "last heap section [IOSd heap 3]" in the command "show proc mem platform detailed name iosd smaps", which seems to be constant in the last 3 weeks:

5650cba16000-5650cc245000 rw-p 00000000 00:00 0 [heap]
Size: 8380 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 8004 kB

Pss: 8004 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 8004 kB
Referenced: 8004 kB
Anonymous: 8004 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
THPeligible:

-------------------------

#sh platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 4.14% 100% 90% 95% H
DRAM 7140MB(45%) 15618MB 90% 95% H
TMPFS 378MB(2%) 15618MB 40% 50% H

------------------------------------------------------------------------

#sh process memory platform sort location r0
System memory: 15993664K total, 7319236K used, 8674428K free,
Lowest: 8634300K
Pid Text Data Stack Dynamic RSS Name
----------------------------------------------------------------------
5916 283193 2232744 136 464 2232744 linux_iosd-imag
26115 209 743448 136 138284 743448 fed main event
2139 1183 268872 1312 53136 268872 iomd
31113 1183 268044 1308 53052 268044 iomd
25844 1183 266856 1312 53076 266856 iomd
22132 473 262964 136 5268 262964 dbm
30373 3389 260792 136 1308 260792 confd.smp
27043 91 242652 136 8516 242652 pubd
27695 346 211924 136 368 211924 ndbmand
20792 1183 210464 1312 3580 210464 iomd
17718 180 208088 136 18404 208088 sessmgrd
28644 463 188344 168 5188 188344 cman_fp
18062 823 178904 136 32376 178904 smand
21382 7434 159876 136 3496 159876 fman_rp
22414 62 146472 136 224 146472 cli_agent
27500 8381 140480 136 4752 140480 fman_fp_image
18621 421 110728 136 2628 110728 repm
9162 213 94108 136 3536 94108 btman
28377 234 69180 136 296 69180 dmiauthd
6485 2026 58480 136 7872 58480 cmand
2521 8 51860 136 9392 51860 python3
16478 938 43652 136 4836 43652 cmcc
3815 938 42696 136 4032 42696 cmcc
8365 938 42572 136 4212 42572 cmcc
20798 1673 39596 136 1892 39596 install_mgr
17416 71 38960 136 1984 38960 psd
11669 938 37776 136 2224 37776 cmcc
19231 938 37752 136 2444 37752 cmcc
17730 938 37244 136 2444 37244 cmcc
14055 938 36972 136 2052 36972 cmcc
29152 213 30672 136 1928 30672 btman
10787 213 30184 136 1936 30184 btman
13549 213 29708 136 1948 29708 btman
21050 494 29536 488 3820 29536 hman
7489 213 29084 136 1944 29084 btman
16039 213 28832 136 1816 28832 btman
18229 213 28612 136 1936 28612 btman
18941 213 28436 136 1816 28436 btman
19996 213 28192 136 1816 28192 btman
6153 472 27648 132 1800 27648 libvirtd
12655 49 25720 136 1384 25720 service_mgr
18344 49 25616 136 1744 25616 service_mgr
1 986 25328 1032 16524 25328 systemd
1974 49 25304 136 1384 25304 service_mgr
4818 49 25224 136 1384 25224 service_mgr
8392 155 25084 136 1884 25084 lman
26413 49 25076 136 1488 25076 service_mgr
7705 49 24344 136 1392 24344 service_mgr
16832 49 23724 136 1364 23724 service_mgr
14199 49 23556 136 1364 23556 service_mgr
9506 49 23268 136 1364 23268 service_mgr
16693 494 23160 136 1912 23160 hman
12846 494 23120 136 1912 23120 hman
5494 484 23064 136 280 23064 stack_mgr
18636 494 22972 136 1912 22972 hman
15474 494 22948 136 1912 22948 hman
3132 494 22908 136 1912 22908 hman
7205 494 22720 136 1912 22720 hman
28082 494 22708 136 1916 22708 hman
10030 494 22616 136 1912 22616 hman
8060 118 20772 136 1848 20772 keyman
19296 230 20164 136 1764 20164 tms
19684 906 19356 136 16684 19356 ncd.sh
16116 906 18352 132 15052 18352 auto_upgrade_cl
8696 266 18304 136 412 18304 nif_mgr
22859 319 17756 136 384 17756 ncsshd
29545 319 17452 136 384 17452 ncsshd
2100 84 16632 136 1136 16632 epc_ws_liaison
9000 81 16168 136 1644 16168 plogd
29083 959 15524 136 1088 15524 ncsshd_bp
7059 59 13884 136 1132 13884 tamd_proc
23783 906 13752 136 10964 13752 periodic.sh
7377 64 13276 136 1572 13276 tams_proc
6109 85 12120 132 396 12120 virtlogd
7672 56 11840 136 840 11840 tam_svcs_esg_cf
32758 906 11796 132 8936 11796 pvp.sh
31630 906 11768 132 8936 11768 pvp.sh
1608 906 11740 132 8932 11740 pvp.sh
4004 906 11716 132 8932 11716 pvp.sh
31664 1663 11208 136 832 11208 nginx
4605 906 11200 132 8328 11200 psvp.sh
4921 906 10464 132 7372 10464 pvp.sh
16041 906 10240 132 6856 10240 kernel_ftrace.s
15639 906 10240 132 7120 10240 chasync.sh
25226 906 9988 132 6788 9988 pvp.sh
26723 59 9940 136 396 9940 pttcd
6087 906 9792 132 6392 9792 droputil.sh
2671 906 9640 132 6640 9640 pvp.sh
--More--

---------------------------------------------- I also tried to do some debugs ------------------------------

show memory debug leaks chunks
Tracekey : 1#4e390fd4a4f98a4332a419097708d807

Adding blocks for GD...

lsmpi_io memory

Address Size PID Alloc-Proc Name Alloc_pc

Chunk Elements:

Address Size Parent Name Alloc_pc

reserve Processor memory

Address Size PID Alloc-Proc Name Alloc_pc

Chunk Elements:

Address Size Parent Name Alloc_pc

Processor memory

Address Size PID Alloc-Proc Name Alloc_pc
7FC2833E7CD0 160 577 SNMP ENGINE SNMP Server Address :5650B368D000+7D1B809
7FC2836E88B0 208 577 SNMP ENGINE SNMP Server Info :5650B368D000+7D1B7EB
7FC289ACB760 152 577 SNMP ENGINE SNMP Server Info :5650B368D000+7D1B7EB
7FC289AD3C00 152 577 SNMP ENGINE SNMP Server Address :5650B368D000+7D1B809
7FC28A529EF8 160 577 SNMP ENGINE SNMP Server Address :5650B368D000+7D1B809
7FC28B94C0C8 216 577 SNMP ENGINE SNMP Server Info :5650B368D000+7D1B7EB

Chunk Elements:

Address Size Parent Name Alloc_pc
7FC27846FC00 28 7FC28387C978 (MallocLite) :5650B368D000+6733D86
7FC278474890 28 7FC28387C978 (MallocLite) :5650B368D000+6733D86
7FC278477258 28 7FC28387C978 (MallocLite) :5650B368D000+6733D86
7FC27BB7A518 58 7FC27BB7BD40 (FMANRP msg chun) NA
7FC27BB7A558 58 7FC27BB7BD40 (FMANRP msg chun) NA
7FC27BB7A598 58 7FC27BB7BD40 (FMANRP msg chun) NA
7FC27BB7A5D8 58 7FC27BB7BD40 (FMANRP msg chun) NA
7FC27BB7A618 58 7FC27BB7BD40 (FMANRP msg chun) NA
7FC27BB7A658 58 7FC27BB7BD40 (FMANRP msg chun) NA
7FC27BB7A698 58 7FC27BB7BD40 (FMANRP msg chun) NA
7FC27BB7A718 58 7FC27BB7BD40 (FMANRP msg chun) NA

KR,

Gábor

[Leo Laohoo]

---

@fgabo87 wrote:

22132 473 262964 136 5268 262964 dbm
 27043 91 242652 136 8516 242652 pubd

---

These two are unusually high. 

Is there a DNAC appliance around?

[fgabo87]

---

@Leo Laohoo wrote:

---

@fgabo87 wrote:

22132 473 262964 136 5268 262964 dbm
 27043 91 242652 136 8516 242652 pubd

---

These two are unusually high. 

Is there a DNAC appliance around?

---

Yes, this SW is also part of the fabric. 

[Leo Laohoo]

Turn off telemetry (DNAC) and see if the memory utilization eases.

[fgabo87]

---

@Leo Laohoo wrote:

Turn off telemetry (DNAC) and see if the memory utilization eases.

---

Are you suspecting issues with pubd or dbm ? How much is the avarage memory consuption for the dbm and pubd ? Or, may I ask you , why you are suspecting a problem regarding Telemetry ?

BR,

[Leo Laohoo]

DNAC will cause a lot of CPU hogs and memory leaks.  The one of many processes, pubd, is usually attributed to DNAC.  Bug IDs will never mention anything about DNAC, however, the name will be substituted with the word "telemetry".  

[fgabo87]

Hi Leo !

Yes, that is actually correct, DNAC has had many surprises so far, but this time I am still not convinced ...   

I have checked the processes this morning again, and it seems pubd and dbm eat a bit less memory, however linux_iosd eats more (+3.2Mbyte). Fed main event eats +1M this morning.  I am still on the fence... but we will see how it works out....

#sh process memory platform sort location r0
System memory: 15993664K total, 7322600K used, 8671064K free,
Lowest: 8630836K
Pid Text Data Stack Dynamic RSS Name
----------------------------------------------------------------------
5916 283193 2236028 136 464 2236028 linux_iosd-imag
26115 209 744568 136 138284 744568 fed main event
2139 1183 268440 1312 53136 268440 iomd
31113 1183 267884 1308 53052 267884 iomd
25844 1183 267284 1312 53076 267284 iomd
22132 473 262904 136 5268 262904 dbm
30373 3389 256528 136 1308 256528 confd.smp
27043 91 242528 136 8516 242528 pubd
27695 346 211952 136 368 211952 ndbmand
20792 1183 210232 1312 3580 210232 iomd
17718 180 207696 136 18404 207696 sessmgrd
28644 463 188344 168 5188 188344 cman_fp
18062 823 178044 136 32124 178044 smand
21382 7434 160064 136 3496 160064 fman_rp
22414 62 146500 136 224 146500 cli_agent
27500 8381 140764 136 4752 140764 fman_fp_image
18621 421 110728 136 2628 110728 repm
9162 213 82340 136 3536 82340 btman
28377 234 69180 136 296 69180 dmiauthd
6485 2026 58840 136 7872 58840 cmand
2521 8 51860 136 9392 51860 python3
16478 938 43652 136 4836 43652 cmcc
3815 938 42900 136 4032 42900 cmcc
8365 938 42776 136 4212 42776 cmcc
20798 1673 39596 136 1892 39596 install_mgr
17416 71 38960 136 1984 38960 psd
11669 938 37776 136 2224 37776 cmcc
19231 938 37752 136 2444 37752 cmcc
17730 938 37244 136 2444 37244 cmcc
14055 938 36972 136 2052 36972 cmcc
7489 213 31192 136 1944 31192 btman
29152 213 31064 136 1928 31064 btman
10787 213 30844 136 1936 30844 btman
18229 213 30256 136 1936 30256 btman
21050 494 30088 488 3820 30088 hman
13549 213 29488 136 1948 29488 btman
16039 213 28844 136 1816 28844 btman
18941 213 28448 136 1816 28448 btman
19996 213 28204 136 1816 28204 btman
6153 472 27648 132 1800 27648 libvirtd