%EVENTLIB-3-CPUHOG: Switch 1 R0/0: smd: undefined: 568ms, Traceback=1

[Paajf]

Hello community,

I have noticed in the logs these errors.

Apr 19 2022 15:27:09.036 MESZ: %IOSXE_INFRA-6-PROCPATH_CLIENT_HOG: IOS shim client 'ngwc fed bipc' has taken 2301 msec (runtime: 569 msec) to process a 'unknown' message
Apr 19 2022 15:27:18.929 MESZ: %PLATFORM_INFRA-5-IOS_INTR_OVER_LIMIT: IOS thread disabled interrupt for 18 msec

Does anyone know what these errors are about?

Thanks.

[Leo Laohoo]

Please provide the complete output to the following commands:

• sh version
• sh platform resources
• sh platform software status con brief

[Paajf]

Hi Leo,

Thanks for the feedback and the outputs follow in order.

Cisco IOS XE Software, Version 16.06.06
Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 11-Apr-19 02:24 by mcpre

**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 4.30% 100% 90% 95% H
DRAM 1139MB(29%) 3889MB 90% 95% H

Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.11 0.15 0.20
2-RP0 Healthy 0.66 0.51 0.45
3-RP0 Healthy 0.21 0.23 0.20
4-RP0 Healthy 0.30 0.26 0.29

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3983088 1167280 (29%) 2815808 (71%) 1537848 (39%)
2-RP0 Warning 3983088 3657112 (92%) 325976 ( 8%) 4529912 (114%)
3-RP0 Healthy 3983088 2445592 (61%) 1537496 (39%) 2800864 (70%)
4-RP0 Healthy 3983088 3431428 (86%) 551660 (14%) 4263132 (107%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 3.40 3.40 0.00 93.18 0.00 0.00 0.00
1 3.60 3.00 0.00 93.40 0.00 0.00 0.00
2 2.70 3.40 0.00 93.90 0.00 0.00 0.00
3 5.50 3.20 0.00 91.30 0.00 0.00 0.00
2-RP0 0 6.00 3.00 0.00 90.80 0.00 0.20 0.00
1 6.70 2.70 0.00 90.50 0.00 0.10 0.00
2 6.50 2.60 0.00 90.90 0.00 0.00 0.00
3 6.80 2.20 0.00 91.00 0.00 0.00 0.00
3-RP0 0 1.99 1.19 0.00 96.80 0.00 0.00 0.00
1 0.70 0.30 0.00 98.99 0.00 0.00 0.00
2 6.10 1.30 0.00 92.60 0.00 0.00 0.00
3 1.49 0.19 0.00 98.30 0.00 0.00 0.00
4-RP0 0 4.30 2.30 0.00 93.30 0.00 0.10 0.00
1 5.09 0.89 0.00 94.00 0.00 0.00 0.00
2 3.59 0.99 0.00 95.40 0.00 0.00 0.00
3 3.49 1.29 0.00 95.10 0.00 0.09 0.00

[Leo Laohoo]

---

@Paajf wrote:

2-RP0 Warning 3983088 3657112 (92%) 325976 ( 8%) 4529912 (114%)
4-RP0 Healthy 3983088 3431428 (86%) 551660 (14%) 4263132 (107%)

---

Please post the "first page" output to the command "sh platform software process memory r0 all".

I want to see the complete output to the command "sh version".  

NOTE:  Remove the hostname but leave the rest alone.  

[Paajf]

Hi Leo,

LAB-F001#sh ver

Cisco IOS XE Software, Version 16.06.06

Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.6, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Thu 11-Apr-19 02:24 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.

All rights reserved.  Certain components of Cisco IOS-XE software are

licensed under the GNU General Public License ("GPL") Version 2.0.  The

software code licensed under GPL Version 2.0 is free software that comes

with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such

GPL code under the terms of GPL Version 2.0.  For more details, see the

documentation or "License Notice" file accompanying the IOS-XE software,

or the applicable URL provided on the flyer accompanying the IOS-XE

software.

ROM: IOS-XE ROMMON

BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE SOFTWARE (P)

LAB-F001 uptime is 1 year, 47 weeks, 5 days, 4 hours, 27 minutes

Uptime for this control processor is 1 year, 47 weeks, 5 days, 4 hours, 36 minutes

System returned to ROM by SSO Switchover

System restarted at 19:20:30 MESZ Thu May 28 2020

System image file is "flash:packages.conf"

Last reload reason: Power Failure or Unknown

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Technology Package License Information:

-----------------------------------------------------------------

Technology-package                   Technology-package

Current             Type             Next reboot

------------------------------------------------------------------

lanbasek9           Permanent        lanbasek9

cisco WS-C3850-48P (MIPS) processor (revision AB0) with 852917K/6147K bytes of memory.

Processor board ID FCW2046D1B3

2 Virtual Ethernet interfaces

208 Gigabit Ethernet interfaces

16 Ten Gigabit Ethernet interfaces

2048K bytes of non-volatile configuration memory.

4194304K bytes of physical memory.

257008K bytes of Crash Files at crashinfo:.

257008K bytes of Crash Files at crashinfo-3:.

257008K bytes of Crash Files at crashinfo-4:.

1550272K bytes of Flash at flash:.

1550272K bytes of Flash at flash-3:.

1550272K bytes of Flash at flash-4:.

0K bytes of WebUI ODM Files at webui:.

257008K bytes of Crash Files at crashinfo-1:.

1550272K bytes of Flash at flash-1:.

-------------------------------------------------------------

LAB-F001#sh platform software process memory switch active r0 all
Pid VIRT RSS PSS Heap Shared Private Name
------------------------------------------------------------------------------------
1 8000 4392 3465 1428 1016 3376 systemd
95 12428 5044 4235 132 860 4184 systemd-journal
105 5540 1972 992 328 1064 908 systemd-udevd
945 6044 3408 1899 1704 1532 1876 pman.sh
991 9600 1840 547 148 1300 540 rotee
1034 5792 3188 1679 1452 1532 1656 pman.sh
1101 5964 3272 1766 1628 1524 1748 brelay_console.
1746 8588 1844 551 148 1300 544 rotee
1916 8708 1840 547 148 1300 540 rotee
1965 9600 1840 547 148 1300 540 rotee
2050 6128 3424 1922 1792 1520 1904 brelay.sh
2919 825552 164388 53249 8744 120528 43860 fman_fp_image
3416 820412 230044 92120 268 161736 68308 smd
3486 4468 1548 246 132 1316 232 libvirtd.sh
3487 6796 4132 2609 2460 1544 2588 droputil.sh
3490 6808 4160 2626 2472 1556 2604 reflector.sh
3494 4864 2216 706 528 1528 688 oom.sh
3499 17168 1928 856 132 1180 748 virtlogd
3501 5144 3144 2775 132 372 2772 klogd
3507 4468 1752 340 132 1432 320 bexec.sh
3510 6176 1128 531 132 648 480 rpcbind
3520 212368 3952 2577 132 1504 2448 libvirtd
3526 4468 920 274 132 652 268 boothelper_evt.
3529 3372 984 227 132 812 172 xinetd
3533 2776 640 136 132 508 132 inotifywait
3557 3844 1484 823 132 716 768 rpc.statd
3628 5476 2876 2272 132 664 2212 rpc.mountd
3649 6104 3432 1930 1764 1520 1912 btelnet.sh
3916 8584 1832 547 148 1292 540 rotee
3939 2612 568 124 132 472 96 sleep
4044 8584 1848 559 148 1296 552 rotee
4051 8584 1848 559 148 1296 552 rotee
4086 2776 704 165 132 544 160 inotifywait
4100 5676 2948 1438 1340 1528 1420 iptbl.sh
4404 8584 1832 547 148 1292 540 rotee
4426 2776 684 149 132 540 144 inotifywait
4812 12660 2724 929 172 1832 892 btelnet
5468 3372 1064 261 132 868 196 xinetd
7339 6044 3412 1903 1704 1532 1880 pman.sh
7612 8588 1840 547 148 1300 540 rotee
7849 6044 3416 1907 1704 1532 1884 pman.sh
8102 699048 119048 24808 2396 98924 20124 plogd
8343 8708 1844 551 148 1300 544 rotee
9095 774812 165612 45844 5736 137004 28608 repm
10614 7812 5200 3690 3476 1528 3672 rollback_timer.
10616 3372 1100 230 132 928 172 xinetd
10626 3372 1132 247 132 948 184 xinetd
10976 4472 1712 311 132 1416 296 stack_sntp.sh
10985 4260 864 209 132 672 192 sntp
11324 8584 1836 547 148 1296 540 rotee
11339 2776 672 144 132 532 140 inotifywait
11453 6044 3408 1903 1704 1528 1880 pman.sh
11743 6044 3412 1903 1704 1532 1880 pman.sh
11882 6716 4104 2582 2380 1540 2564 psvp.sh
11901 8708 1872 549 148 1332 540 rotee
11992 8704 1840 547 148 1300 540 rotee
11999 2776 680 152 132 532 148 inotifywait
12004 7080 4444 2919 2744 1548 2896 pvp.sh
12121 8704 1852 559 148 1300 552 rotee
12193 2776 680 152 132 532 148 inotifywait
12251 8708 1872 549 148 1332 540 rotee
12283 6044 3416 1907 1704 1532 1884 pman.sh
12294 6040 3408 1894 1700 1532 1876 pman.sh
12471 6052 3416 1907 1712 1532 1884 pman.sh
12581 8588 1848 555 148 1300 548 rotee
12738 8708 1872 549 148 1332 540 rotee
12741 6052 3416 1907 1712 1532 1884 pman.sh
12744 2780 636 136 132 504 132 inotifywait
12747 971648 187872 55436 7688 152716 35156 cli_agent
12865 6052 3408 1899 1712 1532 1876 pman.sh
12879 701008 118172 23825 1224 99004 19168 cmm
12893 8588 1844 551 148 1300 544 rotee
13015 2780 632 132 132 504 128 inotifywait
13110 6044 3420 1902 1704 1536 1884 pman.sh
13135 9904 7272 5747 5564 1544 5728 auto_upgrade_cl
13206 8708 1872 549 148 1332 540 rotee
13230 8588 1840 547 148 1300 540 rotee
13271 2620 592 137 132 488 104 sleep
13312 911680 254512 94085 23884 200872 53640 dbm
13317 2620 592 157 132 448 144 sleep
13578 731888 165396 47444 3212 130496 34900 fman_rp
13604 16588 3128 911 148 2264 864 auto_upgrade_se
13666 8708 1844 551 148 1300 544 rotee
13697 6044 3416 1907 1704 1532 1884 pman.sh
13892 707588 121848 26351 1472 100480 21368 bt_logger
13897 6044 3408 1903 1704 1528 1880 pman.sh
13953 8708 1888 565 148 1332 556 rotee
14181 8588 1852 559 148 1300 552 rotee
14188 6740 4064 2541 2400 1544 2520 btrace_rotate.s
14381 8708 1840 547 148 1300 540 rotee
14434 2085764 697920 553526 120 174344 523576 linux_iosd-imag
14440 6048 3416 1898 1708 1536 1880 pman.sh
14761 6044 3408 1899 1704 1532 1876 pman.sh
14805 8708 1880 587 148 1300 580 rotee
14888 8708 1844 551 148 1300 544 rotee
15002 6048 3416 1907 1708 1532 1884 pman.sh
15058 1950904 1369920 1272748 1247440 102584 1267336 hman
15130 6044 3412 1903 1704 1532 1880 pman.sh
15226 8588 1840 547 148 1300 540 rotee
15258 8588 1880 557 148 1332 548 rotee
15461 10164 7504 2424 5824 6872 632 issu_stack.sh
15498 6048 3412 1894 1708 1536 1876 pman.sh
15610 702036 122276 24834 1152 102660 19616 psd
15649 8708 1840 547 148 1300 540 rotee
15746 6708 4048 2511 2372 1560 2488 pvp.sh
15762 6044 3412 1903 1704 1532 1880 pman.sh
15962 692836 115768 23222 432 97208 18560 keyman
16030 8708 1880 557 148 1332 548 rotee
16102 8708 1848 555 148 1300 548 rotee
16352 700948 120476 25262 1916 100200 20276 lman
16379 8704 1868 575 148 1300 568 rotee
16399 875092 184776 85601 29928 105052 79724 smand
16473 2776 680 152 132 532 148 inotifywait
16563 8708 1856 563 148 1300 556 rotee
16681 10168 7596 6056 5828 1564 6032 ncd.sh
16689 6044 3416 1907 1704 1532 1884 pman.sh
16736 5788 3192 1674 1452 1536 1656 pman.sh
16901 6048 3416 1907 1708 1532 1884 pman.sh
16949 8708 1872 549 148 1332 540 rotee
17054 5788 3188 1679 1452 1532 1656 pman.sh
17282 715284 130300 33631 9224 101840 28460 tms
17301 8708 1840 547 148 1300 540 rotee
17595 6048 3416 1898 1708 1536 1880 pman.sh
17612 8588 1840 547 148 1300 540 rotee
17813 8708 1848 555 148 1300 548 rotee
17900 6236 3568 2045 1896 1544 2024 btrace_rotate.s
18154 8708 1920 627 148 1300 620 rotee
18232 6044 3416 1907 1704 1532 1884 pman.sh
18390 8588 1840 547 148 1300 540 rotee
18596 701372 115032 23395 268 96212 18820 nif_mgr
18680 8708 1876 583 148 1300 576 rotee
18751 2563684 312312 201903 71580 120776 191536 fed main event
18812 8708 1856 563 148 1300 556 rotee
18966 8860 6196 4682 4520 1532 4664 periodic.sh
19186 6052 3416 1907 1712 1532 1884 pman.sh
19377 1305444 188848 83752 6664 113532 75316 platform_mgr
19454 6044 3416 1898 1704 1536 1880 pman.sh
19582 8708 1924 631 148 1300 624 rotee
19586 8588 1836 547 148 1296 540 rotee
20016 8588 1852 559 148 1300 552 rotee
20090 6052 3416 1907 1712 1532 1884 pman.sh
20417 2780 684 156 132 532 152 inotifywait
20452 1574576 181920 79237 3004 110392 71528 sif_mgr
20538 4984 2216 819 644 1412 804 sort_files_by_i
20603 8708 1844 551 148 1300 544 rotee
20998 701476 123280 26093 1788 102688 20592 stack_mgr
21991 6120 3420 1914 1784 1524 1896 in.telnetd.sh
21996 6120 3420 1914 1784 1524 1896 in.telnetd.sh
22323 14164 2808 821 172 2048 760 in.telnetd
22324 14164 2808 821 172 2048 760 in.telnetd
22325 6124 3420 1914 1788 1524 1896 brelay.sh
22326 6124 3420 1914 1788 1524 1896 brelay.sh
22698 4464 1752 336 132 1436 316 bexec.sh
22701 4464 1752 336 132 1436 316 bexec.sh
22706 6140 3356 1920 1804 1452 1904 bexecute.sh
22707 6140 3356 1920 1804 1452 1904 bexecute.sh
22898 5560 2892 1379 1224 1532 1360 iosd_console_at
22903 5560 2888 1379 1224 1528 1360 iosd_console_at
23136 8584 1872 549 148 1332 540 rotee
23140 8584 1872 549 148 1332 540 rotee
27312 6120 3420 1914 1784 1524 1896 in.telnetd.sh
27637 14164 2808 821 172 2048 760 in.telnetd
27638 6124 3420 1914 1788 1524 1896 brelay.sh
27668 2780 684 156 132 532 152 inotifywait
27745 2776 712 173 132 544 168 inotifywait
27808 4464 1752 336 132 1436 316 bexec.sh
27811 6140 3356 1920 1804 1452 1904 bexecute.sh
27854 2780 804 196 132 616 188 inotifywait
27857 10032 6420 2268 5692 5972 448 issu_stack.sh
27868 2780 668 149 132 524 144 inotifywait
27869 10032 6512 2266 5692 6072 440 issu_stack.sh
27878 2780 684 156 132 532 152 inotifywait
27928 5560 2892 1379 1224 1532 1360 iosd_console_at
28219 8584 1872 549 148 1332 540 rotee

[Leo Laohoo]

Can I see the "first page" output to the following commands: 

• sh proc memory platform sort location switch 2 r0
• sh proc memory platform sort location switch 4 r0

[Paajf]

Hi Leo,

#sh proc memory platform sort location switch 2 r0
System memory: 3983088K total, 3658636K used, 324452K free,
Lowest: 324452K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
18751 116 312308 136 71580 312308 2563684 fed main event
14434 177364 698276 0 120 698276 2085764 linux_iosd-imag
15058 519 1369980 136 1247440 1369980 1950904 hman
20452 308 181920 136 3004 181920 1574576 sif_mgr
19377 1011 188848 136 6664 188848 1305444 platform_mgr
12747 287 187872 136 7688 187872 971648 cli_agent
13312 170 254512 136 23884 254512 911680 dbm
16399 649 184784 136 29876 184784 875040 smand
2919 8823 164388 136 8744 164388 825552 fman_fp_image
3416 127 230452 0 268 230452 820412 smd
9095 430 165708 136 5736 165708 774812 repm
13578 8482 165396 136 3212 165396 731888 fman_rp
17282 251 130300 136 9224 130300 715284 tms
13892 38 121848 136 1472 121848 707588 bt_logger
15610 40 122276 136 1152 122276 702036 psd
20998 418 123280 136 1788 123280 701476 stack_mgr
18596 200 115032 136 268 115032 701372 nif_mgr
12879 114 118172 136 1224 118172 701008 cmm
16352 147 120476 136 1916 120476 700948 lman
8102 45 119048 136 2396 119048 699048 plogd
15962 73 116232 136 432 116232 692836 keyman
3520 604 3952 132 132 3952 212368 libvirtd
3499 753 1928 132 132 1928 17168 virtlogd
13604 7 3128 136 148 3128 16588 auto_upgrade_se
27637 56 2808 132 172 2808 14164 in.telnetd
22324 56 2808 132 172 2808 14164 in.telnetd
22323 56 2808 132 172 2808 14164 in.telnetd
4812 118 2724 136 172 2724 12660 btelnet
95 314 5044 132 132 5044 12428 systemd-journal
16681 974 7596 136 5828 7596 10168 ncd.sh
15461 974 7504 136 5824 7504 10164 issu_stack.sh
27869 974 6512 136 5692 6512 10032 issu_stack.sh
27857 974 6420 136 5692 6420 10032 issu_stack.sh
13135 974 7272 136 5564 7272 9904 auto_upgrade_cl
1965 7 1840 132 148 1840 9600 rotee
991 7 1840 132 148 1840 9600 rotee
18966 974 6196 136 4520 6196 8860 periodic.sh
20603 7 1844 136 148 1844 8708 rotee
19582 7 1924 136 148 1924 8708 rotee
18812 7 1856 136 148 1856 8708 rotee
18680 7 1880 136 148 1880 8708 rotee
18154 7 1920 136 148 1920 8708 rotee
17813 7 1848 136 148 1848 8708 rotee
17301 7 1840 136 148 1840 8708 rotee
16949 7 1872 136 148 1872 8708 rotee
16563 7 1856 136 148 1856 8708 rotee
16102 7 1848 136 148 1848 8708 rotee
16030 7 1880 136 148 1880 8708 rotee
15649 7 1840 136 148 1840 8708 rotee
14888 7 1844 136 148 1844 8708 rotee
14805 7 1884 136 148 1884 8708 rotee
14381 7 1840 136 148 1840 8708 rotee
13953 7 1888 136 148 1888 8708 rotee
13666 7 1844 136 148 1844 8708 rotee
13206 7 1872 136 148 1872 8708 rotee
12738 7 1872 136 148 1872 8708 rotee
12251 7 1872 136 148 1872 8708 rotee
11901 7 1872 136 148 1872 8708 rotee
8343 7 1844 136 148 1844 8708 rotee
1916 7 1840 136 148 1840 8708 rotee
16379 7 1868 132 148 1868 8704 rotee
12121 7 1852 132 148 1852 8704 rotee
11992 7 1840 132 148 1840 8704 rotee
20016 7 1852 136 148 1852 8588 rotee
19586 7 1836 136 148 1836 8588 rotee
18390 7 1840 136 148 1840 8588 rotee
17612 7 1840 136 148 1840 8588 rotee
15258 7 1880 136 148 1880 8588 rotee
15226 7 1840 136 148 1840 8588 rotee
14181 7 1852 136 148 1852 8588 rotee
13230 7 1840 136 148 1840 8588 rotee
12893 7 1844 136 148 1844 8588 rotee
12581 7 1848 136 148 1848 8588 rotee
7612 7 1840 136 148 1840 8588 rotee
1746 7 1844 136 148 1844 8588 rotee
28219 7 1872 132 148 1872 8584 rotee
23140 7 1872 132 148 1872 8584 rotee
23136 7 1872 132 148 1872 8584 rotee
11324 7 1836 132 148 1836 8584 rotee
4404 7 1832 132 148 1832 8584 rotee
4051 7 1848 132 148 1848 8584 rotee
4044 7 1848 132 148 1848 8584 rotee
3916 7 1832 132 148 1832 8584 rotee
1 1398 4392 132 1428 4392 8000 systemd
10614 974 5200 132 3476 5200 7812 rollback_timer.
12004 974 4444 132 2744 4444 7080 pvp.sh
3490 974 4160 132 2472 4160 6808 reflector.sh
3487 974 4132 132 2460 4132 6796 droputil.sh
14188 974 4064 136 2400 4064 6740 btrace_rotate.s
11882 974 4104 132 2380 4104 6716 psvp.sh
15746 974 4048 132 2372 4048 6708 pvp.sh
17900 974 3568 136 1896 3568 6236 btrace_rotate.s
3510 49 1128 132 132 1128 6176 rpcbind
27811 974 3356 132 1804 3356 6140 bexecute.sh
22707 974 3356 132 1804 3356 6140 bexecute.sh
22706 974 3356 132 1804 3356 6140 bexecute.sh
2050 974 3424 132 1792 3424 6128 brelay.sh
27638 974 3420 132 1788 3420 6124 brelay.sh
22326 974 3420 132 1788 3420 6124 brelay.sh
22325 974 3420 132 1788 3420 6124 brelay.sh
27312 974 3420 132 1784 3420 6120 in.telnetd.sh
21996 974 3420 132 1784 3420 6120 in.telnetd.sh
21991 974 3420 132 1784 3420 6120 in.telnetd.sh
3649 974 3432 136 1764 3432 6104 btelnet.sh
20090 974 3416 136 1712 3416 6052 pman.sh
19186 974 3416 136 1712 3416 6052 pman.sh
12865 974 3408 136 1712 3408 6052 pman.sh
12741 974 3416 136 1712 3416 6052 pman.sh
12471 974 3416 136 1712 3416 6052 pman.sh
17595 974 3416 136 1708 3416 6048 pman.sh
16901 974 3416 136 1708 3416 6048 pman.sh
15498 974 3412 136 1708 3412 6048 pman.sh
15002 974 3416 136 1708 3416 6048 pman.sh
14440 974 3416 136 1708 3416 6048 pman.sh
19454 974 3416 136 1704 3416 6044 pman.sh
18232 974 3416 136 1704 3416 6044 pman.sh
16689 974 3416 136 1704 3416 6044 pman.sh
15762 974 3412 136 1704 3412 6044 pman.sh
15130 974 3412 136 1704 3412 6044 pman.sh
14761 974 3408 136 1704 3408 6044 pman.sh
13897 974 3408 136 1704 3408 6044 pman.sh
13697 974 3416 136 1704 3416 6044 pman.sh
13110 974 3420 136 1704 3420 6044 pman.sh
12283 974 3416 136 1704 3416 6044 pman.sh
11743 974 3412 136 1704 3412 6044 pman.sh
11453 974 3408 136 1704 3408 6044 pman.sh
7849 974 3416 136 1704 3416 6044 pman.sh
7339 974 3412 136 1704 3412 6044 pman.sh
945 974 3408 136 1704 3408 6044 pman.sh
12294 974 3408 136 1700 3408 6040 pman.sh
1101 974 3272 132 1628 3272 5964 brelay_console.
1034 974 3188 136 1452 3188 5792 pman.sh
17054 974 3188 132 1452 3188 5788 pman.sh
16736 974 3192 132 1452 3192 5788 pman.sh
4100 974 2948 132 1340 2948 5676 iptbl.sh
27928 974 2892 132 1224 2892 5560 iosd_console_at
22903 974 2888 132 1224 2888 5560 iosd_console_at
22898 974 2892 132 1224 2892 5560 iosd_console_at
105 417 1972 132 328 1972 5540 systemd-udevd
3628 109 2876 132 132 2876 5476 rpc.mountd
3501 28 3144 132 132 3144 5144 klogd
20538 974 2216 136 644 2216 4984 sort_files_by_i
3494 974 2216 132 528 2216 4864 oom.sh
10976 974 1712 136 132 1712 4472 stack_sntp.sh
3526 974 920 132 132 920 4468 boothelper_evt.
3507 974 1752 136 132 1752 4468 bexec.sh
3486 974 1548 132 132 1548 4468 libvirtd.sh
27808 974 1752 132 132 1752 4464 bexec.sh
22701 974 1752 132 132 1752 4464 bexec.sh
22698 974 1752 132 132 1752 4464 bexec.sh
10985 58 864 136 132 864 4260 sntp
3557 77 1484 132 132 1484 3844 rpc.statd
10626 170 1132 132 132 1132 3372 xinetd
10616 170 1100 132 132 1100 3372 xinetd
5468 170 1064 132 132 1064 3372 xinetd
3529 170 984 132 132 984 3372 xinetd
27878 22 684 136 132 684 2780 inotifywait
27868 22 668 136 132 668 2780 inotifywait
27854 22 804 136 132 804 2780 inotifywait
27668 22 684 136 132 684 2780 inotifywait
20417 22 684 136 132 684 2780 inotifywait
9405 22 636 136 132 636 2780 inotifywait
9306 22 632 136 132 632 2780 inotifywait
27745 22 712 132 132 712 2776 inotifywait
16473 22 680 132 132 680 2776 inotifywait
12193 22 680 132 132 680 2776 inotifywait
11999 22 680 132 132 680 2776 inotifywait
11339 22 672 132 132 672 2776 inotifywait
4426 22 684 132 132 684 2776 inotifywait
4086 22 704 132 132 704 2776 inotifywait
3533 22 640 132 132 640 2776 inotifywait
9639 43 592 136 132 592 2620 sleep
9310 43 592 136 132 592 2620 sleep
3939 43 568 132 132 568 2612 sleep

-----------------------------------------------------------

#sh proc memory platform sort location switch 4 r0
System memory: 3983088K total, 3431896K used, 551192K free,
Lowest: 551192K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
18722 116 309756 136 71580 309756 2563500 fed main event
29203 177364 674196 136 120 674196 2084472 linux_iosd-imag
15080 519 1369816 136 1247408 1369816 1950872 hman
20473 308 182012 136 3060 182012 1574632 sif_mgr
19451 1011 188420 136 6656 188420 1303892 platform_mgr
28130 170 247940 136 22280 247940 903932 dbm
27568 287 183544 136 7548 183544 895668 cli_agent
30935 649 174592 136 20212 174592 865376 smand
4089 8823 164644 136 8832 164644 825640 fman_fp_image
8666 127 216936 0 268 216936 811612 smd
7869 430 160236 136 5456 160236 761972 repm
28372 8482 155912 136 1808 155912 725972 fman_rp
31916 251 126220 136 6360 126220 712420 tms
13949 38 121620 136 1324 121620 710612 bt_logger
30227 40 122264 136 1152 122264 702036 psd
20996 418 122784 136 1760 122784 701448 stack_mgr
18522 200 115032 136 268 115032 701372 nif_mgr
16354 147 120272 136 1908 120272 700940 lman
27693 114 118092 136 1084 118092 700868 cmm
7318 45 118900 136 2224 118900 698876 plogd
15929 73 116692 136 432 116692 692836 keyman
3541 604 3964 132 132 3964 212368 libvirtd
3498 753 1928 132 132 1928 17168 virtlogd
13613 7 3128 136 148 3128 16588 auto_upgrade_se
24068 118 2724 136 172 2724 12660 btelnet
95 314 8316 132 132 8316 12428 systemd-journal
13160 974 7752 136 6092 7752 10432 auto_upgrade_cl
16660 974 7596 136 5828 7596 10168 ncd.sh
15492 974 7492 136 5824 7492 10164 issu_stack.sh
27785 974 6512 136 5692 6512 10032 issu_stack.sh
27779 974 6420 136 5692 6420 10032 issu_stack.sh
23265 7 1872 132 148 1872 9600 rotee
23054 7 1872 132 148 1872 9600 rotee
18928 974 6196 136 4520 6196 8860 periodic.sh
31558 7 1872 136 148 1872 8708 rotee
30605 7 1880 136 148 1880 8708 rotee
28767 7 1884 136 148 1884 8708 rotee
28011 7 1872 136 148 1872 8708 rotee
27583 7 1872 136 148 1872 8708 rotee
27119 7 1872 136 148 1872 8708 rotee
26766 7 1872 136 148 1872 8708 rotee
20579 7 1844 136 148 1844 8708 rotee
19612 7 1924 136 148 1924 8708 rotee
18851 7 1852 136 148 1852 8708 rotee
18653 7 1868 136 148 1868 8708 rotee
18160 7 1920 136 148 1920 8708 rotee
17767 7 1848 136 148 1848 8708 rotee
17316 7 1840 136 148 1840 8708 rotee
16509 7 1848 136 148 1848 8708 rotee
16086 7 1848 136 148 1848 8708 rotee
15611 7 1840 136 148 1840 8708 rotee
14906 7 1844 136 148 1844 8708 rotee
14752 7 1888 136 148 1888 8708 rotee
14398 7 1840 136 148 1840 8708 rotee
13642 7 1844 136 148 1844 8708 rotee
7410 7 1844 136 148 1844 8708 rotee
3369 7 1840 136 148 1840 8708 rotee
16418 7 1868 132 148 1868 8704 rotee
12126 7 1852 132 148 1852 8704 rotee
11996 7 1840 132 148 1840 8704 rotee
29868 7 1880 136 148 1880 8588 rotee
20017 7 1848 136 148 1848 8588 rotee
19557 7 1836 136 148 1836 8588 rotee
18388 7 1840 136 148 1840 8588 rotee
17578 7 1840 136 148 1840 8588 rotee
15188 7 1840 136 148 1840 8588 rotee
14215 7 1852 136 148 1852 8588 rotee
13244 7 1840 136 148 1840 8588 rotee
12902 7 1844 136 148 1844 8588 rotee
12583 7 1848 136 148 1848 8588 rotee
8050 7 1844 136 148 1844 8588 rotee
6919 7 1840 136 148 1840 8588 rotee
11329 7 1836 132 148 1836 8584 rotee
4405 7 1832 132 148 1832 8584 rotee
4049 7 1844 132 148 1844 8584 rotee
4045 7 1848 132 148 1848 8584 rotee
3893 7 1832 132 148 1832 8584 rotee
1 1398 4460 132 1476 4460 8048 systemd
10632 974 5200 132 3476 5200 7812 rollback_timer.
12008 974 4448 132 2740 4448 7076 pvp.sh
3493 974 4156 132 2472 4156 6808 reflector.sh
3505 974 4132 132 2460 4132 6796 droputil.sh
14190 974 4052 136 2408 4052 6748 btrace_rotate.s
11885 974 4104 132 2380 4104 6716 psvp.sh
15747 974 4048 132 2372 4048 6708 pvp.sh
17893 974 3564 136 1896 3564 6236 btrace_rotate.s
3497 49 1144 132 132 1144 6176 rpcbind
23286 974 3424 132 1788 3424 6124 brelay.sh
23684 974 3432 136 1764 3432 6104 btelnet.sh
27676 974 3408 136 1712 3408 6052 pman.sh
26326 974 3416 136 1712 3416 6052 pman.sh
19181 974 3416 136 1712 3416 6052 pman.sh
12742 974 3416 136 1712 3416 6052 pman.sh
12478 974 3416 136 1712 3416 6052 pman.sh
29612 974 3416 136 1708 3416 6048 pman.sh
17631 974 3416 136 1708 3416 6048 pman.sh
16922 974 3416 136 1708 3416 6048 pman.sh
15524 974 3412 136 1708 3412 6048 pman.sh
14430 974 3416 136 1708 3416 6048 pman.sh
31281 974 3416 136 1704 3416 6044 pman.sh
30362 974 3412 136 1704 3412 6044 pman.sh
28508 974 3416 136 1704 3416 6044 pman.sh
27159 974 3416 136 1704 3416 6044 pman.sh
26619 974 3412 136 1704 3412 6044 pman.sh
20085 974 3416 136 1704 3416 6044 pman.sh
19471 974 3416 136 1704 3416 6044 pman.sh
18260 974 3416 136 1704 3416 6044 pman.sh
15131 974 3412 136 1704 3412 6044 pman.sh
14764 974 3408 136 1704 3408 6044 pman.sh
13887 974 3412 136 1704 3412 6044 pman.sh
13090 974 3420 136 1704 3420 6044 pman.sh
7684 974 3408 136 1704 3408 6044 pman.sh
7092 974 3416 136 1704 3416 6044 pman.sh
6514 974 3412 136 1704 3412 6044 pman.sh
12298 974 3408 136 1700 3408 6040 pman.sh
23083 974 3272 132 1628 3272 5964 brelay_console.
2779 974 3188 136 1452 3188 5792 pman.sh
17050 974 3188 132 1452 3188 5788 pman.sh
16753 974 3192 132 1452 3192 5788 pman.sh
4102 974 2948 132 1340 2948 5676 iptbl.sh
3621 109 2880 132 132 2880 5476 rpc.mountd
105 417 1932 132 264 1932 5476 systemd-udevd
3500 28 3144 132 132 3144 5144 klogd
20594 974 2216 136 644 2216 4984 sort_files_by_i
3502 974 2216 132 528 2216 4864 oom.sh
25861 974 1712 136 132 1712 4472 stack_sntp.sh
23656 974 1752 136 132 1752 4468 bexec.sh
3546 974 920 132 132 920 4468 boothelper_evt.
3508 974 1548 132 132 1548 4468 libvirtd.sh
25878 58 856 136 132 856 4260 sntp
3543 77 1484 132 132 1484 3844 rpc.statd
10634 170 1100 132 132 1100 3372 xinetd
10630 170 1132 132 132 1132 3372 xinetd
5464 170 1064 132 132 1064 3372 xinetd
3504 170 984 132 132 984 3372 xinetd
27790 22 684 136 132 684 2780 inotifywait
27783 22 668 136 132 668 2780 inotifywait
27776 22 740 136 132 740 2780 inotifywait
27598 22 684 136 132 684 2780 inotifywait
16752 22 632 136 132 632 2780 inotifywait
16656 22 636 136 132 636 2780 inotifywait
3091 22 684 136 132 684 2780 inotifywait
27657 22 712 132 132 712 2776 inotifywait
16512 22 680 132 132 680 2776 inotifywait
12197 22 680 132 132 680 2776 inotifywait
12003 22 680 132 132 680 2776 inotifywait
11346 22 672 132 132 672 2776 inotifywait
4430 22 684 132 132 684 2776 inotifywait
4087 22 704 132 132 704 2776 inotifywait
3564 22 640 132 132 640 2776 inotifywait
17177 43 592 136 132 592 2620 sleep
17113 43 592 136 132 592 2620 sleep
3914 43 568 132 132 568 2612 sleep

[Leo Laohoo · Accepted solution]

---

@Paajf wrote:

Pid     Text    Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
18751    116  312308 136 71580 312308 2563684 fed main event
14434 177364  698276 0 120 698276 2085764 linux_iosd-imag
15058    519 1369980 136 1247440 1369980 1950904 hman
20452    308  181920 136 3004 181920 1574576 sif_mgr

---

Raise a TAC Case and get them to confirm if this is CSCvj24700 or CSCvo21122.  

You may need to upgrade the firmware of the entire stack to prevent this memory leak from coming back.