Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1607
Views
0
Helpful
7
Replies

Catalyst 3850 High Memory platform

TharasuKoonharn36667
Level 1
Level 1

‎09-25-2022 05:37 AM

Hi,

We have stack switch WS-C3850-48T-S are running version 16.3.7

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 56 WS-C3850-48T 16.3.7 CAT3K_CAA-UNIVERSALK9 BUNDLE
   2 56 WS-C3850-48T 16.3.7 CAT3K_CAA-UNIVERSALK9 BUNDLE

We found the alert high MEM on solarwind, and then checked on switch.

3850-48-01#show processes memory
Processor Pool Total: 886321088 Used: 335837328 Free: 550483760
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

3850-48-01#show processes memory platform sorted
System memory: 3980144K total, 3493056K used, 487088K free,
Lowest: 487088K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
1722 3847 272900 132 18204 272900 2644916 wcm
15716 288 308408 132 102156 308408 2479656 fed main event
4892 147940 1312256 136 80 1312256 2460004 linux_iosd-imag
16395 363 140568 136 2072 140568 1362976 sif_mgr
15474 989 140716 136 1852 140716 1090720 platform_mgr
2762 283 158232 136 5560 158232 776804 cli_agent
32446 542 219504 136 134488 219504 627708 smand
27090 7620 119408 132 6752 119408 617904 fman_fp_image
3456 250 139556 136 4264 139556 581696 dbm
32639 128 138816 0 220 138816 579724 smd
13155 1357 20772 136 3900 20772 573960 nginx
12195 1357 83500 136 3900 83500 571612 nginx
3749 8367 118756 136 3000 118756 528668 fman_rp
1280 247 90204 136 6640 90204 506692 tms
9716 38 81692 136 936 81692 500588 bt_logger
3042 117 81684 136 1120 81684 499748 cmm
14083 200 79624 136 272 79624 496888 nif_mgr
10865 506 88216 136 2144 88216 496252 hman
31061 41 83916 136 1316 83916 493172 psd
17131 381 85260 136 1512 85260 493128 stack_mgr
12189 154 84740 136 1992 84740 492752 lman
30697 46 81932 136 1896 81932 490512 plogd
21219 63 81040 132 956 81040 490124 epc_ws_liaison
17328 24 79688 136 380 79688 488316 swift_sampled
11793 53 80292 136 396 80292 481224 keyman
17950 34 77624 136 532 77624 479144 torchd
31652 507 64144 132 3288 64144 258816 repm
1108 483 1964 132 132 1964 169288 libvirtd
95 234 4668 132 8700 4668 21004 systemd-journal
31736 7 2808 136 148 2808 13316 rotee
11744 7 2796 136 148 2796 13316 rotee
10638 7 2828 136 148 2828 13316 rotee
4144 7 2808 136 148 2808 13316 rotee
32077 7 2808 132 148 2808 13312 rotee
30969 7 2808 132 148 2808 13312 rotee
26359 7 2808 132 148 2808 13312 rotee
16519 7 2808 132 148 2808 13312 rotee
15603 7 2788 132 148 2788 13312 rotee
15384 7 2808 132 148 2808 13312 rotee
14611 7 2808 132 148 2808 13312 rotee
14351 7 2808 132 148 2808 13312 rotee
14124 7 2856 132 148 2856 13312 rotee
13978 7 2800 132 148 2800 13312 rotee
12540 7 2804 132 148 2804 13312 rotee
12368 7 2808 132 148 2808 13312 rotee
11879 7 2808 132 148 2808 13312 rotee
11445 7 2808 132 148 2808 13312 rotee
10705 7 2796 132 148 2796 13312 rotee
10065 7 2808 132 148 2808 13312 rotee
9381 7 2844 132 148 2844 13312 rotee
7758 7 2816 132 148 2816 13312 rotee
7616 7 2788 132 148 2788 13312 rotee
3356 7 2808 132 148 2808 13312 rotee
2884 7 2808 132 148 2808 13312 rotee
2421 7 2808 132 148 2808 13312 rotee
2104 7 2808 132 148 2808 13312 rotee
1301 7 2808 132 148 2808 13312 rotee
794 7 2808 132 148 2808 13312 rotee
10038 7 2800 136 148 2800 13196 rotee
32193 7 2796 132 148 2796 13192 rotee
30296 7 2808 132 148 2808 13192 rotee
29985 7 2808 132 148 2808 13192 rotee
20916 7 2808 132 148 2808 13192 rotee
17594 7 2792 132 148 2792 13192 rotee
16867 7 2792 132 148 2792 13192 rotee
15920 7 2788 132 148 2788 13192 rotee
15851 7 2800 132 148 2800 13192 rotee
13713 7 2792 132 148 2792 13192 rotee
12956 7 2792 132 148 2792 13192 rotee
11044 7 2792 132 148 2792 13192 rotee
8975 7 2792 132 148 2792 13192 rotee
8612 7 2792 132 148 2792 13192 rotee
8301 7 2796 132 148 2796 13192 rotee
7184 7 2784 132 148 2784 13192 rotee
1994 7 2760 132 148 2760 13192 rotee
1627 7 2796 132 148 2796 13192 rotee
1587 7 2796 132 148 2796 13192 rotee
1528 7 2760 132 148 2760 13192 rotee
9379 7 2780 136 148 2780 13132 auto_upgrade_se
30899 52 2580 132 172 2580 11712 in.telnetd
12512 968 7088 132 5404 7088 9568 ncd.sh
11343 968 6924 136 5272 6924 9440 issu_stack.sh
8941 968 6800 136 5148 6800 9316 auto_upgrade_cl
23097 968 5900 136 5128 5900 9296 issu_stack.sh
1 1248 4368 132 1300 4368 8324 systemd
14818 968 5780 136 4108 5780 8212 periodic.sh
7636 968 4272 132 2620 4272 6720 pvp.sh
7504 968 3880 132 2244 3880 6408 psvp.sh
1041 968 3960 132 2292 3960 6392 reflector.sh
1040 968 3944 132 2292 3944 6392 droputil.sh
14717 968 3852 132 2248 3852 6348 btrace_rotate.s
11772 968 3964 132 2248 3964 6348 pvp.sh
9975 968 3896 136 2216 3896 6320 btrace_rotate.s
17276 968 3416 132 1832 3416 5996 pman.sh
15792 968 3416 132 1832 3416 5996 pman.sh
14605 968 3416 132 1832 3416 5996 pman.sh
12901 968 3416 132 1832 3416 5996 pman.sh
10504 968 3416 132 1832 3416 5996 pman.sh
9575 968 3416 132 1832 3416 5996 pman.sh
8414 968 3416 132 1832 3416 5996 pman.sh
8160 968 3416 132 1832 3416 5996 pman.sh
3813 968 3416 132 1832 3416 5996 pman.sh
2394 968 3416 132 1832 3416 5996 pman.sh
1995 968 3416 132 1832 3416 5996 pman.sh
536 968 3416 132 1832 3416 5996 pman.sh
29674 968 3416 132 1720 3416 5884 pman.sh
10881 968 3412 132 1720 3412 5884 pman.sh
31220 968 3412 132 1716 3412 5880 pman.sh
30291 968 3400 132 1716 3400 5880 pman.sh
16169 968 3416 132 1716 3416 5880 pman.sh
13303 968 3416 132 1716 3416 5880 pman.sh
12413 968 3412 132 1716 3412 5880 pman.sh
11425 968 3412 132 1716 3412 5880 pman.sh
2899 968 3412 132 1716 3412 5880 pman.sh
1815 968 3416 132 1716 3416 5880 pman.sh
31542 968 3408 132 1712 3408 5876 pman.sh
29851 968 3412 132 1712 3412 5876 pman.sh
11226 968 3416 132 1712 3416 5876 pman.sh
10149 968 3412 132 1712 3412 5876 pman.sh
7994 968 3412 132 1712 3412 5876 pman.sh
15144 968 3412 132 1708 3412 5872 pman.sh
8737 968 3412 132 1708 3412 5872 pman.sh
926 968 3404 132 1708 3404 5872 pman.sh
30913 968 3244 132 1676 3244 5776 brelay.sh
30134 968 3244 132 1676 3244 5776 in.telnetd.sh
25557 968 3208 132 1584 3208 5748 pman.sh
20489 968 3200 132 1584 3200 5748 pman.sh
13511 968 3204 132 1584 3204 5748 pman.sh
13059 968 3204 132 1584 3204 5748 pman.sh
31622 968 2792 132 1212 2792 5312 iosd_console_at
1682 968 2820 132 1196 2820 5296 iptbl.sh
104 285 1800 132 196 1800 5200 systemd-udevd
16536 968 2080 136 644 2080 4684 sort_files_by_i
6893 968 2184 132 528 2184 4628 rollback_timer.
1044 968 2140 132 528 2140 4628 oom.sh
1106 968 928 132 132 928 4232 boothelper_evt.
31615 968 1588 132 132 1588 4168 mcp_chvrf.sh
31566 968 1680 132 132 1680 4168 bexec.sh
28677 968 1648 132 132 1648 4168 stack_sntp.sh
24080 968 1580 132 132 1580 4168 mcp_chvrf.sh
6890 968 1580 132 132 1580 4168 mcp_chvrf.sh
1065 968 1476 132 132 1476 4168 libvirtd.sh
28695 56 832 132 132 832 3944 sntp
1175 102 1156 132 132 1156 3328 rpc.mountd
1089 45 920 132 132 920 3136 rpcbind
24089 173 1028 132 132 1028 3068 xinetd
24072 173 988 132 132 988 3068 xinetd
6897 173 1016 132 132 1016 3068 xinetd
1052 173 932 132 132 932 3068 xinetd
1113 66 904 132 132 904 2724 rpc.statd
8428 21 676 136 132 676 2568 inotifywait
23107 21 680 132 132 680 2564 inotifywait
23054 21 688 132 132 688 2564 inotifywait
23048 21 680 132 132 680 2564 inotifywait
12741 21 676 132 132 676 2564 inotifywait
7899 21 676 132 132 676 2564 inotifywait
7632 21 676 132 132 676 2564 inotifywait
1640 21 684 132 132 684 2564 inotifywait
24157 21 624 136 132 624 2504 inotifywait
23402 21 624 132 132 624 2500 inotifywait
23096 21 788 132 132 788 2500 inotifywait
7212 21 668 132 132 668 2500 inotifywait
2027 21 672 132 132 672 2500 inotifywait
1116 21 640 132 132 640 2500 inotifywait
24026 39 552 136 132 552 2340 sleep
23882 39 552 132 132 552 2336 sleep
1580 39 532 132 132 532 2336 sleep

Is there normal behavior, and can we reduce it?

Thank you in advance.

0 Helpful
7 Replies 7

TharasuKoonharn36667
Level 1
Level 1

‎09-25-2022 08:16 AM

Please see the output below.
3850T-48-01#show platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 3.60% 100% 90% 95% H
DRAM 3412MB(87%) 3886MB 90% 95% H

0 Helpful

Leo Laohoo
Hall of Fame
Hall of Fame

‎09-25-2022 04:45 PM - edited ‎09-25-2022 06:31 PM

@TharasuKoonharn36667 wrote:
3850-48-01#show processes memory platform sorted​

Stop using that command.  The switch is already on 16.X.X and the above command is not just irrelevant but does not provide a complete picture.  

And I have a good indication what is the cause of the high memory:  keyman

Post the complete output to the following commands: 

  1. sh platform software status control-processor brief
  2. sh processes memory platform sorted location switch active r0
  3. sh processes memory platform sorted location switch standby r0

NOTE:  For #2 and #3, please provide the output of the "first page".  I do not see the complete output.  

@TharasuKoonharn36667 wrote:
11793 53 80292 136 396 80292 481224 keyman

That is way too high already.  The "average" (keyman proces) is 10k (or <15k).  

If my suspicion is correct then I know, exactly, what is causing the keyman process to go "mental" and I know an effective workaround that does not require a reload or a firmware upgrade.  

0 Helpful

TharasuKoonharn36667
Level 1
Level 1
In response to Leo Laohoo

‎09-25-2022 08:58 PM

Hi @Leo Laohoo ,

Thank you for your response.

Please see the attached.

Thank you,

Preview file
30 KB
0 Helpful

Leo Laohoo
Hall of Fame
Hall of Fame
In response to TharasuKoonharn36667

‎09-25-2022 09:11 PM - edited ‎09-25-2022 09:11 PM

@TharasuKoonharn36667 wrote:
   Pid    Text      Data   Stack   Dynamic       RSS     Total              Name  
--------------------------------------------------------------------------------
  1722    3847    273560     132     18204    273560   2644916               wcm

This stack has encountered two (2) bugs:   

  1. Memory leak is due to the WCM process (wireless controller module) and this is a well-known bug, CSCvo28206.
  2. Read this:  FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality

 

0 Helpful

TharasuKoonharn36667
Level 1
Level 1
In response to Leo Laohoo

‎09-25-2022 10:15 PM

@Leo Laohoo 

According to bug, CSCvo28206.

There is no workaround and also no fixed releases.

What should we do on our case?

Thank you,

0 Helpful

Leo Laohoo
Hall of Fame
Hall of Fame
In response to TharasuKoonharn36667

‎09-25-2022 10:43 PM - edited ‎09-25-2022 10:45 PM

Upgrade to 16.3.11 and apply the QuoVadis workaround.

0 Helpful

TharasuKoonharn36667
Level 1
Level 1
In response to Leo Laohoo

‎09-28-2022 10:09 PM

@Leo Laohoo 

Thank you for your support.

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card