Catalyst 3850 memory utilization issue

[ShafiEM90390]

Looks like My Catalyst WS-C3850-48P has memory utilization issue. Its running on 16.3.8 version. Can anyone suggest if it's need an upgrade & if yes, to which version?

Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.8, RELEASE SOFTWARE (fc3)

switch#sh platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 3.80% 100% 90% 95% H
DRAM 3479MB(89%) 3866MB 90% 95% H

switch#sh processes memory sorted
Processor Pool Total: 886309824 Used: 307340528 Free: 578969296
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 323261912 56785400 244119192 0 0 *Init*
72 0 80659106776 116669568 31277400 0 893172 IOSD ipc task
344 0 3938520 5760 3989760 849828 0 EEM ED Syslog
1 0 1599424 0 1644424 0 0 Chunk Manager
353 0 1501688 29104 1517584 0 0 EEM Server
146 0 3939768 10760 1163008 0 0 CWAN OIR Handler

[balaji.bandi]

i see some EEM Scripts running, try to disable for some time see is the memory consumption reduced, also if this switch long uptime try cold reboot.

IOS XE 16.9.8 has improvements - possible upgrade and also collect below information before upgrade.

1) Collect the below output periodically and check whether the "Size" counter increases over time for "platform_mgr" process. Eg:

switch# show platform software process list switch active R0 sort memory

2) Enable memory allocation tracking to look for signs of a "callsite" (function) leaking memory as follows:

a) Enable the memory allocation tracking debug on the active switch.

switch# debug platform software memory platform-mgr switch active R0 alloc callsite start

b) Wait approximately 15 minutes, then run the following command.

switch# show platform software memory platform-mgr switch active R0 alloc callsite brief

c) When done, disable the memory allocation tracking debug.

switch# debug platform software memory platform-mgr switch active R0 alloc callsite stop

[ShafiEM90390]

Thankyou for the suggestions. I have rebooted the switch & adding logs after reboot. Let me know still the EEM script to be disabled & if yes, how can I achieve that.

switch#sh event manager statistics server
EEM Queue Information
Triggered Dropped Queue Queue Average
Client Events Events Size Max Run Time
-------------------------------------------------------------------------------
Call Home 0 0 0 64 0.000
EEM Applets 0 0 0 64 0.000
EEM IOS .sh Scripts 0 0 0 128 0.000
EEM Tcl Scripts 0 0 0 64 0.000

============================================

switch#sh processes memory sorted
Processor Pool Total: 886309824 Used: 308498024 Free: 577811800
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 323015304 56527160 244111392 0 0 *Init*
72 0 57964744 2818840 31263160 0 1458240 IOSD ipc task
344 0 3938440 5680 3989760 849828 0 EEM ED Syslog
352 2 39981928 37981888 2289488 0 0 SSH Process
353 0 1533392 33592 1527808 0 0 EEM Server
1 0 1462552 0 1507552 0 0 Chunk Manager
146 0 3939768 10760 1163008 0 0 CWAN OIR Handler
0 0 0 0 526856 0 0 *MallocLite*
4 0 512784 22968 477960 0 0 RF Slave Main Th
0 0 2771624 2416696 446736 17486679 0 *Dead*
345 0 380872 5680 432192 72316 0 EEM ED Generic
323 0 310728 1160 378568 0 0 Crypto IKEv2
10 0 46375648 45000920 230704 43017544 41651660 Pool Manager
306 0 3513480 3178448 205360 0 0 Crypto CA
244 0 70160 448 186712 0 0 AN
357 0 231992 77104 180032 0 0 Call Home proces

==============================================

switch#sh processes memory platform sorted location switch active r0
System memory: 3959056K total, 2024924K used, 1934132K free,
Lowest: 1934132K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
15602 288 305288 132 101836 305288 2479324 fed main event
32362 147914 648720 136 80 648720 1828224 linux_iosd-imag
16597 363 138908 136 1992 138908 1365760 sif_mgr
15453 988 139892 136 1832 139892 1090724 platform_mgr
30208 283 157884 136 5560 157884 775732 cli_agent
27603 542 215788 136 131748 215788 624916 smand
26599 7621 118748 132 6120 118748 617220 fman_fp_image
30957 250 143928 136 4304 143928 583748 dbm
28055 128 146128 0 220 146128 582088 smd
12027 1357 16796 136 1172 16796 570748 nginx
10993 1357 81572 136 1172 81572 568832 nginx
31292 8368 114184 136 2440 114184 524400 fman_rp
28847 247 88616 132 6376 88616 506372 tms
9700 38 81332 136 792 81332 499876 bt_logger
30529 117 81244 136 1120 81244 497644 cmm
14155 200 79604 136 272 79604 496836 nif_mgr
10854 506 86636 136 2084 86636 496140 hman
17026 381 84180 136 1588 84180 493152 stack_mgr
817 41 83364 136 1260 83364 493064 psd
12142 154 82448 136 1636 82448 492344 lman
26057 46 81116 136 1684 81116 490248 plogd
17343 24 79540 136 380 79540 488264 swift_sampled
11716 53 79772 136 396 79772 481172 keyman
17915 34 77476 136 532 77476 479092 torchd
26756 507 59544 132 2828 59544 254068 repm
1065 483 3488 132 132 3488 169288 libvirtd
31677 7 2872 136 148 2872 13316 rotee
10569 7 2820 136 148 2820 13316 rotee
10557 7 2792 136 148 2792 13316 rotee
30876 7 2788 132 148 2788 13312 rotee

switch#sh run | in applet
switch#

[Reza Sharifi]

Hi,

Most memory issues are caused by software bugs. The latest MD version for the 3850 series is "cat3k_caa-universalk9.16.12.08.SPA.bin"

An upgrade should take care of memory leak. Also, you want to install the "Cisco cli analyzer" tool to get more info and the bugs on the OS you are currently running.

https://cway.cisco.com/docs/cisco-cli-analyzer/2.0/Download_and_Install_the_Cisco_CLI_Analyzer.htm

HTH

[MHM Cisco World]

EEM ED Syslog <<- this eat the memory 
can you share the EEM script you use 

[ShafiEM90390]

Thanks for the reply. Can you help me find the EEM script? 

switch#sh event manager statistics server
switch#sh run | sec applet
switch#EEM Queue Information
Triggered Dropped Queue Queue Average
Client Events Events Size Max Run Time
-------------------------------------------------------------------------------
Call Home 0 0 0 64 0.000
EEM Applets 0 0 0 64 0.000
EEM IOS .sh Scripts 0 0 0 128 0.000
EEM Tcl Scripts 0 0 0 64 0.000

switch#sh run | sec Applet
switch#

[Leo Laohoo]

Post the complete output to the following command: 

1. sh platform software status con brief
2. sh process memory platform sort location switch active r0 < -- Post only the first "page"

[ShafiEM90390]

Adding logs below. Monitored for some time & found platform_mgr  data is not increasing.

switch#sh platform software status control-processor brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.65 0.54 0.44

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3959056 2024616 (51%) 1934440 (49%) 2714328 (69%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 2.50 1.00 0.00 96.50 0.00 0.00 0.00
1 2.30 0.80 0.00 96.89 0.00 0.00 0.00
2 4.59 2.49 0.00 92.90 0.00 0.00 0.00
3 2.39 0.49 0.00 97.00 0.00 0.09 0.00

switch#

switch#sh processes memory platform sorted location switch active r0
System memory: 3959056K total, 2024504K used, 1934552K free,
Lowest: 1934552K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
15602 288 305288 132 101836 305288 2479324 fed main event
32362 147914 648432 136 80 648432 1828224 linux_iosd-imag
16597 363 138908 136 1992 138908 1365760 sif_mgr
15453 988 139916 136 1832 139916 1090724 platform_mgr
30208 283 156576 136 5560 156576 775732 cli_agent
27603 542 214824 136 131748 214824 624916 smand
26599 7621 118748 132 6120 118748 617220 fman_fp_image
30957 250 142628 136 4304 142628 583748 dbm
28055 128 146128 0 220 146128 582088 smd
12027 1357 16796 136 1172 16796 570748 nginx
10993 1357 81572 136 1172 81572 568832 nginx
31292 8368 114184 136 2440 114184 524400 fman_rp
28847 247 88616 132 6376 88616 506372 tms
9700 38 81332 136 792 81332 499876 bt_logger
30529 117 81244 136 1120 81244 497644 cmm
14155 200 79604 136 272 79604 496836 nif_mgr
10854 506 86376 136 2084 86376 496140 hman
17026 381 84300 136 1588 84300 493152 stack_mgr
817 41 83364 136 1260 83364 493064 psd
12142 154 82448 136 1636 82448 492344 lman
26057 46 81116 136 1684 81116 490248 plogd
17343 24 79540 136 380 79540 488264 swift_sampled
11716 53 80724 136 396 80724 481172 keyman
17915 34 77476 136 532 77476 479092 torchd
26756 507 59544 132 2828 59544 254068 repm
1065 483 3488 132 132 3488 169288 libvirtd
31677 7 2872 136 148 2872 13316 rotee
10569 7 2820 136 148 2820 13316 rotee
10557 7 2792 136 148 2792 13316 rotee
30876 7 2788 132 148 2788 13312 rotee
30371 7 2788 132 148 2788 13312 rotee
29841 7 2788 132 148 2788 13312 rotee
29531 7 2788 132 148 2788 13312 rotee
28491 7 2788 132 148 2788 13312 rotee
27268 7 2788 132 148 2788 13312 rotee

[Leo Laohoo]

---

@ShafiEM90390 wrote:

15602 288 305288 132 101836 305288 2479324 fed main event
32362 147914 648432 136 80 648432 1828224 linux_iosd-imag​

---

Those are exceptionally high. 

I recommend upgrading to 16.3.11.

---

@ShafiEM90390 wrote:

11716 53 80724 136 396 80724 481172 keyman​

---

This process is high because of FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality.  Apply the workaround.