Re: Cisco 9300 Facing High Memory utilization.

Pankaj.bandewar1 · ‎12-26-2023

Hello Experts,

we are facing 9300 Facing High Memory utilization for our multiple switches running Code 17.3.3.

Below are some Commands output

=============

Dec 27 04:11:48.282: %PLATFORM-3-ELEMENT_CRITICAL: Switch 2 R0/0: smand: 1/RP/0: Used Memory value 94% exceeds critical level 90%
Dec 27 04:12:28.294: %PLATFORM-3-ELEMENT_CRITICAL: Switch 1 R0/0: smand: 1/RP/0: Used Memory value 94% exceeds critical level 90%

=============

0001#show platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 2.81% 100% 90% 95% H
DRAM 7158MB(94%) 7575MB 85% 90% C
TMPFS 254MB(3%) 7575MB 40% 50% H

Application Resources:
=======================
CPU:
Quota: 25(Percentage)
Available: 18(Percentage)
VCPU:
Count: 2
Memory:
Quota: 2048(MB)
Available: 1548(MB)
Storage space:
Total: 3904(MB)
Available: 2162(MB)

0001#

---

0001#show processes memory sorted
Processor Pool Total: 1363338588 Used: 446928976 Free: 916409612
reserve P Pool Total: 102404 Used: 88 Free: 102316
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 334368088 79683760 225843096 0 0 *Init*
151 0 640890544 597090944 32338360 0 0 Per-minute Jobs
311 0 1089392193728 1138450809584 32321008 0 0 nbar-sdavc-sched
326 0 100591778792 70700253224 27125896 0 0 HTTP CORE
4 0 36912408 12344536 22849800 0 0 RF Slave Main Th
0 0 0 0 19429440 0 0 *MallocLite*
83 0 273208022336 38538371824 18832072 0 0 IOSD ipc task
0 0 112981230584 112206750312 18520248 24110359 1233588 *Dead*
260 64 43505600 37971480 5023912 0 0 iosp_vty_100001_
501 0 14983464 10878080 4147328 849828 0 EEM ED Syslog
1 0 5146136 427864 4105768 0 0 Chunk Manager
149 0 950984836536 940708909440 2191848 0 0 SAMsgThread
522 0 5403599696 5400838888 1979504 491296 0 EEM Server

===

More Command output is attached in file .

I really appreciate community's help in this Holiday season.

Thank you 🙂

Leo Laohoo · ‎12-27-2023

@Pankaj.bandewar1 wrote:

Dec 27 04:11:48.282: %PLATFORM-3-ELEMENT_CRITICAL: Switch 2 R0/0: smand: 1/RP/0: Used Memory value 94% exceeds critical level 90%
Dec 27 04:12:28.294: %PLATFORM-3-ELEMENT_CRITICAL: Switch 1 R0/0: smand: 1/RP/0: Used Memory value 94% exceeds critical level 90%

Post the complete output to the command "sh platform software status conn brief".

Pankaj.bandewar1 · ‎12-27-2023

this command is not supported

0001#sh platform software status conn brief
^
% Invalid input detected at '^' marker.

Pankaj.bandewar1 · ‎12-27-2023

Instead, i captured below output does that works

---

0001#sh platform software status control-processor brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.33 0.25 0.20
2-RP0 Healthy 0.23 0.18 0.13

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Critical 7757632 7329760 (94%) 427872 ( 6%) 9486916 (122%)
2-RP0 Healthy 7757632 3401024 (44%) 4356608 (56%) 3613516 (47%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 2.09 0.79 0.00 97.00 0.00 0.09 0.00
1 1.60 0.70 0.00 97.69 0.00 0.00 0.00
2 1.19 0.99 0.00 97.70 0.00 0.09 0.00
3 0.90 0.60 0.00 98.49 0.00 0.00 0.00
4 1.00 1.10 0.00 97.89 0.00 0.00 0.00
5 1.60 1.50 0.00 96.89 0.00 0.00 0.00
6 1.20 0.90 0.00 97.90 0.00 0.00 0.00
7 2.10 0.70 0.00 97.19 0.00 0.00 0.00
2-RP0 0 0.50 0.10 0.00 99.39 0.00 0.00 0.00
1 0.90 0.30 0.00 98.79 0.00 0.00 0.00
2 0.90 0.20 0.00 98.89 0.00 0.00 0.00
3 0.60 0.20 0.00 99.20 0.00 0.00 0.00
4 0.50 0.10 0.00 99.39 0.00 0.00 0.00
5 0.60 0.30 0.00 99.10 0.00 0.00 0.00
6 1.60 0.40 0.00 97.99 0.00 0.00 0.00
7 0.40 0.20 0.00 99.40 0.00 0.00 0.00

0001#

Leo Laohoo · ‎12-27-2023

@Pankaj.bandewar1 wrote:
multiple switches running Code 17.3.3.

Please read FN72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality.

Pankaj.bandewar1 · ‎12-27-2023

gone through this link did not found any direct relevant support ,

instead i found a bug CSCwa85525 : Bug Search Tool (cisco.com)

but still the process dosent holds that chunk of memory

0 0 0 0 19429440 0 0 *MallocLite*

Leo Laohoo · ‎12-27-2023

Please read FN72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality.