Objective:
This article is to provide better (and understandable) information to non-Cisco readers. For example, Bug ID does not include any useful command(s) to help non-Cisco individuals troubleshoot. Bug ID also states "Catalyst 9K", however, nowhere in the "Products" section is Catalyst 9K ever mentioned.
Terminology Used:
- Switch - a physical unit;
- Stack - a logical unit made up of multiple switches
NOTE:
- This bug only affects Catalyst 9300 in a stack (>8 weeks uptime).
- This bug DOES NOT affect stand-alone Catalyst 9300 switch.
- At the time of writing, I am unable to determine if Catalyst 9400/9600, VSS or standalone, are affected or not.
- At the time of writing, I am unable to determine if Catalyst 9500, Switch Virtual or standalone, are affected or not.
- At the time of writing, I am unable to determine if Catalyst 9200 are affected or not.
Troubleshooting Commands:
There are three (3) commands to check. First two commands have the same output. First command will give the output to the switch master while the other command will provide the output based on the switch member. Third command provides a breakdown of each Core:
1. show processes cpu platform monitor - This command will provide the output from the switch stack master only.
2. show platform software process slot switch <1 to 16> r0 monitor - This command provides the option to see the output of individual switch member.
Switch#sh plat software process slot switch 2 r0 monitor
top - 12:24:15 up 176 days, 15:57, 0 users, load average: 1.50, 1.51, 1.54
Tasks: 381 total, 2 running, 379 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.8 us, 2.2 sy, 0.0 ni, 90.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 7711304 total, 3502396 free, 1732816 used, 2476092 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 4082480 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21472 root 20 0 1261380 62812 36520 R 100.0 0.8 93455:20 sessmgrd
14829 root 20 0 3944352 221848 21212 S 36.8 2.9 85399:36 fed main e+
12261 root 20 0 247504 11380 7416 S 10.5 0.1 9464:46 btman
32758 root 20 0 20376 2580 1960 R 10.5 0.0 0:00.05 top
11904 root 20 0 5269456 18528 7320 S 5.3 0.2 694:11.89 cmand
< -- CUT -- >
Normal value of the "sessmgrd" process is <0%. Example (above) is showing as "100.0".
3. show processes cpu platform sorted location switch <1 to 16> r0 - Breakdown of the state of each Core.
Switch#show processes cpu platform sorted location switch 2 r0
CPU utilization for five seconds: 18%, one minute: 22%, five minutes: 19%
Core 0: CPU utilization for five seconds: 12%, one minute: 14%, five minutes: 13%
Core 1: CPU utilization for five seconds: 100%, one minute: 37%, five minutes: 12%
Core 2: CPU utilization for five seconds: 4%, one minute: 13%, five minutes: 12%
Core 3: CPU utilization for five seconds: 7%, one minute: 13%, five minutes: 8%
Core 4: CPU utilization for five seconds: 0%, one minute: 8%, five minutes: 7%
Core 5: CPU utilization for five seconds: 0%, one minute: 63%, five minutes: 49%
Core 6: CPU utilization for five seconds: 6%, one minute: 12%, five minutes: 17%
Core 7: CPU utilization for five seconds: 16%, one minute: 13%, five minutes: 35%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
21472 20867 100% 100% 99% R 66844 sessmgrd
14829 13594 33% 35% 34% S 228440 fed main event
12261 12026 9% 10% 10% S 11368 btman
10138 9751 1% 1% 1% S 12752 sif_mgr
9873 9342 1% 3% 3% S 730532 linux_iosd-imag
< -- CUT -- >
Normal value of the "sessmgrd" process is <0%. Example (above) is showing as "100.0". Core #1 is in a state of "hot spin" or "hot".
Additional Notes:
- These commands replaces the "classic" Cisco IOS command "sh process cpu sort | exclude 0.00".
- Output is sorted based on CPU level.
