08-25-2016 11:03 PM - edited 03-08-2019 07:09 AM
Experts,
we have encountered similar cases in one of our sites.
device information:
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 32 WS-C3850-24T 03.02.03.SE cat3k_caa-universalk9 INSTALL
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 32 WS-C3850-24P 03.03.05SE cat3k_caa-universalk9 INSTALL
High CPU caused by pdsd, iosd processes.
sample capture from device:
switch#Show process CPU sorted | ex 0.0
Core 0: CPU utilization for five seconds: 98%; one minute: 90%; five minutes: 91%
Core 1: CPU utilization for five seconds: 89%; one minute: 56%; five minutes: 73%
Core 2: CPU utilization for five seconds: 97%; one minute: 97%; five minutes: 80%
Core 3: CPU utilization for five seconds: 23%; one minute: 53%; five minutes: 55%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
6092 3359485 13453456 1739 22.87 23.34 23.12 0 pdsd
9173 3749773 35001568 125 1.82 1.43 1.47 0 iosd
6133 1139697 12560142 212 0.34 0.15 0.13 0 snmp_subagent
6117 1929734 25961533 950 0.15 0.12 0.10 0 cpumemd
#--- switch Show process cpu history Execute at 2016-08-26 9:17:00
switch#Show process cpu history
History information for system:
777777777777777777777777777777777777777777777777777777777777
555556666655555555555555544444555554444444444555555555566666
100
90 ************************************ ***
80 ************************* ***** ***************
70 ************************************************************
60 ************************************************************
50 ************************************************************
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
777777777777777777777777777777777777777777777777777777777777
667666555668567666677667776766775665656676566665666565766675
100
90
80 ***********#**#*#*####*##*****#*******#*#**##*#*******#*****
70 ############################################################
60 ############################################################
50 ############################################################
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
777777777777777777779788777777787777777777778777777777777777787777777777
777777778778797877773813777787727777777687775777779878877877707777877878
100
90 * *
80 ************************************************************************
70 ########################################################################
60 ########################################################################
50 ########################################################################
40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
Hopefully, we could have a sound advice on what's going on.
Norix S.
08-25-2016 11:19 PM
1 32 WS-C3850-24T 03.02.03.SE cat3k_caa-universalk9 INSTALL
Wow. IOS version 3.2.3 has been released since 24 Sep 2013. I'd upgrade the IOS first before doing any troubleshooting.
09-06-2016 06:56 AM
Hi,
I have this problem too, except mine is reporting at 100% utilization - even on our Network Monitoring software.
We don't necessarily want to upgrade the IOS - any sort of work around you've found?
-Ian
09-06-2016 07:23 AM
Please provide the result of
#show processes cpu detailed process iosd sort | ex 0.0
#show processes cpu detailed process pdsd sort | ex 0.0
sidenote: no related bug but check this out
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuj91008
Thank you.
09-06-2016 08:39 AM
Switch3#show process cpu det proc iosd sort | e 0.0
Core 0: CPU utilization for five seconds: 95%; one minute: 97%; five minutes: 98%
Core 1: CPU utilization for five seconds: 95%; one minute: 98%; five minutes: 98%
Core 2: CPU utilization for five seconds: 94%; one minute: 96%; five minutes: 96%
Core 3: CPU utilization for five seconds: 99%; one minute: 97%; five minutes: 96%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
8570 L 409865 2922905 190 2.07 1.29 1.15 0 iosd
8570 L 0 8570 2558103 2229519 0 1.83 1.16 1.02 0 iosd
8570 L 2 9212 2748744 1569935 0 0.20 0.11 0.10 0 iosd.fastpath
398 I 688850 3393052 0 1.11 0.33 0.33 0 PDU DISPATCHER
399 I 2246684 3924748 0 0.66 0.55 0.66 0 SNMP ENGINE
413 I 360 171 0 0.22 0.44 0.11 3 SSH Process
400 I 1528153 6824237 0 0.22 0.22 0.22 0 IP SNMP
59 I 3109301 6155702 0 0.11 0.11 0.11 0 ARP Snoop
98 I 1295594 2172321 0 0.11 0.22 0.11 0 PLATFORM_MGR SPI in
Switch3#show process cpu det proc pdsd sort | e 0.0
Core 0: CPU utilization for five seconds: 97%; one minute: 98%; five minutes: 98%
Core 1: CPU utilization for five seconds: 98%; one minute: 98%; five minutes: 98%
Core 2: CPU utilization for five seconds: 99%; one minute: 97%; five minutes: 97%
Core 3: CPU utilization for five seconds: 97%; one minute: 97%; five minutes: 96%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
6243 L 1944481 7127380 730 13.08 18.83 20.42 0 pdsd
6243 L 2 8462 3203594 3908112 0 20.58 18.96 20.41 0 pdsd
I think the other member of my team may have just opened a TAC case, but he (a senior member) also pointed out the fed process was taking up a lot
Switch3#show process cpu det proc fed sort | e 0.0
Core 0: CPU utilization for five seconds: 93%; one minute: 98%; five minutes: 98%
Core 1: CPU utilization for five seconds: 99%; one minute: 98%; five minutes: 98%
Core 2: CPU utilization for five seconds: 97%; one minute: 96%; five minutes: 96%
Core 3: CPU utilization for five seconds: 99%; one minute: 96%; five minutes: 96%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
5705 L 4132537 3126866 439 49.83 48.78 48.46 1088 fed
5705 L 3 9176 2226953 1194949 0 24.64 24.35 24.09 0 PunjectRx
5705 L 2 9094 3981651 1596517 0 16.12 17.67 17.70 0 Xcvr
5705 L 0 9331 1071351 2216321 0 7.44 5.33 5.20 0 XcvrPoll
5705 L 2 6147 1400833 5255049 0 1.18 0.97 0.98 0 fed-ots-nfl
5705 L 0 9177 586851 1234595 0 0.25 0.14 0.16 0 PunjectTx
09-08-2016 04:22 PM
6133 1139697 12560142 212 0.34 0.15 0.13 0 snmp_subagent
398 I 688850 3393052 0 1.11 0.33 0.33 0 PDU DISPATCHER
399 I 2246684 3924748 0 0.66 0.55 0.66 0 SNMP ENGINE
400 I 1528153 6824237 0 0.22 0.22 0.22 0 IP SNMP
The output shows SNMP is responsible for the majority of the CPU usage. Looks like its bug CSCuh88726. This can be resolved by upgrading your ios version to recommended version 3.6.5E.
CSCuh88726: SNMP High CPU when polling lldpXMedLocMediaPolicyEntry
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3se/release_notes/OL_30562_01.html
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuh88726/?referring_site=bugquickviewredir
Please rate if helpful.
Thank you.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: