6133 1139697 12560142 212 0

Norix S · ‎08-25-2016

Experts,

we have encountered similar cases in one of our sites.

device information:

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 32 WS-C3850-24T 03.02.03.SE cat3k_caa-universalk9 INSTALL

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 32 WS-C3850-24P 03.03.05SE cat3k_caa-universalk9 INSTALL

High CPU caused by pdsd, iosd processes.

sample capture from device:

switch#Show process CPU sorted | ex 0.0
Core 0: CPU utilization for five seconds: 98%; one minute: 90%; five minutes: 91%
Core 1: CPU utilization for five seconds: 89%; one minute: 56%; five minutes: 73%
Core 2: CPU utilization for five seconds: 97%; one minute: 97%; five minutes: 80%
Core 3: CPU utilization for five seconds: 23%; one minute: 53%; five minutes: 55%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
6092 3359485 13453456 1739 22.87 23.34 23.12 0 pdsd
9173 3749773 35001568 125 1.82 1.43 1.47 0 iosd
6133 1139697 12560142 212 0.34 0.15 0.13 0 snmp_subagent
6117 1929734 25961533 950 0.15 0.12 0.10 0 cpumemd

#--- switch Show process cpu history Execute at 2016-08-26 9:17:00
switch#Show process cpu history

History information for system:

777777777777777777777777777777777777777777777777777777777777
555556666655555555555555544444555554444444444555555555566666
100
90 ************************************ ***
80 ************************* ***** ***************
70 ************************************************************
60 ************************************************************
50 ************************************************************
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

777777777777777777777777777777777777777777777777777777777777
667666555668567666677667776766775665656676566665666565766675
100
90
80 ***********#**#*#*####*##*****#*******#*#**##*#*******#*****
70 ############################################################
60 ############################################################
50 ############################################################
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

777777777777777777779788777777787777777777778777777777777777787777777777
777777778778797877773813777787727777777687775777779878877877707777877878
100
90 * *
80 ************************************************************************
70 ########################################################################
60 ########################################################################
50 ########################################################################
40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

Hopefully, we could have a sound advice on what's going on.

Norix S.

Leo Laohoo · ‎08-25-2016

1 32 WS-C3850-24T 03.02.03.SE cat3k_caa-universalk9 INSTALL

Wow. IOS version 3.2.3 has been released since 24 Sep 2013. I'd upgrade the IOS first before doing any troubleshooting.

ifritch.fritchy1 · ‎09-06-2016

Hi,

I have this problem too, except mine is reporting at 100% utilization - even on our Network Monitoring software.

We don't necessarily want to upgrade the IOS - any sort of work around you've found?

-Ian

Austin Sabio · ‎09-06-2016

Please provide the result of

#show processes cpu detailed process iosd sort | ex 0.0

#show processes cpu detailed process pdsd sort | ex 0.0

sidenote: no related bug but check this out

CSCuj91008: Cat3850 running at 30+% CPU while 'doing nothing'

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuj91008

Thank you.

ifritch.fritchy1 · ‎09-06-2016

Switch3#show process cpu det proc iosd sort | e 0.0
Core 0: CPU utilization for five seconds: 95%; one minute: 97%; five minutes: 98%
Core 1: CPU utilization for five seconds: 95%; one minute: 98%; five minutes: 98%
Core 2: CPU utilization for five seconds: 94%; one minute: 96%; five minutes: 96%
Core 3: CPU utilization for five seconds: 99%; one minute: 97%; five minutes: 96%
PID    T C TID    Runtime(ms) Invoked uSecs 5Sec      1Min     5Min     TTY   Process
                                               (%)       (%)      (%)
8570   L           409865      2922905 190    2.07      1.29    1.15    0     iosd
8570   L 0 8570   2558103     2229519 0      1.83      1.16    1.02    0     iosd
8570   L 2 9212   2748744     1569935 0      0.20      0.11    0.10    0     iosd.fastpath
398    I           688850      3393052 0      1.11      0.33    0.33    0       PDU DISPATCHER
399    I           2246684     3924748 0      0.66      0.55    0.66    0       SNMP ENGINE
413    I           360         171     0      0.22      0.44    0.11    3       SSH Process
400    I           1528153     6824237 0      0.22      0.22    0.22    0       IP SNMP
59     I           3109301     6155702 0      0.11      0.11    0.11    0       ARP Snoop
98     I           1295594     2172321 0      0.11      0.22    0.11    0       PLATFORM_MGR SPI in

Switch3#show process cpu det proc pdsd sort | e 0.0
Core 0: CPU utilization for five seconds: 97%; one minute: 98%; five minutes: 98%
Core 1: CPU utilization for five seconds: 98%; one minute: 98%; five minutes: 98%
Core 2: CPU utilization for five seconds: 99%; one minute: 97%; five minutes: 97%
Core 3: CPU utilization for five seconds: 97%; one minute: 97%; five minutes: 96%
PID    T C TID    Runtime(ms) Invoked uSecs 5Sec      1Min     5Min     TTY   Process
                                               (%)       (%)      (%)
6243   L           1944481     7127380 730    13.08     18.83   20.42   0     pdsd
6243   L 2 8462   3203594     3908112 0      20.58     18.96   20.41   0     pdsd

I think the other member of my team may have just opened a TAC case, but he (a senior member) also pointed out the fed process was taking up a lot

Switch3#show process cpu det proc fed sort | e 0.0
Core 0: CPU utilization for five seconds: 93%; one minute: 98%; five minutes: 98%
Core 1: CPU utilization for five seconds: 99%; one minute: 98%; five minutes: 98%
Core 2: CPU utilization for five seconds: 97%; one minute: 96%; five minutes: 96%
Core 3: CPU utilization for five seconds: 99%; one minute: 96%; five minutes: 96%
PID    T C TID    Runtime(ms) Invoked uSecs 5Sec      1Min     5Min     TTY   Process
                                               (%)       (%)      (%)
5705   L           4132537     3126866 439    49.83     48.78   48.46   1088 fed
5705   L 3 9176   2226953     1194949 0      24.64     24.35   24.09   0     PunjectRx
5705   L 2 9094   3981651     1596517 0      16.12     17.67   17.70   0     Xcvr
5705   L 0 9331   1071351     2216321 0      7.44      5.33    5.20    0     XcvrPoll
5705   L 2 6147   1400833     5255049 0      1.18      0.97    0.98    0     fed-ots-nfl
5705   L 0 9177   586851      1234595 0      0.25      0.14    0.16    0     PunjectTx

Austin Sabio · ‎09-08-2016

6133 1139697 12560142 212 0.34 0.15 0.13 0 snmp_subagent

398    I           688850      3393052 0      1.11      0.33    0.33    0       PDU DISPATCHER
399    I           2246684     3924748 0      0.66      0.55    0.66    0       SNMP ENGINE
400    I           1528153     6824237 0      0.22      0.22    0.22    0       IP SNMP

The output shows SNMP is responsible for the majority of the CPU usage. Looks like its bug CSCuh88726. This can be resolved by upgrading your ios version to recommended version 3.6.5E.

CSCuh88726: SNMP High CPU when polling lldpXMedLocMediaPolicyEntry

Symptom:

LLDP snmp mib walk times out with cpu hog

Conditions:
SNMP walk of OID 1.0.8802.1.1.2.1.5.4795

Workaround:
Exclude the following mib "lldpXMedMIB" using the CLI command:
snmp-server view restrict lldpXMedMIB excluded

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3se/release_notes/OL_30562_01.html

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuh88726/?referring_site=bugquickviewredir

Please rate if helpful.

Thank you.

High CPU caused by pdsd, iosd processes (3850 switches)