High CPU utilisation -3850 stack switch

[ad27singh]

Hi,

we have a 2 3850 stack switches with high utilisation and I will high appreciate if you could help to interpret the output from the commands.

#show processes cpu sorted
Load for five secs: 0%/0%; one minute: 1%; five minutes: 1%
No time source, *05:17:01.912 AEST Mon Apr 10 2017
Core 0: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 1: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 2: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 3: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 100%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5635 4023094 34582089 2885 74.50 69.71 69.83 1088 fed
5637 968593 64678942 1101 24.96 29.71 29.54 0 stack-mgr
6293 3318937 36186476 1184 0.15 0.14 0.15 0 cpumemd
10796 3124651 24714172 117 0.15 0.27 0.27 0 iosd
5330 2447428 35517514 91 0.05 0.02 0.03 0 system_mgr
5636 1077198 20602658 19 0.05 0.02 0.01 0 platform_mgr
6290 1666552 67871864 84 0.05 0.05 0.05 0 ffm
1 90310 22172325 4 0.00 0.00 0.00 0 init
2 0 138 0 0.00 0.00 0.00 0 kthreadd
3 418440 49548845 8 0.00 0.00 0.00 0 migration/0
4 0 3 0 0.00 0.00 0.00 0 sirq-high/0
5 1857987 28218081 15 0.00 0.00 0.00 0 sirq-timer/0
6 30 1956 15 0.00 0.00 0.00 0 sirq-net-tx/0
7 4160760 33222305 64 0.00 0.00 0.00 0 sirq-net-rx/0
8 20150 4468036 4 0.00 0.00 0.00 0 sirq-block/0
9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo
10 0 54 0 0.00 0.00 0.00 0 sirq-tasklet/0
11 3074888 29817786 5 0.00 0.00 0.00 0 sirq-sched/0
12 56700 1026597 55 0.00 0.00 0.00 0 sirq-hrtimer/0
13 4052040 21101271 52 0.00 0.00 0.00 0 sirq-rcu/0
14 412670 91088062 4 0.00 0.00 0.00 0 desched/0
15 533540 68847105 7 0.00 0.00 0.00 0 migration/1
16 0 15 0 0.00 0.00 0.00 0 sirq-high/1
17 1944850 22343111 2 0.00 0.00 0.00 0 sirq-timer/1
18 40 3022 13 0.00 0.00 0.00 0 sirq-net-tx/1
19 359878 31110959 42 0.00 0.01 0.04 0 sirq-net-rx/1
20 9690 1661298 5 0.00 0.00 0.00 0 sirq-block/1
21 0 20 0 0.00 0.00 0.00 0 sirq-block-iopo
22 0 21 0 0.00 0.00 0.00 0 sirq-tasklet/1
23 2322132 30216480 2 0.00 0.00 0.00 0 sirq-sched/1
24 0 819089 0 0.00 0.00 0.00 0 sirq-hrtimer/1
25 250897 16095561 7 0.00 0.00 0.00 0 sirq-rcu/1
26 541550 11182285 4 0.00 0.00 0.00 0 desched/1
27 528260 61480386 8 0.00 0.00 0.00 0 migration/2
28 0 27 0 0.00 0.00 0.00 0 sirq-high/2
29 2296330 10638868 0 0.00 0.00 0.00 0 sirq-timer/2
30 20 2629 7 0.00 0.00 0.00 0 sirq-net-tx/2
31 2249515 30971088 34 0.00 0.00 0.02 0 sirq-net-rx/2
32 3070 770003 3 0.00 0.00 0.00 0 sirq-block/2
33 0 32 0 0.00 0.00 0.00 0 sirq-block-iopo
34 0 33 0 0.00 0.00 0.00 0 sirq-tasklet/2
35 1165680 26090587 0 0.00 0.00 0.00 0 sirq-sched/2
36 0 726228 0 0.00 0.00 0.00 0 sirq-hrtimer/2
37 4071750 75001069 0 0.00 0.00 0.00 0 sirq-rcu/2

#show proc cpu detailed process iosd sort | ex 0.00
Load for five secs: 3%/0%; one minute: 1%; five minutes: 1%
No time source, *05:52:10.429 AEST Mon Apr 10 2017
Core 0: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 1: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 2: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 3: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
10796 L 3167591 2471683 117 0.83 0.32 0.27 0 iosd
10796 L 3 10796 2639747 4640025 0 0.83 0.28 0.24 0 iosd
6 I 4141748 1882306 0 2.99 0.44 0.33 0 Check heaps
311 I 2268888 7175766 0 0.11 0.11 0.11 0 QoS stats process

#show proc cpu detailed process fed sort | ex 0.00
Load for five secs: 0%/0%; one minute: 1%; five minutes: 1%
No time source, *05:52:31.051 AEST Mon Apr 10 2017
Core 0: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 1: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 2: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 3: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
5635 L 1280979 3458500 2885 68.39 67.18 67.48 1088 fed
5635 L 0 6089 4037297 1433410 0 22.73 16.87 17.13 0 fed-ots-main
5635 L 3 11340 1074700 1687361 0 17.85 15.55 15.69 0 Xcvr
5635 L 2 11397 806245 1079690 0 14.79 18.12 16.98 0 Replenish
5635 L 1 11399 4129381 1098857 0 10.90 14.23 14.74 0 PunjectRx
5635 L 2 11836 3288991 2955545 0 1.58 1.97 2.50 0 XcvrPoll
5635 L 0 6092 3068177 7021271 0 0.34 0.31 0.30 0 fed-ots-nfl
5635 L 3 11400 1888616 9729526 0 0.15 0.10 0.09 0 PunjectTx
5635 L 0 6120 3976454 2381474 0 0.05 0.02 0.03 0 IntrDrv

[Julio E. Moisa]

Hi

Apparently these are the processes with more cpu utilization, the issue is recognized as a bug, please check this link: https://quickview.cloudapps.cisco.com/quickview/bug/CSCuo14511

5635 4023094 34582089 2885 74.50 69.71 69.83 1088 fed
5637 968593 64678942 1101 24.96 29.71 29.54 0 stack-mgr

Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst 3850 Series Switch and is responsible for all hardware programming/forwarding.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html

Please rate the comment if it is useful

:-)