High CPU utilization on 3850 switch stack member

[zstamm]

I recently upgraded a switch stack from 03.07.04.E to 16.3.7 due to a number of bugs on the previous firmware, including high CPU utilization. I checked our monitoring software (Orion) after the install, and found that CPU #3 was still at 99%. Using the command 'show processes cpu' does not show that the CPU is being taxed.

CPU utilization for five seconds: 15%/8%; one minute: 16%; five minutes: 16%

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 34 WS-C3850-24XS 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL
2 56 WS-C3850-48U 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL
3 56 WS-C3850-48U 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL

It appears it is measuring the CPU for switch 3. First, I was under the impression that all of the processing was performed by the stack master. Why would switch 3 have high CPU usage? Second, is there a way to get more diagnostics on switch #3 regarding CPU usage?

[balaji.bandi]

Can you post the output of below commands :

#show switch detail

#show processes cpu platform

#show processes cpu | ex 0.00

[zstamm]

Here is the output of the commands that you requested:

XXXXXX#show switch detail
Switch/Stack Mac Address : 0056.2ba9.3a80 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 0056.2ba9.3a80 13 V02 Ready
2 Standby 94d4.6946.1200 11 V05 Ready
3 Member 94d4.698c.6180 9 V05 Ready

Stack Port Status Neighbors
Switch# Port 1 Port 2 Port 1 Port 2
--------------------------------------------------------
1 OK OK 2 3
2 OK OK 3 1
3 OK OK 1 2

XXXXXX#show processes cpu platform
CPU utilization for five seconds: 8%, one minute: 9%, five minutes: 10%
Core 0: CPU utilization for five seconds: 10%, one minute: 12%, five minutes: 14%
Core 1: CPU utilization for five seconds: 8%, one minute: 9%, five minutes: 10%
Core 2: CPU utilization for five seconds: 9%, one minute: 9%, five minutes: 10%
Core 3: CPU utilization for five seconds: 7%, one minute: 8%, five minutes: 10%
Core 4: CPU utilization for five seconds: 5%, one minute: 8%, five minutes: 10%
Core 5: CPU utilization for five seconds: 5%, one minute: 7%, five minutes: 10%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
1 0 0% 0% 0% S 8536064 systemd
2 0 0% 0% 0% S 0 kthreadd
3 2 0% 0% 0% S 0 ksoftirqd/0
5 2 0% 0% 0% S 0 kworker/0:0H
7 2 0% 0% 0% S 0 migration/0
8 2 0% 0% 0% S 0 rcu_bh
9 2 0% 0% 0% S 0 rcu_sched
10 2 0% 0% 0% S 0 rcuc/0
11 2 0% 0% 0% S 0 rcuc/1
12 2 0% 0% 0% S 0 migration/1
13 2 0% 0% 0% S 0 ksoftirqd/1
15 2 0% 0% 0% S 0 kworker/1:0H
16 2 0% 0% 0% S 0 rcuc/2
17 2 0% 0% 0% S 0 migration/2
18 2 0% 0% 0% S 0 ksoftirqd/2
20 2 0% 0% 0% S 0 kworker/2:0H
21 2 0% 0% 0% R 0 rcuc/3
22 2 0% 0% 0% S 0 migration/3
23 2 0% 0% 0% S 0 ksoftirqd/3
24 2 0% 0% 0% S 0 kworker/3:0
25 2 0% 0% 0% S 0 kworker/3:0H
26 2 0% 0% 0% S 0 rcuc/4
27 2 0% 0% 0% S 0 migration/4
28 2 0% 0% 0% S 0 ksoftirqd/4
29 2 0% 0% 0% S 0 kworker/4:0
30 2 0% 0% 0% S 0 kworker/4:0H
31 2 0% 0% 0% S 0 rcuc/5
32 2 0% 0% 0% S 0 migration/5
33 2 0% 0% 0% S 0 ksoftirqd/5
34 2 0% 0% 0% S 0 kworker/5:0
35 2 0% 0% 0% S 0 kworker/5:0H
36 2 0% 0% 0% S 0 khelper
37 2 0% 0% 0% S 0 kdevtmpfs
38 2 0% 0% 0% S 0 netns
39 2 0% 0% 0% S 0 irq/60-octeon-h
40 2 0% 0% 0% S 0 irq/89-octeon-h
41 2 0% 0% 0% S 0 irq/90-octeon-h
42 2 0% 0% 0% S 0 irq/94-octeon-h
43 2 0% 0% 0% S 0 irq/91-octeon-h
44 2 0% 0% 0% S 0 irq/92-octeon-h
45 2 0% 0% 0% S 0 irq/93-octeon-h
46 2 0% 0% 0% S 0 irq/95-octeon-h
47 2 0% 0% 0% S 0 irq/127-octeon-
48 2 0% 0% 0% S 0 irq/96-octeon-h
49 2 0% 0% 0% S 0 irq/98-octeon-h
50 2 0% 0% 0% S 0 irq/99-octeon-h
51 2 0% 0% 0% S 0 writeback
52 2 0% 0% 0% S 0 bioset
53 2 0% 0% 0% S 0 crypto
54 2 0% 0% 0% S 0 kblockd
55 2 0% 0% 0% S 0 ata_sff
56 2 0% 0% 0% S 0 khubd
57 2 0% 0% 0% S 0 md
58 2 0% 0% 0% S 0 edac-poller
59 2 0% 0% 0% S 0 rpciod
62 2 0% 0% 0% S 0 kworker/1:1
63 2 0% 0% 0% S 0 kswapd0
64 2 0% 0% 0% S 0 ksmd
65 2 0% 0% 0% S 0 fsnotify_mark
66 2 0% 0% 0% S 0 nfsiod
73 2 0% 0% 0% S 0 scsi_eh_0
78 2 0% 0% 0% S 0 octeon-ethernet
79 2 0% 0% 0% S 0 irq/97-octeon-h
90 2 0% 0% 0% S 0 deferwq
91 2 0% 0% 0% S 0 kworker/0:1
105 1 0% 0% 0% S 12734464 systemd-journal
113 1 0% 0% 0% S 5328896 systemd-udevd
114 2 0% 0% 0% S 0 kworker/2:1
194 2 0% 0% 0% S 0 scsi_eh_1
195 2 0% 0% 0% S 0 usb-storage
197 2 0% 0% 0% S 0 kworker/0:2
201 2 0% 0% 0% S 0 kworker/2:2
218 2 0% 0% 0% S 0 kworker/3:1H
222 2 0% 0% 0% S 0 kworker/2:1H
225 2 0% 0% 0% S 0 kworker/4:1H
227 2 0% 0% 0% S 0 kworker/4:2
230 2 0% 0% 0% S 0 kworker/0:1H
235 2 0% 0% 0% S 0 kworker/1:1H
237 2 0% 0% 0% S 0 kworker/1:2
239 2 0% 0% 0% S 0 kworker/5:1H
246 2 0% 0% 0% S 0 kworker/3:2
380 2 0% 0% 0% S 0 kworker/5:2
477 2 0% 0% 0% S 0 loop0
514 2 0% 0% 0% S 0 loop1
534 2 0% 0% 0% S 0 loop2
599 2 0% 0% 0% S 0 loop3
619 2 0% 0% 0% S 0 loop4
660 2 0% 0% 0% S 0 loop5
680 2 0% 0% 0% S 0 loop6
703 2 0% 0% 0% S 0 loop7
753 2 0% 0% 0% S 0 loop8
773 2 0% 0% 0% S 0 loop9
854 31941 2% 2% 2% S 269312000 repm
898 2 0% 0% 0% S 0 loop10
1049 1 0% 0% 0% S 6545408 droputil.sh
1050 1 0% 0% 0% S 6545408 reflector.sh
1054 1 0% 0% 0% S 4739072 oom.sh
1065 1 0% 0% 0% S 3141632 xinetd
1068 1 0% 0% 0% S 4268032 libvirtd.sh
1084 1 0% 0% 0% S 3211264 rpcbind
1115 1 0% 0% 0% S 2789376 rpc.statd
1123 1068 0% 0% 0% S 173350912 libvirtd
1132 1 0% 0% 0% S 4333568 boothelper_evt.
1135 2 0% 0% 0% S 0 lockd
1136 2 0% 0% 0% S 0 nfsd
1137 2 0% 0% 0% S 0 nfsd
1138 2 0% 0% 0% S 0 nfsd
1139 2 0% 0% 0% S 0 nfsd
1140 1132 0% 0% 0% S 2560000 inotifywait
1141 2 0% 0% 0% S 0 nfsd
1142 2 0% 0% 0% S 0 nfsd
1143 2 0% 0% 0% S 0 nfsd
1144 2 0% 0% 0% S 0 nfsd
1199 1 0% 0% 0% S 3407872 rpc.mountd
1492 1 0% 0% 0% S 13508608 rotee
1517 1054 0% 0% 0% S 2392064 sleep
1532 1 0% 0% 0% S 13508608 rotee
1651 1 0% 0% 0% S 13508608 rotee
1678 1049 0% 0% 0% S 2625536 inotifywait
1692 1 0% 0% 0% S 5423104 iptbl.sh
2010 1 0% 0% 0% S 13508608 rotee
2040 1692 0% 0% 0% S 2560000 inotifywait
2109 14436 0% 0% 0% S 5885952 pman.sh
2295 2 0% 0% 0% S 0 lsmpi-refill
2296 2 0% 1% 1% S 0 lsmpi-xmit
2297 2 0% 0% 1% S 0 lsmpi-rx
2407 1 0% 0% 0% S 13631488 rotee
2558 2109 0% 0% 0% S 4268032 shfwd.sh
2564 2558 0% 0% 0% S 1392562176 python2.7
3334 26333 0% 0% 0% S 5914624 in.telnetd.sh
3748 3334 0% 0% 0% S 11993088 in.telnetd
3760 3748 0% 0% 0% S 5914624 brelay.sh
4096 3760 0% 0% 0% S 4268032 bexec.sh
4110 4096 0% 0% 0% S 4268032 mcp_chvrf.sh
4115 4110 0% 0% 0% S 5439488 iosd_console_at
4129 12427 0% 0% 0% S 4268032 stack_sntp.sh
4143 4129 0% 0% 0% S 4038656 sntp
4598 1 0% 0% 0% S 13508608 rotee
4954 7757 0% 0% 0% S 6139904 pman.sh
5053 26333 0% 0% 0% S 5914624 in.telnetd.sh
5269 10102 0% 0% 0% S 2564096 inotifywait
5301 7757 0% 0% 0% S 6139904 pman.sh
5396 13831 0% 0% 0% S 2392064 sleep
5593 1 0% 0% 0% S 13631488 rotee
5819 17232 0% 0% 0% S 2560000 inotifywait
5917 15481 0% 0% 0% S 2392064 sleep
5965 5053 0% 0% 0% S 11993088 in.telnetd
5985 5965 0% 0% 0% S 5914624 brelay.sh
6045 1 0% 0% 0% S 13631488 rotee
6125 7757 0% 0% 0% S 6139904 pman.sh
6720 4954 0% 0% 0% S 795496448 cli_agent
6802 1 0% 0% 0% S 13631488 rotee
6864 5985 0% 0% 0% S 4268032 bexec.sh
6886 7757 0% 0% 0% S 6139904 pman.sh
6905 6864 0% 0% 0% S 4268032 mcp_chvrf.sh
6909 6905 0% 0% 0% S 5439488 iosd_console_at
6953 5301 0% 0% 0% S 511737856 cmm
7342 1 0% 0% 0% S 4268032 mcp_chvrf.sh
7345 1 0% 0% 0% S 4739072 rollback_timer.
7351 7342 0% 0% 0% S 3141632 xinetd
7394 1 0% 0% 0% S 13631488 rotee
7520 1 0% 0% 0% S 13508608 rotee
7567 6125 0% 0% 0% S 595648512 dbm
7581 1 0% 0% 0% S 6561792 psvp.sh
7627 1 0% 0% 0% S 13508608 rotee
7658 7345 0% 0% 0% S 2560000 inotifywait
7737 1 0% 0% 0% S 13631488 rotee
7753 7581 0% 0% 0% S 2625536 inotifywait
7757 1 0% 0% 0% S 6877184 pvp.sh
7833 6886 0% 0% 0% S 548839424 fman_rp
7879 1 0% 0% 0% S 13631488 rotee
8019 7757 0% 0% 0% S 2625536 inotifywait
8048 7757 0% 0% 0% S 6021120 pman.sh
8115 7757 0% 0% 0% S 6008832 pman.sh
8280 7757 0% 0% 0% S 6139904 pman.sh
8417 1 0% 0% 0% S 13508608 rotee
8430 1 0% 0% 0% S 13635584 rotee
8537 7757 0% 0% 0% S 6139904 pman.sh
8734 1 0% 0% 0% S 13508608 rotee
8846 7757 0% 0% 0% S 6017024 pman.sh
9078 8115 0% 0% 0% S 9539584 auto_upgrade_cl
9094 1 0% 0% 0% S 13508608 rotee
9353 7757 0% 0% 0% S 6017024 pman.sh
9477 1 0% 0% 0% S 13631488 rotee
9570 8280 0% 0% 0% S 13447168 auto_upgrade_se
9672 7757 0% 0% 0% S 6139904 pman.sh
9740 8048 16% 20% 23% S 1846304768 linux_iosd-imag
9768 1 0% 0% 0% S 13512704 rotee
9894 8537 0% 0% 0% S 510590976 bt_logger
9988 7757 0% 0% 0% S 6017024 pman.sh
10102 8846 0% 0% 0% S 6582272 btrace_rotate.s
10152 7757 0% 0% 0% S 6139904 pman.sh
10168 1 0% 0% 0% S 13631488 rotee
10228 7757 0% 0% 0% S 6012928 pman.sh
10314 1 0% 0% 0% S 13512704 rotee
10327 9353 0% 0% 0% S 504950784 psd
10431 1 0% 0% 0% S 13635584 rotee
10578 7757 0% 0% 0% S 6139904 pman.sh
10644 1 0% 0% 0% S 13631488 rotee
10792 1 0% 0% 0% S 13631488 rotee
10861 1 0% 0% 0% S 13635584 rotee
11008 7757 0% 0% 0% S 6025216 pman.sh
11027 9672 0% 0% 0% R 508231680 hman
11113 9988 0% 0% 0% S 644444160 smand
11155 1 0% 0% 0% S 13508608 rotee
11342 7757 0% 0% 0% S 6021120 pman.sh
11413 7757 0% 0% 0% S 6139904 pman.sh
11424 10228 0% 0% 0% S 593870848 smd
11457 10152 0% 0% 0% S 9654272 issu_stack.sh
11559 1 0% 0% 0% S 13631488 rotee
11674 1 0% 0% 0% S 13631488 rotee
11914 7757 0% 0% 0% S 6008832 pman.sh
11929 1 0% 0% 0% S 13631488 rotee
11939 10578 0% 0% 0% S 492773376 keyman
12064 11413 0% 0% 0% S 519397376 tms
12287 1 0% 0% 0% S 13631488 rotee
12319 11008 0% 0% 0% S 504651776 lman
12419 7757 0% 0% 0% S 6139904 pman.sh
12427 11342 0% 0% 0% S 9797632 ncd.sh
12446 1 0% 0% 0% S 13631488 rotee
12672 11914 0% 0% 0% S 3784294400 wcm
12722 1 0% 0% 0% S 13508608 rotee
12835 7757 0% 0% 0% S 6139904 pman.sh
12971 9078 0% 0% 0% S 2629632 inotifywait
13191 1 0% 0% 0% S 13631488 rotee
13207 7757 0% 0% 0% S 6139904 pman.sh
13257 12419 0% 0% 0% S 508813312 nif_mgr
13281 1 0% 0% 0% S 13508608 rotee
13627 1 0% 0% 0% S 13631488 rotee
13828 7757 0% 0% 0% S 6139904 pman.sh
13831 12835 0% 0% 0% S 8134656 periodic.sh
14194 7757 0% 0% 0% S 6017024 pman.sh
14209 1 0% 0% 0% S 13631488 rotee
14359 13207 0% 0% 0% S 1239363584 platform_mgr
14436 1 0% 0% 0% S 6500352 pvp.sh
14519 1 0% 0% 0% S 13508608 rotee
14735 1 0% 0% 0% S 13508608 rotee
14847 2 0% 0% 0% S 0 kworker/u12:0
14906 7757 0% 0% 0% S 6139904 pman.sh
15086 1 0% 0% 0% S 13631488 rotee
15240 13828 3% 3% 3% S 1518071808 sif_mgr
15384 7757 0% 0% 0% S 6017024 pman.sh
15481 14194 0% 0% 0% S 4796416 sort_files_by_i
15499 14436 0% 0% 0% S 2625536 inotifywait
15585 1 0% 0% 0% S 13631488 rotee
15644 7757 0% 0% 0% S 6139904 pman.sh
15926 14436 0% 0% 0% S 5885952 pman.sh
15946 1 0% 0% 0% S 13635584 rotee
15973 1 0% 0% 0% S 13508608 rotee
16332 14436 0% 0% 0% S 5885952 pman.sh
16411 15644 0% 0% 0% S 585330688 nginx
16554 14906 0% 0% 0% S 505135104 stack_mgr
16685 1 0% 0% 0% S 13631488 rotee
16971 15384 0% 0% 0% S 500035584 swift_sampled
16999 1 0% 0% 0% S 13631488 rotee
17022 7757 0% 0% 0% S 6139904 pman.sh
17232 15926 0% 0% 0% S 6238208 btrace_rotate.s
17369 16411 0% 0% 0% S 586911744 nginx
17541 1 0% 0% 0% S 13508608 rotee
17684 16332 25% 29% 32% S 2721173504 fed main event
17856 1 0% 0% 0% S 13631488 rotee
18000 17022 0% 0% 0% S 490643456 torchd
20139 7757 0% 0% 0% S 6021120 pman.sh
20407 7757 0% 0% 0% S 6139904 pman.sh
20584 1 0% 0% 0% S 13512704 rotee
20928 1 0% 0% 0% S 13512704 rotee
21266 20139 0% 0% 0% S 496267264 tamd_proc
21460 20407 0% 0% 0% S 497156096 tams_proc
22156 14436 0% 0% 0% S 5885952 pman.sh
22582 1 0% 0% 0% S 13508608 rotee
22873 22156 0% 0% 0% S 501886976 epc_ws_liaison
24502 2 0% 0% 0% S 0 kworker/u12:2
25213 1050 0% 0% 0% S 2625536 inotifywait
25243 12427 0% 0% 0% S 2625536 inotifywait
25691 11457 0% 0% 0% S 2560000 inotifywait
25693 11457 0% 0% 0% S 9519104 issu_stack.sh
25707 11457 0% 0% 0% S 2625536 inotifywait
26315 1 0% 0% 0% S 3141632 xinetd
26324 1 0% 0% 0% S 4268032 mcp_chvrf.sh
26333 26324 0% 0% 0% S 3141632 xinetd
28140 14436 0% 0% 0% S 5885952 pman.sh
28704 1 0% 0% 0% S 13631488 rotee
29411 28140 0% 0% 0% S 637480960 fman_fp_image
31704 7757 0% 0% 0% S 6025216 pman.sh
31941 7757 0% 0% 0% S 6021120 pman.sh
32049 1 0% 0% 0% S 13508608 rotee
32406 1 0% 0% 0% S 13631488 rotee
32676 31704 0% 0% 0% S 502435840 plogd

XXXXXX# show processes cpu | ex 0.00
CPU utilization for five seconds: 18%/10%; one minute: 17%; five minutes: 18%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
35 1311337 4219458 310 0.79% 0.92% 0.93% 0 ARP Input
66 158448 744816 212 0.15% 0.10% 0.10% 0 Net Background
73 456960 2008971 227 0.23% 0.34% 0.32% 0 IOSD ipc task
89 120078 727937 164 0.15% 0.09% 0.08% 0 cpf_process_tpQ
105 63098 366791 172 0.07% 0.04% 0.05% 0 ARP HA
116 855418 14150706 60 1.43% 1.04% 1.20% 0 IOSXE-RP Punt Se
135 397219 471304 842 0.31% 0.26% 0.26% 0 PLFM-MGR IPC pro
136 75836 36398 2083 0.07% 0.05% 0.06% 0 FEP background p
139 8562175 890329 9616 0.15% 0.44% 0.40% 0 NGWC L2M
190 91080 2209969 41 0.07% 0.05% 0.06% 0 VRRS Main thread
196 211176 294542 716 0.15% 0.13% 0.15% 0 CDP Protocol
204 115410 4235705 27 0.07% 0.07% 0.07% 0 IPAM Manager
207 106164 4235701 25 0.07% 0.06% 0.07% 0 IP ARP Retry Age
208 333455 2009415 165 0.39% 0.33% 0.34% 0 IP Input
226 2808420 1817805 1544 2.31% 1.99% 1.95% 0 Spanning Tree
227 213896 1451011 147 0.07% 0.13% 0.13% 0 UDLD
312 167810 710277 236 0.15% 0.15% 0.16% 0 SISF Switcher Th
313 180680 1744718 103 0.07% 0.15% 0.15% 0 SISF Main Thread
332 80385 14551 5524 0.07% 0.05% 0.05% 0 QoS stats proces
434 28830 182416 158 0.07% 0.03% 0.02% 0 DHCPD Receive
526 1343386 32835187 40 1.03% 0.91% 0.90% 0 IP SLAs XOS Even
XXXXXX#

[balaji.bandi]

May be i was over looked, as per your output..i do not see any issue here..all looks normal..

where exactly you see the issue 99%?

[zstamm]

I see it in Orion, our monitoring software.  So it is getting it from SNMP.

[balaji.bandi]

it would nice if you can post the screen shot of that to have a look, let me know what  Orion version ?

[zstamm]

Here is a screenshot of what I am seeing.

[balaji.bandi]

To be honestly, i have used Solarwinds for many years, some it not report as expected. due to Database stuck not able to co-up with many devices in the List to Monitor.

Since have seen command level all working as expected.

Time to un-manage the device after 5min Manage the device and test it.

[joseph.h.nguyen]

I agree with BB that I too seen mismatched data between systems.  I would look into your Orion.  The link below supports BB's perspective and maybe the answer for you.

https://thwack.solarwinds.com/thread/118235

Either that or you may have to update your switch to the latest version.  If that doesn't work, you may have to work with Solarwinds.