Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1212
Views
0
Helpful
6
Replies

Switching High CPU troubleshooting on cisco 3850

ratha chum
Level 1
Level 1

‎09-09-2019 12:58 AM

What is service is consume CPU utilization?

 

#show processes cpu
Core 0: CPU utilization for five seconds: 95%; one minute: 84%; five minutes: 83%
Core 1: CPU utilization for five seconds: 69%; one minute: 48%; five minutes: 55%
Core 2: CPU utilization for five seconds: 26%; one minute: 37%; five minutes: 39%
Core 3: CPU utilization for five seconds: 27%; one minute: 51%; five minutes: 44%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 1650 1055 1563 0.00 0.00 0.00 0 init
2 0 124 0 0.00 0.00 0.00 0 kthreadd
3 357980 31002638 11 0.00 0.04 0.04 0 migration/0
4 0 3 0 0.00 0.00 0.00 0 sirq-high/0
5 317200 418583733 15 0.00 0.00 0.00 0 sirq-timer/0
6 0 8 0 0.00 0.00 0.00 0 sirq-net-tx/0
7 20 31091 0 0.00 0.00 0.00 0 sirq-net-rx/0
8 4560 1130720 4 0.00 0.00 0.00 0 sirq-block/0
9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo
10 110 22571 4 0.00 0.00 0.00 0 sirq-tasklet/0
11 1406995 127737908 7 0.00 0.00 0.00 0 sirq-sched/0
12 24980 679127 36 0.00 0.00 0.00 0 sirq-hrtimer/0
13 2459054 386030819 51 0.00 0.00 0.00 0 sirq-rcu/0
14 281930 48680670 5 0.00 0.00 0.00 0 desched/0
15 912740 81939360 11 0.00 0.00 0.00 0 migration/1
16 0 15 0 0.00 0.00 0.00 0 sirq-high/1
17 2585045 421177288 2 0.00 0.02 0.01 0 sirq-timer/1
18 0 21 0 0.00 0.00 0.00 0 sirq-net-tx/1
19 100 178331 0 0.00 0.00 0.00 0 sirq-net-rx/1
20 5740 849480 6 0.00 0.00 0.00 0 sirq-block/1
21 0 20 0 0.00 0.00 0.00 0 sirq-block-iopo
22 0 21 0 0.00 0.00 0.00 0 sirq-tasklet/1
23 409732 192572290 2 0.00 0.00 0.00 0 sirq-sched/1
24 0 562645 0 0.00 0.00 0.00 0 sirq-hrtimer/1
25 2452310 365932018 5 0.00 0.00 0.00 0 sirq-rcu/1
26 396890 61768449 6 0.00 0.00 0.00 0 desched/1
27 1736700 120663411 14 0.00 0.00 0.00 0 migration/2
28 0 27 0 0.00 0.00 0.00 0 sirq-high/2
29 1209670 352585715 0 0.00 0.01 0.00 0 sirq-timer/2
30 0 29 0 0.00 0.00 0.00 0 sirq-net-tx/2
31 100 272190 0 0.00 0.00 0.00 0 sirq-net-rx/2
32 1910 331011 5 0.00 0.00 0.00 0 sirq-block/2
33 0 32 0 0.00 0.00 0.00 0 sirq-block-iopo
34 0 33 0 0.00 0.00 0.00 0 sirq-tasklet/2
35 603380 176650451 0 0.00 0.00 0.00 0 sirq-sched/2
36 0 474975 0 0.00 0.00 0.00 0 sirq-hrtimer/2
37 2703580 325458230 0 0.00 0.00 0.00 0 sirq-rcu/2
38 497720 89008177 5 0.00 0.00 0.00 0 desched/2
39 2139300 89284609 23 0.00 0.00 0.00 0 migration/3
40 0 39 0 0.00 0.00 0.00 0 sirq-high/3
41 420 332008359 0 0.00 0.01 0.04 0 sirq-timer/3
42 0 43 0 0.00 0.00 0.00 0 sirq-net-tx/3
43 90 221403 0 0.00 0.00 0.00 0 sirq-net-rx/3
44 1350 206149 6 0.00 0.00 0.00 0 sirq-block/3
45 0 44 0 0.00 0.00 0.00 0 sirq-block-iopo
46 0 45 0 0.00 0.00 0.00 0 sirq-tasklet/3
47 260 167186790 0 0.00 0.00 0.00 0 sirq-sched/3
48 0 411797 0 0.00 0.00 0.00 0 sirq-hrtimer/3
49 200 317243363 0 0.00 0.00 0.00 0 sirq-rcu/3
50 631840 110182643 5 0.00 0.00 0.00 0 desched/3
51 881140 45791454 19 0.00 0.00 0.00 0 events/0
52 202650 44916852 4 0.00 0.00 0.03 0 events/1
53 44100 42166393 1 0.00 0.00 0.00 0 events/2
54 1190 83774441 0 0.00 0.04 0.00 0 events/3
55 30 2971 10 0.00 0.00 0.00 0 khelper
58 0 55 0 0.00 0.00 0.00 0 netns
60 0 60 0 0.00 0.00 0.00 0 async/mgr
158 281080 7027568 39 0.00 0.00 0.00 0 sync_supers
160 149260 8433508 17 0.00 0.00 0.00 0 bdi-default
162 29670 935308 31 0.00 0.00 0.00 0 kblockd/0
163 4530 714036 6 0.00 0.00 0.00 0 kblockd/1
164 1120 276642 4 0.00 0.00 0.00 0 kblockd/2
165 430 171584 2 0.00 0.00 0.00 0 kblockd/3
171 0 69 0 0.00 0.00 0.00 0 ata/0
172 0 68 0 0.00 0.00 0.00 0 ata/1
173 0 65 0 0.00 0.00 0.00 0 ata/2
174 0 66 0 0.00 0.00 0.00 0 ata/3
175 0 67 0 0.00 0.00 0.00 0 ata_aux
183 0 70 0 0.00 0.00 0.00 0 khubd
209 0 88 0 0.00 0.00 0.00 0 rpciod/0
210 0 88 0 0.00 0.00 0.00 0 rpciod/1
211 0 75 0 0.00 0.00 0.00 0 rpciod/2
212 0 78 0 0.00 0.00 0.00 0 rpciod/3
228 0 77 0 0.00 0.00 0.00 0 kswapd0
229 0 78 0 0.00 0.00 0.00 0 aio/0
230 0 79 0 0.00 0.00 0.00 0 aio/1
231 0 80 0 0.00 0.00 0.00 0 aio/2
232 0 81 0 0.00 0.00 0.00 0 aio/3
233 0 82 0 0.00 0.00 0.00 0 nfsiod
235 0 83 0 0.00 0.00 0.00 0 crypto/0
236 0 84 0 0.00 0.00 0.00 0 crypto/1
237 0 85 0 0.00 0.00 0.00 0 crypto/2
238 0 86 0 0.00 0.00 0.00 0 crypto/3
788 14020 2791939 5 0.00 0.00 0.02 0 pata_octeon_cf
789 0 99 0 0.00 0.00 0.00 0 scsi_eh_0
795 7450 982 7586 0.00 0.00 0.00 0 mtdblockd
824 2149290 71306259 30 0.00 0.00 0.00 0 octeon-ethernet
858 0 96 0 0.00 0.00 0.00 0 kstriped
860 0 97 0 0.00 0.00 0.00 0 ksnapd
938 410 1239 330 0.00 0.00 0.00 0 udevd
1254 194680 8433489 23 0.00 0.00 0.00 0 flush-8:0
1574 210 4404 47 0.00 0.00 0.00 0 loop1
1599 60 4265 14 0.00 0.00 0.00 0 loop2
1624 50 2139 23 0.00 0.00 0.00 0 loop3
1649 110 2893 38 0.00 0.00 0.00 0 loop4
1674 110 2442 45 0.00 0.00 0.00 0 loop5
1759 20 609 32 0.00 0.00 0.00 0 loop6
2478 0 1036 0 0.00 0.00 0.00 0 udevd
2479 0 1013 0 0.00 0.00 0.00 0 udevd
3339 120 1109 108 0.00 0.00 0.00 0 dbus-daemon
3717 150 1573 95 0.00 0.00 0.00 0 klogd
3729 0 974 0 0.00 0.00 0.00 0 portmap
3798 10 955 10 0.00 0.00 0.00 0 sshd
3810 0 946 0 0.00 0.00 0.00 0 xinetd
3814 10 947 10 0.00 0.00 0.00 0 xinetd
3848 0 112 0 0.00 0.00 0.00 0 lockd
3849 240 11829 20 0.00 0.00 0.00 0 nfsd
3850 300 11829 25 0.00 0.00 0.00 0 nfsd
3851 250 11831 21 0.00 0.00 0.00 0 nfsd
3852 270 11831 22 0.00 0.00 0.00 0 nfsd
3853 240 11833 20 0.00 0.00 0.00 0 nfsd
3854 280 11832 23 0.00 0.00 0.00 0 nfsd
3855 240 11834 20 0.00 0.00 0.00 0 nfsd
3856 310 11836 26 0.00 0.00 0.00 0 nfsd
3862 0 977 0 0.00 0.00 0.00 0 rpc.mountd
5412 2108110 208742352 92 0.05 0.02 0.03 0 system_mgr
5641 1032850 46378918 22 0.00 0.00 0.00 1088 slproc
5717 379773 363247097 1470 27.0 27.0 27.0 1088 fed
5718 1099380 328652777 19 0.05 0.09 0.09 0 platform_mgr
5719 2763268 208279396 1044 3.98 3.83 3.81 0 stack-mgr
6228 3989622 84189964 98 0.00 0.00 0.00 0 idope.py
6344 3523160 170816213 20 0.00 0.00 0.00 0 ns_oir_proxy
6345 3516880 170825170 20 0.00 0.01 0.03 0 profiled
6348 3616940 3961425 913 0.00 0.02 0.02 0 oom_poll.sh
6349 3539582 295978087 26 0.00 0.01 0.03 0 obfld
6350 3492210 170825303 20 0.00 0.01 0.04 0 consoled
6351 2112845 214508901 49 0.00 0.01 0.00 0 console_relay
6352 3488230 170825231 20 0.00 0.03 0.04 0 csprovider
6353 3502260 171246751 20 0.00 0.00 0.00 0 system_mgr_cli
6354 4143305 348212037 36 0.00 0.02 0.00 0 plogd
6355 3514250 170825835 20 0.00 0.00 0.03 0 psdprov
6356 3611930 346867418 1995 19.1 19.9 20.3 0 pdsd
6357 278242 179972800 25 0.00 0.01 0.03 0 gold_slave
6360 2878242 175742353 40 0.00 0.01 0.16 0 osinfo-provider
6361 3517650 170825272 20 0.00 0.02 0.00 0 oscore_p
6362 159692 175334014 25 0.00 0.00 0.01 0 netd
6363 2734220 208705915 95 0.00 0.00 0.03 0 mem_mgmt
6364 458372 212963537 22 0.00 0.00 0.00 0 mgmte_tap
6365 3538200 170851791 20 0.00 0.01 0.03 0 installer
6366 2007635 249426825 42 0.00 0.00 0.04 0 ffm
6368 1442098 206410606 610 0.00 0.49 0.48 0 eicored
6370 3539530 172235276 20 0.00 0.00 0.00 0 dtmgr
6371 3229786 212714593 1186 0.14 0.15 0.14 0 cpumemd
6377 445132 213666493 22 0.00 0.00 0.02 0 liin_tap
6388 3539920 171047581 20 0.00 0.01 0.04 0 cli_agent
6392 3171552 539907702 13 0.05 0.01 0.01 0 snmp_subagent
6394 228772 212972890 21 0.00 0.01 0.01 0 erm_p
6396 171922 212973456 20 0.00 0.00 0.00 0 ngdumper_provid
6397 152752 212973282 20 0.00 0.00 0.00 0 sysinfo_p
6411 1033380 46364749 22 0.00 0.00 0.00 0 fs_cmd
8605 440 1412 311 0.00 0.00 0.00 0 rollback_timer.
8809 0 1402 0 0.00 0.00 0.00 0 rollback_timer.
8811 190 1402 135 0.00 0.00 0.00 0 xlogger
8819 0 1406 0 0.00 0.00 0.00 0 inotifywait
8992 150 1175 127 0.00 0.00 0.00 0 nova_sntp.sh
9142 0 1173 0 0.00 0.00 0.00 0 nova_sntp.sh
9143 160 1173 136 0.00 0.00 0.00 0 xlogger
9146 0 1179 0 0.00 0.00 0.00 0 sntp
9383 4125241 161088652 30 0.10 0.09 0.10 0 wcm
9384 3530496 146313973 20 0.00 0.03 0.03 0 table_mgr
9385 80 1420 56 0.00 0.00 0.00 0 wnweb.sh
9386 3504670 170829083 20 0.00 0.01 0.03 0 iifd
9387 4038839 258393296 156 4.17 3.82 3.30 0 iosd
9388 3538250 171253293 20 0.00 0.03 0.02 0 ha_mgr
10550 2148955 208784774 51 0.00 0.00 0.00 0 wnweb_paster.py
10551 370 1420 260 0.00 0.00 0.00 0 xlogger
24611 0 3961426 0 0.00 0.00 0.00 0 sleep
24710 10 519758360 0 0.00 0.00 0.00 0 sh

 

Thanks,

Labels:
0 Helpful
6 Replies 6

Georg Pauwen
VIP
VIP

‎09-09-2019 01:38 AM

Hello,

 

post the full running configuration of the switch, as well as the output of:

 

show proc cpu | exclude 0.00

 

so that you can only see the processes that actually use CPU cycles.

 

From what I can see, the 'iosd' and 'fed' processes are high, which usually means high interrupts.

 

Also, check the troubleshooting procedure outlined in the link below:

 

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html

0 Helpful

ratha chum
Level 1
Level 1
In response to Georg Pauwen

‎09-09-2019 02:19 AM

show processes cpu sort | exclude 0.0
Core 0: CPU utilization for five seconds: 82%; one minute: 81%; five minutes: 82%
Core 1: CPU utilization for five seconds: 96%; one minute: 50%; five minutes: 50%
Core 2: CPU utilization for five seconds: 26%; one minute: 30%; five minutes: 41%
Core 3: CPU utilization for five seconds: 15%; one minute: 60%; five minutes: 48%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5717 251676 36349739 1470 27.04 26.93 26.95 1088 fed
6356 2591563 34696505 1995 20.22 20.64 20.58 0 pdsd
5719 2946368 20829842 1044 3.94 3.84 3.81 0 stack-mgr
9387 172322 26025372 156 3.70 3.24 3.24 0 iosd
5718 1111530 32945969 19 0.14 0.12 0.11 0 platform_mgr
6371 3252906 21273399 1186 0.14 0.14 0.14 0 cpumemd
9383 4144981 16118078 30 0.10 0.10 0.10 0 wcm
0 Helpful

Leo Laohoo
Hall of Fame
Hall of Fame

‎09-09-2019 02:02 AM

Post the complete output to the following commands:
1. sh proc cpu sort | e 0.00; and
2. sh version
0 Helpful

ratha chum
Level 1
Level 1
In response to Leo Laohoo

‎09-09-2019 02:20 AM

show processes cpu sort | exclude 0.0
Core 0: CPU utilization for five seconds: 82%; one minute: 81%; five minutes: 82%
Core 1: CPU utilization for five seconds: 96%; one minute: 50%; five minutes: 50%
Core 2: CPU utilization for five seconds: 26%; one minute: 30%; five minutes: 41%
Core 3: CPU utilization for five seconds: 15%; one minute: 60%; five minutes: 48%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5717 251676 36349739 1470 27.04 26.93 26.95 1088 fed
6356 2591563 34696505 1995 20.22 20.64 20.58 0 pdsd
5719 2946368 20829842 1044 3.94 3.84 3.81 0 stack-mgr
9387 172322 26025372 156 3.70 3.24 3.24 0 iosd
5718 1111530 32945969 19 0.14 0.12 0.11 0 platform_mgr
6371 3252906 21273399 1186 0.14 0.14 0.14 0 cpumemd
9383 4144981 16118078 30 0.10 0.10 0.10 0 wcm


#show version
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.02.03.SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Mon 23-Sep-13 18:24 by prod_rel_team



Cisco IOS-XE software, Copyright (c) 2005-2013 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.



ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.18, RELEASE SOFTWARE (P)

SW-CORE-DC1 uptime is 1 year, 17 weeks, 4 days, 1 hour, 32 minutes
Uptime for this control processor is 1 year, 17 weeks, 4 days, 1 hour, 35 minutes
System returned to ROM by reload at 01:30:31 ICT Fri Mar 23 2018
System restarted at 14:53:18 ICT Wed May 9 2018
System image file is "flash:packages.conf"
Last reload reason: Reload command



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Ipbase
License Type: Permanent
Next reload license Level: Ipbase

cisco WS-C3850-48T (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FCW1901D01B
3 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of at webui:.

Base Ethernet MAC Address : 00:ca:e5:6b:36:00
Motherboard Assembly Number : 73-16296-07
Motherboard Serial Number : FOC20094VHU
Model Revision Number : Z0
Motherboard Revision Number : B0

0 Helpful

Georg Pauwen
VIP
VIP
In response to ratha chum

‎09-09-2019 03:37 AM

Hello,

 

what do you have in your configuration (post 'show run) ? Anything such as access list logging could punt packets to the CPU. 

0 Helpful
Customers Also Viewed These Support Documents