Solved: Can you please post the

bluesea2010 · ‎06-09-2015

Hi,

Cpu goes sometimes 100% , average is 40%

stack members are 3850 switch ,

Just migrated from 3560 , pvst switches to 3850

History information for system:

111111111111111111111111111111111111111111111111111111111111
666669999988888555557777799999888885555599999888885555588888
100
90
80
70
60
50
40
30
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

112222112222211121122212222221122222211121222221221222212222
981137981000199969910292423008822655598909111029009020094254
100
90
80
70
60
50
40
30 * * **** *
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

262222676777665676677226667666679
888676517035872969910677550695738
100 *
90 *
80 * * *
70 * ******** ****** **********
60 * ***#**** ****#* **********
50 * *#*###****##*#* *########*
40 * *#*###****##*#* *########*
30 *#****######***###*#***#########*
20 #################################
10 #################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

History information for core 0:

222222222211111111111111122222333331111122222222222222222222
111111111188888777778888899999555555555522222111110000077777
100
90
80
70
60
50
40 *****
30 ********** *****
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

332233324322332322333223333233233333322323222323332223324332
529720660098606167130753854512447392185165789190117891870068
100
90
80
70
60
50
40 * * * * ** * * * * * *
30 ****************************** **##*********************#***
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

474443677778776787777446666777779
630158972472110555244009999120438
100 *
90 * *
80 * ** *** *
70 * ******** ****** **********
60 * *#*###****#**#* ********#*
50 ** * *#*###****##*#* *########*
40 ******##*###****##*#****########*
30 *#****########*######**##########
20 #################################
10 #################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

History information for core 1:

111112222211111111111111122222111111111122222222221111111111
444444444444444555558888800000444446666633333333332222255555
100
90
80
70
60
50
40
30
20 ***** *************** *************** *****
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

222222223232222232222222222232222322222332222222232222222332
353869371518146224457617743033497188694115955352624935616008
100
90
80
70
60
50
40
30 * *** ***** * * *** ** * ******* ****** * ** * ** ****
20 ###############################################*############
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

373334777778776787777347777777789
348530264640540417388115014252318
100 *
90 *
80 * * ** ** ** * * **
70 * ******** ****#* **********
60 * *#*##******#*#* ********#*
50 * *#*###****##*#* *########*
40 *** *##*###****##*#* **########*
30 *#****######***###*#***#########*
20 #################################
10 #################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

History information for core 2:

111111111111111 11111111111111111111111111111111111
888885555533333666669999966666444444444499999555557777777777
100
90
80
70
60
50
40
30
20 ***** ***** ***** ********************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

122223222222312242222222321232223322323122222222232221122332
978555937020382224104866258218731382232862067291118068980407
100
90
80
70
60
50
40 * *
30 ****** * * * ***** *** *** * * * ** * ** * * ***
20 *########**####*##***#####*####*#####***##*##*##*##*#**#*###
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

1
373433677778775787777337767676770
434022953371643314051353091909040
100 *
90 *
80 * *** * * *
70 * ******** ****** **********
60 * ***#**** ****#* **********
50 * *#*###****##*#* *########*
40 * * *#*###****##*#* **########*
30 ******######***###*#***#########*
20 ##**#################*###########
10 #################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

History information for core 3:

222221111122222111112222211111 1111111111111111111111111
000005555566666333331111111111777777777722222111111111122222
100
90
80
70
60
50
40
30 *****
20 *************** ***** *****
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

222122212332222212222222222212222323223211222222212231212232
626805472603111982670124313092283671040589763454591419292134
100
90
80
70
60
50
40 * *
30 * * * ** * ** * *** ** ** * * * *
20 *****##**#*****#**##**##******######**#***#***#***#*#*****##
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

1
373343677778664687776337766676780
513751971210999920419120099706110
100 *
90 *
80 * * * **
70 * ******** ****** **********
60 * ***#**** ****#* **********
50 * * *#*###****##*#* ********#*
40 ** ** *#*###****##*#* *########*
30 ******######****##*#***#########*
20 *#****###############**##########
10 #################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

Leo Laohoo · ‎06-09-2015

Can you please post the complete output to the following commands:

1. sh version;

2. sh proc cpu sort | ex 0.00

View solution in original post

Leo Laohoo · ‎06-09-2015

Can you please post the complete output to the following commands:

1. sh version;

2. sh proc cpu sort | ex 0.00

bluesea2010 · ‎06-09-2015

#sh processes cpu | ex 0.00
Core 0: CPU utilization for five seconds: 59%; one minute: 57%; five minutes: 54%
Core 1: CPU utilization for five seconds: 56%; one minute: 60%; five minutes: 57%
Core 2: CPU utilization for five seconds: 67%; one minute: 61%; five minutes: 57%
Core 3: CPU utilization for five seconds: 57%; one minute: 56%; five minutes: 52%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5292 54610 639383 85 0.04 0.02 0.02 0 system_mgr
5597 759444 372231262 232 30.1 30.8 29.2 1088 fed
5598 369730 27063253 13 0.14 0.13 0.12 0 platform_mgr
5599 1976880 45001533 43 0.59 0.53 0.53 0 stack-mgr
6248 32340 1054798 30 0.04 0.01 0.04 0 plogd
6250 139072 74279418 59 1.83 1.60 1.53 0 pdsd
6258 184080 1479662 124 0.14 0.06 0.05 0 ffm
6259 612190 856876 714 0.41 0.14 0.08 0 eicored
6261 779670 665480 1171 0.14 0.15 0.14 0 cpumemd
6271 14900 577690 25 1.74 0.70 0.82 0 cli_agent
12709 659860 30941381 21 0.18 0.13 0.12 0 wcm
12710 26470 4265807 6 0.09 0.02 0.01 34817 table_mgr
12713 3955895 115341067 667 24.0 24.2 22.7 34816 iosd

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.02.03.SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Mon 23-Sep-13 18:24 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2013 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.2, RELEASE SOFTWARE (P)

SCR-G-W5 uptime is 1 day, 11 hours, 4 minutes
Uptime for this control processor is 1 day, 11 hours, 8 minutes
System returned to ROM by reload
System restarted at 21:24:44 UTC Mon Jun 8 2015
System image file is "flash:packages.conf"
Last reload reason: Reload command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Ipbase
License Type: Permanent
Next reload license Level: Ipbase

cisco WS-C3850-48P (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FCW1911D0Z1
2 Virtual Ethernet interfaces
208 Gigabit Ethernet interfaces
16 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
250456K bytes of Crash Files at crashinfo-2:.
250456K bytes of Crash Files at crashinfo-3:.
250456K bytes of Crash Files at crashinfo-4:.
1609272K bytes of Flash at flash:.
1609272K bytes of Flash at flash-2:.
1609272K bytes of Flash at flash-3:.
1609272K bytes of Flash at flash-4:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of Dummy USB Flash at usbflash0-2:.
0K bytes of Dummy USB Flash at usbflash0-3:.
0K bytes of Dummy USB Flash at usbflash0-4:.
0K bytes of at webui:.

Base Ethernet MAC Address : 40:a6:e8:de:d9:00
Motherboard Assembly Number : 73-14442-10
Motherboard Serial Number :
Model Revision Number : S0
Motherboard Revision Number : A0
Model Number : WS-C3850-48P
System Serial Number :

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48P 03.02.03.SE cat3k_caa-universalk9 INSTALL
2 56 WS-C3850-48P 03.02.03.SE cat3k_caa-universalk9 INSTALL
3 56 WS-C3850-48P 03.02.03.SE cat3k_caa-universalk9 INSTALL
4 56 WS-C3850-48P 03.02.03.SE cat3k_caa-universalk9 INSTALL

Switch 02
---------
Switch uptime : 1 day, 11 hours, 8 minutes
Base Ethernet MAC Address : 40:a6:e8:de:dc:00
Motherboard Assembly Number : 73-14442-10
Motherboard Serial Number :
Model Revision Number : S0
Motherboard Revision Number : A0
Model Number : WS-C3850-48P
System Serial Number :

Switch 03
---------
Switch uptime : 1 day, 11 hours, 8 minutes
Base Ethernet MAC Address : 40:a6:e8:33:aa:80
Motherboard Assembly Number : 73-14442-10
Motherboard Serial Number :
Model Revision Number : S0
Motherboard Revision Number : A0
Model Number : WS-C3850-48P
System Serial Number :

Switch 04
---------
Switch uptime : 1 day, 11 hours, 8 minutes
Base Ethernet MAC Address : 88:90:8d:1a:01:80
Motherboard Assembly Number : 73-14442-10
Motherboard Serial Number :
Model Revision Number : S0
Motherboard Revision Number : A0
Model Number : WS-C3850-48P
System Serial Number :

Configuration register is 0x102

Thanks

bluesea2010 · ‎06-09-2015

Hi

Another interesting thing , it seems phone calls are getting dropped , Cisco phones connected to the affected switches facing the issue . I think when cpu goes high , caller cannot listen to the other party

Thanks

Leo Laohoo · ‎06-09-2015

I think you need to upgrade your IOS. Try 3.6.X.

bluesea2010 · ‎06-14-2015

Did no luck

Leo Laohoo · ‎06-14-2015

Ok, you're running 3.6.X? Post the output to the following commands:

1. sh version;

2. sh proc cpu sort | ex 0.00

Charles Hill · ‎06-10-2015

Verify that command "ip cef accounting non-recursive" is not in the config.

It's not supported and will cause cpu spikes on a 3850.

CSCuq08184

bluesea2010 · ‎06-10-2015

Hi ,

There is no command like ""ip cef accounting non-recursive" in the config

Thanks

bluesea2010 · ‎06-09-2015

Hi ,

Sorry by mistake i clicked on 'correct answer '

Now don't no how o revert

Thanks

Do I have a loop in the stack ?