08-19-2010 03:34 AM - edited 03-06-2019 12:33 PM
Hello,
i have a problem on C3560-48PS Switch. It have spontaneously a CPU usage of 100%.
Version 12.2(53)SE2
Image C3560-IPBASEK9-M
Switch#show processes cpu history
11111 11111 1111111
8000008888899999000008888899999666669999977777777772222200
100
90
80
70
60
50
40
30
20
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1111111411111551311111111251111111111111111113121111111111
4768767787677036189757767146768887676897767788876866679768
100
90
80
70
60
50 * ** *
40 * ** * *
30 * ** * * * *
20 ******#*****##*###******##****##**#**###**#####*#***##**#
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1 1
5557555555055555505548555855555555455555555553555555556565555555555555
4414204230002132401095220100122040411321320027101311419360123311269472
100 * *
90 * * *
80 * * * *
70 * * * * * * *
60 * * * * * * * ** *
50 ********************************** ********** ************************
40 **********************************************************************
30 **********************************************************************
20 ######################**#*###################*######################**
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
When i look at the processes, i find "REP LSL Hello PP" at over 7,01%
Switch#show processes cpu sorted
CPU utilization for five seconds: 9%/0%; one minute: 14%; five minutes: 15%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
248 173 10068 17 0.95% 0.12% 0.02% 1 Virtual Exec
197 59494892 56599704 1051 0.79% 0.52% 0.53% 0 Spanning Tree
234 70914044 21489129 3299 0.79% 0.55% 0.56% 0 Marvell wk-a Pow
125 45757496 251358992 182 0.47% 0.38% 0.36% 0 Hulc LED Process
34 16809346 8573236 1960 0.31% 0.14% 0.12% 0 Per-Second Jobs
219 8201334892686047037 0 0.31% 6.39% 7.01% 0 REP LSL Hello PP
35 6436343 430927 14936 0.31% 0.04% 0.00% 0 Per-minute Jobs
211 754 13644 55 0.15% 0.01% 0.00% 0 TCP Timer
191 1109086 8875447 124 0.15% 0.03% 0.00% 0 IP Input
134 5563918 3448322 1613 0.15% 0.03% 0.01% 0 HRPC qos request
11 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API
10 993484 5008527 198 0.00% 0.00% 0.00% 0 ARP Input
13 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
I got this evaluation for one week. And I have the process for the 100% CPU load is not yet discovered.
With've noticed that the load to 8% - 9% decrease as soon as I connect via Telnet and stay looged in over 5 minutes.
Thanks in advance
08-19-2010 06:57 AM
Haven't done it myself yet but did you consider writing an EEM script that checks the CPU load periodically and if it goes beyond 90% or so sends you an email with the content of 'sh proc cpu sorted'?
I do not know how much the script adds to the CPU load. you might want to try this by changing the interval slowly observing the CPU load at the same time.
HTH
mat
08-19-2010 04:38 PM
EEM script would work.
Also, attach your config. As you see its a fast spike, it doesn't increase the avg utilization. It may be hard to catch, but mabey try removing some features and waiting to see if it happens.
08-20-2010 12:41 AM
Thanks for the Answers,
i find an interessting inI found an interesting information. And this problem is officially known, but unfortunately no solution.
CSCth24278 (Catalyst 2960-S switch)
The CPU utilization on the switch remains high (50 to 60 percent) when the switch is not being accessed by a telnet or a console session. When you telnet or console into the switch, the CPU utilization goes down.
There is no workaround.
I have the same problem at 40x 3560 Devices with 12.2(53)SE2
After a downgrade to the stable 12.2(46), the problem has gone away. And i have enought Features for our Enviroment.
But the tip with the EEM I find really helpful and interessting for other troubleshootings
Thanks for the help
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide