04-07-2010 06:42 AM - edited 03-06-2019 10:29 AM
Ok this is just weird.
I have a stack of 2 3750's (WS-C3750G-24TS-S1U) both running IOS 12.2(46)SE.
During the work day these switches are constantly at 62% CPU utilization. (this information comes from my monitoring system)
Because i wanted to find out what the main reason for this was, i logged on to the switch with ssh and constantly did a sh proc cpu sort 5sec. to find out what process the switches were working on.
Now for the strange part: When i was logged on to the switch the cpu would drop dramaticly to just 27%.
When i logged OUT of the switch i saw the cpu increase immediately (again in the monitoring system)
At first i thought it would have somthing to do with logging. So i turned off all possible logging & Debugging, but the issue remains the same.
Because a picture is worth a thousand words i added a picture from my monitoring system
Does anyone have an explanation or a solution for this behaviour?
Thanks,
Pim
04-07-2010 11:48 PM
Ok this is just weird.
I have a stack of 2 3750's (WS-C3750G-24TS-S1U) both running IOS 12.2(46)SE.
During the work day these switches are constantly at 62% CPU utilization. (this information comes from my monitoring system)
Because i wanted to find out what the main reason for this was, i logged on to the switch with ssh and constantly did a sh proc cpu sort 5sec. to find out what process the switches were working on.
Now for the strange part: When i was logged on to the switch the cpu would drop dramaticly to just 27%.
When i logged OUT of the switch i saw the cpu increase immediately (again in the monitoring system)
At first i thought it would have somthing to do with logging. So i turned off all possible logging & Debugging, but the issue remains the same.
Because a picture is worth a thousand words i added a picture from my monitoring system
Does anyone have an explanation or a solution for this behaviour?
Thanks,
Pim
Hi Pim,
Check out the below link on troubleshootin high CPU on Cisco 3750 switches
http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml
Hope to Help !!
Ganesh.H
Remember to rate the helpful post
04-08-2010 01:37 AM
Hello Ganesh,
Thanks for the reply.
I know the document but it does not explain to me why the switch is 35% busier when i am NOT logged in with a VTY line.
Pim
04-08-2010 02:06 AM
Hello Ganesh,
Thanks for the reply.
I know the document but it does not explain to me why the switch is 35% busier when i am NOT logged in with a VTY line.
Pim
Hi Pim,
In order to troubleshoot the high CPU problem we need to check the following by issueing the commnd in switch at high CPU time
First check which process is taking high cpu by command show process cpu then we can come to some point for further analysis.
Hope to Help !!
Ganesh.H
Remember to rate helpful post
04-08-2010 02:50 AM
Hello Ganesh,
Well that's just the problem. i tried that (read my original post) but when i log on to the switch with ssh or the console port the cpu drops immediately 35%.
The only way i can see the cpu is at 62% is through my monitoring system.
When is do a sh proc cpu sorted 1min immediately after i login the busiest process does not consume more than 0.17%. (See below)
So my guess would be that the the 35% Extra is at Interrupt level.
That still does not explain why it's not doing that when i login to the switch either with ssh or console port.
sh proc cpu sort 1min
CPU utilization for five seconds: 27%/22%; one minute: 57%; five minutes: 59%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
137 41077777 5652281 7267 0.00% 0.14% 0.15% 0 HQM Stack Proces
267 43759037 18032643 2426 0.00% 0.14% 0.07% 0 SNMP ENGINE
138 37829575 22594904 1674 0.15% 0.13% 0.15% 0 HRPC qos request
192 47496746 300953302 157 0.00% 0.11% 0.17% 0 Spanning Tree
4 45436147 3602719 12611 0.00% 0.10% 0.11% 0 Check heaps
95 14105374 44922008 313 0.00% 0.06% 0.05% 0 hpm counter proc
265 9971220 35293026 282 0.00% 0.06% 0.01% 0 IP SNMP
04-08-2010 04:09 AM
Ok the cpu utilization comes definately from the interrupt level.
I just logged in and immediately did a sh proc cpu sort 5sec twice in a row and see the results:
sh proc cpu sort 5sec
CPU utilization for five seconds: 45%/39%; one minute: 61%; five minutes: 62%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
137 41086091 5653308 7267 0.15% 0.15% 0.15% 0 HQM Stack Proces
59 67289051516009344 4 0.15% 0.03% 0.00% 0 Fifo Error Detec
138 37838979 22599012 1674 0.15% 0.05% 0.06% 0 HRPC qos request
192 47509352 301009620 157 0.15% 0.38% 0.25% 0 Spanning Tree
130 23577882 120024363 196 0.15% 0.04% 0.04% 0 HL3U bkgrd proce
sh proc cpu sort 5sec
CPU utilization for five seconds: 26%/20%; one minute: 56%; five minutes: 60%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
137 41086107 5653310 7267 0.15% 0.13% 0.15% 0 HQM Stack Proces
192 47509369 301009757 157 0.15% 0.33% 0.25% 0 Spanning Tree
63 6008588 31366068 191 0.15% 0.03% 0.00% 0 hrpc <- response
95 14109188 44930185 314 0.15% 0.06% 0.05% 0 hpm counter proc
40 5023895 5657153 888 0.15% 0.03% 0.00% 0 Compute load avg
Now for the question why does it drop so drasticly when the second i logon to the switch?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide