03-25-2011 05:29 AM
Hi all,
We have the problem on many 2960G Switches that the CPU is sporadically high for quite a long period of time
show logg:
000530: Jan 1 09:37:54.798: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util): 65/76%, 4/0%, 55/0%
000531: Jan 1 11:00:37.743: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 19%/0%.
000532: Jan 3 11:00:38.589: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util): 156/65%, 79/2%, 48/2%
000533: Jan 3 15:00:36.545: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 46%/0%.
000534: Jan 5 15:00:32.450: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 87%/1%, Top 3 processes(Pid/Util): 156/68%, 154/2%, 29/1%
000535: Jan 5 17:40:42.772: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000536: Jan 7 15:00:48.501: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 80%/1%, Top 3 processes(Pid/Util): 156/61%, 154/3%, 79/2%
000537: Jan 7 15:41:47.419: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000538: Jan 8 17:42:26.970: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util): 65/80%, 55/0%, 20/0%
000539: Jan 8 19:00:54.550: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 8%/0%.
000540: Jan 10 15:00:37.643: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/0%, Top 3 processes(Pid/Util): 156/63%, 154/3%, 79/2%
000541: Jan 10 19:00:35.659: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 42%/0%.
000542: Jan 11 23:00:31.699: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/1%, Top 3 processes(Pid/Util): 156/62%, 154/3%, 79/2%
000543: Jan 12 03:00:39.773: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 29%/0%.
000544: Jan 19 11:00:32.371: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/1%, Top 3 processes(Pid/Util): 156/63%, 154/3%, 79/2%
000545: Jan 19 15:00:35.372: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/0%.
000546: Jan 22 18:51:11.496: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/0%, Top 3 processes(Pid/Util): 65/76%, 116/0%, 42/0%
000547: Jan 22 19:00:37.408: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
000548: Jan 24 03:00:36.361: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util): 156/65%, 48/2%, 154/2%
000549: Jan 24 07:00:34.329: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 44%/0%.
000550: Jan 26 19:53:29.406: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 90%/0%, Top 3 processes(Pid/Util): 65/84%, 29/0%, 55/0%
000551: Jan 26 23:00:37.345: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 46%/0%.
000552: Jan 27 11:00:31.317: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util): 156/69%, 154/2%, 33/2%
000553: Jan 27 15:00:34.335: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/0%.
000554: Jan 29 23:00:31.407: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/1%, Top 3 processes(Pid/Util): 156/64%, 154/2%, 79/2%
000555: Jan 30 03:00:34.450: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000556: Feb 1 04:56:41.639: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/0%, Top 3 processes(Pid/Util): 65/75%, 55/0%, 83/0%
000557: Feb 1 07:00:33.490: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 49%/1%.
000558: Feb 6 07:00:28.202: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/1%, Top 3 processes(Pid/Util): 156/65%, 154/2%, 48/2%
000559: Feb 6 07:00:38.218: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000560: Feb 11 08:02:44.294: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 95%/0%, Top 3 processes(Pid/Util): 65/88%, 116/0%, 55/0%
000561: Feb 11 11:00:41.370: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 36%/0%.
000562: Feb 15 01:04:56.229: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util): 65/77%, 29/0%, 55/0%
000563: Feb 15 03:00:52.332: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000564: Feb 15 14:05:21.120: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 84%/0%, Top 3 processes(Pid/Util): 65/79%, 55/0%, 83/0%
000565: Feb 15 15:00:41.417: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 6%/0%.
000566: Feb 24 17:10:50.976: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util): 65/79%, 20/0%, 83/0%
000567: Feb 24 18:11:01.786: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
The Process are:
65 5701590 114171 49939 0.00% 0.00% 0.00% 0 crypto sw pk pro
156 5204927 1587723 3278 0.00% 0.44% 0.85% 0 SNMP ENGINE
We have Cisco Works with default polling interval and user tracking timers.
Does anyone konw the reson for this issue and and how to solve it.
TIA,
Emanuel
03-25-2011 07:18 PM
Post the show version and "show stack 156" output from this switch when the SNMP ENGINE is taking up a lot of CPU.
03-26-2011 10:50 AM
Hi Joseph,
Thx for your quike answer.
Here the show version:
------------------ show version ------------------
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(25)SEE4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Mon 16-Jul-07 02:53 by myl
Image text-base: 0x00003000, data-base: 0x00CC0000
ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)
l2sw-036 uptime is 23 weeks, 1 day, 22 hours, 44 minutes
System returned to ROM by power-on
System restarted at 09:14:15 MEST Mon Oct 4 2010
System image file is "flash:/c2960-lanbasek9-mz.122-25.SEE4.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
cisco WS-C2960G-48TC-L (PowerPC405) processor (revision E0) with 61440K/4088K bytes of memory.
Processor board ID FOC1230X5YX
Last reset from power-on
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:22:BD:14:34:80
Motherboard assembly number : 73-10300-09
Power supply part number : 341-0098-02
Motherboard serial number : FOC123035L7
Power supply serial number : DCA122793LV
Model revision number : E0
Motherboard revision number : A0
Model number : WS-C2960G-48TC-L
System serial number : FOC1230X5YX
Top Assembly Part Number : 800-27071-04
Top Assembly Revision Number : A0
Version ID : V04
CLEI Code Number : COM4A10BRC
Hardware Board Revision Number : 0x01
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 48 WS-C2960G-48TC-L 12.2(25)SEE4 C2960-LANBASEK9-M
Configuration register is 0xF
It also happens with 12.2(44)SE6
I scheduled the event for "show stack 156" and will post it asap.
Thanks,
Emanuel
04-11-2011 01:51 AM
I have now the Output ofshow stack 156
Switch#show stack 156
Process 156: IP SNMP
Stack segment 0x1D4DB90 - 0x1D50A70
FP: 0x1D50A20, RA: 0xA0483C
FP: 0x1D50A60, RA: 0xC710C
FP: 0x1D50A68, RA: 0xA3B270
FP: 0x0, RA: 0xA317D0
The strange thing is, that CPU is not really high, but no falling trap is seen:
000167: Apr 29 06:24:31.563: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 89%/0%, Top 3 processes(Pid/Util): 150/82%, 90/0%, 55/0%
000168: Apr 29 08:01:10.831: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 32%/0%.
000630: Apr 1 12:00:49.778: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/0%, Top 3 processes(Pid/Util): 158/64%, 156/3%, 79/2%
Switch#sh clock
13:02:11.075 MEST Fri Apr 1 2011
Switch#show proc cpu
CPU utilization for five seconds: 5%/0%; one minute: 5%; five minutes: 5%
Switch#show proc cpu history
5555555555555555555555555555555555555555555555555666655555
100
90
80
70
60
50
40
30
20
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
2 22
6666556761656665665566666533666666666566556665566555666666
100
90
80
70
60
50
40
30
20 * **
10 ##################*######################*################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1
8532635353366224531462615015632273437324541373246122632243426214522172
1179020815345405608075069083337907548660727497666976926872196971078808
100 *
90 *
80 * * * *
70 * * * * * * * * * * *
60 * * ** * * * ** * * * * * * * * *
50 ** * * * ** ** * * ** ** * * * * * ** * * * * *
40 *** * **** ** ** ** * ** ** *** ** *** ** ** * * * * ** *
30 ************* *** **** ** *************** ****** ************ **** **
20 **********************************************************************
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
04-11-2011 10:39 PM
This trace shows a completely idle IP SNMP process. You'll need to capture the stack at the moment IP SNMP is taking up a large amount of CPU. I'd suggest to use EEM to do this, but EEM is not available on the Cat2K switches. That means you'll need to be fast.
05-23-2011 10:39 PM
We installed several Switches (also 2960) with a different IOS (122-50.SE5) and the prtoblem is not seen.
The CPURISING and FALLING traps also appearing but it seems more reliable, at least most of the time.
Here is the log:
000260: May 2 03:50:53.693: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 84%/0%, Top 3 processes(Pid/Util): 217/78%, 79/0%, 110/0%
000261: May 2 04:00:49.634: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
000262: May 2 12:00:40.506: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/0%, Top 3 processes(Pid/Util): 191/56%, 4/7%, 106/4%
000263: May 2 13:51:16.063: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 6%/0%.
000264: May 3 20:00:38.995: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/0%, Top 3 processes(Pid/Util): 191/63%, 106/3%, 70/2%
000265: May 3 23:52:11.086: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
000266: May 4 00:32:09.914: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 95%/0%, Top 3 processes(Pid/Util): 191/79%, 106/4%, 50/2%
000267: May 4 00:33:05.078: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 6%/0%.
000320: May 11 00:32:07.921: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 90%/1%, Top 3 processes(Pid/Util): 191/73%, 106/5%, 156/1%
000321: May 11 00:33:03.185: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 22%/0%.
000322: May 18 00:32:11.516: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 80%/0%, Top 3 processes(Pid/Util): 191/63%, 106/5%, 4/2%
000323: May 18 00:33:06.798: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 18%/0%.
I think we are facing a cosmetic problem here.
During the interval of a CPURISING and a CPUFALLING trap the switch has no high CPU ussage. In my opinion just the FALLING Trap is seen way to late. Does anyone konw something about it ? / ever heard ?
05-23-2011 11:04 PM
The problem may be that the spikes are happening too quickly to catch. Typically, we'd use EEM to capture the show stack output, but the 2900s don't support full EEM. If you can be logged in on the device at the time the spike occurs, seeing that show stack for PID 191 would be telling. I don't see any known bugs for bogus CPURISINGTHRESHOLD messages, so either this is a new issue, or something is taxing the SNMP engine.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: