cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2075
Views
0
Helpful
6
Replies
e.hoehn
Beginner

WS-C2960G process cpu threshold not falling for a long period

Hi all,

We have the problem on many 2960G Switches that the CPU is sporadically high for quite a long period of time

show logg:

000530: Jan  1 09:37:54.798: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util):  65/76%, 4/0%, 55/0%
000531: Jan  1 11:00:37.743: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 19%/0%.
000532: Jan  3 11:00:38.589: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util):  156/65%, 79/2%, 48/2%
000533: Jan  3 15:00:36.545: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 46%/0%.
000534: Jan  5 15:00:32.450: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 87%/1%, Top 3 processes(Pid/Util):  156/68%, 154/2%, 29/1%
000535: Jan  5 17:40:42.772: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000536: Jan  7 15:00:48.501: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 80%/1%, Top 3 processes(Pid/Util):  156/61%, 154/3%, 79/2%
000537: Jan  7 15:41:47.419: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000538: Jan  8 17:42:26.970: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util):  65/80%, 55/0%, 20/0%
000539: Jan  8 19:00:54.550: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 8%/0%.
000540: Jan 10 15:00:37.643: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/0%, Top 3 processes(Pid/Util):  156/63%, 154/3%, 79/2%
000541: Jan 10 19:00:35.659: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 42%/0%.
000542: Jan 11 23:00:31.699: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/1%, Top 3 processes(Pid/Util):  156/62%, 154/3%, 79/2%
000543: Jan 12 03:00:39.773: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 29%/0%.
000544: Jan 19 11:00:32.371: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/1%, Top 3 processes(Pid/Util):  156/63%, 154/3%, 79/2%
000545: Jan 19 15:00:35.372: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/0%.
000546: Jan 22 18:51:11.496: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/0%, Top 3 processes(Pid/Util):  65/76%, 116/0%, 42/0%
000547: Jan 22 19:00:37.408: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
000548: Jan 24 03:00:36.361: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util):  156/65%, 48/2%, 154/2%
000549: Jan 24 07:00:34.329: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 44%/0%.
000550: Jan 26 19:53:29.406: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 90%/0%, Top 3 processes(Pid/Util):  65/84%, 29/0%, 55/0%
000551: Jan 26 23:00:37.345: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 46%/0%.
000552: Jan 27 11:00:31.317: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util):  156/69%, 154/2%, 33/2%
000553: Jan 27 15:00:34.335: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/0%.
000554: Jan 29 23:00:31.407: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/1%, Top 3 processes(Pid/Util):  156/64%, 154/2%, 79/2%
000555: Jan 30 03:00:34.450: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000556: Feb  1 04:56:41.639: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/0%, Top 3 processes(Pid/Util):  65/75%, 55/0%, 83/0%
000557: Feb  1 07:00:33.490: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 49%/1%.
000558: Feb  6 07:00:28.202: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/1%, Top 3 processes(Pid/Util):  156/65%, 154/2%, 48/2%
000559: Feb  6 07:00:38.218: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000560: Feb 11 08:02:44.294: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 95%/0%, Top 3 processes(Pid/Util):  65/88%, 116/0%, 55/0%
000561: Feb 11 11:00:41.370: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 36%/0%.
000562: Feb 15 01:04:56.229: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 83%/0%, Top 3 processes(Pid/Util):  65/77%, 29/0%, 55/0%
000563: Feb 15 03:00:52.332: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.
000564: Feb 15 14:05:21.120: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 84%/0%, Top 3 processes(Pid/Util):  65/79%, 55/0%, 83/0%
000565: Feb 15 15:00:41.417: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 6%/0%.
000566: Feb 24 17:10:50.976: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util):  65/79%, 20/0%, 83/0%
000567: Feb 24 18:11:01.786: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 5%/0%.

The Process are:

  65     5701590    114171      49939  0.00%  0.00%  0.00%   0 crypto sw pk pro

156     5204927   1587723       3278  0.00%  0.44%  0.85%   0 SNMP ENGINE    

We have Cisco Works with default polling interval and user tracking timers.

Does anyone konw the reson for this issue and and how to solve it.

TIA,

Emanuel

6 REPLIES 6
Joe Clarke
Hall of Fame Cisco Employee

Post the show version and "show stack 156" output from this switch when the SNMP ENGINE is taking up a lot of CPU.

Hi Joseph,

Thx for your quike answer.

Here the show version:

------------------ show version ------------------

Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(25)SEE4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Mon 16-Jul-07 02:53 by myl
Image text-base: 0x00003000, data-base: 0x00CC0000

ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)

l2sw-036 uptime is 23 weeks, 1 day, 22 hours, 44 minutes
System returned to ROM by power-on
System restarted at 09:14:15 MEST Mon Oct 4 2010
System image file is "flash:/c2960-lanbasek9-mz.122-25.SEE4.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960G-48TC-L (PowerPC405) processor (revision E0) with 61440K/4088K bytes of memory.
Processor board ID FOC1230X5YX
Last reset from power-on
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:22:BD:14:34:80
Motherboard assembly number     : 73-10300-09
Power supply part number        : 341-0098-02
Motherboard serial number       : FOC123035L7
Power supply serial number      : DCA122793LV
Model revision number           : E0
Motherboard revision number     : A0
Model number                    : WS-C2960G-48TC-L
System serial number            : FOC1230X5YX
Top Assembly Part Number        : 800-27071-04
Top Assembly Revision Number    : A0
Version ID                      : V04
CLEI Code Number                : COM4A10BRC
Hardware Board Revision Number  : 0x01


Switch   Ports  Model              SW Version              SW Image           
------   -----  -----              ----------              ----------         
*    1   48     WS-C2960G-48TC-L   12.2(25)SEE4            C2960-LANBASEK9-M 


Configuration register is 0xF

It also happens with 12.2(44)SE6

I scheduled the event for "show stack 156" and will post it asap.

Thanks,

Emanuel

I have now the Output ofshow stack 156

Switch#show stack 156
Process 156:  IP SNMP
  Stack segment 0x1D4DB90 - 0x1D50A70
  FP: 0x1D50A20, RA: 0xA0483C
  FP: 0x1D50A60, RA: 0xC710C
  FP: 0x1D50A68, RA: 0xA3B270
  FP: 0x0, RA: 0xA317D0

The strange thing is, that CPU is not really high, but no falling trap is seen:

000167: Apr 29 06:24:31.563: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 89%/0%, Top 3 processes(Pid/Util):  150/82%, 90/0%, 55/0%
000168: Apr 29 08:01:10.831: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 32%/0%.
000630: Apr  1 12:00:49.778: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/0%, Top 3 processes(Pid/Util):  158/64%, 156/3%, 79/2%


Switch#sh clock
13:02:11.075 MEST Fri Apr 1 2011

Switch#show proc cpu
CPU utilization for five seconds: 5%/0%; one minute: 5%; five minutes: 5%

Switch#show proc cpu history


     5555555555555555555555555555555555555555555555555666655555
100
90
80
70
60
50
40
30
20
10  **********************************************************
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)


              2                22
     6666556761656665665566666533666666666566556665566555666666
100
90
80
70
60
50
40
30
20           *                **
10  ##################*######################*################
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                              1
     8532635353366224531462615015632273437324541373246122632243426214522172
     1179020815345405608075069083337907548660727497666976926872196971078808
100                           *
90                           *
80  *                        *          *       *
70  *           *       *    *      *   *       *   *   *       *       *
60  *   *      **   *   * * **  *   *   *   *   *   *   *       *       *
50  **  * * *  **  **   * * ** **   * * *   *   *  **   *   *   *   *   *
40  *** * **** **  **  ** * ** **   *** ** ***  ** **   *   * * *  **   *
30  *************  *** **** ** *************** ****** ************ **** **
20  **********************************************************************
10  ######################################################################
    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
              0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Joe Clarke
Hall of Fame Cisco Employee

This trace shows a completely idle IP SNMP process.  You'll need to capture the stack at the moment IP SNMP is taking up a large amount of CPU.  I'd suggest to use EEM to do this, but EEM is not available on the Cat2K switches.  That means you'll need to be fast.

We installed several Switches (also 2960) with a different IOS (122-50.SE5) and the prtoblem is not seen.

The CPURISING and FALLING traps also appearing but it seems more reliable, at least most of the time.

Here is the log:

000260: May  2 03:50:53.693: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 84%/0%, Top 3 processes(Pid/Util):  217/78%, 79/0%, 110/0%
000261: May  2 04:00:49.634: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
000262: May  2 12:00:40.506: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 82%/0%, Top 3 processes(Pid/Util):  191/56%, 4/7%, 106/4%
000263: May  2 13:51:16.063: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 6%/0%.
000264: May  3 20:00:38.995: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/0%, Top 3 processes(Pid/Util):  191/63%, 106/3%, 70/2%
000265: May  3 23:52:11.086: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
000266: May  4 00:32:09.914: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 95%/0%, Top 3 processes(Pid/Util):  191/79%, 106/4%, 50/2%
000267: May  4 00:33:05.078: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 6%/0%.
000320: May 11 00:32:07.921: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 90%/1%, Top 3 processes(Pid/Util):  191/73%, 106/5%, 156/1%
000321: May 11 00:33:03.185: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 22%/0%.
000322: May 18 00:32:11.516: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 80%/0%, Top 3 processes(Pid/Util):  191/63%, 106/5%, 4/2%
000323: May 18 00:33:06.798: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 18%/0%.

I think we are facing a cosmetic problem here.

During the interval of a CPURISING and a CPUFALLING trap the switch has no high CPU ussage. In my opinion just the FALLING Trap is seen way to late. Does anyone konw something about it ? / ever heard ?

Joe Clarke
Hall of Fame Cisco Employee

The problem may be that the spikes are happening too quickly to catch.  Typically, we'd use EEM to capture the show stack output, but the 2900s don't support full EEM.  If you can be logged in on the device at the time the spike occurs, seeing that show stack for PID 191 would be telling.  I don't see any known bugs for bogus CPURISINGTHRESHOLD messages, so either this is a new issue, or something is taxing the SNMP engine.