2950 is experiencing high CPU utilization

vipinrajrc · ‎12-10-2011

Hi Experts,

I am experiencing high cpu utilization in my 29501 switch. please see the copied sh process cpu output.

CPU utilization for five seconds: 99%/95%; one minute: 89%; five minutes: 37%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 0 2 0 0.00% 0.00% 0.00% 0 Chunk Manager

2 2484 242827 10 0.00% 0.00% 0.00% 0 Load Meter

3 2584 2100 1230 0.00% 0.00% 0.00% 0 SpanTree Helper

4 16888 22547063 0 0.00% 0.00% 0.00% 0 Next Page

5 346608 123642 2803 0.00% 0.01% 0.00% 0 Check heaps

6 1088 588 1850 0.00% 0.00% 0.00% 0 Pool Manager

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

8 88 4 22000 0.00% 0.00% 0.00% 0 Entity MIB API

9 21520 242809 88 0.00% 0.00% 0.00% 0 HC Counter Timer

10 1034464 2816159 367 0.15% 0.07% 0.07% 0 ARP Input

11 4860 44848 108 0.00% 0.00% 0.00% 0 Net Input

12 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

13 747948 500812 1493 0.07% 0.04% 0.02% 0 Net Background

14 85980 305900 281 0.00% 0.04% 0.01% 0 Logger

15 94108 1196713 78 0.00% 0.04% 0.05% 0 TTY Background

16 54336 1196722 45 0.00% 0.00% 0.00% 0 Per-Second Jobs

17 60984 242833 251 0.07% 0.03% 0.01% 0 Compute load avg

18 174232 20285 8589 0.00% 0.01% 0.00% 0 Per-minute Jobs

19 0 105 0 0.00% 0.00% 0.00% 0 AggMgr Process

20 0 2 0 0.00% 0.00% 0.00% 0 Calhoun Storm Co

21 9035560 29107827 310 0.60% 0.54% 0.63% 0 Calhoun LED Proc

22 9546192 2375105 4019 0.52% 0.50% 0.65% 0 Calhoun Statisti

23 8900 5904889 1 0.00% 0.00% 0.00% 0 HMATM Learn proc

24 2736720 27441737 99 0.00% 0.03% 0.05% 0 CalhounPM

25 90692 5905374 15 0.15% 0.13% 0.06% 0 Link Status Proc

26 0 1 0 0.00% 0.00% 0.00% 0 CalhounInsIpopti

27 0 1 0 0.00% 0.00% 0.00% 0 Calhoun DHCP Sno

28 427848 325596 1314 0.00% 0.02% 0.00% 0 CDP Protocol

29 140 10145 13 0.00% 0.00% 0.00% 0 DHCP Snooping

30 9775828 767336 12739 0.15% 0.31% 0.20% 0 IP Input

31 407984 443833 919 0.07% 0.03% 0.00% 0 DTP Protocol

32 6672 121439 54 0.00% 0.00% 0.00% 0 Ethchnl

33 24 57 421 0.00% 0.00% 0.00% 0 VLAN Manager

34 0 2 0 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

35 572 1764 324 0.00% 0.00% 0.00% 0 VMATM Callback

36 7746824 2763240 2803 0.38% 0.24% 0.27% 0 Spanning Tree

37 12224 1217532 10 0.07% 0.01% 0.00% 0 UDLD

38 520 20293 25 0.00% 0.00% 0.00% 0 Port-Security

39 12252 15504 790 0.00% 0.00% 0.00% 0 DHCPD Receive

40 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer

41 1812 4322 419 0.00% 0.00% 0.00% 0 RARP Input

42 2240 10353 216 0.00% 0.00% 0.00% 0 TCP Timer

43 220 100 2200 0.00% 0.00% 0.00% 0 TCP Protocols

44 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

45 224 1249 179 0.00% 0.00% 0.00% 0 L2MM

46 0 1 0 0.00% 0.00% 0.00% 0 MRD

47 2496 7692 324 0.00% 0.00% 0.00% 0 IGMPSN

48 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr

49 2668 8851 301 0.00% 0.00% 0.00% 0 IGMP Snooping Re

50 20172 94525 213 0.00% 0.00% 0.00% 0 IP Multicast Dat

51 0 2 0 0.00% 0.00% 0.00% 0 L2TRACE SERVER

52 332 15219 21 0.00% 0.00% 0.00% 0 Cluster L2

53 7952 121440 65 0.00% 0.00% 0.00% 0 Cluster RARP

54 15008 152109 98 0.00% 0.00% 0.00% 0 Cluster Base

56 728 10145 71 0.00% 0.00% 0.00% 0 DHCPD Timer

57 0 2 0 0.00% 0.00% 0.00% 0 Router Autoconf

58 2044 553 3696 0.00% 0.00% 0.00% 0 Syslog Traps

59 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF

60 243536 4047528 60 0.07% 0.04% 0.02% 0 PM Callback

61 152 112 1357 0.00% 0.00% 0.00% 0 VTP Trap Process

62 14164 344831 41 0.00% 0.00% 0.00% 0 DHCPD Database

63 34708 1196738 29 0.00% 0.00% 0.00% 0 Calhoun EnvMon P

64 4 2 2000 0.00% 0.00% 0.00% 0 STP STACK TOPOLO

65 0 2 0 0.00% 0.00% 0.00% 0 STP FAST TRANSIT

66 0 2 0 0.00% 0.00% 0.00% 0 STP STACK TOPOLO

67 0 2 0 0.00% 0.00% 0.00% 0 CSRT RAPID TRANS

68 1828 258 7085 0.00% 0.16% 0.17% 1 Virtual Exec

70 0 3 0 0.00% 0.00% 0.00% 0 SNMP Timers

71 35893256 483793 74191 0.84% 0.63% 0.34% 0 IP SNMP

72 143380 242289 591 0.15% 0.11% 0.07% 0 PDU DISPATCHER

73 947612 242317 3910 0.68% 0.64% 0.38% 0 SNMP ENGINE

74 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

75 3968 2071 1915 0.00% 0.00% 0.00% 0 SNMP Traps

Before 1 month it is usually 3%. But now suddenly it is showing 99%. in the process also no of the process is above 1%. all the process are showing 0.xx%. How can i solve the issue.

there is another switch in the infra, which is conencted to this switch. That is a small-medium switch with 24 PoE port. so all the IP phones and around 10 PC s are connected to the small-medium switch.

ASA is conencted to the 2950switch. So all the machines are getting internet through the 2950 switch only.

So what may the issue of this high CPU utilization.

Please advice.

Thanks

Vipin

Thanks and Regards, Vipin

smogra · ‎12-10-2011

the cpu is not high because of any processes. it is the interrupt percentage that is high (95%).

you need to analyse the traffic that is flowing in and why it is hitting cpu.

1. check the logs for any suspicious error messages.

2. check if the spanning-tree is stable.

-Sweta

vipinrajrc · ‎12-12-2011

Hi Sweta,

I will check and update.

Thanks and Regards, Vipin

vipinrajrc · ‎12-12-2011

Hi Sweta,

please see the below output of "sh logg". I think it is due to NTP synchronization mismatch.

Is this is the cause of the high CPU utilization on some time.?

2w1d: xmt D2901053.E25BBA81 (00:32:03.884 IST Mon Dec 12 2011)

2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)

2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s

tate to down

2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s

tate to up

2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s

tate to down

2w1d: %LINK-3-UPDOWN: Interface FastEthernet0/7, changed state to down

2w1d: %LINK-3-UPDOWN: Interface FastEthernet0/7, changed state to up

2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s

tate to up

2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)

2w1d: NTP: xmit packet to 203.200.188.4:

2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64

2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: xmt D2903209.E22BEA2F (02:55:53.883 IST Mon Dec 12 2011)

2w1d: NTP: xmit packet to 210.212.45.242:

2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64

2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: xmt D2903213.E22AE1A0 (02:56:03.883 IST Mon Dec 12 2011)

2w1d: NTP: xmit packet to 203.200.188.4:

2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64

2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: xmt D2903249.E22C37D9 (02:56:57.883 IST Mon Dec 12 2011)

2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)

2w1d: NTP: xmit packet to 203.200.188.4:

2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64

2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: xmt D2903289.E229E00F (02:58:01.883 IST Mon Dec 12 2011)

2w1d: NTP: xmit packet to 203.200.188.4:

2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64

2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: xmt D29032C9.E22A5B5B (02:59:05.883 IST Mon Dec 12 2011)

2w1d: NTP: xmit packet to 203.200.188.4:

2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64

2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: xmt D2903309.E228BECB (03:00:09.883 IST Mon Dec 12 2011)

2w1d: NTP: xmit packet to 203.200.188.4:

2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64

2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)

2w1d: xmt D2903349.E228D9CB (03:01:13.883 IST Mon Dec 12 2011)

2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)

2w2d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/10, changed

state to down

2w2d: %LINK-3-UPDOWN: Interface FastEthernet0/10, changed state to down

2w1d: xmt D2901053.E25BBA81 (00:32:03.884 IST Mon Dec 12 2011)
2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)
2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s
tate to down
2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s
tate to up
2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s
tate to down
2w1d: %LINK-3-UPDOWN: Interface FastEthernet0/7, changed state to down
2w1d: %LINK-3-UPDOWN: Interface FastEthernet0/7, changed state to up
2w1d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed s
tate to up
2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)
2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)
2w1d: NTP: xmit packet to 203.200.188.4:
2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64
2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: xmt D2903209.E22BEA2F (02:55:53.883 IST Mon Dec 12 2011)
2w1d: NTP: xmit packet to 210.212.45.242:
2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64
2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: xmt D2903213.E22AE1A0 (02:56:03.883 IST Mon Dec 12 2011)
2w1d: NTP: xmit packet to 203.200.188.4:
2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64
2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: xmt D2903249.E22C37D9 (02:56:57.883 IST Mon Dec 12 2011)
2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)
2w1d: NTP: xmit packet to 203.200.188.4:
2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64
2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: xmt D2903289.E229E00F (02:58:01.883 IST Mon Dec 12 2011)
2w1d: NTP: xmit packet to 203.200.188.4:
2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64
2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: xmt D29032C9.E22A5B5B (02:59:05.883 IST Mon Dec 12 2011)
2w1d: NTP: xmit packet to 203.200.188.4:
2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64
2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: xmt D2903309.E228BECB (03:00:09.883 IST Mon Dec 12 2011)
2w1d: NTP: xmit packet to 203.200.188.4:
2w1d: leap 3, mode 3, version 3, stratum 0, ppoll 64
2w1d: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
2w1d: ref 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: org 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: rec 00000000.00000000 (19:00:00.000 IST Thu Dec 31 1899)
2w1d: xmt D2903349.E228D9CB (03:01:13.883 IST Mon Dec 12 2011)
2w1d: %SYS-5-CONFIG_I: Configured from console by vty0 (10.254.254.25)
2w2d: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/10, changed
state to down
2w2d: %LINK-3-UPDOWN: Interface FastEthernet0/10, changed state to down

here we have only two switches. so do i need to check spaning-tree???

Thanks

Vipin

Thanks and Regards, Vipin

zabeelmusa · ‎12-13-2011

your switch seems to be processing a lot of traffic you would need to check which interfaces are highly utilized

Sh int | inc (up|txload) or replace the transmit with the receive.

Sent from Cisco Technical Support iPhone App