High CPU edge router

zyang · ‎08-06-2013

My edge router is pegged at 80% and interrupts at 8%. None of the processes show up as being high. What is the next step to troubleshoot this? How do I find out what is using all this CPU? below is the show proc cpu sort command. and show proc cpu history.

CEF is enabled.

CPU utilization for five seconds: 78%/7%; one minute: 78%; five minutes: 77%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

65 3659260336 390589358 9368 5.14% 5.82% 5.82% 0 IP Input

103 2935808 123190216 23 0.16% 0.14% 0.14% 0 TCP Timer

118 929632 220585332 4 0.08% 0.01% 0.00% 0 Inspect process

71 21172428 121535238 174 0.08% 0.03% 0.02% 0 BGP I/O

226 4480 2834 1580 0.08% 0.09% 0.06% 196 SSH Process

18 9694672 79550472 121 0.08% 0.00% 0.00% 0 EEM ED Syslog

104 350964 429432 817 0.08% 0.07% 0.08% 0 TCP Protocols

8 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

9 6484 3767064 1 0.00% 0.00% 0.00% 0 Environmental mo

10 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

11 2399092 9123552 262 0.00% 0.00% 0.00% 0 ARP Input

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

13 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

14 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

15 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

16 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers

6 96624 119331 809 0.00% 0.00% 0.00% 0 Pool Manager

12 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

19 73864 5650559 13 0.00% 0.00% 0.00% 0 HC Counter Timer

5 96382516 13484167 7147 0.00% 0.06% 0.05% 0 Check heaps

21 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers

22 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa

23 0 2 0 0.00% 0.00% 0.00% 0 SMART

24 272820 113013132 2 0.00% 0.00% 0.00% 0 GraphIt

25 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

26 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

27 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

28 44 7793 5 0.00% 0.00% 0.00% 0 IP RACL Ager

4 0 1 0 0.00% 0.00% 0.00% 0 EDDRI_MAIN

17 8 2 4000 0.00% 0.00% 0.00% 0 Entity MIB API

20 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

32 4 2 2000 0.00% 0.00% 0.00% 0 IDB Work

33 1502748 148454142 10 0.00% 0.01% 0.00% 0 Logger

34 400944 112974797 3 0.00% 0.00% 0.00% 0 TTY Background

35 2679652 113013290 23 0.00% 0.04% 0.05% 0 Per-Second Jobs

36 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process

29 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

38 104 25 4160 0.00% 0.00% 0.00% 0 ESWPPM

39 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con

40 7460 1263048 5 0.00% 0.00% 0.00% 0 Time Range Proce

41 0 1 0 0.00% 0.00% 0.00% 0 dev_device_inser

42 0 1 0 0.00% 0.00% 0.00% 0 dev_device_remov

43 0 2 0 0.00% 0.00% 0.00% 0 SM Monitor

44 0 2 0 0.00% 0.00% 0.00% 0 FLEX DNLD MAIN

45 32 213 150 0.00% 0.00% 0.00% 0 IGMP Snooping Pr

46 32 219 146 0.00% 0.00% 0.00% 0 IGMP Snooping Re

47 0 1 0 0.00% 0.00% 0.00% 0 Platform Compact

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

48 6268 3767065 1 0.00% 0.00% 0.00% 0 Call Management

49 12 23 521 0.00% 0.00% 0.00% 0 Switch Link Moni

50 191172 112974880 1 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

51 0 2 0 0.00% 0.00% 0.00% 0 DTP Protocol

3 2202200 175147099 12 0.00% 0.03% 0.02% 0 Spanning Tree

53 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon

30 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

55 0 2 0 0.00% 0.00% 0.00% 0 Dot11 Mgmt & Ass

56 9471696 423966835 22 0.00% 0.01% 0.00% 0 BGP Router

31 350220 50938407 6 0.00% 0.00% 0.00% 0 Net Background

58 0 2 0 0.00% 0.00% 0.00% 0 Dot11 auth Dot1x

37 0 1 0 0.00% 0.00% 0.00% 0 Token Daemon

60 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

61 29376 417852 70 0.00% 0.00% 0.00% 0 AAA Server

62 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc

63 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr

64 1376428 18054357 76 0.00% 0.00% 0.00% 0 CDP Protocol

2 2662872 22602652 117 0.00% 0.04% 0.05% 0 Load Meter

66 36236 5410033 6 0.00% 0.00% 0.00% 0 ICMP event handl

67 4 1072 3 0.00% 0.00% 0.00% 0 TurboACL

68 0 2 0 0.00% 0.00% 0.00% 0 TurboACL chunk

69 39752 449634 88 0.00% 0.00% 0.00% 0 MOP Protocols

70 0 3 0 0.00% 0.00% 0.00% 0 PPP Hooks

54 220916 112974945 1 0.00% 0.00% 0.00% 0 linktest

72 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager

73 26788 15068075 1 0.00% 0.00% 0.00% 0 SSS Test Client

74 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana

75 1297980 441409045 2 0.00% 0.01% 0.00% 0 SSS Feature Time

76 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage

77 0 1 0 0.00% 0.00% 0.00% 0 L2X Socket proce

78 0 1 0 0.00% 0.00% 0.00% 0 L2X SSS manager

79 0 2 0 0.00% 0.00% 0.00% 0 L2TP mgmt daemon

80 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

81 0 2 0 0.00% 0.00% 0.00% 0 EAPoUDP Process

82 0 1 0 0.00% 0.00% 0.00% 0 IPv6 RIB Redistr

83 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA

84 167740 3252239 51 0.00% 0.00% 0.00% 0 IP Background

85 38212 1914227 19 0.00% 0.00% 0.00% 0 IP RIB Update

86 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route

87 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP

88 1225764 180111159 6 0.00% 0.01% 0.00% 0 CEF process

89 284216 112931728 2 0.00% 0.00% 0.00% 0 Socket Timers

90 8 1766 4 0.00% 0.00% 0.00% 0 L2MM

91 40 220 181 0.00% 0.00% 0.00% 0 MRD

92 0 43 0 0.00% 0.00% 0.00% 0 IGMPSN

93 2624420 225863448 11 0.00% 0.03% 0.02% 0 DHCPD Receive

94 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute

95 0 2 0 0.00% 0.00% 0.00% 0 RLM groups Proce

96 0 2 0 0.00% 0.00% 0.00% 0 SNMP Timers

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

97 4 2 2000 0.00% 0.00% 0.00% 0 SCTP Main Proces

98 0 1 0 0.00% 0.00% 0.00% 0 IUA Main Process

99 385808 112974915 3 0.00% 0.00% 0.00% 0 RUDPV1 Main Proc

100 0 1 0 0.00% 0.00% 0.00% 0 bsm_timers

101 180908 112974905 1 0.00% 0.00% 0.00% 0 bsm_xmt_proc

102 0 1 0 0.00% 0.00% 0.00% 0 CES Client SVC R

52 0 15 0 0.00% 0.00% 0.00% 0 dot1x

1 664 848 783 0.00% 0.00% 0.00% 0 Chunk Manager

105 0 1 0 0.00% 0.00% 0.00% 0 COPS

106 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder

107 28160 1883493 14 0.00% 0.00% 0.00% 0 IP Cache Ager

108 95744 1883547 50 0.00% 0.00% 0.00% 0 Adj Manager

109 4 2 2000 0.00% 0.00% 0.00% 0 ATM OAM Input

110 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER

111 145828 473546 307 0.00% 0.00% 0.00% 0 HTTP CORE

112 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

57 0 2 0 0.00% 0.00% 0.00% 0 Dot11 aaa proces

59 0 1 0 0.00% 0.00% 0.00% 0 Dot11 Mac Auth

115 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

116 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS

117 33750761127424862 2 0.00% 0.03% 0.02% 0 RBSCP Background

113 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

119 4036 941775 4 0.00% 0.00% 0.00% 0 DHCPD Timer

120 976 376711 2 0.00% 0.00% 0.00% 0 Authentication P

121 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B

122 0 1 0 0.00% 0.00% 0.00% 0 IPS Timer

123 0 2 0 0.00% 0.00% 0.00% 0 SDEE Management

124 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Inspect Tim

125 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc

126 8 3 2666 0.00% 0.00% 0.00% 0 Crypto HW Proc

127 16764 4520532 3 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_

128 163600 41820278 3 0.00% 0.00% 0.00% 0 FLEX DSPRM MAIN

129 58628 41820276 1 0.00% 0.00% 0.00% 0 FLEX DSP KEEPALI

130 30796160 7434014 4142 0.00% 0.01% 0.00% 0 BGP Scanner

131 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve

132 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

133 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr

134 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

135 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

136 51864 417852 124 0.00% 0.01% 0.00% 0 LOCAL AAA

137 0 2 0 0.00% 0.00% 0.00% 0 TPLUS

138 0 2 0 0.00% 0.00% 0.00% 0 VSP_MGR

139 0 3 0 0.00% 0.00% 0.00% 0 Crypto WUI

140 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support

141 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_HTSP

142 0 2 0 0.00% 0.00% 0.00% 0 VPM_MWI_BACKGROU

143 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_R2

144 268 125571 2 0.00% 0.00% 0.00% 0 FB/KS Log HouseK

145 0 2 0 0.00% 0.00% 0.00% 0 EPHONE MWI BG Pr

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

146 0 1 0 0.00% 0.00% 0.00% 0 CCSWVOICE

147 0 264 0 0.00% 0.00% 0.00% 0 HSRP Common

148 0 1 0 0.00% 0.00% 0.00% 0 http client proc

149 88 77 1142 0.00% 0.00% 0.00% 0 CCH323_CT

150 0 1 0 0.00% 0.00% 0.00% 0 CCH323_DNS

151 4 2 2000 0.00% 0.00% 0.00% 0 CCAAL2_CT

152 0 2 0 0.00% 0.00% 0.00% 0 CCFRF11_CT

153 1832264 509987 3592 0.00% 0.00% 0.00% 0 CCSIP_SPI_CONTRO

154 0 1 0 0.00% 0.00% 0.00% 0 CCSIP_DNS

155 596392 797031 748 0.00% 0.00% 0.00% 0 CCSIP_UDP_SOCKET

156 32 197 162 0.00% 0.00% 0.00% 0 CCSIP_TCP_SOCKET

157 143952 1965788 73 0.00% 0.00% 0.00% 0 traffic_shape

158 0 1 0 0.00% 0.00% 0.00% 0 QOS_MODULE_MAIN

159 0 1 0 0.00% 0.00% 0.00% 0 RPMS_PROC_MAIN

160 0 1 0 0.00% 0.00% 0.00% 0 VoIP AAA

161 4 12 333 0.00% 0.00% 0.00% 0 crypto engine pr

162 32 4 8000 0.00% 0.00% 0.00% 0 Crypto CA

163 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL

164 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL

165 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc

166 2336 18312 127 0.00% 0.00% 0.00% 0 Crypto ACL

167 0 2 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce

168 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT

169 0 4 0 0.00% 0.00% 0.00% 0 Crypto IKMP

170 17404 5650563 3 0.00% 0.00% 0.00% 0 IPSEC key engine

171 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key

172 0 1 0 0.00% 0.00% 0.00% 0 Crypto PAS Proc

173 0 2 0 0.00% 0.00% 0.00% 0 Key Proc

174 0 1 0 0.00% 0.00% 0.00% 0 GDOI GM Process

175 232944 113013138 2 0.00% 0.00% 0.00% 0 Crypto Device Up

176 0 2 0 0.00% 0.00% 0.00% 0 Multi-ISA Event

177 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Cleanu

178 183644 32010030 5 0.00% 0.00% 0.00% 0 PM Callback

179 1412192 564928991 2 0.00% 0.01% 0.00% 0 Atheros LED Ctro

180 0 2 0 0.00% 0.00% 0.00% 0 Control-plane ho

181 0 1 0 0.00% 0.00% 0.00% 0 DATA Transfer Pr

182 0 1 0 0.00% 0.00% 0.00% 0 DATA Collector

183 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

184 2848500 38010126 74 0.00% 0.01% 0.00% 0 HSRP IPv4

185 0 3 0 0.00% 0.00% 0.00% 0 EEM ED CLI

186 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Counter

187 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Interface

188 0 3 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD

189 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Memory-th

190 0 2 0 0.00% 0.00% 0.00% 0 EEM ED None

191 0 2 0 0.00% 0.00% 0.00% 0 EEM ED OIR

192 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Resource

193 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP

194 9476 1883644 5 0.00% 0.00% 0.00% 0 EEM ED Timer

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

195 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Track

196 90136 24341533 3 0.00% 0.00% 0.00% 0 EEM Server

197 20436 11300918 1 0.00% 0.00% 0.00% 0 RMON Recycle Pro

198 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se

199 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

200 20 2 10000 0.00% 0.00% 0.00% 0 VLAN Manager

201 2956 1883549 1 0.00% 0.00% 0.00% 0 DHCPD Database

202 244488 112846050 2 0.00% 0.00% 0.00% 0 trunk conditioni

203 0 1 0 0.00% 0.00% 0.00% 0 trunk conditioni

204 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc

205 5445956 74591605 73 0.00% 0.04% 0.05% 0 Syslog

206 0 1 0 0.00% 0.00% 0.00% 0 VPDN Scal

207 239888 6311492 38 0.00% 0.00% 0.00% 0 IP SLAs Event Pr

208 0 21 0 0.00% 0.00% 0.00% 0 Net Input

209 569360 22602683 25 0.00% 0.00% 0.00% 0 Compute load avg

210 32977268 1909947 17266 0.00% 0.03% 0.00% 0 Per-minute Jobs

211 96992 3919547 24 0.00% 0.00% 0.00% 0 CEF Scanner

212 1887592 9886 190935 0.00% 0.00% 0.00% 0 crypto sw pk pro

213 0 1 0 0.00% 0.00% 0.00% 0 tHUB

214 0 2 0 0.00% 0.00% 0.00% 0 tENM

215 460 2876 159 0.00% 0.00% 0.00% 0 SSH Event handle

216 228 433 526 0.00% 0.00% 0.00% 0 TCP Listener

217 53965516 117099189 460 0.00% 0.05% 0.05% 0 IP SNMP

218 14740996 57836129 254 0.00% 0.01% 0.00% 0 PDU DISPATCHER

219 91337708 57836954 1579 0.00% 0.10% 0.08% 0 SNMP ENGINE

220 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

221 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps

222 546544 113189644 4 0.00% 0.00% 0.00% 0 NTP

223 1012 15 67466 0.00% 0.00% 0.00% 194 SSH Process

224 1000 8 125000 0.00% 0.00% 0.00% 195 SSH Process

225 115968 10818533 10 0.00% 0.00% 0.00% 0 Track

114 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background

8877777888887777788888777777777777777777777777788888888888

6677777000006666622222555556666688888999995555533333444442

100

90 **

80 ************************************************************

70 ************************************************************

60 ************************************************************

50 ************************************************************

40 ************************************************************

30 ************************************************************

20 ************************************************************

10 ************************************************************

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)

8888888887888988998899889989889889888988899888988899999898

4847203239886078007726971074980981769266821888098815122919

100 * *

90 * * **************************************************

80 ####******############################*###*#################

70 ############################################################

60 ############################################################

50 ############################################################

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

9998998999899889888876875998695562463239453245566645555633323429353622

5608971144811769712891133388496979120709760766093791332097290899182589

100 ** ** * * * * *

90 ****** ********** * *** * * *

80 ##******************* * *** * * *

70 ###*##*#########***** ** *#* * * * * * * *

60 #################*#***** *#****** * * * * *** * * * *

50 ####################******##***** * *** ************ * * * *

40 #####################**#**####*** ** *** *****#******** * * * * **

30 ########################**#####**************###########****************

20 ########################*#######***#***#*****###########****************

10 ########################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

zyang · ‎08-06-2013

Also the interface for outbound traffic is:

5 minute input rate 627000 bits/sec, 182 packets/sec

5 minute output rate 493000 bits/sec, 201 packets/sec

Which doesn't suggest alot of traffic flowing through.

There is a huge ACL and FW inspection is on. Though with that little traffic, I don't see why it could cause such high cpu loads.

Thanks and any help would be appreciated in diagnosing the problem.

jawad-mukhtar · ‎08-06-2013

enable netflow on Router.

All thing will be clear to you, which port and ip using cpu.

***Do Rate Helpful Posts***

Jawad