cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
798
Views
0
Helpful
2
Replies

High CPU edge router

zyang
Beginner
Beginner

My edge router is pegged at 80% and interrupts at 8%.  None of the processes show up as being high.  What is the next step to troubleshoot this?  How do I find out what is using all this CPU?  below is the show proc cpu sort command.  and show proc cpu history.

CEF is enabled.

CPU utilization for five seconds: 78%/7%; one minute: 78%; five minutes: 77%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  65  3659260336 390589358       9368  5.14%  5.82%  5.82%   0 IP Input

103     2935808 123190216         23  0.16%  0.14%  0.14%   0 TCP Timer

118      929632 220585332          4  0.08%  0.01%  0.00%   0 Inspect process

  71    21172428 121535238        174  0.08%  0.03%  0.02%   0 BGP I/O

226        4480      2834       1580  0.08%  0.09%  0.06% 196 SSH Process

  18     9694672  79550472        121  0.08%  0.00%  0.00%   0 EEM ED Syslog

104      350964    429432        817  0.08%  0.07%  0.08%   0 TCP Protocols

   8           0         1          0  0.00%  0.00%  0.00%   0 OIR Handler

   9        6484   3767064          1  0.00%  0.00%  0.00%   0 Environmental mo

  10           0         1          0  0.00%  0.00%  0.00%   0 Crash writer

  11     2399092   9123552        262  0.00%  0.00%  0.00%   0 ARP Input

   7           0         2          0  0.00%  0.00%  0.00%   0 Timers

  13           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit

  14           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

  15           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager

  16           0         2          0  0.00%  0.00%  0.00%   0 DDR Timers

   6       96624    119331        809  0.00%  0.00%  0.00%   0 Pool Manager

  12           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer

  19       73864   5650559         13  0.00%  0.00%  0.00%   0 HC Counter Timer

   5    96382516  13484167       7147  0.00%  0.06%  0.05%   0 Check heaps

  21           0         1          0  0.00%  0.00%  0.00%   0 RO Notify Timers

  22           0         1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa

  23           0         2          0  0.00%  0.00%  0.00%   0 SMART

  24      272820 113013132          2  0.00%  0.00%  0.00%   0 GraphIt

  25           0         2          0  0.00%  0.00%  0.00%   0 Dialer event

  26           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect

  27           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client

  28          44      7793          5  0.00%  0.00%  0.00%   0 IP RACL Ager

   4           0         1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN

  17           8         2       4000  0.00%  0.00%  0.00%   0 Entity MIB API

  20           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun

  32           4         2       2000  0.00%  0.00%  0.00%   0 IDB Work

  33     1502748 148454142         10  0.00%  0.01%  0.00%   0 Logger

  34      400944 112974797          3  0.00%  0.00%  0.00%   0 TTY Background

  35     2679652 113013290         23  0.00%  0.04%  0.05%   0 Per-Second Jobs

  36           0         1          0  0.00%  0.00%  0.00%   0 AggMgr Process

  29           0         1          0  0.00%  0.00%  0.00%   0 Inode Table Dest

  38         104        25       4160  0.00%  0.00%  0.00%   0 ESWPPM

  39           0         2          0  0.00%  0.00%  0.00%   0 Eswilp Storm Con

  40        7460   1263048          5  0.00%  0.00%  0.00%   0 Time Range Proce

  41           0         1          0  0.00%  0.00%  0.00%   0 dev_device_inser

  42           0         1          0  0.00%  0.00%  0.00%   0 dev_device_remov

  43           0         2          0  0.00%  0.00%  0.00%   0 SM Monitor

  44           0         2          0  0.00%  0.00%  0.00%   0 FLEX DNLD MAIN

  45          32       213        150  0.00%  0.00%  0.00%   0 IGMP Snooping Pr

  46          32       219        146  0.00%  0.00%  0.00%   0 IGMP Snooping Re

  47           0         1          0  0.00%  0.00%  0.00%   0 Platform Compact

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  48        6268   3767065          1  0.00%  0.00%  0.00%   0 Call Management

  49          12        23        521  0.00%  0.00%  0.00%   0 Switch Link Moni

  50      191172 112974880          1  0.00%  0.00%  0.00%   0 PI MATM Aging Pr

  51           0         2          0  0.00%  0.00%  0.00%   0 DTP Protocol

   3     2202200 175147099         12  0.00%  0.03%  0.02%   0 Spanning Tree

  53           0         1          0  0.00%  0.00%  0.00%   0 L2X Data Daemon

  30           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd

  55           0         2          0  0.00%  0.00%  0.00%   0 Dot11 Mgmt & Ass

  56     9471696 423966835         22  0.00%  0.01%  0.00%   0 BGP Router

  31      350220  50938407          6  0.00%  0.00%  0.00%   0 Net Background

  58           0         2          0  0.00%  0.00%  0.00%   0 Dot11 auth Dot1x

  37           0         1          0  0.00%  0.00%  0.00%   0 Token Daemon

  60           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R

  61       29376    417852         70  0.00%  0.00%  0.00%   0 AAA Server

  62           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc

  63           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr

  64     1376428  18054357         76  0.00%  0.00%  0.00%   0 CDP Protocol

   2     2662872  22602652        117  0.00%  0.04%  0.05%   0 Load Meter

  66       36236   5410033          6  0.00%  0.00%  0.00%   0 ICMP event handl

  67           4      1072          3  0.00%  0.00%  0.00%   0 TurboACL

  68           0         2          0  0.00%  0.00%  0.00%   0 TurboACL chunk

  69       39752    449634         88  0.00%  0.00%  0.00%   0 MOP Protocols

  70           0         3          0  0.00%  0.00%  0.00%   0 PPP Hooks

  54      220916 112974945          1  0.00%  0.00%  0.00%   0 linktest

  72           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager

  73       26788  15068075          1  0.00%  0.00%  0.00%   0 SSS Test Client

  74           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana

  75     1297980 441409045          2  0.00%  0.01%  0.00%   0 SSS Feature Time

  76           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage

  77           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce

  78           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager

  79           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon

  80           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana

  81           0         2          0  0.00%  0.00%  0.00%   0 EAPoUDP Process

  82           0         1          0  0.00%  0.00%  0.00%   0 IPv6 RIB Redistr

  83           0         2          0  0.00%  0.00%  0.00%   0 KRB5 AAA

  84      167740   3252239         51  0.00%  0.00%  0.00%   0 IP Background

  85       38212   1914227         19  0.00%  0.00%  0.00%   0 IP RIB Update

  86           0         2          0  0.00%  0.00%  0.00%   0 PPP IP Route

  87           0         2          0  0.00%  0.00%  0.00%   0 PPP IPCP

  88     1225764 180111159          6  0.00%  0.01%  0.00%   0 CEF process

  89      284216 112931728          2  0.00%  0.00%  0.00%   0 Socket Timers

  90           8      1766          4  0.00%  0.00%  0.00%   0 L2MM

  91          40       220        181  0.00%  0.00%  0.00%   0 MRD

  92           0        43          0  0.00%  0.00%  0.00%   0 IGMPSN

  93     2624420 225863448         11  0.00%  0.03%  0.02%   0 DHCPD Receive

  94           0         1          0  0.00%  0.00%  0.00%   0 IP Traceroute

  95           0         2          0  0.00%  0.00%  0.00%   0 RLM groups Proce

  96           0         2          0  0.00%  0.00%  0.00%   0 SNMP Timers

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  97           4         2       2000  0.00%  0.00%  0.00%   0 SCTP Main Proces

  98           0         1          0  0.00%  0.00%  0.00%   0 IUA Main Process

  99      385808 112974915          3  0.00%  0.00%  0.00%   0 RUDPV1 Main Proc

100           0         1          0  0.00%  0.00%  0.00%   0 bsm_timers

101      180908 112974905          1  0.00%  0.00%  0.00%   0 bsm_xmt_proc

102           0         1          0  0.00%  0.00%  0.00%   0 CES Client SVC R

  52           0        15          0  0.00%  0.00%  0.00%   0 dot1x

   1         664       848        783  0.00%  0.00%  0.00%   0 Chunk Manager

105           0         1          0  0.00%  0.00%  0.00%   0 COPS

106           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder

107       28160   1883493         14  0.00%  0.00%  0.00%   0 IP Cache Ager

108       95744   1883547         50  0.00%  0.00%  0.00%   0 Adj Manager

109           4         2       2000  0.00%  0.00%  0.00%   0 ATM OAM Input

110           0         2          0  0.00%  0.00%  0.00%   0 ATM OAM TIMER

111      145828    473546        307  0.00%  0.00%  0.00%   0 HTTP CORE

112           0         1          0  0.00%  0.00%  0.00%   0 RARP Input

  57           0         2          0  0.00%  0.00%  0.00%   0 Dot11 aaa proces

  59           0         1          0  0.00%  0.00%  0.00%   0 Dot11 Mac Auth

115           0         2          0  0.00%  0.00%  0.00%   0 PPP Bind

116           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS

117     33750761127424862          2  0.00%  0.03%  0.02%   0 RBSCP Background

113           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall

119        4036    941775          4  0.00%  0.00%  0.00%   0 DHCPD Timer

120         976    376711          2  0.00%  0.00%  0.00%   0 Authentication P

121           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B

122           0         1          0  0.00%  0.00%  0.00%   0 IPS Timer

123           0         2          0  0.00%  0.00%  0.00%   0 SDEE Management

124           0         1          0  0.00%  0.00%  0.00%   0 IPv6 Inspect Tim

125           0         2          0  0.00%  0.00%  0.00%   0 URL filter proc

126           8         3       2666  0.00%  0.00%  0.00%   0 Crypto HW Proc

127       16764   4520532          3  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_

128      163600  41820278          3  0.00%  0.00%  0.00%   0 FLEX DSPRM MAIN

129       58628  41820276          1  0.00%  0.00%  0.00%   0 FLEX DSP KEEPALI

130    30796160   7434014       4142  0.00%  0.01%  0.00%   0 BGP Scanner

131           0         2          0  0.00%  0.00%  0.00%   0 AAA Cached Serve

132           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA

133           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr

134           0         1          0  0.00%  0.00%  0.00%   0 Key chain liveke

135           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA

136       51864    417852        124  0.00%  0.01%  0.00%   0 LOCAL AAA

137           0         2          0  0.00%  0.00%  0.00%   0 TPLUS

138           0         2          0  0.00%  0.00%  0.00%   0 VSP_MGR

139           0         3          0  0.00%  0.00%  0.00%   0 Crypto WUI

140           0         2          0  0.00%  0.00%  0.00%   0 Crypto Support

141           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_HTSP

142           0         2          0  0.00%  0.00%  0.00%   0 VPM_MWI_BACKGROU

143           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_R2

144         268    125571          2  0.00%  0.00%  0.00%   0 FB/KS Log HouseK

145           0         2          0  0.00%  0.00%  0.00%   0 EPHONE MWI BG Pr

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

146           0         1          0  0.00%  0.00%  0.00%   0 CCSWVOICE

147           0       264          0  0.00%  0.00%  0.00%   0 HSRP Common

148           0         1          0  0.00%  0.00%  0.00%   0 http client proc

149          88        77       1142  0.00%  0.00%  0.00%   0 CCH323_CT

150           0         1          0  0.00%  0.00%  0.00%   0 CCH323_DNS

151           4         2       2000  0.00%  0.00%  0.00%   0 CCAAL2_CT

152           0         2          0  0.00%  0.00%  0.00%   0 CCFRF11_CT

153     1832264    509987       3592  0.00%  0.00%  0.00%   0 CCSIP_SPI_CONTRO

154           0         1          0  0.00%  0.00%  0.00%   0 CCSIP_DNS

155      596392    797031        748  0.00%  0.00%  0.00%   0 CCSIP_UDP_SOCKET

156          32       197        162  0.00%  0.00%  0.00%   0 CCSIP_TCP_SOCKET

157      143952   1965788         73  0.00%  0.00%  0.00%   0 traffic_shape

158           0         1          0  0.00%  0.00%  0.00%   0 QOS_MODULE_MAIN

159           0         1          0  0.00%  0.00%  0.00%   0 RPMS_PROC_MAIN

160           0         1          0  0.00%  0.00%  0.00%   0 VoIP AAA

161           4        12        333  0.00%  0.00%  0.00%   0 crypto engine pr

162          32         4       8000  0.00%  0.00%  0.00%   0 Crypto CA

163           0         1          0  0.00%  0.00%  0.00%   0 Crypto PKI-CRL

164           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL

165           0         1          0  0.00%  0.00%  0.00%   0 encrypt proc

166        2336     18312        127  0.00%  0.00%  0.00%   0 Crypto ACL

167           0         2          0  0.00%  0.00%  0.00%   0 CRYPTO QoS proce

168           0         1          0  0.00%  0.00%  0.00%   0 Crypto INT

169           0         4          0  0.00%  0.00%  0.00%   0 Crypto IKMP

170       17404   5650563          3  0.00%  0.00%  0.00%   0 IPSEC key engine

171           0         1          0  0.00%  0.00%  0.00%   0 IPSEC manual key

172           0         1          0  0.00%  0.00%  0.00%   0 Crypto PAS Proc

173           0         2          0  0.00%  0.00%  0.00%   0 Key Proc

174           0         1          0  0.00%  0.00%  0.00%   0 GDOI GM Process

175      232944 113013138          2  0.00%  0.00%  0.00%   0 Crypto Device Up

176           0         2          0  0.00%  0.00%  0.00%   0 Multi-ISA Event

177           0         1          0  0.00%  0.00%  0.00%   0 Multi-ISA Cleanu

178      183644  32010030          5  0.00%  0.00%  0.00%   0 PM Callback

179     1412192 564928991          2  0.00%  0.01%  0.00%   0 Atheros LED Ctro

180           0         2          0  0.00%  0.00%  0.00%   0 Control-plane ho

181           0         1          0  0.00%  0.00%  0.00%   0 DATA Transfer Pr

182           0         1          0  0.00%  0.00%  0.00%   0 DATA Collector

183           0         2          0  0.00%  0.00%  0.00%   0 AAA SEND STOP EV

184     2848500  38010126         74  0.00%  0.01%  0.00%   0 HSRP IPv4

185           0         3          0  0.00%  0.00%  0.00%   0 EEM ED CLI

186           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Counter

187           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Interface

188           0         3          0  0.00%  0.00%  0.00%   0 EEM ED IOSWD

189           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Memory-th

190           0         2          0  0.00%  0.00%  0.00%   0 EEM ED None

191           0         2          0  0.00%  0.00%  0.00%   0 EEM ED OIR

192           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Resource

193           0         2          0  0.00%  0.00%  0.00%   0 EEM ED SNMP

194        9476   1883644          5  0.00%  0.00%  0.00%   0 EEM ED Timer

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

195           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Track

196       90136  24341533          3  0.00%  0.00%  0.00%   0 EEM Server

197       20436  11300918          1  0.00%  0.00%  0.00%   0 RMON Recycle Pro

198           0         2          0  0.00%  0.00%  0.00%   0 RMON Deferred Se

199           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps

200          20         2      10000  0.00%  0.00%  0.00%   0 VLAN Manager

201        2956   1883549          1  0.00%  0.00%  0.00%   0 DHCPD Database

202      244488 112846050          2  0.00%  0.00%  0.00%   0 trunk conditioni

203           0         1          0  0.00%  0.00%  0.00%   0 trunk conditioni

204           0         2          0  0.00%  0.00%  0.00%   0 EEM Policy Direc

205     5445956  74591605         73  0.00%  0.04%  0.05%   0 Syslog

206           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal

207      239888   6311492         38  0.00%  0.00%  0.00%   0 IP SLAs Event Pr

208           0        21          0  0.00%  0.00%  0.00%   0 Net Input

209      569360  22602683         25  0.00%  0.00%  0.00%   0 Compute load avg

210    32977268   1909947      17266  0.00%  0.03%  0.00%   0 Per-minute Jobs

211       96992   3919547         24  0.00%  0.00%  0.00%   0 CEF Scanner

212     1887592      9886     190935  0.00%  0.00%  0.00%   0 crypto sw pk pro

213           0         1          0  0.00%  0.00%  0.00%   0 tHUB

214           0         2          0  0.00%  0.00%  0.00%   0 tENM

215         460      2876        159  0.00%  0.00%  0.00%   0 SSH Event handle

216         228       433        526  0.00%  0.00%  0.00%   0 TCP Listener

217    53965516 117099189        460  0.00%  0.05%  0.05%   0 IP SNMP

218    14740996  57836129        254  0.00%  0.01%  0.00%   0 PDU DISPATCHER

219    91337708  57836954       1579  0.00%  0.10%  0.08%   0 SNMP ENGINE

220           0         1          0  0.00%  0.00%  0.00%   0 SNMP ConfCopyPro

221           0         1          0  0.00%  0.00%  0.00%   0 SNMP Traps

222      546544 113189644          4  0.00%  0.00%  0.00%   0 NTP

223        1012        15      67466  0.00%  0.00%  0.00% 194 SSH Process

224        1000         8     125000  0.00%  0.00%  0.00% 195 SSH Process

225      115968  10818533         10  0.00%  0.00%  0.00%   0 Track

114           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background

    8877777888887777788888777777777777777777777777788888888888

    6677777000006666622222555556666688888999995555533333444442

100

90 **

80 ************************************************************

70 ************************************************************

60 ************************************************************

50 ************************************************************

40 ************************************************************

30 ************************************************************

20 ************************************************************

10 ************************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)

    8888888887888988998899889989889889888988899888988899999898

    4847203239886078007726971074980981769266821888098815122919

100                      *                             *

90  * *      **************************************************

80 ####******############################*###*#################

70 ############################################################

60 ############################################################

50 ############################################################

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

    9998998999899889888876875998695562463239453245566645555633323429353622

    5608971144811769712891133388496979120709760766093791332097290899182589

100 **  **         *          *  *         *                       *

90 ****** **********  *     *** *         *                       *

80 ##******************* *  *** *         *                       *

70 ###*##*#########***** ** *#* *  *      *       * *             *   *

60 #################*#***** *#******  *   * *   * ***     *       * * *

50 ####################******##*****  *   ***  ************     * * * *

40 #####################**#**####*** **   ***  *****#********   * * * *  **

30 ########################**#####**************###########****************

20 ########################*#######***#***#*****###########****************

10 ########################################################################

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

2 Replies 2

zyang
Beginner
Beginner

Also the interface for outbound traffic is:

  5 minute input rate 627000 bits/sec, 182 packets/sec

  5 minute output rate 493000 bits/sec, 201 packets/sec

Which doesn't suggest alot of traffic flowing through.

There is a huge ACL and FW inspection is on.  Though with that little traffic, I don't see why it could cause such high cpu loads. 

Thanks and any help would be appreciated in diagnosing the problem.

enable netflow on Router.

All thing will be clear to you, which port and ip using cpu.

***Do Rate Helpful Posts***

Jawad

Jawad
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: