7613 High CPU utilization

[Samer Ghishan]

We have a Cisco 7613 router and we are using it to aggregate the customer on through ethernet and E1s. The CPU utilization reached 84 % since the last couple of weeks. Below is the " show proc cpu sorted" command.

CPU utilization for five seconds: 80%/62%; one minute: 75%; five minutes: 75%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

462   8974379402129299664        421 10.95% 12.86% 12.97%   0 PPP Events

196   2583163922560587415          0  4.79%  2.94%  2.72%   0 IP Input

  10    85066040 395965254        214  0.63%  0.52%  0.48%   0 ARP Input

494    14400404    667101      21586  0.47%  0.22%  0.21%   0 MFI LFD Stats Pr

417    19016200  93700037        202  0.23%  0.09%  0.08%   0 OSPF-100 Router

492     5042360   2414006       2088  0.15%  0.06%  0.06%   0 Tag Control

175     1557312  11981435        129  0.07%  0.00%  0.00%   0 esw_vlan_stat_pr

266    12123656  31468460        385  0.07%  0.15%  0.12%   0 IP RIB Update

332    19359304   8098769       2390  0.07%  0.08%  0.08%   0 HIDDEN VLAN Proc

  38     1681700  12622832        133  0.07%  0.00%  0.00%   0 EnvMon

  25     5926416  37994469        155  0.07%  0.01%  0.00%   0 IPC Seat Manager

198     7727228  88461084         87  0.07%  0.04%  0.01%   0 ADJ resolve proc

284     6642508  16428072        404  0.07%  0.04%  0.02%   0 IPC LC Message H

  13           0         1          0  0.00%  0.00%  0.00%   0 ATM ASYNC PROC

  12           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer

  11       15220  16892514          0  0.00%  0.00%  0.00%   0 ARP Background

   7    48159076   3303297      14579  0.00%  0.18%  0.21%   0 Check heaps

   9           0         2          0  0.00%  0.00%  0.00%   0 Timers

   8          12       104        115  0.00%  0.00%  0.00%   0 Pool Manager

  14           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

  15           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager

  22        2544  16193096          0  0.00%  0.00%  0.00%   0 IPC Periodic Tim

  16         108       514        210  0.00%  0.00%  0.00%   0 Entity MIB API

  24        1648  16193103          0  0.00%  0.00%  0.00%   0 IPC Deferred Por

  17      515384  16205021         31  0.00%  0.00%  0.00%   0 EEM ED Syslog

  18           0         1          0  0.00%  0.00%  0.00%   0 IFS Agent Manage

  27       32108   3239140          9  0.00%  0.00%  0.00%   0 IPC Loadometer

  19          36    269937          0  0.00%  0.00%  0.00%   0 IPC Dynamic Cach

  29           0         1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa

  20         564   3239111          0  0.00%  0.00%  0.00%   0 IPC Service NonC

  31           0         1          0  0.00%  0.00%  0.00%   0 CEF MIB API

  32         208     10650         19  0.00%  0.00%  0.00%   0 DDR Timers

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  21          12        92        130  0.00%  0.00%  0.00%   0 PF_Split Sync Pr

  34           0         1          0  0.00%  0.00%  0.00%   0 SENSOR-MGR event

  23       46320  73892193          0  0.00%  0.00%  0.00%   0 IPC Managed Time

  36           0         1          0  0.00%  0.00%  0.00%   0 Crash writer

  37           0         1          0  0.00%  0.00%  0.00%   0 Exception contro

  26        5864     23052        254  0.00%  0.00%  0.00%   0 IPC Seat Control

  39        1860  16193067          0  0.00%  0.00%  0.00%   0 GraphIt

  40         148      1707         86  0.00%  0.00%  0.00%   0 rf proxy rp agen

  41          76       633        120  0.00%  0.00%  0.00%   0 rf proxy message

  42          56       354        158  0.00%  0.00%  0.00%   0 Entity MIB C6k P

  43           0         3          0  0.00%  0.00%  0.00%   0 client_entity_se

  44           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect

  28           0         1          0  0.00%  0.00%  0.00%   0 RO Notify Timers

  46           8       144         55  0.00%  0.00%  0.00%   0 Cat6k SNMP

  47        2552     52929         48  0.00%  0.00%  0.00%   0 Cat6k SNMP Trap

  30         104   1619602          0  0.00%  0.00%  0.00%   0 Compute SRP rate

  49           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd

   6           0         1          0  0.00%  0.00%  0.00%   0 PF Redun ICC Req

  51          12       259         46  0.00%  0.00%  0.00%   0 IDB Work

  52      118564  29227057          4  0.00%  0.00%  0.00%   0 Logger

  33           0        10          0  0.00%  0.00%  0.00%   0 ifIndex Receive

  54      198892  17640814         11  0.00%  0.00%  0.00%   0 Per-Second Jobs

  55     4359588    310322      14048  0.00%  0.02%  0.00%   0 Per-minute Jobs

  56         792     13746         57  0.00%  0.00%  0.00%   0 IF-MGR control p

  57         508    553902          0  0.00%  0.00%  0.00%   0 IF-MGR event pro

  35           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun

  59          40       941         42  0.00%  0.00%  0.00%   0 Env action

  60        1668  16193087          0  0.00%  0.00%  0.00%   0 EOBC ETSEC FREEZ

  61           0         2          0  0.00%  0.00%  0.00%   0 Flash ICC Server

  45           0         1          0  0.00%  0.00%  0.00%   0 Snmp ICC Process

   4           4       426          9  0.00%  0.00%  0.00%   0 Retransmission o

  64     2553496 142155332         17  0.00%  0.00%  0.00%   0 Net Input

  65           4        24        166  0.00%  0.00%  0.00%   0 draco-oir-proces

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  66           0         6          0  0.00%  0.00%  0.00%   0 FR LMI

   3           8        34        235  0.00%  0.00%  0.00%   0 VTEMPLATE Backgr

  68           8        17        470  0.00%  0.00%  0.00%   0 HA_INCR_SYNC_PRO

  69     2884516  21659843        133  0.00%  0.00%  0.00%   0 IFCOM Msg Hdlr

  70           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

   2       61828   3239135         19  0.00%  0.00%  0.00%   0 Load Meter

  72        1472   4557393          0  0.00%  0.00%  0.00%   0 BGP Scheduler

  73           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  74           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  75           4        20        200  0.00%  0.00%  0.00%   0 OBFL Cfg Dispatc

  76       57340   8373659          6  0.00%  0.00%  0.00%   0 IP SLAs Sync Pro

  77           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  78           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  79           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  80           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  81      231772 748291146          0  0.00%  0.00%  0.00%   0 IP SLA Responder

  82           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  48        2568  16193095          0  0.00%  0.00%  0.00%   0 Dynamic ARP Insp

  84           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  85           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  86           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  87        1052   1079735          0  0.00%  0.00%  0.00%   0 SCP async: Draco

   5           4        11        363  0.00%  0.00%  0.00%   0 IPC ISSU Dispatc

  89           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  90           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  91           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  92           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  93           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  94           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  95           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  96           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  97           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  50       81332  16234879          5  0.00%  0.00%  0.00%   0 Net Background

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  53        3024  16193061          0  0.00%  0.00%  0.00%   0 TTY Background

100           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

101           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

102           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

103           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

104           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

  58           0         7          0  0.00%  0.00%  0.00%   0 RPC rf_rfmib_not

106           0        11          0  0.00%  0.00%  0.00%   0 RPC Sync

107           0         3          0  0.00%  0.00%  0.00%   0 RPC rpc-slave-17

108           0         2          0  0.00%  0.00%  0.00%   0 SCP Multicast

109           0     26994          0  0.00%  0.00%  0.00%   0 simba

110           4         8        500  0.00%  0.00%  0.00%   0 MWAM CONFIG ON S

111           0         1          0  0.00%  0.00%  0.00%   0 SAMI CONFIG ON S

112           0         1          0  0.00%  0.00%  0.00%   0 CHKPT Test clien

113           0         8          0  0.00%  0.00%  0.00%   0 DHCPC HA

114           0         8          0  0.00%  0.00%  0.00%   0 DHCPD HA

115           0         1          0  0.00%  0.00%  0.00%   0 MDR SM Proxy Sla

   1          32       329         97  0.00%  0.00%  0.00%   0 Chunk Manager

117           0         3          0  0.00%  0.00%  0.00%   0 CWAN HA VLAN SYN

118           4        29        137  0.00%  0.00%  0.00%   0 CWAN IF EVENT HA

119           0         3          0  0.00%  0.00%  0.00%   0 CWAN APS HA Proc

120           0         3          0  0.00%  0.00%  0.00%   0 CWAN SRP HA Proc

121           0         1          0  0.00%  0.00%  0.00%   0 Inode Table Dest

122       62184  16189969          3  0.00%  0.00%  0.00%   0 Heartbeat Proces

  62           8        21        380  0.00%  0.00%  0.00%   0 ICC Nego

  63           0     27001          0  0.00%  0.00%  0.00%   0 itasca

125           0         1          0  0.00%  0.00%  0.00%   0 Const ICC except

  67       13360   1349966          9  0.00%  0.00%  0.00%   0 svclc

127      300232  27312172         10  0.00%  0.00%  0.00%   0 ICC Async mcast

128           0         1          0  0.00%  0.00%  0.00%   0 ICC_REQ Debug Pr

129           0         1          0  0.00%  0.00%  0.00%   0 ICC FC Process

130           0         1          0  0.00%  0.00%  0.00%   0 ICC FC Test Send

131           0         1          0  0.00%  0.00%  0.00%   0 ICC FC Test Low

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

132           0         1          0  0.00%  0.00%  0.00%   0 PF Sync process

  71        8060  63225798          0  0.00%  0.00%  0.00%   0 CCM Subscriber P

134           0         4          0  0.00%  0.00%  0.00%   0 CWRP SPA TSM HA

135           4         4       1000  0.00%  0.00%  0.00%   0 Switch IP Host T

136           0         1          0  0.00%  0.00%  0.00%   0 EM Action CNS

137           8        23        347  0.00%  0.00%  0.00%   0 PF_Init Process

138           0        28          0  0.00%  0.00%  0.00%   0 RPC pf-split-rp

139           4        28        142  0.00%  0.00%  0.00%   0 RPC idprom-MP

140        2540  10464681          0  0.00%  0.00%  0.00%   0 Transport Port A

141           0         1          0  0.00%  0.00%  0.00%   0 BACK CHECK

  83           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

143           0         8          0  0.00%  0.00%  0.00%   0 RFS server proce

144       79324   3299712         24  0.00%  0.00%  0.00%   0 Compute load avg

145          68       887         76  0.00%  0.00%  0.00%   0 CHKPT rcv MSG pr

146           0         2          0  0.00%  0.00%  0.00%   0 cpf_process_msg_

147      610728  12721085         48  0.00%  0.00%  0.00%   0 cpf_process_ipcQ

148           0        19          0  0.00%  0.00%  0.00%   0 IPC Apps Task

149     8996444 139866962         64  0.00%  0.03%  0.04%   0 IFCOM Msg Hdlr

150           0        28          0  0.00%  0.00%  0.00%   0 AggMgr Process

151           0         2          0  0.00%  0.00%  0.00%   0 CWAN ICC

152           0     27002          0  0.00%  0.00%  0.00%   0 SFF8472

153           0         1          0  0.00%  0.00%  0.00%   0 Remote Console P

154           0         1          0  0.00%  0.00%  0.00%   0 CCB SEND RETRY A

155           0         1          0  0.00%  0.00%  0.00%   0 POS APS Event Pr

156       57036    737530         77  0.00%  0.00%  0.00%   0 CWAN OIR Handler

157           0         2          0  0.00%  0.00%  0.00%   0 CWAN Egress SACL

158           0         1          0  0.00%  0.00%  0.00%   0 DSX3MIB ll handl

159           4        15        266  0.00%  0.00%  0.00%   0 FPD Management P

160           0         1          0  0.00%  0.00%  0.00%   0 FPD Action Proce

161           0         2          0  0.00%  0.00%  0.00%   0 const FPD ICC pr

162           0         1          0  0.00%  0.00%  0.00%   0 dwdm RPC RP proc

163           0         2          0  0.00%  0.00%  0.00%   0 netclk_process

164           4         4       1000  0.00%  0.00%  0.00%   0 netclk_ha_proces

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

165           0         7          0  0.00%  0.00%  0.00%   0 SCP async: LCP#7

  88           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

167        3788    277108         13  0.00%  0.00%  0.00%   0 CMFI RP process

168        1980  16193083          0  0.00%  0.00%  0.00%   0 CWAN CHOCX PROCE

169     1187460   6538083        181  0.00%  0.00%  0.00%   0 HAL Mailbox

170        3000  16193084          0  0.00%  0.00%  0.00%   0 CWAN CHDSX PROCE

171           0         1          0  0.00%  0.00%  0.00%   0 CT3 Mailbox

172           0         1          0  0.00%  0.00%  0.00%   0 CE3 Mailbox

173           0         2          0  0.00%  0.00%  0.00%   0 MIP Mailbox

174           0         1          0  0.00%  0.00%  0.00%   0 Earl NDE Task

  98           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

176           0        10          0  0.00%  0.00%  0.00%   0 Gemini L3 forwar

177           0         1          0  0.00%  0.00%  0.00%   0 CWRP SPA ATM OIR

178       37128 504301944          0  0.00%  0.00%  0.00%   0 SPA CHOCX APS Pr

179           4         1       4000  0.00%  0.00%  0.00%   0 spa_plim_fsvr

180           0         1          0  0.00%  0.00%  0.00%   0 Async write proc

181           0         1          0  0.00%  0.00%  0.00%   0 Async write proc

182           0         1          0  0.00%  0.00%  0.00%   0 Async write proc

183           0         1          0  0.00%  0.00%  0.00%   0 Async write proc

184          76    269937          0  0.00%  0.00%  0.00%   0 DFS flush period

185        2496     75217         33  0.00%  0.00%  0.00%   0 AAA Server

186           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc

187       47644 505437006          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr

188           0         1          0  0.00%  0.00%  0.00%   0 AAA System Acct

189           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B

190           0     53988          0  0.00%  0.00%  0.00%   0 IP Admin SM Proc

191           0        15          0  0.00%  0.00%  0.00%   0 CEF RP IPC Backg

192           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R

193          44    135025          0  0.00%  0.00%  0.00%   0 DHCP Snooping

194           0         2          0  0.00%  0.00%  0.00%   0 CEF switching ba

195        1204    541859          2  0.00%  0.00%  0.00%   0 IP ARP Adjacency

  99           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

197       20876  15263011          1  0.00%  0.00%  0.00%   0 ICMP event handl

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

105           0         1          0  0.00%  0.00%  0.00%   0 IPC ISSU Version

199           0         1          0  0.00%  0.00%  0.00%   0 PPCP RP Stats Ba

200           0         1          0  0.00%  0.00%  0.00%   0 IPv6 Echo event

201     1377024  10543075        130  0.00%  0.00%  0.00%   0 PPP SIP

202           0         2          0  0.00%  0.00%  0.00%   0 PPP Bind

203       14820     16020        925  0.00%  0.00%  0.00%   0 PPP IP Route

204      391240   5142228         76  0.00%  0.00%  0.00%   0 IP SLAs Sync Pro

205           0         3          0  0.00%  0.00%  0.00%   0 HDLC HA

206       81960   3221977         25  0.00%  0.00%  0.00%   0 ARP HA

207      909996  43478071         20  0.00%  0.00%  0.00%   0 Spanning Tree

208           4         5        800  0.00%  0.00%  0.00%   0 EAPoUDP Process

116       44044    269912        163  0.00%  0.00%  0.00%   0 RF CWAN HA Proce

210      482716   2999577        160  0.00%  0.00%  0.00%   0 SSS Manager

211           0         1          0  0.00%  0.00%  0.00%   0 SSS Policy Manag

212        3084    142329         21  0.00%  0.00%  0.00%   0 SSS Feature Mana

213        8684  63265246          0  0.00%  0.00%  0.00%   0 SSS Feature Time

214           0         1          0  0.00%  0.00%  0.00%   0 SG DPM

215           0         1          0  0.00%  0.00%  0.00%   0 Timer handler fo

216           0         1          0  0.00%  0.00%  0.00%   0 Prepaid response

217           0         1          0  0.00%  0.00%  0.00%   0 Timed Policy act

218           0         1          0  0.00%  0.00%  0.00%   0 AAA response han

219           0         1          0  0.00%  0.00%  0.00%   0 AAA EPD HANDLER

220           0         1          0  0.00%  0.00%  0.00%   0 PM EPD API

221           0         1          0  0.00%  0.00%  0.00%   0 AC Switch

222           0         2          0  0.00%  0.00%  0.00%   0 EFP Errd

223           0         1          0  0.00%  0.00%  0.00%   0 LSP Tunnel FRR

224           0         4          0  0.00%  0.00%  0.00%   0 MPLS Auto-Tunnel

225           0         1          0  0.00%  0.00%  0.00%   0 O-UNI Client Msg

226           0         3          0  0.00%  0.00%  0.00%   0 cv6 per prefix a

227       12756    998400         12  0.00%  0.00%  0.00%   0 CCM

228           0         3          0  0.00%  0.00%  0.00%   0 CCM IPC flow con

229           0      2994          0  0.00%  0.00%  0.00%   0 DMLP Counter

230           0         2          0  0.00%  0.00%  0.00%   0 802.1x Accountin

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

330       29700    595512         49  0.00%  0.00%  0.00%   0 Collection proce

331           0         5          0  0.00%  0.00%  0.00%   0 Const Rate Limit

209      158832   6660651         23  0.00%  0.00%  0.00%   0 SSM connection m

333           0         1          0  0.00%  0.00%  0.00%   0 FR XDR manager

334           0         1          0  0.00%  0.00%  0.00%   0 CWAN LTL manager

335         844   1756529          0  0.00%  0.00%  0.00%   0 CWAN-RP SCP Proc

336          12        32        375  0.00%  0.00%  0.00%   0 CWAN-RP SCP Inpu

337           0         4          0  0.00%  0.00%  0.00%   0 DHCP Snooping HA

338           0         2          0  0.00%  0.00%  0.00%   0 EEM ED CLI

339          72       215        334  0.00%  0.00%  0.00%   0 IPHC RP Process

340           4       171         23  0.00%  0.00%  0.00%   0 L2TRACE SERVER

341           0         1          0  0.00%  0.00%  0.00%   0 MLSM-CC Process

342        1196    276422          4  0.00%  0.00%  0.00%   0 CWPA QoS

343           0         2          0  0.00%  0.00%  0.00%   0 Const TCAM Mgr R

344           0         2          0  0.00%  0.00%  0.00%   0 MWAM_CONSOLE Tim

345           0         1          0  0.00%  0.00%  0.00%   0 Container Proces

346         316   3954152          0  0.00%  0.00%  0.00%   0 IP SIP Process

347           0         2          0  0.00%  0.00%  0.00%   0 IP Inband Sessio

348           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr

349        5180     75847         68  0.00%  0.00%  0.00%   0 LOCAL AAA

350           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA

351           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA

352     3236876  92848343         34  0.00%  0.01%  0.00%   0 OSPF-100 Hello

353           0         2          0  0.00%  0.00%  0.00%   0 TPLUS

354           0         1          0  0.00%  0.00%  0.00%   0 IDMGR CORE

355           4       205         19  0.00%  0.00%  0.00%   0 MPLS Auto Mesh P

356           0         1          0  0.00%  0.00%  0.00%   0  Laminar Icc Eve

357           0         2          0  0.00%  0.00%  0.00%   0 SLB Main CPU pro

358           0         1          0  0.00%  0.00%  0.00%   0 DATA Transfer Pr

359           0         1          0  0.00%  0.00%  0.00%   0 DATA Collector

360       10556   1068799          9  0.00%  0.00%  0.00%   0 AAA SEND STOP EV

361           0        10          0  0.00%  0.00%  0.00%   0 EEM ED Resource

362           0        19          0  0.00%  0.00%  0.00%   0 EEM ED Track

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

363           0        25          0  0.00%  0.00%  0.00%   0 L3MM Process

364           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps

365           0        12          0  0.00%  0.00%  0.00%   0 RPC aapi_rp

366          52      1125         46  0.00%  0.00%  0.00%   0 RPC capi_rp

367           0         3          0  0.00%  0.00%  0.00%   0 RPC chapi_rp

368          24       781         30  0.00%  0.00%  0.00%   0 RPC dot1x_aaa_rp

369           0         2          0  0.00%  0.00%  0.00%   0 RPC ether_oam_sp

370           0         2          0  0.00%  0.00%  0.00%   0 RPC epld_sp2rp

371           0         2          0  0.00%  0.00%  0.00%   0 RPC gvrp_sp2rp

372           8        74        108  0.00%  0.00%  0.00%   0 RPC idbman_sp2mp

373           0         1          0  0.00%  0.00%  0.00%   0 LC Registry Expo

374       86740   1978325         43  0.00%  0.00%  0.00%   0 RPC online_diag_

375           0        46          0  0.00%  0.00%  0.00%   0 RPC online_wan_d

376          92       156        589  0.00%  0.00%  0.00%   0 RPC pagp_switch_

377          92       693        132  0.00%  0.00%  0.00%   0 RPC pm_switch_sp

378           0         2          0  0.00%  0.00%  0.00%   0 RPC psecure_rp

379           0         2          0  0.00%  0.00%  0.00%   0 RPC pt_switch_sp

380          16       692         23  0.00%  0.00%  0.00%   0 RPC qnq_switch_s

381           0         2          0  0.00%  0.00%  0.00%   0 RPC rf_cli_sp2rp

382           0         1          0  0.00%  0.00%  0.00%   0 SONET Traps

383       15144 134793030          0  0.00%  0.00%  0.00%   0 FR Broadcast Out

384           0         2          0  0.00%  0.00%  0.00%   0 VTPMIB EDIT BUFF

385           0         2          0  0.00%  0.00%  0.00%   0 RPC snmp_notify

386           4        45         88  0.00%  0.00%  0.00%   0 RPC snmp_stackli

387           0         2          0  0.00%  0.00%  0.00%   0 RPC span_switch_

388         188      2047         91  0.00%  0.00%  0.00%   0 RPC stp_sp2mp_ap

389           0         1          0  0.00%  0.00%  0.00%   0 EIGRP-IPv4: Rout

390           0         2          0  0.00%  0.00%  0.00%   0 Module enable-di

391          16        45        355  0.00%  0.00%  0.00%   0 RPC c6k_power_rp

392           0        44          0  0.00%  0.00%  0.00%   0 RPC c6k_rp_envir

393           0         1          0  0.00%  0.00%  0.00%   0 CONST AFL

394           0         2          0  0.00%  0.00%  0.00%   0 RPC cwan_ltl_sp2

395         208   2696331          0  0.00%  0.00%  0.00%   0 RP Error Detecti

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

396           0         2          0  0.00%  0.00%  0.00%   0 RPC error_detect

397           0         1          0  0.00%  0.00%  0.00%   0 HW Capacity ICC

398           0         2          0  0.00%  0.00%  0.00%   0 RPC itasca_rp

399           0         1          0  0.00%  0.00%  0.00%   0 RP L2 Applicatio

400           0         8          0  0.00%  0.00%  0.00%   0 LACP-MP Process

401           0        36          0  0.00%  0.00%  0.00%   0 Online Diag EEM

402           0         8          0  0.00%  0.00%  0.00%   0 PAgP-MP Process

403           0         2          0  0.00%  0.00%  0.00%   0 RPC pclc_g2_agsm

404           4        17        235  0.00%  0.00%  0.00%   0 RPC pclc_g2_fw_r

405           8        61        131  0.00%  0.00%  0.00%   0 Logger ICC Proce

406           0        10          0  0.00%  0.00%  0.00%   0 RPC pf_issu_sp2r

407           0         2          0  0.00%  0.00%  0.00%   0 RPC pikachu_rp

408           0         2          0  0.00%  0.00%  0.00%   0 FR ARP

409         732      6107        119  0.00%  0.00%  0.00%   0 RPC pm-mp

410       40784  89666588          0  0.00%  0.00%  0.00%   0 PM Callback

411    56145644 547247522        102  0.00%  0.25%  0.24%   0 Port manager per

412         472   3239108          0  0.00%  0.00%  0.00%   0 Max CPU utilizat

413           0        16          0  0.00%  0.00%  0.00%   0 RPC lcc_sp

414           0         9          0  0.00%  0.00%  0.00%   0 RPC fabman_sp

415           0      4500          0  0.00%  0.00%  0.00%   0 rp to lc clock u

416      145900   1091962        133  0.00%  0.00%  0.00%   0 SPA ENTITY Proce

231           0        22          0  0.00%  0.00%  0.00%   0 Dot1x Mgr Proces

418           0         2          0  0.00%  0.00%  0.00%   0 SPAN Capture

419           0        14          0  0.00%  0.00%  0.00%   0 EC-Card-Cfg

420           0         2          0  0.00%  0.00%  0.00%   0 RPC sw_backup_in

421       16796    209750         80  0.00%  0.00%  0.00%   0 RPC sw_vlan_rp

422           0         4          0  0.00%  0.00%  0.00%   0 RPC udld_rp

423       13328    105062        126  0.00%  0.00%  0.00%   0 RPC vtp_const_ap

424           0         2          0  0.00%  0.00%  0.00%   0 RPC rep_sp2rp_ap

425           0         2          0  0.00%  0.00%  0.00%   0 RPC dot1ad_sp2mp

426           0         2          0  0.00%  0.00%  0.00%   0 Rommon upgrade p

427          60    269976          0  0.00%  0.00%  0.00%   0 Call Home Timer

428           4         5        800  0.00%  0.00%  0.00%   0 CWAN OIR IPC Rea

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

429           0      4499          0  0.00%  0.00%  0.00%   0 PF Clock Process

430           0         3          0  0.00%  0.00%  0.00%   0 EEM ED Counter

431           0         3          0  0.00%  0.00%  0.00%   0 EM ED GOLD

432           0         3          0  0.00%  0.00%  0.00%   0 EEM ED Interface

433           0         3          0  0.00%  0.00%  0.00%   0 EEM ED IOSWD

434           0         3          0  0.00%  0.00%  0.00%   0 EEM ED None

435           0         3          0  0.00%  0.00%  0.00%   0 EEM ED OIR

436           0        11          0  0.00%  0.00%  0.00%   0 EEM ED RF

437           0         3          0  0.00%  0.00%  0.00%   0 EEM ED SNMP

438          32    405772          0  0.00%  0.00%  0.00%   0 EEM ED Timer

439           0         3          0  0.00%  0.00%  0.00%   0 EEM ED Test

440           0         3          0  0.00%  0.00%  0.00%   0 EEM ED Config

441           0         3          0  0.00%  0.00%  0.00%   0 EEM ED Env

442           0         2          0  0.00%  0.00%  0.00%   0 Call Home proces

443           0         1          0  0.00%  0.00%  0.00%   0 IP SLAs Ethernet

444           0         1          0  0.00%  0.00%  0.00%   0 IP SLA AutoEther

445         128    808857          0  0.00%  0.00%  0.00%   0 IP SLA MPLSLM Pr

446        3908    276358         14  0.00%  0.00%  0.00%   0 DRMI Config Disp

447      216500   1220568        177  0.00%  0.00%  0.00%   0 Syslog

448           4        20        200  0.00%  0.00%  0.00%   0 CWAN Bridge task

449           0         3          0  0.00%  0.00%  0.00%   0 QM MRM Process o

450         244       756        322  0.00%  0.00%  0.00%   0 QM Process on RP

451          24       268         89  0.00%  0.00%  0.00%   0 QM Reply Process

452           0         2          0  0.00%  0.00%  0.00%   0 QM Timer ACL Pro

453         144   1295816          0  0.00%  0.00%  0.00%   0 DHCP Snooping Sw

454        2600   3239099          0  0.00%  0.00%  0.00%   0 Track

455           0        21          0  0.00%  0.00%  0.00%   0 CONST PC POLICYM

456           0         1          0  0.00%  0.00%  0.00%   0 Online Diag CNS

457           0         1          0  0.00%  0.00%  0.00%   0 Online Diag CNS

458          20    269937          0  0.00%  0.00%  0.00%   0 VACL Logger

459           0        11          0  0.00%  0.00%  0.00%   0 REP RP Proc

460           0         2          0  0.00%  0.00%  0.00%   0 SIP10G QM Proces

461       78956 505435229          0  0.00%  0.00%  0.00%   0 PPP manager

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

246         748      5206        143  0.00%  0.00%  0.00%   0 HWIF QoS Process

463       18836  16194066          1  0.00%  0.00%  0.00%   0 Multilink PPP

464          52      5834          8  0.00%  0.00%  0.00%   0 Multilink event

465          16    224926          0  0.00%  0.00%  0.00%   0 FR FRAGMENTATION

466      1499002021680101          0  0.00%  0.00%  0.00%   0 IP SLAs Event Pr

467          24       377         63  0.00%  0.00%  0.00%   0 Kron Process

468    16103552 195012892         82  0.00%  0.11%  0.05%   0 IP SNMP

469     4469844 100146655         44  0.00%  0.01%  0.00%   0 PDU DISPATCHER

470    24507380 103031899        237  0.00%  0.13%  0.07%   0 SNMP ENGINE

471           0         2          0  0.00%  0.00%  0.00%   0 IP SNMPV6

472           0         1          0  0.00%  0.00%  0.00%   0 SNMP ConfCopyPro

473           0         1          0  0.00%  0.00%  0.00%   0 SNMP Traps

474         604    674773          0  0.00%  0.00%  0.00%   0 FR TUNNEL

475           0       175          0  0.00%  0.00%  0.00%   0 RSVP

476           0         1          0  0.00%  0.00%  0.00%   0 FRF9 manager

477           0         1          0  0.00%  0.00%  0.00%   0 FRF9 timed event

478        2248   2159465          1  0.00%  0.00%  0.00%   0 SCP async: Draco

479          68   1349536          0  0.00%  0.00%  0.00%   0 compute load per

480      402924   3833922        105  0.00%  0.00%  0.00%   0 LDP

481      610560   6534577         93  0.00%  0.00%  0.00%   0 IP SLAs Sync Pro

482      270776   3421106         79  0.00%  0.00%  0.00%   0 IP SLAs Sync Pro

483      104060   7524408         13  0.00%  0.00%  0.00%   0 IP SLAs Sync Pro

484         440    316533          1  0.00%  0.00%  0.00%   0 SCP async: Draco

485      999588  15730539         63  0.00%  0.00%  0.00%   0 IFCOM Msg Hdlr

486      809420  22421252         36  0.00%  0.00%  0.00%   0 BGP Router

487        3908    167221         23  0.00%  0.00%  0.00%   0 BGP I/O

488      355644     85887       4140  0.00%  0.00%  0.00%   0 BGP Scanner

489           0        79          0  0.00%  0.00%  0.00%   0 BGP Event

490           4         6        666  0.00%  0.00%  0.00%   0 TCP Listener

491        1484    934268          1  0.00%  0.00%  0.00%   0 LSD Main Proc

247           0         3          0  0.00%  0.00%  0.00%   0 EEM Policy Direc

493        4940     77804         63  0.00%  0.00%  0.00%   0 IPRM

252           0         1          0  0.00%  0.00%  0.00%   0 Switch IP Admiss

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

495         104    311210          0  0.00%  0.00%  0.00%   0 MFI LFD Timer Pr

496           4         2       2000  0.00%  0.00%  0.00%   0 MFI LFD Main Pro

497         820     20962         39  0.00%  0.00%  0.00%   0 Tagcon Addr

499         144       193        746  0.00%  0.06%  0.01%   1 Virtual Exec

500       60724   2129416         28  0.00%  0.00%  0.00%   0 TDP Hello

501         684   2303045          0  0.00%  0.00%  0.00%   0 MPLS HC Counter

502        1380         7     197142  0.00%  0.00%  0.00%   0 IPRM Collection

503           0         1          0  0.00%  0.00%  0.00%   0 LSP Verification

504        2324     77698         29  0.00%  0.00%  0.00%   0 LDP Background

[Reza Sharifi]

what version of IOS are you running?

[Samer Ghishan]

we are running this version

c7600rsp72043-advipservices-mz.122-33.SRD4.bin

[Peter Paluch]

Samer,

According to the output, your router is currently high in the interrupt level CPU load. That can be caused by several reasons - please have a look at this document which deals with these issues in more detail:

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

I am sorry for being unable to guide you more here as I have almost no experiences with these high-end devices. Still, this document could be a good start.

Best regards,

Peter

[Giuseppe Larosa]

Hello Samer,

your device is processing a lot of traffic on the main cpu

see:

CPU utilization for five seconds: 80%>>>62%<<<<; one minute: 75%; five minutes: 75%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

462   8974379402129299664        421 10.95% 12.86% 12.97%   0 PPP Events

most of cpu usage is caused by interrupts ( 62%) and 13% is used by PPP

see

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml

and

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml#utilities

and troubleshooting high cpu usage caused by interrupts

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

Are you using forms of multilink PPP over the E1 links?

the system should not allow to configure bundles of E1 that are in different PAs or different SPAs

Hope to help

Giuseppe

[andtoth]

Hi,

The device is utilizing the CPU due to interrupts which usually means packets are being software switched by the CPU.

Have a look at the following documentation which was originally written for 6500 switches but since they are similar with 7600 platform, it should give you some useful ideas.

Catalyst 6500/6000 Switch High CPU Utilization

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml

Also, try to think of recent changes in the configuration which might have caused the increase in the CPU utilization and try matching them with the exceptions mentioned in the documentation when a packet needs special processing.

If you cannot solve the issue following the documentation, you might want to consider opening a TAC case so that an engineer can have a closer look and assist you in finding the root cause of the problem.

Best regards,

Andras

[andtoth]

Sorry, double post.

[andtoth]

Sorry, double post.