High CPU utilization in a CISCO GGSN (CISCO 7604 ios 12.4)

socrates_2000 · ‎07-27-2011

We have a CISCO 7604 Router (GGSN) in our customer site. Since two weeks are observing high CPU utilization (even touching 100 percent). We performed a SNMP configuration on this routers 2 weeks back as well. But SNMP process is not consuming much CPU resources. The following is the output of the "show proc cpu" and "show proc cpu history. Please help me with a fix.

------------------ show process cpu ------------------

CPU utilization for five seconds: 50%/15%; one minute: 49%; five minutes: 51%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 1436 36481 39 0.00% 0.00% 0.00% 0 Chunk Manager

2 19452 332170 58 0.00% 0.00% 0.00% 0 Load Meter

3 0 1 0 0.00% 0.00% 0.00% 0 chkpt message ha

4 16658292 808257 20610 1.99% 0.69% 0.79% 0 Check heaps

5 29224 727491 40 0.00% 0.00% 0.00% 0 Pool Manager

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

7 132 27660 4 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

8 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

9 4228 1656478 2 0.00% 0.00% 0.00% 0 IPC Periodic Tim

10 3704 1656472 2 0.00% 0.00% 0.00% 0 IPC Deferred Por

11 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager

12 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure

13 3886236 76765855 50 0.31% 0.31% 0.31% 0 ARP Input

14 108124 1725807 62 0.00% 0.02% 0.01% 0 ARP Background

15 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

16 1390416 586910 2369 0.00% 0.06% 0.05% 0 AAA high-capacit

17 0 627 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

18 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

19 0 3 0 0.00% 0.00% 0.00% 0 DDR Timers

20 0 4 0 0.00% 0.00% 0.00% 0 Entity MIB API

21 8972 395087 22 0.00% 0.00% 0.00% 0 HC Counter Timer

22 0 2 0 0.00% 0.00% 0.00% 0 PrstVbl

23 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

24 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers

25 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa

26 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

27 1852640 76128016 24 0.00% 0.00% 0.00% 0 EEM ED Syslog

28 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

29 4252 1660846 2 0.00% 0.00% 0.00% 0 GraphIt

30 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

31 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

32 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

33 1328 168742 7 0.00% 0.00% 0.00% 0 Net Background

34 0 3 0 0.00% 0.00% 0.00% 0 IDB Work

35 1539012 22576268 68 0.00% 0.00% 0.00% 0 Logger

36 290440 1656417 175 0.07% 0.04% 0.05% 0 TTY Background

37 19464 1660934 11 0.00% 0.00% 0.00% 0 Per-Second Jobs

38 0 1 0 0.00% 0.00% 0.00% 0 DHCPD HA

39 0 1 0 0.00% 0.00% 0.00% 0 DHCPC HA

40 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

41 4 4 1000 0.00% 0.00% 0.00% 0 rf task

42 8 698 11 0.00% 0.00% 0.00% 0 Net Input

43 8200 332283 24 0.00% 0.00% 0.00% 0 Compute load avg

44 122488 29919 4093 0.07% 0.00% 0.00% 0 Per-minute Jobs

45 56 12127 4 0.00% 0.00% 0.00% 0 Transport Port A

46 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process

47 0 1 0 0.00% 0.00% 0.00% 0 RF_INTERDEV_DELA

48 0 1 0 0.00% 0.00% 0.00% 0 RF_INTERDEV_SCTP

49 0 2 0 0.00% 0.00% 0.00% 0 SAMI IPCP retran

50 8 49 163 0.00% 0.00% 0.00% 0 SAMI_EOBC_MAC_PR

51 4496 165902 27 0.00% 0.00% 0.00% 0 SAMI Health Moni

52 0 3 0 0.00% 0.00% 0.00% 0 SAMI LCP Notific

53 1080 379616 2 0.00% 0.00% 0.00% 0 ddr monitor

54 0 1 0 0.00% 0.00% 0.00% 0 Bouncer Nitrox C

55 0 1 0 0.00% 0.00% 0.00% 0 CF_INTERDEV_SCTP

56 0 2 0 0.00% 0.00% 0.00% 0 CEF switching ba

57 3305384 51171651 64 0.31% 0.25% 0.28% 0 ADJ resolve proc

58 0 2 0 0.00% 0.00% 0.00% 0 SSM connection m

59 0 2 0 0.00% 0.00% 0.00% 0 PSD Client

60 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

61 13411480 93461095 143 0.15% 0.20% 0.22% 0 AAA Server

62 14048596 96595345 145 0.79% 0.82% 0.85% 0 AAA ACCT Proc

63 574536 20618540 27 0.15% 0.11% 0.10% 0 ACCT Periodic Pr

64 3712 165967 22 0.00% 0.00% 0.00% 0 CDP Protocol

65 4724 22879 206 0.00% 0.00% 0.00% 0 IP ARP Adjacency

66 0 1 0 0.00% 0.00% 0.00% 0 IP ARP Retry Age

67 65305420 759669253 85 3.75% 3.61% 3.51% 0 IP Input

68 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl

69 0 14 0 0.00% 0.00% 0.00% 0 TurboACL

70 0 2 0 0.00% 0.00% 0.00% 0 TurboACL chunk

71 0 1 0 0.00% 0.00% 0.00% 0 LSP Tunnel FRR

72 0 1 0 0.00% 0.00% 0.00% 0 MPLS Auto-Tunnel

73 0 3 0 0.00% 0.00% 0.00% 0 PPP Hooks

75 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager

76 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana

77 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Time

78 4 2 2000 0.00% 0.00% 0.00% 0 Spanning Tree

79 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

80 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Echo event

81 2908 252338 11 0.00% 0.00% 0.00% 0 CEF background p

82 2537204 28269 89752 0.00% 0.07% 0.09% 0 IP Background

83 13182572 78885015 167 0.71% 0.71% 0.74% 0 IP RIB Update

84 23576 928237 25 0.00% 0.00% 0.00% 0 CEF: IPv4 proces

85 0 5 0 0.00% 0.00% 0.00% 0 ADJ background

86 0 3 0 0.00% 0.00% 0.00% 0 Flow Exporter Ti

87 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

88 0 1 0 0.00% 0.00% 0.00% 0 GTP Proxy PPP

89 0 1 0 0.00% 0.00% 0.00% 0 GTP Proxy PPP De

90 620 49247 12 0.00% 0.00% 0.00% 0 GPRS-CF

91 0 10 0 0.00% 0.00% 0.00% 0 SCTP Main Proces

92 0 1 0 0.00% 0.00% 0.00% 0 CHKPT EXAMPLE

93 0 1 0 0.00% 0.00% 0.00% 0 CHKPT DevTest

94 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage

95 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route

96 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP

97 0 3 0 0.00% 0.00% 0.00% 0 SNMP Timers

98 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute

99 1524 84712 17 0.00% 0.00% 0.00% 0 TCP Timer

100 488 11514 42 0.00% 0.00% 0.00% 0 TCP Protocols

101 0 1 0 0.00% 0.00% 0.00% 0 COPS

102 24 5547 4 0.00% 0.00% 0.00% 0 HTTP CORE

103 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

104 0 1 0 0.00% 0.00% 0.00% 0 LAPB Process

105 0 2 0 0.00% 0.00% 0.00% 0 LFDp Input Proc

106 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

107 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background

108 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

109 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS

110 0 1 0 0.00% 0.00% 0.00% 0 MQC Flow Event B

111 681092 348012352 1 0.15% 0.15% 0.15% 0 HQF Shaper Backg

112 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon

113 3180 829143 3 0.00% 0.00% 0.00% 0 BOUNCER_CONSOLE

114 0 2 0 0.00% 0.00% 0.00% 0 bouncer_TTY Back

115 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Inspect Tim

116 0 1 0 0.00% 0.00% 0.00% 0 Crypto HW Proc

117 0 2 0 0.00% 0.00% 0.00% 0 NOR_RTP

118 0 2 0 0.00% 0.00% 0.00% 0 cpf_process_msg_

119 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve

120 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

121 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr

122 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

123 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

124 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA

125 0 15 0 0.00% 0.00% 0.00% 0 MPLS Auto Mesh P

126 0 2 0 0.00% 0.00% 0.00% 0 Diameter Base pr

127 0 1 0 0.00% 0.00% 0.00% 0 Diameter Transpo

128 0 2 0 0.00% 0.00% 0.00% 0 TPLUS

129 36472 16382665 2 0.07% 0.00% 0.00% 0 fastblk backgrou

130 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support

131 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT

132 0 3 0 0.00% 0.00% 0.00% 0 Crypto IKE Dispa

133 0 3 0 0.00% 0.00% 0.00% 0 Crypto IKMP

134 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key

135 476 82945 5 0.00% 0.00% 0.00% 0 IPSEC key engine

136 0 1 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce

137 0 26 0 0.00% 0.00% 0.00% 0 Crypto ACL

138 4 1 4000 0.00% 0.00% 0.00% 0 Crypto PAS Proc

139 52 9 5777 0.00% 0.00% 0.00% 0 crypto engine pr

140 4 4 1000 0.00% 0.00% 0.00% 0 Crypto CA

141 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL

142 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc

143 0 2 0 0.00% 0.00% 0.00% 0 crypto sw pk pro

144 0 2 0 0.00% 0.00% 0.00% 0 Lawful Intercept

145 0 2 0 0.00% 0.00% 0.00% 0 LI messaging

146 588 55397 10 0.00% 0.00% 0.00% 0 IPv6 IDB

147 200 27661 7 0.00% 0.00% 0.00% 0 IPv6 Input

148 0 1 0 0.00% 0.00% 0.00% 0 IPv6 ND

149 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Address

150 32428 6643366 4 0.00% 0.00% 0.00% 0 MLD

151 0 3 0 0.00% 0.00% 0.00% 0 IPv6 RIB Event H

152 0 2 0 0.00% 0.00% 0.00% 0 Control-plane ho

153 10093684 148742879 67 0.23% 0.25% 0.24% 0 AAA SEND STOP EV

154 0 1 0 0.00% 0.00% 0.00% 0 cpf_process_tpQ

155 372 165859 2 0.00% 0.00% 0.00% 0 RMON Recycle Pro

156 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se

157 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

158 0 1 0 0.00% 0.00% 0.00% 0 DATA Transfer Pr

159 0 1 0 0.00% 0.00% 0.00% 0 DATA Collector

160 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Resource

161 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Track

162 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Routing

163 0 1 0 0.00% 0.00% 0.00% 0 VPDN Test

164 1469176 22311637 65 0.00% 0.00% 0.00% 0 Syslog

165 4 270 14 0.00% 0.00% 0.00% 0 EEM Server

166 0 3 0 0.00% 0.00% 0.00% 0 EEM ED CLI

167 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Counter

168 0 3 0 0.00% 0.00% 0.00% 0 EM ED GOLD

169 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Interface

170 0 3 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD

171 0 3 0 0.00% 0.00% 0.00% 0 EEM ED None

172 0 3 0 0.00% 0.00% 0.00% 0 EEM ED OIR

173 0 3 0 0.00% 0.00% 0.00% 0 EEM ED SNMP

174 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Test

175 184 45180 4 0.00% 0.00% 0.00% 0 EEM ED Timer

176 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc

177 0 3 0 0.00% 0.00% 0.00% 0 EEM ED RF

178 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP Noti

179 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Config

180 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Env

181 10148 4129972 2 0.00% 0.00% 0.00% 0 DUART Background

182 24 188 127 0.00% 0.00% 0.00% 3 Virtual Exec

183 0 1 0 0.00% 0.00% 0.00% 0 Key Proc

184 338568532 180775847 1872 16.30% 15.29% 15.44% 0 GTP Management

185 68 553 122 0.00% 0.00% 0.00% 0 Collection proce

186 0 1 0 0.00% 0.00% 0.00% 0 GPRS DNS Managem

187 783080 345563 2266 0.00% 0.01% 0.00% 0 GTP PDP Cleanup

188 209011156 183022403 1141 5.99% 6.57% 7.14% 0 GPRS RADIUS Mana

189 0 1 0 0.00% 0.00% 0.00% 0 GPRS DCCA Client

190 484188 6619977 73 0.07% 0.04% 0.02% 0 DHCP Client

191 357640 3700177 96 0.00% 0.01% 0.00% 0 GPRS DHCP Manage

192 8917144 1580555 5641 0.23% 0.56% 0.55% 0 GPRS DHCP Client

193 5556 988 5623 0.00% 0.00% 0.00% 0 GPRS Charging

194 8872 21447 413 0.00% 0.00% 0.00% 0 GPRS Charging Tr

195 0 1 0 0.00% 0.00% 0.00% 0 GGSN iSCSI proce

196 60563196 483294801 125 1.83% 2.06% 2.17% 0 RADIUS

197 24054624 436402697 55 1.67% 1.56% 1.47% 0 GTP I/O

198 233272 51329053 4 0.00% 0.04% 0.05% 0 PPP manager

199 180528 51329049 3 0.07% 0.02% 0.00% 0 PPP Events

200 3552 1657387 2 0.00% 0.00% 0.00% 0 Multilink PPP

201 303256 9218097 32 0.00% 0.02% 0.05% 0 IP SNMP

202 93368 1636934 57 0.00% 0.00% 0.00% 0 PDU DISPATCHER

203 11566196 1677628 6894 0.00% 0.00% 0.58% 0 SNMP ENGINE

204 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6

205 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

206 45660976 74294763 614 0.15% 0.16% 0.18% 0 SNMP Traps

207 0 95 0 0.00% 0.00% 0.00% 0 MPLS MIB traps

208 17860 1705409 10 0.00% 0.05% 0.46% 0 NTP

209 376 27724 13 0.00% 0.00% 0.00% 0 Time Range Proce

210 20 3 6666 0.00% 0.00% 0.00% 0 VTEMPLATE Backgr

211 2500 300 8333 0.00% 0.33% 0.28% 4 Virtual Exec

212 382108 1745612 218 0.00% 0.00% 0.00% 0 AAA aux

------------------ show process cpu history ------------------

GGSNLAG01 04:11:01 PM Friday Jul 15 2011 GMT

555554444444444555555555544444444444444455555555555555544444

000008888844444000001111188888888887777700000555554444477777

100

90

80

70

60 *****

50 ********** *********************************************

40 ************************************************************

30 ************************************************************

20 ************************************************************

10 ************************************************************

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per second (last 60 seconds)

565555555995555555565565985555555565556995555555555555985555

598858449996645404208616916785557925352994377598459745907778

100 ** * ** *

90 ** * ** *

80 ** ** ** **

70 * ** ** ** **

60 ****** *##** * *****#*********** **## ***** *** *#*****

50 ############################################################

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

1 1 1

999999099987999909909999977997775444344577777787777777765543445677877777

999999099964999909909999904041008810937928503144647364378128070007243052

100 ********** *************

90 *********** ************* **

80 *********** ************* ** ** * * * * ** *

70 *****************###************ *#************** *##*****

60 ****##**##******###############** *###############** *########

50 ##########******#################* **#################* **#########

40 ##########*****###################****####################****##########

30 ##########****##########################################################

20 ###########*############################################################

10 ########################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

bhornilesh · ‎07-27-2011

Hi Ajith,

Please find following URL & do the necessary things on your router for reduce CPU utilization.

http://www.debianadmin.com/fix-for-high-cpu-usage-when-snmp-is-enabled-in-cisco-routers.html