Re: High CPU utilization on 7200 Router due to interrupts

Shaheen Sidana · ‎12-05-2012

Hi Everybody,

I have on 7200 series router with NPE-G1 module which is facing high CPU utilization. I have not found any particular process causing this high CPU utilization as it is caused due to interrupts. I have already enabled fast switching by "ip cef" command. Please suggest how to normalize the utilization as it is impacting the network and causing slowness. Please find below the output of show process cpu and also find attached the show tech of the device. Also let me know if any other output is required.

TCISL-GGCOLO-7206-RO#sh proc cpu sorted

CPU utilization for five seconds: 75%/71%; one minute: 70%; five minutes: 69%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

184 3311658521741691145 190 1.47% 1.41% 1.41% 0 IP Input

41 351217284 21287490 16498 1.14% 1.23% 1.22% 0 Per-Second Jobs

25 76426548 963268136 79 0.24% 0.29% 0.29% 0 ARP Input

287 1505324 629990778 2 0.16% 0.10% 0.08% 0 PPP Events

37 28118300 16690436 1684 0.16% 0.11% 0.10% 0 Net Background

193 140 2417 57 0.16% 0.02% 0.00% 2 Virtual Exec

113 431068 20149940 21 0.08% 0.04% 0.06% 0 hal_periodic 1

286 1082160 628735884 1 0.08% 0.11% 0.10% 0 PPP manager

224 4720 20151164 0 0.08% 0.00% 0.00% 0 RUDPV1 Main Proc

207 1174992 36046711 32 0.08% 0.10% 0.08% 0 CEF process

177 545544 20149939 27 0.08% 0.05% 0.06% 0 hal_periodic 2

274 4786700 4074008 1174 0.08% 0.04% 0.06% 0 Compute load avg

186 1293884 20640618 62 0.08% 0.05% 0.06% 0 CDP Protocol

300 1078960 23234478 46 0.08% 0.00% 0.00% 0 OSPF-1 Router

15 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

14 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

17 4630400 20156773 229 0.00% 0.00% 0.00% 0 EnvMon

13 0 81 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

12 700 6153 113 0.00% 0.00% 0.00% 0 AAA high-capacit

16 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers

21 2544 20146727 0 0.00% 0.00% 0.00% 0 IPC Periodic Tim

11 0 2 0 0.00% 0.00% 0.00% 0 ATM VC Auto Crea

23 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager

24 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure

18 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

10 0 2 0 0.00% 0.00% 0.00% 0 ATM AutoVC Perio

27 504 12210 41 0.00% 0.00% 0.00% 0 DDR Timers

19 32 335833 0 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

20 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

22 2216 20146739 0 0.00% 0.00% 0.00% 0 IPC Deferred Por

31 19080 20149945 0 0.00% 0.00% 0.00% 0 GraphIt

32 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

33 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

36 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

6 60596 222285 272 0.00% 0.00% 0.00% 0 Pool Manager

38 0 2 0 0.00% 0.00% 0.00% 0 IDB Work

39 1680 35559 47 0.00% 0.00% 0.00% 0 Logger

40 4016 20146730 0 0.00% 0.00% 0.00% 0 TTY Background

5 27720652 2487467 11144 0.00% 0.06% 0.05% 0 Check heaps

26 6176 4029876 1 0.00% 0.00% 0.00% 0 HC Counter Timer

28 0 5 0 0.00% 0.00% 0.00% 0 Entity MIB API

44 0 1 0 0.00% 0.00% 0.00% 0 POS APS Event Pr

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

45 0 2 0 0.00% 0.00% 0.00% 0 VNM DSPRM MAIN

46 0 1 0 0.00% 0.00% 0.00% 0 Coproc Event Pro

47 0 1 0 0.00% 0.00% 0.00% 0 CES Line Conditi

48 0 2 0 0.00% 0.00% 0.00% 0 Flash MIB Update

29 228 18090 12 0.00% 0.00% 0.00% 0 EEM ED Syslog

50 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/1 bert

30 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

52 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/3 bert

35 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

54 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/5 bert

55 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/6 bert

56 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/7 bert

57 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/8 bert

58 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/9 bert

59 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/10 ber

60 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/11 ber

42 0 1 0 0.00% 0.00% 0.00% 0 CSP Timer

62 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/13 ber

63 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/14 ber

64 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/15 ber

65 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/16 ber

66 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/17 ber

67 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/18 ber

68 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/19 ber

69 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/20 ber

70 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/21 ber

71 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/1 bert

72 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/2 bert

4 0 1 0 0.00% 0.00% 0.00% 0 EDDRI_MAIN

9 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

75 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/5 bert

43 0 1 0 0.00% 0.00% 0.00% 0 SONET alarm time

49 0 1 0 0.00% 0.00% 0.00% 0 Flash Card Oir

78 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/8 bert

79 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/9 bert

80 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/10 ber

81 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/11 ber

82 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/12 ber

83 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/13 ber

84 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/14 ber

85 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/15 ber

86 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/16 ber

87 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/17 ber

88 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/18 ber

51 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/2 bert

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

90 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/20 ber

91 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/21 ber

92 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/1 bert

53 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/4 bert

2 226688 4029991 56 0.00% 0.01% 0.00% 0 Load Meter

95 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/4 bert

96 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/5 bert

61 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.1/12 ber

98 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/7 bert

99 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/8 bert

100 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/9 bert

101 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/10 ber

102 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/11 ber

103 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/12 ber

73 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/3 bert

105 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/14 ber

106 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/15 ber

107 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/16 ber

108 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/17 ber

109 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/18 ber

110 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/19 ber

111 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/20 ber

112 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/21 ber

74 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/4 bert

114 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/1 bert

115 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/2 bert

116 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/3 bert

117 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/4 bert

118 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/5 bert

119 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/6 bert

120 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/7 bert

121 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/8 bert

122 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/9 bert

123 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/10 ber

124 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/11 ber

125 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/12 ber

126 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/13 ber

127 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/14 ber

128 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/15 ber

129 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/16 ber

130 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/17 ber

131 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/18 ber

132 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/19 ber

133 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/20 ber

134 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.1/21 ber

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

135 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/1 bert

136 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/2 bert

137 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/3 bert

8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

139 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/5 bert

140 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/6 bert

141 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/7 bert

142 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/8 bert

3 328 32620 10 0.00% 0.00% 0.00% 0 TACACS+

76 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/6 bert

145 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/11 ber

146 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/12 ber

147 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/13 ber

148 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/14 ber

149 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/15 ber

77 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/7 bert

151 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/17 ber

89 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.2/19 ber

153 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/19 ber

154 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/20 ber

155 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/21 ber

156 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/1 bert

157 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/2 bert

158 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/3 bert

159 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/4 bert

160 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/5 bert

161 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/6 bert

162 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/7 bert

163 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/8 bert

164 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/9 bert

165 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/10 ber

166 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/11 ber

167 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/12 ber

168 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/13 ber

169 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/14 ber

170 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/15 ber

171 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/16 ber

172 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/17 ber

173 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/18 ber

174 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/19 ber

175 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/20 ber

176 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.3/21 ber

93 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/2 bert

178 0 8 0 0.00% 0.00% 0.00% 0 TurboACL

179 0 2 0 0.00% 0.00% 0.00% 0 TurboACL chunk

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

180 984 9503 103 0.00% 0.00% 0.00% 0 AAA Server

181 1100 11699 94 0.00% 0.00% 0.00% 0 AAA ACCT Proc

182 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr

183 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

94 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/3 bert

185 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl

97 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/6 bert

187 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager

188 972 2686633 0 0.00% 0.00% 0.00% 0 SSS Test Client

189 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana

190 78352 78699756 0 0.00% 0.00% 0.00% 0 SSS Feature Time

191 740 13735 53 0.00% 0.00% 0.00% 0 PPP Hooks

192 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

104 0 3 0 0.00% 0.00% 0.00% 0 hal 1/0.3/13 ber

194 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage

195 0 1 0 0.00% 0.00% 0.00% 0 L2X Socket proce

196 0 1 0 0.00% 0.00% 0.00% 0 L2X SSS manager

197 0 2 0 0.00% 0.00% 0.00% 0 L2TP mgmt daemon

198 0 8 0 0.00% 0.00% 0.00% 0 IPv6 RIB Redistr

199 37548 4799266 7 0.00% 0.00% 0.00% 0 IP Background

200 118704 982981 120 0.00% 0.00% 0.00% 0 IP RIB Update

201 0 2 0 0.00% 0.00% 0.00% 0 SNMP Timers

202 61684 24579 2509 0.00% 0.00% 0.00% 0 PPP IP Route

203 2400 24939 96 0.00% 0.00% 0.00% 0 PPP IPCP

204 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input

205 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER

206 318040 338733 938 0.00% 0.00% 0.00% 0 Adj Manager

1 0 65 0 0.00% 0.00% 0.00% 0 Chunk Manager

208 121848 10170577 11 0.00% 0.00% 0.00% 0 TCP Timer

209 5256 5885 893 0.00% 0.00% 0.00% 0 TCP Protocols

210 452088 33290469 13 0.00% 0.00% 0.00% 0 Socket Timers

211 468 67172 6 0.00% 0.00% 0.00% 0 HTTP CORE

212 140 44028 3 0.00% 0.00% 0.00% 0 Transport Port A

213 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute

214 44 466 94 0.00% 0.00% 0.00% 0 RARP Input

215 11471192 66714322 171 0.00% 0.03% 0.05% 0 DHCPD Receive

216 1296 335828 3 0.00% 0.00% 0.00% 0 IP Cache Ager

217 0 1 0 0.00% 0.00% 0.00% 0 COPS

218 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS

219 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

220 56072 201283790 0 0.00% 0.02% 0.02% 0 RBSCP Background

221 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

222 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background

223 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder

138 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/4 bert

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

225 0 1 0 0.00% 0.00% 0.00% 0 bsm_timers

226 1360 20146733 0 0.00% 0.00% 0.00% 0 bsm_xmt_proc

227 0 2 0 0.00% 0.00% 0.00% 0 RLM groups Proce

228 0 3 0 0.00% 0.00% 0.00% 0 gk process

229 0 2 0 0.00% 0.00% 0.00% 0 Border Element p

230 1008 806000 1 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_

231 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA

232 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

233 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

234 53864 94613 569 0.00% 0.00% 0.00% 0 TPLUS

235 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

236 0 2 0 0.00% 0.00% 0.00% 0 VSP_MGR

237 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr

238 0 1 0 0.00% 0.00% 0.00% 0 VoIP AAA

239 0 1 0 0.00% 0.00% 0.00% 0 QOS_MODULE_MAIN

240 0 1 0 0.00% 0.00% 0.00% 0 RPMS_PROC_MAIN

241 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_HTSP

242 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_R2

243 0 1 0 0.00% 0.00% 0.00% 0 CCSWVOICE

244 0 1 0 0.00% 0.00% 0.00% 0 xcpa-driver

245 4 1 4000 0.00% 0.00% 0.00% 0 http client proc

246 1728 2014974 0 0.00% 0.00% 0.00% 0 OSPFv3-10 Hello

247 100 22389 4 0.00% 0.00% 0.00% 0 FB/KS Log HouseK

248 0 2 0 0.00% 0.00% 0.00% 0 EPHONE MWI BG Pr

249 0 2 0 0.00% 0.00% 0.00% 0 Proxy Session Ap

250 384 2014966 0 0.00% 0.00% 0.00% 0 RMON Recycle Pro

251 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se

252 0 1 0 0.00% 0.00% 0.00% 0 SYSMGT Events

253 1080 12735 84 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

254 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

255 0 1 0 0.00% 0.00% 0.00% 0 IMA Traps

256 0 3 0 0.00% 0.00% 0.00% 0 EEM ED CLI

257 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Counter

258 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Interface

259 0 3 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD

260 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Memory-th

261 0 2 0 0.00% 0.00% 0.00% 0 EEM ED None

262 0 2 0 0.00% 0.00% 0.00% 0 EM ED OIR

263 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP

264 128 335855 0 0.00% 0.00% 0.00% 0 EEM ED Timer

265 1096 4041466 0 0.00% 0.00% 0.00% 0 EEM Server

266 0 1 0 0.00% 0.00% 0.00% 0 RMON Packets

267 1960 17865 109 0.00% 0.00% 0.00% 0 Syslog

268 0 1 0 0.00% 0.00% 0.00% 0 VPDN Scal

269 2468 20136413 0 0.00% 0.00% 0.00% 0 trunk conditioni

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

270 0 1 0 0.00% 0.00% 0.00% 0 trunk conditioni

271 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc

272 37660 19608 1920 0.00% 0.00% 0.07% 3 Virtual Exec

273 2334936 69759518 33 0.00% 0.01% 0.00% 0 Net Input

143 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/9 bert

275 7435728 375772 19787 0.00% 0.01% 0.00% 0 Per-minute Jobs

276 452 167918 2 0.00% 0.00% 0.00% 0 CC-API_VCM

277 79220 201307791 0 0.00% 0.03% 0.02% 0 CCPROXY_CT

278 311348 699974 444 0.00% 0.00% 0.00% 0 CEF Scanner

280 0 2 0 0.00% 0.00% 0.00% 0 IP Flow Backgrou

281 92 335896 0 0.00% 0.00% 0.00% 0 IPv6 Input

282 2100 4904125 0 0.00% 0.00% 0.00% 0 IPv6 IDB

283 516 115849 4 0.00% 0.00% 0.00% 0 IPv6 ND

284 148220 80588022 1 0.00% 0.01% 0.00% 0 MLD

285 41720 19757276 2 0.00% 0.00% 0.00% 0 OSPFv3-10 Router

144 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/10 ber

150 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/16 ber

288 3132 20147912 0 0.00% 0.00% 0.00% 0 Multilink PPP

289 623396 8131126 76 0.00% 0.00% 0.00% 0 OSPF-1 Hello

290 304 321 947 0.00% 0.00% 0.00% 0 TCP Listener

291 1066944 6757845 157 0.00% 0.00% 0.00% 0 IP SNMP

292 454052 3841146 118 0.00% 0.00% 0.00% 0 PDU DISPATCHER

293 2206124 3846304 573 0.00% 0.00% 0.00% 0 SNMP ENGINE

294 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6

295 939228 49998 18785 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

296 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps

297 170172 20492075 8 0.00% 0.00% 0.00% 0 NTP

298 256 167917 1 0.00% 0.00% 0.00% 0 DHCPD Timer

299 84 335838 0 0.00% 0.00% 0.00% 0 DHCPD Database

152 0 3 0 0.00% 0.00% 0.00% 0 hal 2/0.2/18 ber

301 711820 57447570 12 0.00% 0.00% 0.00% 0 BGP Router

302 2376008 9713766 244 0.00% 0.00% 0.00% 0 BGP I/O

303 1627216 1349715 1205 0.00% 0.00% 0.00% 0 BGP Scanner

TCISL-GGCOLO-7206-RO#

Giuseppe Larosa · ‎12-06-2012

Hello Shaheen,

follow the below specific document for high cpu caused by interrupts

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

Hope to help

Giuseppe

Shaheen Sidana · ‎12-06-2012

Hii,

Thanks for the link. I have already gone through this document but nothing found relevant which can solve the issue. Please go through the show tech once and suggest what I can do to lower down the utilization.

Thanks ,

Shaheen

Raju Sekharan · ‎12-06-2012

7200 is a CEF router. So if the traffic rate goes high, you will see the CPU utilization going high because of interrupts because they use CPU to switch the traffic

As per below link, the cef switching performacne for NPE-G1 is 521 Mbps

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

your box is switching more than 400 Mbps of traffic

GigabitEthernet0/1 is up, line protocol is up

30 second input rate 317693000 bits/sec, 42778 packets/sec

30 second output rate 111276000 bits/sec, 34165 packets/sec

GigabitEthernet0/3 is up, line protocol is up

5 minute input rate 93029000 bits/sec, 29066 packets/sec

5 minute output rate 270483000 bits/sec, 35920 packets/sec

So what you are sseeing should be an expected behavior for the amount of traffic you have

Thanks

Raju

Joseph W. Doherty · ‎12-06-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I agree with Raju, CPU usage appears reasonable for traffic volume.

If the CPU routinely doesn't top 75%, you might have other cause(s) for your "slowness" issue.

AmitLohumi · ‎12-06-2012

Hi All,

Following are a a coupls of CEF outputs from the same router :

###################################Show cef not-cef-switched######################
TCISL-GGCOLO-7206-RO#Show cef not-cef-switched
CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP 114933 0 0 0 269738 257 0 0

##################################Show cef drop##################################
TCISL-GGCOLO-7206-RO#Show cef drop
CEF Drop Statistics
Slot Encap_fail Unresolved Unsupported No_route No_adj ChkSum_Err
RP 2486523203 4 139268 8313115 0 2721

We can see a large no. of 'Encap_fail' packets which would indicate the no. of packets throttled by CEF after exceeding the limit for packets punted to the processor due to missing CEF adjacency information. Would this not indicate malicious traffic from amy of the overloaded ports ?? I am also looking at Bug ID : CSCso80549.

Kindly suggest your views on this .

Thanks

Amit

Joseph W. Doherty · ‎12-07-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

My view would be since the encap_fail indicates a possible CEF adjacency issue, you might examine Technote:

http://www.cisco.com/en/US/tech/tk827/tk831/technologies_tech_note09186a0080094303.shtml and see if there's anything that helps. Otherwise, if you have a support contract, you might open a case with TAC.

AmitLohumi · ‎12-08-2012

Hi Joseph,

Thanks for sharing your views. I was going through this document which does not seem to suggest a straightforward way to troubleshoot this problem. What do you think about this Bug that seems to match the symptoms in this case. ?

Regards,

Amit

Giuseppe Larosa · ‎12-10-2012

Hello Amit,

the router has many subinterfaces each of them with a policer applied inbound and a policer applied outbound of different rates

The aggregate traffic volume is high as noted by Raju and I agree that given the software only nature of C7200 with NPE-G1 your device is not behaving bad but as it can be expected.

About the bug you are mentioning. I find this bug description to be vague.

your router is running

disk2:c7200-is-mz.124-16b.bin

Release 12.4(16)B is not listed among the affected IOS images in the list of affected IOS versions.

Given the number of subinterfaces defined with policers and the level of aggregate traffic it looks like that the router is behaving correctly.

the order of packets received and sent on GE interrfaces is like the following from show tech

Gi0/1

17336889990 packets input

15158090407 packets output

Gi0/2

1317444924 packets input

1623446401 packets output

Gi0/3

12719037739 packets input

14364366054 packets output

The rate of enc failure compared to packets input on gi0/1

2486523203 /17336889990 = 0.14%

Even if the number looks like great it is not in comparison to total packets passing on interfaces

So this can be seen as a confirmation the trouble described in the bug should not apply to your router.

Hope to help

Giuseppe

AmitLohumi · ‎12-12-2012

Hi Giuseppe,

Many thanks for your detailed analysis. Have a query though.

If the CEF switching limit for NPE-G1 has not yet been crossed over, why should there be any interrupts going on to the CPU on account of the high volume of traffic.

Regards,

Amit

AmitLohumi · ‎12-12-2012

Would it be because CEF switching s being implemented in Software without dedicated ASICs ?

Regards,

Amit

Raju Sekharan · ‎12-12-2012

Hi Amit,

7200 is a CEF router. It doesn't use any ASIC to forward the packet. Any CEF router uses interrupts to switch the packets

Thanks

Raju

AmitLohumi · ‎12-12-2012

Hi raj,

What would you say of the possibility of a DOS attack, since the utilization has shot up without any major changesbeing made. Do you think using the 'Scheduler Allocate' would help here without any negative side-affects or do we need to go for a traffic analysis exercise using Wireshark to identify the source of this unexpected traffic ?

Regards,

Amit

Raju Sekharan · ‎12-12-2012

Hi Amit,

My suggestion would be to sniff the traffic if you are seeing increased traffc rate on any of the interfaces

Thanks

Raju

Joseph W. Doherty · ‎12-13-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Amit, you seem to discounting the possibility that the high interrupt CPU might be "normal" as it might be just caused by your traffic loading.

Two things that might help demonstrate this could be your root cause, if you can graph overall CPU utilization with overall traffic volume transiting the router, you might see a very strong relationship.

Second, pick a traffic generator of your choice and push some synthetic traffic through your router. Watch interrupt CPU as you push different volumes of such test traffic. Again, you likely see interrupt CPU correspond with the amount of test traffic.