Solved: Re: Core Router crashing!

tdotvix1982 · ‎06-30-2011

Hi folks,

For one of our bigger clients recently for the past couple of days our router started crashing. The router model is ISR 3825 and the following is what I have loaded in the inventory:

NAME: "3825 chassis", DESCR: "3825 chassis"

PID: CISCO3825 , VID: V03 , SN: xxxxxxxxxx

NAME: "WAN Interface Card - Serial 2T on Slot 0 SubSlot 0", DESCR: "WAN Interface Card - Serial 2T"

PID: WIC-2T , VID: V01, SN: xxxxxxxxxx

NAME: "WAN Interface Card - Serial 2T on Slot 0 SubSlot 1", DESCR: "WAN Interface Card - Serial 2T"

PID: WIC-2T , VID: V01, SN: xxxxxxxxxx

NAME: "WAN Interface Card - Serial (1T) on Slot 0 SubSlot 2", DESCR: "WAN Interface Card - Serial (1T)"

PID: WIC-1T= , VID: 1.1, SN: xxxxxxxxxx

NAME: "4 Port FE Switch on Slot 0 SubSlot 3", DESCR: "4 Port FE Switch"

PID: HWIC-4ESW , VID: V01 , SN: xxxxxxxxxx

NAME: "Virtual Private Network (VPN) Module on Slot 0", DESCR: "Encryption AIM Element"

PID: AIM-VPN/EPII-PLUS , VID: V01 , SN: xxxxxxxxxx

NAME: "NME-16ES-1G-P: EtherSwitch SM 16 10/100T PoE + 1 GE on Slot 1", DESCR: "NME-16ES-1G-P: EtherSwitch SM 16 10/100T PoE + 1 GE"

PID: NME-16ES-1G-P , VID: V01 , SN: xxxxxxxxxx

We have the following technologies running on our clients network (over 50 Medium sized branches) over the WAN connected to the branches:

DMVPN (with IPSEC Encryption)

Multi-Area OSPF

At peak times router utilization touches 70 percent or so. Our CPU utilization for the past 60 minutes is:

100 **

90 **

80 ***

70 * * * * *** ##*

60 *** **** *** *##* #*#** *** **# ###**** * ****

50 * ####*#########*#*#########**####*###*#########*#*#*

40 ####################################################*

30 ####################################################*

20 #####################################################

10 #####################################################

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

Following is the end of the output description:

003134: Jun 29 12:35:39: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has

invalid spi for destaddr=192.168.173.102, prot=50, spi=0x11C9ADA(18651866),

srcaddr=192.168.170.174

003135: Jun 29 12:35:40: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for

ipnat entry. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,

-Traceback= 0x616CB6D0 0x601027A4

003136: Jun 29 12:35:50: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for

ipnat entry. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,

-Traceback= 0x616CB6D0 0x601027A4CMD: 'show ip in tbri' 12:35:56 UTC Wed Jun 29 2011

003137: Jun 29 12:36:00: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed

from 0x60102EC8, alignment 8

Pool: Processor Free: 436172 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x616CB6D0 0x600F0EBC

0x600F6D84 0x600F7430 0x60102ED0 0x601025D4 0x60102760

003138: Jun 29 12:36:00: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for

ipnat entry. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,

-Traceback= 0x616CB6D0 0x601027A4CMD: 'show run | inc xxxxxx' 12:36:03 UTC Wed Jun

29 2011

CMD: 'show run | inc ip host' 12:36:09 UTC Wed Jun 29 2011

003139: Jun 29 12:36:10: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for

ipnat node. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,

-Traceback= 0x616CB6D0 0x601027A4

003140: Jun 29 12:36:11: %OSPF-5-ADJCHG: Process 100, Nbr 172.16.15.1 on Tunnel0

from FULL to DOWN, Neighbor Down: Dead timer expired

%ALIGN-1-FATAL: Illegal access to a low address 12:36:11 UTC Wed Jun 29 2011

addr=0x0, pc=0x6179394C , ra=0x62F91F90 , sp=0x681AB2C8

%ALIGN-1-FATAL: Illegal access to a low address 12:36:11 UTC Wed Jun 29 2011

addr=0x0, pc=0x6179394C , ra=0x62F91F90 , sp=0x681AB2C8

12:36:11 UTC Wed Jun 29 2011: TLB (store) exception, CPU signal 10, PC = 0x6179394C

My bad, I forgot to include the show version output in the message earlier:

xxxxxxxxxxxxxxxxx#show version

Cisco IOS Software, 3800 Software (C3825-ADVIPSERVICESK9-M), Version 12.4(15)T9, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Compiled Tue 28-Apr-09 17:45 by prod_rel_team

ROM: System Bootstrap, Version 12.3(11r)T2, RELEASE SOFTWARE (fc1)

xxxxxxxxxxxxxxxxxxxx uptime is 1 hour, 7 minutes

System returned to ROM by bus error at PC 0x6179394C, address 0xB0D0B0D at 11:29:29 UTC Thu Jun 30 2011

System restarted at 11:31:10 UTC Thu Jun 30 2011

System image file is "flash:c3825-advipservicesk9-mz.124-15.T9.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Cisco 3825 (revision 1.1) with 218112K/44032K bytes of memory.

Processor board ID FCZ104673ES

4 FastEthernet interfaces

3 Gigabit Ethernet interfaces

5 Serial(sync/async) interfaces

1 terminal line

2 Virtual Private Network (VPN) Modules

DRAM configuration is 64 bits wide with parity enabled.

479K bytes of NVRAM.

62720K bytes of ATA System CompactFlash (Read/Write)

Configuration register is 0x2102

If you require more information do let me know please and I will try and get that for you. Your help in this regard would be highly appreciated. I get the feeling that router is out of memory...

Message was edited by: Waqas Raza

Latchum Naidu · ‎06-30-2011

Hi

It seems like because of high cpu utilizaton.
Check what causing the high cpu utilization, do you have NAT running on this. If yes i would suggest you to check the NAT translations "sh ip nat tra" and identify which source is doing more (some of them may do on port 21 which will cause high cpu utilization)

Also provide the complete "sh log" and "sh proc cpu sor 5min" output

Please rate the helpfull posts.
Regards,
Naidu.

View solution in original post

Shaikh Aman Uddin · ‎06-30-2011

Hi,

If you performing NAT translation, we can limit the translation per host basis. using command "ip nat trans max".

log the memory output and check it with cisco output itnerpreter.

Please rate my reply.

Thanks,

Aman

View solution in original post

Latchum Naidu · ‎06-30-2011

Hi

It seems like because of high cpu utilizaton.
Check what causing the high cpu utilization, do you have NAT running on this. If yes i would suggest you to check the NAT translations "sh ip nat tra" and identify which source is doing more (some of them may do on port 21 which will cause high cpu utilization)

Also provide the complete "sh log" and "sh proc cpu sor 5min" output

Please rate the helpfull posts.
Regards,
Naidu.

tdotvix1982 · ‎06-30-2011

Hi Naidu,

Following is the Process output for 5 minutes followed by the sh logging output. I also checked the ' sh ip nat translations' and there are no ports being translated on 21:

xxxxxxxxxxxxxxxxxx#show processes cpu sorted 5min

CPU utilization for five seconds: 89%/51%; one minute: 78%; five minutes: 70%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

272 142088 116400 1220 21.87% 14.01% 8.62% 578 Virtual Exec

109 169640 440446 385 5.44% 4.30% 3.09% 0 IP Input

229 179984 473314 380 3.09% 2.62% 2.63% 0 Crypto IKMP

282 114544 10516 10892 2.11% 2.43% 2.36% 0 IP NAT Ager

43 85152 4508 18889 1.30% 1.26% 1.27% 0 Per-Second Jobs

115 63696 61341 1038 1.38% 1.16% 1.12% 579 SSH Process

283 45996 394453 116 0.32% 0.33% 0.38% 0 NAT MIB Helper

228 25612 72741 352 0.32% 0.28% 0.30% 0 Crypto IKE Dispa

5 17300 929 18622 2.44% 0.32% 0.22% 0 Check heaps

36 13992 47909 292 0.16% 0.17% 0.18% 0 OSPF-100 Hello

39 8088 6424 1259 0.00% 0.12% 0.12% 0 Net Background

236 9452 285111 33 0.08% 0.07% 0.08% 0 Key Proc

104 708 134922 5 0.08% 0.06% 0.08% 0 ACCT Periodic Pr

108 580 134918 4 0.00% 0.04% 0.06% 0 IP ARP Retry Age

235 4840 8369 578 0.00% 0.04% 0.05% 0 Crypto PAS Proc

2 416 900 462 0.00% 0.04% 0.05% 0 Load Meter

171 484 43908 11 0.08% 0.03% 0.04% 0 RBSCP Background

135 696 8501 81 0.00% 0.03% 0.02% 0 CEF process

286 2360 6011 392 0.00% 0.02% 0.01% 0 NHRP

280 9292 1126259 8 0.00% 0.05% 0.01% 580 Virtual Exec

42 48 4478 10 0.08% 0.00% 0.00% 0 TTY Background

174 2112 10676 197 0.08% 0.01% 0.00% 0 OSPF-100 Router

120 612 6798 90 0.08% 0.01% 0.00% 0 Spanning Tree

157 20 76 263 0.08% 0.00% 0.00% 0 IP Cache Ager

24 0 3 0 0.00% 0.00% 0.00% 0 DDR Timers

23 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

25 4 4 1000 0.00% 0.00% 0.00% 0 Entity MIB API

26 40 772 51 0.00% 0.00% 0.00% 0 EEM ED Syslog

29 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers

27 132 1351 97 0.00% 0.00% 0.00% 0 HC Counter Timer

31 0 2 0 0.00% 0.00% 0.00% 0 SMART

28 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

30 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa

34 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

35 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

15 12 4492 2 0.00% 0.00% 0.00% 0 IPC Deferred Por

37 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

32 32 4491 7 0.00% 0.00% 0.00% 0 GraphIt

14 20 4492 4 0.00% 0.00% 0.00% 0 IPC Periodic Tim

40 0 3 0 0.00% 0.00% 0.00% 0 IDB Work

41 44 922 47 0.00% 0.00% 0.00% 0 Logger

21 0 107 0 0.00% 0.00% 0.00% 0 AAA high-capacit

33 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

44 0 1 0 0.00% 0.00% 0.00% 0 IKE HA Mgr

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

38 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

46 0 4 0 0.00% 0.00% 0.00% 0 rf task

20 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

19 72 4698 15 0.00% 0.00% 0.00% 0 ARP Background

49 2048 77 26597 0.00% 0.02% 0.00% 0 Per-minute Jobs

50 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process

18 208 670 310 0.00% 0.00% 0.00% 0 ARP Input

52 0 1 0 0.00% 0.00% 0.00% 0 dev_device_inser

53 0 1 0 0.00% 0.00% 0.00% 0 dev_device_remov

17 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure

22 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

56 0 1 0 0.00% 0.00% 0.00% 0 ARL Table Manage

57 12 3 4000 0.00% 0.00% 0.00% 0 ESWPPM

58 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con

59 0 2 0 0.00% 0.00% 0.00% 0 ESWILPPM

60 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con

61 132 17923 7 0.00% 0.00% 0.00% 0 Netclock Backgro

62 20 302 66 0.00% 0.00% 0.00% 0 SM Monitor

63 0 2 0 0.00% 0.00% 0.00% 0 VNM DSPRM MAIN

64 0 1 0 0.00% 0.00% 0.00% 0 DSPFARM DSP READ

65 0 2 0 0.00% 0.00% 0.00% 0 FLEX DNLD MAIN

66 0 1 0 0.00% 0.00% 0.00% 0 HDV background

67 0 2 0 0.00% 0.00% 0.00% 0 Bryce I2C CMD Qu

16 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager

13 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

70 0 1 0 0.00% 0.00% 0.00% 0 RF_INTERDEV_DELA

45 0 1 0 0.00% 0.00% 0.00% 0 IPSEC HA Mgr

72 0 8 0 0.00% 0.00% 0.00% 0 Ether-Switch RBC

73 0 1 0 0.00% 0.00% 0.00% 0 AAL2CPS TIMER_CU

74 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr

47 104 1050 99 0.00% 0.00% 0.00% 0 Net Input

76 0 151 0 0.00% 0.00% 0.00% 0 Call Management

77 0 1 0 0.00% 0.00% 0.00% 0 CES Line Conditi

48 204 900 226 0.00% 0.01% 0.00% 0 Compute load avg

79 4 2 2000 0.00% 0.00% 0.00% 0 Switch Link Moni

51 0 1 0 0.00% 0.00% 0.00% 0 Token Daemon

81 0 2 0 0.00% 0.00% 0.00% 0 cpf_process_msg_

82 24 4502 5 0.00% 0.00% 0.00% 0 Crypto Device Up

83 0 2 0 0.00% 0.00% 0.00% 0 Multi-ISA Event

84 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Cleanu

85 84 637 131 0.00% 0.00% 0.00% 0 crypto engine pr

54 12 751 15 0.00% 0.00% 0.00% 0 mxt5100

87 20 4491 4 0.00% 0.00% 0.00% 0 linktest

88 0 2 0 0.00% 0.00% 0.00% 0 Dot11 Mgmt & Ass

89 0 2 0 0.00% 0.00% 0.00% 0 Dot11 aaa proces

90 0 76 0 0.00% 0.00% 0.00% 0 pmkid

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

91 0 2 0 0.00% 0.00% 0.00% 0 Dot11 auth Dot1x

92 0 1 0 0.00% 0.00% 0.00% 0 Dot11 Mac Auth

55 0 1 0 0.00% 0.00% 0.00% 0 sal_dpc_process

94 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework

68 216 4 54000 0.00% 0.00% 0.00% 0 USB Startup

96 0 2 0 0.00% 0.00% 0.00% 0 DTP Protocol

97 8 4482 1 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

98 12 450 26 0.00% 0.00% 0.00% 0 EtherChnl

99 0 2 0 0.00% 0.00% 0.00% 0 Ethernet CFM

100 12 4377 2 0.00% 0.00% 0.00% 0 Ethernet Timer C

101 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

102 0 43 0 0.00% 0.00% 0.00% 0 AAA Server

103 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc

12 4 76 52 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

105 276 1127 244 0.00% 0.00% 0.00% 0 CDP Protocol

106 0 2 0 0.00% 0.00% 0.00% 0 Ethernet OAM Pro

107 0 41 0 0.00% 0.00% 0.00% 0 IP ARP Adjacency

11 0 1 0 0.00% 0.00% 0.00% 0 chkpt message ha

10 108 899 120 0.00% 0.00% 0.00% 0 Environmental mo

110 12 1499 8 0.00% 0.00% 0.00% 0 ICMP event handl

111 0 4 0 0.00% 0.00% 0.00% 0 TurboACL

112 0 2 0 0.00% 0.00% 0.00% 0 TurboACL chunk

113 0 39 0 0.00% 0.00% 0.00% 0 MOP Protocols

114 0 3 0 0.00% 0.00% 0.00% 0 PPP Hooks

9 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

116 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager

117 0 601 0 0.00% 0.00% 0.00% 0 SSS Test Client

8 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

119 136 17546 7 0.00% 0.00% 0.00% 0 SSS Feature Time

69 0 2 0 0.00% 0.00% 0.00% 0 VMI Background

121 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

71 0 1 0 0.00% 0.00% 0.00% 0 RF_INTERDEV_SCTP

123 0 1 0 0.00% 0.00% 0.00% 0 AC Switch

124 0 2 0 0.00% 0.00% 0.00% 0 Ethernet LMI

125 236 2 118000 0.00% 0.00% 0.00% 0 EAPoUDP Process

75 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Re

127 0 1 0 0.00% 0.00% 0.00% 0 IPv6 RIB Redistr

128 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA

129 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route

130 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP

131 124 165 751 0.00% 0.00% 0.00% 0 IP Background

132 44 151 291 0.00% 0.00% 0.00% 0 IP RIB Update

133 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute

134 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper

78 0 1 0 0.00% 0.00% 0.00% 0 CF_INTERDEV_SCTP

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

137 44 484 90 0.00% 0.00% 0.00% 0 TCP Timer

138 252 1703 147 0.00% 0.00% 0.00% 0 TCP Protocols

139 0 1 0 0.00% 0.00% 0.00% 0 COPS

6 0 3 0 0.00% 0.00% 0.00% 0 Pool Manager

80 84 7391 11 0.00% 0.00% 0.00% 0 ILPM

142 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

4 0 1 0 0.00% 0.00% 0.00% 0 EDDRI_MAIN

86 56 4502 12 0.00% 0.00% 0.00% 0 Kontrol Common H

145 0 1 0 0.00% 0.00% 0.00% 0 IGMPSN

146 0 2 0 0.00% 0.00% 0.00% 0 RLM groups Proce

147 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon

148 0 2 0 0.00% 0.00% 0.00% 0 PPPoE Flow Contr

149 0 3 0 0.00% 0.00% 0.00% 0 SNMP Timers

150 0 2 0 0.00% 0.00% 0.00% 0 SCTP Main Proces

151 0 1 0 0.00% 0.00% 0.00% 0 IUA Main Process

152 44 4482 9 0.00% 0.00% 0.00% 0 RUDPV1 Main Proc

153 0 1 0 0.00% 0.00% 0.00% 0 bsm_timers

154 20 4491 4 0.00% 0.00% 0.00% 0 bsm_xmt_proc

155 0 1 0 0.00% 0.00% 0.00% 0 CES Client SVC R

156 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder

93 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces

158 40 76 526 0.00% 0.00% 0.00% 0 Adj Manager

159 0 3 0 0.00% 0.00% 0.00% 0 Flow Exporter Ti

160 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input

161 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER

162 0 18 0 0.00% 0.00% 0.00% 0 HTTP CORE

163 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

164 20 1615 12 0.00% 0.00% 0.00% 0 Transport Port A

165 0 1 0 0.00% 0.00% 0.00% 0 LAPB Process

166 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

167 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background

168 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

169 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS

170 0 1 0 0.00% 0.00% 0.00% 0 MQC Flow Event B

95 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework

172 4 1 4000 0.00% 0.00% 0.00% 0 VPDN call manage

173 100 8730 11 0.00% 0.00% 0.00% 0 Inspect process

122 0 76 0 0.00% 0.00% 0.00% 0 SSM connection m

175 0 16 0 0.00% 0.00% 0.00% 0 Authentication P

176 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B

177 0 1 0 0.00% 0.00% 0.00% 0 CHKPT EXAMPLE

178 0 1 0 0.00% 0.00% 0.00% 0 CHKPT DevTest

179 0 1 0 0.00% 0.00% 0.00% 0 IPS Process

180 0 2 0 0.00% 0.00% 0.00% 0 IPS Auto Update

181 0 2 0 0.00% 0.00% 0.00% 0 SDEE Management

182 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Inspect Tim

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

183 0 1 0 0.00% 0.00% 0.00% 0 Select Timers

184 8 2 4000 0.00% 0.00% 0.00% 0 HTTP Process

185 0 2 0 0.00% 0.00% 0.00% 0 CIFS API Process

186 0 2 0 0.00% 0.00% 0.00% 0 CIFS Proxy Proce

187 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc

188 0 3 0 0.00% 0.00% 0.00% 0 Crypto HW Proc

189 100 2 50000 0.00% 0.00% 0.00% 0 CCVPM_HDSPRM

190 8 1664 4 0.00% 0.00% 0.00% 0 FLEX DSPRM MAIN

191 8 1662 4 0.00% 0.00% 0.00% 0 FLEX DSP KEEPALI

192 0 182 0 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_

193 0 4 0 0.00% 0.00% 0.00% 0 HDA DSPRM MAIN

194 0 1 0 0.00% 0.00% 0.00% 0 cpf_process_tpQ

195 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve

196 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

197 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr

198 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

199 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

200 136 988 137 0.00% 0.00% 0.00% 0 LOCAL AAA

201 0 2 0 0.00% 0.00% 0.00% 0 TPLUS

202 0 2 0 0.00% 0.00% 0.00% 0 VSP_MGR

203 0 3 0 0.00% 0.00% 0.00% 0 Crypto WUI

204 876 1463 598 0.00% 0.00% 0.00% 0 Crypto Support

205 0 1 0 0.00% 0.00% 0.00% 0 IPSECv6 PS Proc

206 0 1 0 0.00% 0.00% 0.00% 0 EPM MAIN PROCESS

207 4 1 4000 0.00% 0.00% 0.00% 0 CCVPM_HTSP

208 0 2 0 0.00% 0.00% 0.00% 0 VPM_MWI_BACKGROU

209 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_R2

210 0 1 0 0.00% 0.00% 0.00% 0 EPHONE MWI Refre

211 0 6 0 0.00% 0.00% 0.00% 0 FB/KS Log HouseK

212 0 2 0 0.00% 0.00% 0.00% 0 EPHONE MWI BG Pr

213 0 1 0 0.00% 0.00% 0.00% 0 Skinny HW confer

214 8 151 52 0.00% 0.00% 0.00% 0 VOICE REG BG Pro

215 0 1 0 0.00% 0.00% 0.00% 0 Presence Process

216 0 1 0 0.00% 0.00% 0.00% 0 CCSWVOICE

217 40 488 81 0.00% 0.00% 0.00% 0 Tunnel Security

218 0 1 0 0.00% 0.00% 0.00% 0 http client proc

219 20 184 108 0.00% 0.00% 0.00% 0 Crypto SS Proces

220 0 1 0 0.00% 0.00% 0.00% 0 QOS_MODULE_MAIN

221 0 1 0 0.00% 0.00% 0.00% 0 RPMS_PROC_MAIN

222 0 1 0 0.00% 0.00% 0.00% 0 VoIP AAA

223 0 4 0 0.00% 0.00% 0.00% 0 Crypto CA

224 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL

225 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL

226 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc

227 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT

3 1356 34857 38 0.00% 0.01% 0.00% 0 CRYPTO IKMP IPC

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

126 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track Pr

230 4 1 4000 0.00% 0.00% 0.00% 0 Crypto IKEv2

231 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key

232 456 2299 198 0.00% 0.00% 0.00% 0 IPSEC key engine

233 0 1 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce

234 296 373 793 0.00% 0.00% 0.00% 0 Crypto ACL

118 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana

136 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

237 0 1 0 0.00% 0.00% 0.00% 0 GDOI GM Process

238 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY

239 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY AC

240 228 22425 10 0.00% 0.01% 0.00% 0 Atheros LED Ctro

241 0 2 0 0.00% 0.00% 0.00% 0 Control-plane ho

242 32 2423 13 0.00% 0.00% 0.00% 0 PM Callback

243 0 1 0 0.00% 0.00% 0.00% 0 DATA Transfer Pr

244 0 1 0 0.00% 0.00% 0.00% 0 DATA Collector

245 8 34 235 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

246 0 15 0 0.00% 0.00% 0.00% 0 EEM ED Resource

247 0 25 0 0.00% 0.00% 0.00% 0 EEM ED Track

248 0 450 0 0.00% 0.00% 0.00% 0 RMON Recycle Pro

249 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se

250 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

251 4 141 28 0.00% 0.00% 0.00% 0 Crypto cTCP proc

252 32 4440 7 0.00% 0.00% 0.00% 0 trunk conditioni

253 0 1 0 0.00% 0.00% 0.00% 0 trunk conditioni

254 20 2 10000 0.00% 0.00% 0.00% 0 VLAN Manager

256 0 133 0 0.00% 0.00% 0.00% 0 EEM Server

257 0 12 0 0.00% 0.00% 0.00% 0 EEM ED CLI

258 0 23 0 0.00% 0.00% 0.00% 0 EEM ED Counter

259 0 23 0 0.00% 0.00% 0.00% 0 EEM ED Interface

260 0 23 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD

261 4 12 333 0.00% 0.00% 0.00% 0 EEM ED None

262 0 23 0 0.00% 0.00% 0.00% 0 EEM ED OIR

263 0 23 0 0.00% 0.00% 0.00% 0 EEM ED SNMP

264 0 138 0 0.00% 0.00% 0.00% 0 EEM ED Timer

265 0 13 0 0.00% 0.00% 0.00% 0 EEM Policy Direc

266 288 535 538 0.00% 0.00% 0.00% 0 Syslog

267 0 1 0 0.00% 0.00% 0.00% 0 VPDN Test

268 0 159 0 0.00% 0.00% 0.00% 0 IP SLA MPLSLM Pr

269 0 1 0 0.00% 0.00% 0.00% 0 tHUB

270 8 157 50 0.00% 0.00% 0.00% 0 CEF Scanner

271 2084 148 14081 0.00% 0.00% 0.00% 0 crypto sw pk pro

140 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

273 0 5 0 0.00% 0.00% 0.00% 0 SSH Event handle

274 8 99 80 0.00% 0.00% 0.00% 0 IP SNMP

275 0 1 0 0.00% 0.00% 0.00% 0 PDU DISPATCHER

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

276 700 7 100000 0.00% 0.00% 0.00% 0 SNMP ENGINE

277 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6

278 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

279 928 139 6676 0.00% 0.00% 0.00% 0 SNMP Traps

1 592 864 685 0.00% 0.00% 0.00% 0 Chunk Manager

281 12 897 13 0.00% 0.00% 0.00% 0 Track

141 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

143 4 8 500 0.00% 0.00% 0.00% 0 L2MM

284 0 1 0 0.00% 0.00% 0.00% 0 IP NAT WLAN

285 28 472 59 0.00% 0.00% 0.00% 0 IP VFR proc

144 0 1 0 0.00% 0.00% 0.00% 0 MRD

288 232 1574 147 0.00% 0.00% 0.00% 0 IP SLAs Event Pr

289 524 5913 88 0.00% 0.00% 0.00% 0 NTP

Sh logging output:

Syslog logging: enabled (12 messages dropped, 371 messages rate-limited,

0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: disabled

Monitor logging: level debugging, 0 messages logged, xml disabled,

filtering disabled

Buffer logging: level debugging, 1194 messages logged, xml disabled,

filtering disabled

Logging Exception size (4096 bytes)

Count and timestamp logging messages: disabled

Persistent logging: disabled

No active filter modules.

ESM: 0 messages dropped

Trap logging: level notifications, 1192 message lines logged

Logging to 172.20.10.8 (udp port 514, audit disabled,

authentication disabled, encryption disabled, link up),

1192 message lines logged,

0 message lines rate-limited,

0 message lines dropped-by-MD,

xml disabled, sequence number disabled

filtering disabled

Log Buffer (4096 bytes):

tion id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=00000758

001184: Jun 30 12:44:12: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001185: Jun 30 12:44:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).

001186: Jun 30 12:44:42: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001187: Jun 30 12:44:45: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=421 local=10.0.117.38 remote=10.1.20.202 spi=42D93024 seqno=00001D47

001188: Jun 30 12:45:13: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001189: Jun 30 12:45:43: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001190: Jun 30 12:45:46: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=421 local=10.0.117.38 remote=10.1.20.202 spi=42D93024 seqno=00001FCA

001191: Jun 30 12:45:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down

001192: Jun 30 12:45:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to up

001193: Jun 30 12:46:13: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001194: Jun 30 12:46:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).

001195: Jun 30 12:46:43: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001196: Jun 30 12:46:48: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=00000834

001197: Jun 30 12:47:14: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001198: Jun 30 12:47:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).

001199: Jun 30 12:47:44: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001200: Jun 30 12:47:49: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=000008A2

001201: Jun 30 12:48:12: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=289, sequence number=833

001202: Jun 30 12:48:14: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.64.74,dstadr=10.16.101.1,size=1504,handle=0x5985

001203: Jun 30 12:48:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).

001204: Jun 30 12:48:45: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001205: Jun 30 12:48:50: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=000009B3

001206: Jun 30 12:49:16: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961

001207: Jun 30 12:49:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).

Any help would be much appreciated!

Shaikh Aman Uddin · ‎06-30-2011

Hi,

If you performing NAT translation, we can limit the translation per host basis. using command "ip nat trans max".

log the memory output and check it with cisco output itnerpreter.

Please rate my reply.

Thanks,

Aman

Latchum Naidu · ‎06-30-2011

Hi

The cpu utilization is crossing 90% at times right? whcih caused crashing the device.
The following process are taking more ARP Input, Check heaps, Crypto IKMP...

Do you have seperate AIM ecnryption module on this router? If you dont have I would suggest you to have it on the router so that the crypto process will not burden to the actualy CPU. And you have DMVPN also which do log ot encryptions.

Apart from that I would suggest you to do harden your router....
1. Stop ip unreachables, proxy-arp and directed-broadcast to control the present scenario.

Please rate the helpfull posts.
Regards,
Naidu.

tdotvix1982 · ‎06-30-2011

@Aman, I will definitely take up with our Network Security specialist and see what he has to say. That could very likely be the cause of this. Thanks for the tip mate.

@Naidu, yes we do have an AIM encryption for the necessary offloading of encryption processes. Indeed the router needs to be hardened. Optimization of the device should help however I feel more effective load balancing of traffic needs to be done. Lets hope we can coerce the client to do so.

gerald.suiza · ‎06-30-2011

hmm...looks like the router is being hammered..is the CPU high throughout the day or only at certain times? how many tunnels connect to this router?

tdotvix1982 · ‎06-30-2011

Hi Gerald,

At the moment there are 84 encrypted DMVPN tunnels running on the router. This router is placed at the core of a bank where over 50 branches are terminating. This past week was also the end of the financial calendar hence you could imagine the load on the router. Hopefully, we will have a few days of respite before the problems comes back. We will continue monitoring the load on the router at different times of the day. Thanks for the tip mate.

Regards,

Waqas.

gerald.suiza · ‎07-03-2011

i'd have TAC take a look at this.. I ran into something similar in the past, same router platform, it turned out the router could not take the load. The router crashes (CPU flatlines to 100%) or EIGRP neighbor relationships drop at the time they do the nightly batch load. I short the router is to busy due to the amount of traffic. Iwould try and co-relate the traffic patterns and the high CPU. I also think 84 encrypted tunnels is a lot for a 3825 to handle considering there are other services running on the router...hth...

Joseph W. Doherty · ‎06-30-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

12.4(15)T9 - You might try newer code; now that 15.x is out, why run prior T train?

Would look to resolve:

001198: Jun 30 12:47:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).

Would also investigate the cause processes using much CPU

CPU utilization for five seconds: 89%/51%; one minute: 78%; five minutes: 70%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

272 142088 116400 1220 21.87% 14.01% 8.62% 578 Virtual Exec

etc.

tdotvix1982 · ‎06-30-2011

@Joseph regarding '

001198: Jun 30 12:47:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).'

Yes, that is definitely a slight issue we need to resolve. The mismatch is with a remote device with which we tunnel. Convincing the client to go for a new license...hmm need to sharpen up my sales skills...Virtual Exec sounds kinda mysterious hence would need to read up more about that. Cheers for the help mate.

tdotvix1982 · ‎07-27-2011

Hi guys,

Sorry for not keeping the forum updated about this. The past few weeks have been a bit manic at work. We verified the router config as much as we could, installed extra memory on the router. Even went so far as replacing 3 routers and modifying the design to distribute as much load as we could between the clients' corporate network and their internet connection. Finally, we noticed that NAT entries originating from our LAN by our server were well over the 300,000 mark; we tested by removing the NAT entry/disconnecting the server from our network and instantly things getting smooth. Apparently, for some reason the server which runs some sort of a banking application which notifies customers of transactions via SMS messages (among other services) was trying to make SSH connections to every IP on the global internet routing table. I take it the server has some sort of a malicious piece of software on it or has a bug that is being exploited by an exploit. If anyone else has come across such an issue where your server has been behaving this way then please do post it here. We asked the server/software development team to rebuild the server and install an effective anti-virus software on it to ensure it is secure; the server team followed suit but we are having the same result so far. This probably is a bug in the software code itself. Thanks for all the help with this issue guys. It really helped myself and my team get off to a good start whilst investigating this issue.

Thanks,
Vick

Latchum Naidu · ‎07-28-2011

Hi Vick,

I have experienced the same issue with one of my windows 2003 server.
as per my knowledge experience I would suggest you to control the global NAT tranlsations on your router...this is one way.

The second way is that you can configure one to one nat by allowing required ports and required subnets from outside say (port 22, 443 from subnets 206.206.206.0 126.122.144.0 etc.,) and this way it will not open to all whcih is secure.

To control nat translation you can do by configuring below command in global config mode.

ip nat translation tcp-timeout 600
ip nat translation udp-timeout 600

Please let me know if you want go with the second way so that i can help you in configuration

Please rate the helpfull posts.
Regards,
Naidu.