06-30-2011 12:35 AM - edited 03-04-2019 12:51 PM
Hi folks,
For one of our bigger clients recently for the past couple of days our router started crashing. The router model is ISR 3825 and the following is what I have loaded in the inventory:
NAME: "3825 chassis", DESCR: "3825 chassis"
PID: CISCO3825 , VID: V03 , SN: xxxxxxxxxx
NAME: "WAN Interface Card - Serial 2T on Slot 0 SubSlot 0", DESCR: "WAN Interface Card - Serial 2T"
PID: WIC-2T , VID: V01, SN: xxxxxxxxxx
NAME: "WAN Interface Card - Serial 2T on Slot 0 SubSlot 1", DESCR: "WAN Interface Card - Serial 2T"
PID: WIC-2T , VID: V01, SN: xxxxxxxxxx
NAME: "WAN Interface Card - Serial (1T) on Slot 0 SubSlot 2", DESCR: "WAN Interface Card - Serial (1T)"
PID: WIC-1T= , VID: 1.1, SN: xxxxxxxxxx
NAME: "4 Port FE Switch on Slot 0 SubSlot 3", DESCR: "4 Port FE Switch"
PID: HWIC-4ESW , VID: V01 , SN: xxxxxxxxxx
NAME: "Virtual Private Network (VPN) Module on Slot 0", DESCR: "Encryption AIM Element"
PID: AIM-VPN/EPII-PLUS , VID: V01 , SN: xxxxxxxxxx
NAME: "NME-16ES-1G-P: EtherSwitch SM 16 10/100T PoE + 1 GE on Slot 1", DESCR: "NME-16ES-1G-P: EtherSwitch SM 16 10/100T PoE + 1 GE"
PID: NME-16ES-1G-P , VID: V01 , SN: xxxxxxxxxx
We have the following technologies running on our clients network (over 50 Medium sized branches) over the WAN connected to the branches:
DMVPN (with IPSEC Encryption)
Multi-Area OSPF
At peak times router utilization touches 70 percent or so. Our CPU utilization for the past 60 minutes is:
100 **
90 **
80 ***
70 * * * * *** ##*
60 *** **** *** *##* #*#** *** **# ###**** * ****
50 * ####*#########*#*#########**####*###*#########*#*#*
40 ####################################################*
30 ####################################################*
20 #####################################################
10 #####################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
Following is the end of the output description:
003134: Jun 29 12:35:39: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has
invalid spi for destaddr=192.168.173.102, prot=50, spi=0x11C9ADA(18651866),
srcaddr=192.168.170.174
003135: Jun 29 12:35:40: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for
ipnat entry. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,
-Traceback= 0x616CB6D0 0x601027A4
003136: Jun 29 12:35:50: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for
ipnat entry. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,
-Traceback= 0x616CB6D0 0x601027A4CMD: 'show ip in tbri' 12:35:56 UTC Wed Jun 29 2011
003137: Jun 29 12:36:00: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed
from 0x60102EC8, alignment 8
Pool: Processor Free: 436172 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x616CB6D0 0x600F0EBC
0x600F6D84 0x600F7430 0x60102ED0 0x601025D4 0x60102760
003138: Jun 29 12:36:00: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for
ipnat entry. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,
-Traceback= 0x616CB6D0 0x601027A4CMD: 'show run | inc xxxxxx' 12:36:03 UTC Wed Jun
29 2011
CMD: 'show run | inc ip host' 12:36:09 UTC Wed Jun 29 2011
003139: Jun 29 12:36:10: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for
ipnat node. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1,
-Traceback= 0x616CB6D0 0x601027A4
003140: Jun 29 12:36:11: %OSPF-5-ADJCHG: Process 100, Nbr 172.16.15.1 on Tunnel0
from FULL to DOWN, Neighbor Down: Dead timer expired
%ALIGN-1-FATAL: Illegal access to a low address 12:36:11 UTC Wed Jun 29 2011
addr=0x0, pc=0x6179394C , ra=0x62F91F90 , sp=0x681AB2C8
%ALIGN-1-FATAL: Illegal access to a low address 12:36:11 UTC Wed Jun 29 2011
addr=0x0, pc=0x6179394C , ra=0x62F91F90 , sp=0x681AB2C8
12:36:11 UTC Wed Jun 29 2011: TLB (store) exception, CPU signal 10, PC = 0x6179394C
My bad, I forgot to include the show version output in the message earlier:
xxxxxxxxxxxxxxxxx#show version
Cisco IOS Software, 3800 Software (C3825-ADVIPSERVICESK9-M), Version 12.4(15)T9, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 28-Apr-09 17:45 by prod_rel_team
ROM: System Bootstrap, Version 12.3(11r)T2, RELEASE SOFTWARE (fc1)
xxxxxxxxxxxxxxxxxxxx uptime is 1 hour, 7 minutes
System returned to ROM by bus error at PC 0x6179394C, address 0xB0D0B0D at 11:29:29 UTC Thu Jun 30 2011
System restarted at 11:31:10 UTC Thu Jun 30 2011
System image file is "flash:c3825-advipservicesk9-mz.124-15.T9.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
Cisco 3825 (revision 1.1) with 218112K/44032K bytes of memory.
Processor board ID FCZ104673ES
4 FastEthernet interfaces
3 Gigabit Ethernet interfaces
5 Serial(sync/async) interfaces
1 terminal line
2 Virtual Private Network (VPN) Modules
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
62720K bytes of ATA System CompactFlash (Read/Write)
Configuration register is 0x2102
If you require more information do let me know please and I will try and get that for you. Your help in this regard would be highly appreciated. I get the feeling that router is out of memory...
Message was edited by: Waqas Raza
Solved! Go to Solution.
06-30-2011 12:44 AM
Hi
It seems like because of high cpu utilizaton.
Check what causing the high cpu utilization, do you have NAT running on this. If yes i would suggest you to check the NAT translations "sh ip nat tra" and identify which source is doing more (some of them may do on port 21 which will cause high cpu utilization)
Also provide the complete "sh log" and "sh proc cpu sor 5min" output
Please rate the helpfull posts.
Regards,
Naidu.
06-30-2011 01:01 AM
Hi,
If you performing NAT translation, we can limit the translation per host basis. using command "ip nat trans max".
log the memory output and check it with cisco output itnerpreter.
Please rate my reply.
Thanks,
Aman
06-30-2011 12:44 AM
Hi
It seems like because of high cpu utilizaton.
Check what causing the high cpu utilization, do you have NAT running on this. If yes i would suggest you to check the NAT translations "sh ip nat tra" and identify which source is doing more (some of them may do on port 21 which will cause high cpu utilization)
Also provide the complete "sh log" and "sh proc cpu sor 5min" output
Please rate the helpfull posts.
Regards,
Naidu.
06-30-2011 12:59 AM
Hi Naidu,
Following is the Process output for 5 minutes followed by the sh logging output. I also checked the ' sh ip nat translations' and there are no ports being translated on 21:
xxxxxxxxxxxxxxxxxx#show processes cpu sorted 5min
CPU utilization for five seconds: 89%/51%; one minute: 78%; five minutes: 70%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
272 142088 116400 1220 21.87% 14.01% 8.62% 578 Virtual Exec
109 169640 440446 385 5.44% 4.30% 3.09% 0 IP Input
229 179984 473314 380 3.09% 2.62% 2.63% 0 Crypto IKMP
282 114544 10516 10892 2.11% 2.43% 2.36% 0 IP NAT Ager
43 85152 4508 18889 1.30% 1.26% 1.27% 0 Per-Second Jobs
115 63696 61341 1038 1.38% 1.16% 1.12% 579 SSH Process
283 45996 394453 116 0.32% 0.33% 0.38% 0 NAT MIB Helper
228 25612 72741 352 0.32% 0.28% 0.30% 0 Crypto IKE Dispa
5 17300 929 18622 2.44% 0.32% 0.22% 0 Check heaps
36 13992 47909 292 0.16% 0.17% 0.18% 0 OSPF-100 Hello
39 8088 6424 1259 0.00% 0.12% 0.12% 0 Net Background
236 9452 285111 33 0.08% 0.07% 0.08% 0 Key Proc
104 708 134922 5 0.08% 0.06% 0.08% 0 ACCT Periodic Pr
108 580 134918 4 0.00% 0.04% 0.06% 0 IP ARP Retry Age
235 4840 8369 578 0.00% 0.04% 0.05% 0 Crypto PAS Proc
2 416 900 462 0.00% 0.04% 0.05% 0 Load Meter
171 484 43908 11 0.08% 0.03% 0.04% 0 RBSCP Background
135 696 8501 81 0.00% 0.03% 0.02% 0 CEF process
286 2360 6011 392 0.00% 0.02% 0.01% 0 NHRP
280 9292 1126259 8 0.00% 0.05% 0.01% 580 Virtual Exec
42 48 4478 10 0.08% 0.00% 0.00% 0 TTY Background
174 2112 10676 197 0.08% 0.01% 0.00% 0 OSPF-100 Router
120 612 6798 90 0.08% 0.01% 0.00% 0 Spanning Tree
157 20 76 263 0.08% 0.00% 0.00% 0 IP Cache Ager
24 0 3 0 0.00% 0.00% 0.00% 0 DDR Timers
23 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
25 4 4 1000 0.00% 0.00% 0.00% 0 Entity MIB API
26 40 772 51 0.00% 0.00% 0.00% 0 EEM ED Syslog
29 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers
27 132 1351 97 0.00% 0.00% 0.00% 0 HC Counter Timer
31 0 2 0 0.00% 0.00% 0.00% 0 SMART
28 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
30 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa
34 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
35 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
15 12 4492 2 0.00% 0.00% 0.00% 0 IPC Deferred Por
37 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest
32 32 4491 7 0.00% 0.00% 0.00% 0 GraphIt
14 20 4492 4 0.00% 0.00% 0.00% 0 IPC Periodic Tim
40 0 3 0 0.00% 0.00% 0.00% 0 IDB Work
41 44 922 47 0.00% 0.00% 0.00% 0 Logger
21 0 107 0 0.00% 0.00% 0.00% 0 AAA high-capacit
33 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
44 0 1 0 0.00% 0.00% 0.00% 0 IKE HA Mgr
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
38 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
46 0 4 0 0.00% 0.00% 0.00% 0 rf task
20 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
19 72 4698 15 0.00% 0.00% 0.00% 0 ARP Background
49 2048 77 26597 0.00% 0.02% 0.00% 0 Per-minute Jobs
50 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process
18 208 670 310 0.00% 0.00% 0.00% 0 ARP Input
52 0 1 0 0.00% 0.00% 0.00% 0 dev_device_inser
53 0 1 0 0.00% 0.00% 0.00% 0 dev_device_remov
17 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure
22 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
56 0 1 0 0.00% 0.00% 0.00% 0 ARL Table Manage
57 12 3 4000 0.00% 0.00% 0.00% 0 ESWPPM
58 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con
59 0 2 0 0.00% 0.00% 0.00% 0 ESWILPPM
60 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con
61 132 17923 7 0.00% 0.00% 0.00% 0 Netclock Backgro
62 20 302 66 0.00% 0.00% 0.00% 0 SM Monitor
63 0 2 0 0.00% 0.00% 0.00% 0 VNM DSPRM MAIN
64 0 1 0 0.00% 0.00% 0.00% 0 DSPFARM DSP READ
65 0 2 0 0.00% 0.00% 0.00% 0 FLEX DNLD MAIN
66 0 1 0 0.00% 0.00% 0.00% 0 HDV background
67 0 2 0 0.00% 0.00% 0.00% 0 Bryce I2C CMD Qu
16 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager
13 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
70 0 1 0 0.00% 0.00% 0.00% 0 RF_INTERDEV_DELA
45 0 1 0 0.00% 0.00% 0.00% 0 IPSEC HA Mgr
72 0 8 0 0.00% 0.00% 0.00% 0 Ether-Switch RBC
73 0 1 0 0.00% 0.00% 0.00% 0 AAL2CPS TIMER_CU
74 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr
47 104 1050 99 0.00% 0.00% 0.00% 0 Net Input
76 0 151 0 0.00% 0.00% 0.00% 0 Call Management
77 0 1 0 0.00% 0.00% 0.00% 0 CES Line Conditi
48 204 900 226 0.00% 0.01% 0.00% 0 Compute load avg
79 4 2 2000 0.00% 0.00% 0.00% 0 Switch Link Moni
51 0 1 0 0.00% 0.00% 0.00% 0 Token Daemon
81 0 2 0 0.00% 0.00% 0.00% 0 cpf_process_msg_
82 24 4502 5 0.00% 0.00% 0.00% 0 Crypto Device Up
83 0 2 0 0.00% 0.00% 0.00% 0 Multi-ISA Event
84 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Cleanu
85 84 637 131 0.00% 0.00% 0.00% 0 crypto engine pr
54 12 751 15 0.00% 0.00% 0.00% 0 mxt5100
87 20 4491 4 0.00% 0.00% 0.00% 0 linktest
88 0 2 0 0.00% 0.00% 0.00% 0 Dot11 Mgmt & Ass
89 0 2 0 0.00% 0.00% 0.00% 0 Dot11 aaa proces
90 0 76 0 0.00% 0.00% 0.00% 0 pmkid
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
91 0 2 0 0.00% 0.00% 0.00% 0 Dot11 auth Dot1x
92 0 1 0 0.00% 0.00% 0.00% 0 Dot11 Mac Auth
55 0 1 0 0.00% 0.00% 0.00% 0 sal_dpc_process
94 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework
68 216 4 54000 0.00% 0.00% 0.00% 0 USB Startup
96 0 2 0 0.00% 0.00% 0.00% 0 DTP Protocol
97 8 4482 1 0.00% 0.00% 0.00% 0 PI MATM Aging Pr
98 12 450 26 0.00% 0.00% 0.00% 0 EtherChnl
99 0 2 0 0.00% 0.00% 0.00% 0 Ethernet CFM
100 12 4377 2 0.00% 0.00% 0.00% 0 Ethernet Timer C
101 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
102 0 43 0 0.00% 0.00% 0.00% 0 AAA Server
103 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc
12 4 76 52 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
105 276 1127 244 0.00% 0.00% 0.00% 0 CDP Protocol
106 0 2 0 0.00% 0.00% 0.00% 0 Ethernet OAM Pro
107 0 41 0 0.00% 0.00% 0.00% 0 IP ARP Adjacency
11 0 1 0 0.00% 0.00% 0.00% 0 chkpt message ha
10 108 899 120 0.00% 0.00% 0.00% 0 Environmental mo
110 12 1499 8 0.00% 0.00% 0.00% 0 ICMP event handl
111 0 4 0 0.00% 0.00% 0.00% 0 TurboACL
112 0 2 0 0.00% 0.00% 0.00% 0 TurboACL chunk
113 0 39 0 0.00% 0.00% 0.00% 0 MOP Protocols
114 0 3 0 0.00% 0.00% 0.00% 0 PPP Hooks
9 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
116 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager
117 0 601 0 0.00% 0.00% 0.00% 0 SSS Test Client
8 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler
119 136 17546 7 0.00% 0.00% 0.00% 0 SSS Feature Time
69 0 2 0 0.00% 0.00% 0.00% 0 VMI Background
121 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
71 0 1 0 0.00% 0.00% 0.00% 0 RF_INTERDEV_SCTP
123 0 1 0 0.00% 0.00% 0.00% 0 AC Switch
124 0 2 0 0.00% 0.00% 0.00% 0 Ethernet LMI
125 236 2 118000 0.00% 0.00% 0.00% 0 EAPoUDP Process
75 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Re
127 0 1 0 0.00% 0.00% 0.00% 0 IPv6 RIB Redistr
128 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA
129 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route
130 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP
131 124 165 751 0.00% 0.00% 0.00% 0 IP Background
132 44 151 291 0.00% 0.00% 0.00% 0 IP RIB Update
133 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute
134 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper
78 0 1 0 0.00% 0.00% 0.00% 0 CF_INTERDEV_SCTP
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
137 44 484 90 0.00% 0.00% 0.00% 0 TCP Timer
138 252 1703 147 0.00% 0.00% 0.00% 0 TCP Protocols
139 0 1 0 0.00% 0.00% 0.00% 0 COPS
6 0 3 0 0.00% 0.00% 0.00% 0 Pool Manager
80 84 7391 11 0.00% 0.00% 0.00% 0 ILPM
142 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
4 0 1 0 0.00% 0.00% 0.00% 0 EDDRI_MAIN
86 56 4502 12 0.00% 0.00% 0.00% 0 Kontrol Common H
145 0 1 0 0.00% 0.00% 0.00% 0 IGMPSN
146 0 2 0 0.00% 0.00% 0.00% 0 RLM groups Proce
147 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon
148 0 2 0 0.00% 0.00% 0.00% 0 PPPoE Flow Contr
149 0 3 0 0.00% 0.00% 0.00% 0 SNMP Timers
150 0 2 0 0.00% 0.00% 0.00% 0 SCTP Main Proces
151 0 1 0 0.00% 0.00% 0.00% 0 IUA Main Process
152 44 4482 9 0.00% 0.00% 0.00% 0 RUDPV1 Main Proc
153 0 1 0 0.00% 0.00% 0.00% 0 bsm_timers
154 20 4491 4 0.00% 0.00% 0.00% 0 bsm_xmt_proc
155 0 1 0 0.00% 0.00% 0.00% 0 CES Client SVC R
156 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder
93 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces
158 40 76 526 0.00% 0.00% 0.00% 0 Adj Manager
159 0 3 0 0.00% 0.00% 0.00% 0 Flow Exporter Ti
160 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input
161 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER
162 0 18 0 0.00% 0.00% 0.00% 0 HTTP CORE
163 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
164 20 1615 12 0.00% 0.00% 0.00% 0 Transport Port A
165 0 1 0 0.00% 0.00% 0.00% 0 LAPB Process
166 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
167 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background
168 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind
169 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS
170 0 1 0 0.00% 0.00% 0.00% 0 MQC Flow Event B
95 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework
172 4 1 4000 0.00% 0.00% 0.00% 0 VPDN call manage
173 100 8730 11 0.00% 0.00% 0.00% 0 Inspect process
122 0 76 0 0.00% 0.00% 0.00% 0 SSM connection m
175 0 16 0 0.00% 0.00% 0.00% 0 Authentication P
176 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B
177 0 1 0 0.00% 0.00% 0.00% 0 CHKPT EXAMPLE
178 0 1 0 0.00% 0.00% 0.00% 0 CHKPT DevTest
179 0 1 0 0.00% 0.00% 0.00% 0 IPS Process
180 0 2 0 0.00% 0.00% 0.00% 0 IPS Auto Update
181 0 2 0 0.00% 0.00% 0.00% 0 SDEE Management
182 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Inspect Tim
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
183 0 1 0 0.00% 0.00% 0.00% 0 Select Timers
184 8 2 4000 0.00% 0.00% 0.00% 0 HTTP Process
185 0 2 0 0.00% 0.00% 0.00% 0 CIFS API Process
186 0 2 0 0.00% 0.00% 0.00% 0 CIFS Proxy Proce
187 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc
188 0 3 0 0.00% 0.00% 0.00% 0 Crypto HW Proc
189 100 2 50000 0.00% 0.00% 0.00% 0 CCVPM_HDSPRM
190 8 1664 4 0.00% 0.00% 0.00% 0 FLEX DSPRM MAIN
191 8 1662 4 0.00% 0.00% 0.00% 0 FLEX DSP KEEPALI
192 0 182 0 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_
193 0 4 0 0.00% 0.00% 0.00% 0 HDA DSPRM MAIN
194 0 1 0 0.00% 0.00% 0.00% 0 cpf_process_tpQ
195 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve
196 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA
197 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr
198 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke
199 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA
200 136 988 137 0.00% 0.00% 0.00% 0 LOCAL AAA
201 0 2 0 0.00% 0.00% 0.00% 0 TPLUS
202 0 2 0 0.00% 0.00% 0.00% 0 VSP_MGR
203 0 3 0 0.00% 0.00% 0.00% 0 Crypto WUI
204 876 1463 598 0.00% 0.00% 0.00% 0 Crypto Support
205 0 1 0 0.00% 0.00% 0.00% 0 IPSECv6 PS Proc
206 0 1 0 0.00% 0.00% 0.00% 0 EPM MAIN PROCESS
207 4 1 4000 0.00% 0.00% 0.00% 0 CCVPM_HTSP
208 0 2 0 0.00% 0.00% 0.00% 0 VPM_MWI_BACKGROU
209 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_R2
210 0 1 0 0.00% 0.00% 0.00% 0 EPHONE MWI Refre
211 0 6 0 0.00% 0.00% 0.00% 0 FB/KS Log HouseK
212 0 2 0 0.00% 0.00% 0.00% 0 EPHONE MWI BG Pr
213 0 1 0 0.00% 0.00% 0.00% 0 Skinny HW confer
214 8 151 52 0.00% 0.00% 0.00% 0 VOICE REG BG Pro
215 0 1 0 0.00% 0.00% 0.00% 0 Presence Process
216 0 1 0 0.00% 0.00% 0.00% 0 CCSWVOICE
217 40 488 81 0.00% 0.00% 0.00% 0 Tunnel Security
218 0 1 0 0.00% 0.00% 0.00% 0 http client proc
219 20 184 108 0.00% 0.00% 0.00% 0 Crypto SS Proces
220 0 1 0 0.00% 0.00% 0.00% 0 QOS_MODULE_MAIN
221 0 1 0 0.00% 0.00% 0.00% 0 RPMS_PROC_MAIN
222 0 1 0 0.00% 0.00% 0.00% 0 VoIP AAA
223 0 4 0 0.00% 0.00% 0.00% 0 Crypto CA
224 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL
225 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL
226 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc
227 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT
3 1356 34857 38 0.00% 0.01% 0.00% 0 CRYPTO IKMP IPC
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
126 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track Pr
230 4 1 4000 0.00% 0.00% 0.00% 0 Crypto IKEv2
231 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key
232 456 2299 198 0.00% 0.00% 0.00% 0 IPSEC key engine
233 0 1 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce
234 296 373 793 0.00% 0.00% 0.00% 0 Crypto ACL
118 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana
136 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
237 0 1 0 0.00% 0.00% 0.00% 0 GDOI GM Process
238 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY
239 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY AC
240 228 22425 10 0.00% 0.01% 0.00% 0 Atheros LED Ctro
241 0 2 0 0.00% 0.00% 0.00% 0 Control-plane ho
242 32 2423 13 0.00% 0.00% 0.00% 0 PM Callback
243 0 1 0 0.00% 0.00% 0.00% 0 DATA Transfer Pr
244 0 1 0 0.00% 0.00% 0.00% 0 DATA Collector
245 8 34 235 0.00% 0.00% 0.00% 0 AAA SEND STOP EV
246 0 15 0 0.00% 0.00% 0.00% 0 EEM ED Resource
247 0 25 0 0.00% 0.00% 0.00% 0 EEM ED Track
248 0 450 0 0.00% 0.00% 0.00% 0 RMON Recycle Pro
249 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se
250 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
251 4 141 28 0.00% 0.00% 0.00% 0 Crypto cTCP proc
252 32 4440 7 0.00% 0.00% 0.00% 0 trunk conditioni
253 0 1 0 0.00% 0.00% 0.00% 0 trunk conditioni
254 20 2 10000 0.00% 0.00% 0.00% 0 VLAN Manager
256 0 133 0 0.00% 0.00% 0.00% 0 EEM Server
257 0 12 0 0.00% 0.00% 0.00% 0 EEM ED CLI
258 0 23 0 0.00% 0.00% 0.00% 0 EEM ED Counter
259 0 23 0 0.00% 0.00% 0.00% 0 EEM ED Interface
260 0 23 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD
261 4 12 333 0.00% 0.00% 0.00% 0 EEM ED None
262 0 23 0 0.00% 0.00% 0.00% 0 EEM ED OIR
263 0 23 0 0.00% 0.00% 0.00% 0 EEM ED SNMP
264 0 138 0 0.00% 0.00% 0.00% 0 EEM ED Timer
265 0 13 0 0.00% 0.00% 0.00% 0 EEM Policy Direc
266 288 535 538 0.00% 0.00% 0.00% 0 Syslog
267 0 1 0 0.00% 0.00% 0.00% 0 VPDN Test
268 0 159 0 0.00% 0.00% 0.00% 0 IP SLA MPLSLM Pr
269 0 1 0 0.00% 0.00% 0.00% 0 tHUB
270 8 157 50 0.00% 0.00% 0.00% 0 CEF Scanner
271 2084 148 14081 0.00% 0.00% 0.00% 0 crypto sw pk pro
140 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
273 0 5 0 0.00% 0.00% 0.00% 0 SSH Event handle
274 8 99 80 0.00% 0.00% 0.00% 0 IP SNMP
275 0 1 0 0.00% 0.00% 0.00% 0 PDU DISPATCHER
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
276 700 7 100000 0.00% 0.00% 0.00% 0 SNMP ENGINE
277 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6
278 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
279 928 139 6676 0.00% 0.00% 0.00% 0 SNMP Traps
1 592 864 685 0.00% 0.00% 0.00% 0 Chunk Manager
281 12 897 13 0.00% 0.00% 0.00% 0 Track
141 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
143 4 8 500 0.00% 0.00% 0.00% 0 L2MM
284 0 1 0 0.00% 0.00% 0.00% 0 IP NAT WLAN
285 28 472 59 0.00% 0.00% 0.00% 0 IP VFR proc
144 0 1 0 0.00% 0.00% 0.00% 0 MRD
288 232 1574 147 0.00% 0.00% 0.00% 0 IP SLAs Event Pr
289 524 5913 88 0.00% 0.00% 0.00% 0 NTP
Sh logging output:
Syslog logging: enabled (12 messages dropped, 371 messages rate-limited,
0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 1194 messages logged, xml disabled,
filtering disabled
Logging Exception size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
No active filter modules.
ESM: 0 messages dropped
Trap logging: level notifications, 1192 message lines logged
Logging to 172.20.10.8 (udp port 514, audit disabled,
authentication disabled, encryption disabled, link up),
1192 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Log Buffer (4096 bytes):
tion id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=00000758
001184: Jun 30 12:44:12: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001185: Jun 30 12:44:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).
001186: Jun 30 12:44:42: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001187: Jun 30 12:44:45: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=421 local=10.0.117.38 remote=10.1.20.202 spi=42D93024 seqno=00001D47
001188: Jun 30 12:45:13: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001189: Jun 30 12:45:43: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001190: Jun 30 12:45:46: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=421 local=10.0.117.38 remote=10.1.20.202 spi=42D93024 seqno=00001FCA
001191: Jun 30 12:45:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down
001192: Jun 30 12:45:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to up
001193: Jun 30 12:46:13: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001194: Jun 30 12:46:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).
001195: Jun 30 12:46:43: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001196: Jun 30 12:46:48: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=00000834
001197: Jun 30 12:47:14: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001198: Jun 30 12:47:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).
001199: Jun 30 12:47:44: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001200: Jun 30 12:47:49: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=000008A2
001201: Jun 30 12:48:12: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed
connection id=289, sequence number=833
001202: Jun 30 12:48:14: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.64.74,dstadr=10.16.101.1,size=1504,handle=0x5985
001203: Jun 30 12:48:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).
001204: Jun 30 12:48:45: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001205: Jun 30 12:48:50: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=353 local=10.16.101.1 remote=10.150.132.2 spi=7B50B77D seqno=000009B3
001206: Jun 30 12:49:16: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=10.150.132.2,dstadr=10.16.101.1,size=1512,handle=0x5961
001207: Jun 30 12:49:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).
Any help would be much appreciated!
06-30-2011 01:01 AM
Hi,
If you performing NAT translation, we can limit the translation per host basis. using command "ip nat trans max".
log the memory output and check it with cisco output itnerpreter.
Please rate my reply.
Thanks,
Aman
06-30-2011 01:13 AM
Hi
The cpu utilization is crossing 90% at times right? whcih caused crashing the device.
The following process are taking more ARP Input, Check heaps, Crypto IKMP...
Do you have seperate AIM ecnryption module on this router? If you dont have I would suggest you to have it on the router so that the crypto process will not burden to the actualy CPU. And you have DMVPN also which do log ot encryptions.
Apart from that I would suggest you to do harden your router....
1. Stop ip unreachables, proxy-arp and directed-broadcast to control the present scenario.
Please rate the helpfull posts.
Regards,
Naidu.
06-30-2011 10:19 PM
@Aman, I will definitely take up with our Network Security specialist and see what he has to say. That could very likely be the cause of this. Thanks for the tip mate.
@Naidu, yes we do have an AIM encryption for the necessary offloading of encryption processes. Indeed the router needs to be hardened. Optimization of the device should help however I feel more effective load balancing of traffic needs to be done. Lets hope we can coerce the client to do so.
06-30-2011 01:17 AM
hmm...looks like the router is being hammered..is the CPU high throughout the day or only at certain times? how many tunnels connect to this router?
06-30-2011 10:46 PM
Hi Gerald,
At the moment there are 84 encrypted DMVPN tunnels running on the router. This router is placed at the core of a bank where over 50 branches are terminating. This past week was also the end of the financial calendar hence you could imagine the load on the router. Hopefully, we will have a few days of respite before the problems comes back. We will continue monitoring the load on the router at different times of the day. Thanks for the tip mate.
Regards,
Waqas.
07-03-2011 07:17 PM
i'd have TAC take a look at this.. I ran into something similar in the past, same router platform, it turned out the router could not take the load. The router crashes (CPU flatlines to 100%) or EIGRP neighbor relationships drop at the time they do the nightly batch load. I short the router is to busy due to the amount of traffic. Iwould try and co-relate the traffic patterns and the high CPU. I also think 84 encrypted tunnels is a lot for a 3825 to handle considering there are other services running on the router...hth...
06-30-2011 02:28 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
12.4(15)T9 - You might try newer code; now that 15.x is out, why run prior T train?
Would look to resolve:
001198: Jun 30 12:47:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).
Would also investigate the cause processes using much CPU
CPU utilization for five seconds: 89%/51%; one minute: 78%; five minutes: 70%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
272 142088 116400 1220 21.87% 14.01% 8.62% 578 Virtual Exec
etc.
06-30-2011 11:12 PM
@Joseph regarding '
001198: Jun 30 12:47:16: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on FastEthernet0/3/2 (4), with xxxxxxx.xxxxxx.com FastEthernet1/15 (2).'
Yes, that is definitely a slight issue we need to resolve. The mismatch is with a remote device with which we tunnel. Convincing the client to go for a new license...hmm need to sharpen up my sales skills...Virtual Exec sounds kinda mysterious hence would need to read up more about that. Cheers for the help mate.
07-27-2011 11:42 PM
Hi guys,
Sorry for not keeping the forum updated about this. The past few weeks have been a bit manic at work. We verified the router config as much as we could, installed extra memory on the router. Even went so far as replacing 3 routers and modifying the design to distribute as much load as we could between the clients' corporate network and their internet connection. Finally, we noticed that NAT entries originating from our LAN by our server were well over the 300,000 mark; we tested by removing the NAT entry/disconnecting the server from our network and instantly things getting smooth. Apparently, for some reason the server which runs some sort of a banking application which notifies customers of transactions via SMS messages (among other services) was trying to make SSH connections to every IP on the global internet routing table. I take it the server has some sort of a malicious piece of software on it or has a bug that is being exploited by an exploit. If anyone else has come across such an issue where your server has been behaving this way then please do post it here. We asked the server/software development team to rebuild the server and install an effective anti-virus software on it to ensure it is secure; the server team followed suit but we are having the same result so far. This probably is a bug in the software code itself. Thanks for all the help with this issue guys. It really helped myself and my team get off to a good start whilst investigating this issue.
Thanks,
Vick
07-28-2011 12:22 AM
Hi Vick,
I have experienced the same issue with one of my windows 2003 server.
as per my knowledge experience I would suggest you to control the global NAT tranlsations on your router...this is one way.
The second way is that you can configure one to one nat by allowing required ports and required subnets from outside say (port 22, 443 from subnets 206.206.206.0 126.122.144.0 etc.,) and this way it will not open to all whcih is secure.
To control nat translation you can do by configuring below command in global config mode.
ip nat translation tcp-timeout 600
ip nat translation udp-timeout 600
Please let me know if you want go with the second way so that i can help you in configuration
Please rate the helpfull posts.
Regards,
Naidu.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide