11-20-2013 11:39 PM
Hello!
On C3825 with IOS 15.1(4)M5, I faced with issue:
I see high cpu load by Crypto IKMP proccess:
CPU utilization for five seconds: 92%/11%; one minute: 87%; five minutes: 86%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
295 96696680 625524 154590 76.42% 70.87% 71.46% 0 Crypto IKMP
65 7719140 608642 12682 1.08% 1.32% 1.33% 0 Per-Second Jobs
6 838308 93487 8967 1.00% 0.16% 0.12% 0 Check heaps
139 1534084 6029064 254 0.75% 0.69% 0.62% 0 IP Input
But we haven`t a lot of traffic ~10 Mbit/s.
We have 1 GRE tunnel to branch, and crypto map for remote clients and some ASA`s
I didn`t find any information how I can tshoot this situation.
How I can deal with this? Why CPU load so high?
Router1#show crypto engine accelerator statistic
Device: Onboard VPN
Location: Onboard: 0
:Statistics for encryption device since the last clear
of counters 1325 seconds ago
370618 packets in 370618 packets out
78673429 bytes in 77224979 bytes out
279 paks/sec in 279 paks/sec out
474 Kbits/sec in 466 Kbits/sec out
171515 packets decrypted 199103 packets encrypted
24579216 bytes before decrypt 52645763 bytes encrypted
15116445 bytes decrypted 63556984 bytes after encrypt
0 packets decompressed 0 packets compressed
0 bytes before decomp 0 bytes before comp
0 bytes after decomp 0 bytes after comp
0 packets bypass decompr 0 packets bypass compres
0 bytes bypass decompres 0 bytes bypass compressi
0 packets not decompress 0 packets not compressed
0 bytes not decompressed 0 bytes not compressed
1.0:1 compression ratio 1.0:1 overall
Last 5 minutes:
69175 packets in 69175 packets out
230 paks/sec in 230 paks/sec out
380964 bits/sec in 376846 bits/sec out
2652816 bytes decrypted 9459770 bytes encrypted
71697 Kbits/sec decrypted 255669 Kbits/sec encrypted
1.0:1 compression ratio 1.0:1 overall
Errors:
0 pkts dropped 0 ppq full
0 tx parts overflow 0 rx parts overflow
0 replenishment failure 0 zero len
0 flow inputs bad 0 cmd invalid
0 IPV4 len 0 IPV6 len
0 algor invalid
0 bad shadow particle 0 algor disabled
0 pre tx fail 0 dma error
0 dbit miss 0 pipeline abort
0 failsafe timeout 0 reserv
0 bad sz count 0 bad shdw
0 bad flow tx 0 spi mismatch
0 bad flow rx 0 auth fail
0 udm fs fail 0 pad fail
0 addr limit fixup fail 0 seq fail
0 quad fix sp 0 quad fix mp
0 quad fix cont
Thanks!
11-21-2013 12:17 AM
Typically due to incoming or outgoing negotation requests.
I suggest opening a TAC case to move faster.
show crypto isa sa
show crypto isa stats (hidden)
That's the minimum you should provide, I'd also suggest clearing the isa stats and taking them a few seconds apart while CPU is high.
11-21-2013 12:36 AM
#show crypto isa sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
109.x.x.x 79.105.x.x QM_IDLE 1196 ACTIVE L2L
109.x.x.x 77.41.x.x QM_IDLE 1157 ACTIVE VPNClient
109.x.x.x 85.88.x.x QM_IDLE 1162 ACTIVE L2L
109.x.x.x 2.61.x.x QM_IDLE 1179 ACTIVE L2L
109.x.x.x 178.212.x.x QM_IDLE 1178 ACTIVE L2L
109.x.x.x 178.210.x.x QM_IDLE 1191 ACTIVE VPNClient
109.x.x.x 178.210.x.x QM_IDLE 1186 ACTIVE VPNClient
109.x.x.x 178.210.x.x QM_IDLE 1182 ACTIVE VPNClient
109.x.x.x 178.210.x.x QM_IDLE 1180 ACTIVE VPNClient
109.x.x.x 178.210.x.x QM_IDLE 1177 ACTIVE VPNClient
109.x.x.x 194.186.x.x QM_IDLE 1187 ACTIVE L2L
109.x.x.x 62.32.x.x QM_IDLE 1146 ACTIVE L2LIPv6 Crypto ISAKMP SA
Router1#show crypto isa stats
ISAKMP Process Packet Stats
---------------------------
IKE Received Packets.......276181
IKE Transmit Packets.......278526
IKE Int Q Depth [0]........0
IKE Int Q Peak [0].........0
IKE Int Q Depth [1]........0
IKE Int Q Peak [1].........0
IKE Int Q Depth [2]........0
IKE Int Q Peak [2].........1
IKE Int Q Depth [3]........0
IKE Int Q Peak [3].........12
IKE Int Q Depth [4]........0
IKE Int Q Peak [4].........0
IKE IPC Q Depth............0
IKE IPC Q Peak.............1
IKE P1 Retransmitted.......13
IKE P2 Retransmitted.......143
IKE P1 Rcvd Retransmit.....12
IKE P2 Rcvd Retransmit.....11
IKE Dup Retransmit.........0
Pak too long in queue......0
Packets too long in queues
dropped by IKE Dispatcher..0
NAT Keepalives Received....16
IKE call reenqueue 0
IPSec Node Dead Reasons:
(errored reason mark with *)
No reason ...0
*By Error ...0
By User Command ...0
By Expired Lifetime ...0
No Error ...197
Informational (in) state 1 ...273268
Informational (in) state 2 ...0
Done with xauth request/reply exchange ...117
Transaction mode done ...81
Saved QM no longer needed ...2
IKMP_NO_ERR_NO_TRANS ...0
P2 Re-tx timer expired (CONF_ADDR) ...0
Config mode cleanup ...0
QM done ...0
QM done (commit) ...0
QM done (await) ...944
IKE deleted ...22
*Delete Larval ...1
*Phase 2 err count exceeded ...11
*Decrypt_payload failed ...0
*Invalid payload ...0
*No IV for Transaction ...0
*DELAYED_QM_TIMER expired ...0
*QM no hash ...0
*QM bad hash ...0
*QM not authenticated ...0
*QM rejected ...16
*QM not accepted ...0
*No ke payload ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
*Invalid reason code ...0IKE SA Dead Reason:
No reason ...111
*By error ...9
BY user command ...132
BY expired lifetime ...0
No error ...0
Delete no delete ...0
P1 delete notify (in) ...0
VRF removed from profile ...0
Death by tree-walk ...0
End of ipsec tunnel ...0
IKE SA Lifetime Exceeded ...0
*Receive initial contact ...0
*P1 errcounter exceeded (PEERS_ALIVE_TIMER) ...18
*Needed xauth ...18
*XAUTH fail ...0
*Client cancel xauth prompt ...0
*XAUTH not complete 1 ...8
*XAUTH not complete 2 ...0
*Fail to allocate ip address ...0
*Failed to allocate a connection id ...0
*Phase1 SA policy proposal not accepted ...52
*Recevied fatal informational ...0
*SA err counter exceeded (info) ...0
*Death by retransmission P1 ...0
*Death by retransmission P2 ...0
*Death by retransmission throw ...0
*Encrypt failure ...0
*Delete_me flag/throw ...0
*IKMP_ERR_NO_RETRANS ...0
*gen_ipsec_isakmp_delete but doi isakmp ...0
*QM_TIMER expired ...0
*IKE Fragmentation Failure ...0
* ...0
*Invalid reason code ...0
11-21-2013 12:39 AM
After clear stats
Router1#show crypto isa stats
ISAKMP Process Packet Stats
---------------------------
IKE Received Packets.......61
IKE Transmit Packets.......60
IKE Int Q Depth [0]........0
IKE Int Q Peak [0].........0
IKE Int Q Depth [1]........0
IKE Int Q Peak [1].........0
IKE Int Q Depth [2]........0
IKE Int Q Peak [2].........0
IKE Int Q Depth [3]........0
IKE Int Q Peak [3].........1
IKE Int Q Depth [4]........0
IKE Int Q Peak [4].........0
IKE IPC Q Depth............0
IKE IPC Q Peak.............0
IKE P1 Retransmitted.......0
IKE P2 Retransmitted.......0
IKE P1 Rcvd Retransmit.....0
IKE P2 Rcvd Retransmit.....0
IKE Dup Retransmit.........0
Pak too long in queue......0
Packets too long in queues
dropped by IKE Dispatcher..0
NAT Keepalives Received....0
IKE call reenqueue 0
IPSec Node Dead Reasons:
(errored reason mark with *)
No reason ...0
*By Error ...0
By User Command ...0
By Expired Lifetime ...0
No Error ...0
Informational (in) state 1 ...61
Informational (in) state 2 ...0
Done with xauth request/reply exchange ...0
Transaction mode done ...0
Saved QM no longer needed ...0
IKMP_NO_ERR_NO_TRANS ...0
P2 Re-tx timer expired (CONF_ADDR) ...0
Config mode cleanup ...0
QM done ...0
QM done (commit) ...0
QM done (await) ...0
IKE deleted ...0
*Delete Larval ...0
*Phase 2 err count exceeded ...0
*Decrypt_payload failed ...0
*Invalid payload ...0
*No IV for Transaction ...0
*DELAYED_QM_TIMER expired ...0
*QM no hash ...0
*QM bad hash ...0
*QM not authenticated ...0
*QM rejected ...0
*QM not accepted ...0
*No ke payload ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
* ...0
*Invalid reason code ...0IKE SA Dead Reason:
No reason ...0
*By error ...0
BY user command ...0
BY expired lifetime ...0
No error ...0
Delete no delete ...0
P1 delete notify (in) ...0
VRF removed from profile ...0
Death by tree-walk ...0
End of ipsec tunnel ...0
IKE SA Lifetime Exceeded ...0
*Receive initial contact ...0
*P1 errcounter exceeded (PEERS_ALIVE_TIMER) ...0
*Needed xauth ...0
*XAUTH fail ...0
*Client cancel xauth prompt ...0
*XAUTH not complete 1 ...0
*XAUTH not complete 2 ...0
*Fail to allocate ip address ...0
*Failed to allocate a connection id ...0
*Phase1 SA policy proposal not accepted ...0
*Recevied fatal informational ...0
*SA err counter exceeded (info) ...0
*Death by retransmission P1 ...0
*Death by retransmission P2 ...0
*Death by retransmission throw ...0
*Encrypt failure ...0
*Delete_me flag/throw ...0
*IKMP_ERR_NO_RETRANS ...0
*gen_ipsec_isakmp_delete but doi isakmp ...0
*QM_TIMER expired ...0
*IKE Fragmentation Failure ...0
* ...0
*Invalid reason code ...0
11-21-2013 12:49 AM
First of all you have debugs on ... probably not the best idea ever :-)
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide