3620 with NM-VPN/MP: high CPU util due to many interrupts

[tnatsukawa]

Hi! My Cisco 3620 experiences very high CPU utilization with modest amount of traffic (about 600 kbps) over an ipsec vpn tunnel. The CPU utilization is 98%, and 97% of that is due to interrupts. If I generate more traffic, the router freezes (no response from the console port). I'm not sure where I should go from here...

My 3620 has an NM-VPN/MP module. That's an NM-style hardware VPN accerelator. On the router I only have one 3DES vpn tunnel configured.

If I disable the hardware accerelator, I don't get very many interrupts, but the CPU utilization is very high obviously because the CPU has to encrypt/decrypt packets...

Please help.

Tatsuo

With about 600-kbps traffic:

the3620#sh proc cpu sort 5sec

CPU utilization for five seconds: 98%/97%; one minute: 56%; five minutes: 23%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

3 1456 110 13236 0.68% 0.08% 0.01% 0 Exec

81 1116 1424 783 0.20% 0.04% 0.01% 0 CEF process

21 676 887 762 0.06% 0.01% 0.00% 0 Per-Second Jobs

...

Again, with about 600-kbps traffic, but this time with "no crypto engine accelerator" to disable to hardware accerelator:

the3620#sh proc cpu sort 5sec

CPU utilization for five seconds: 100%/35%; one minute: 41%; five minutes: 11%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

70 35396 127 278708 63.86% 26.04% 6.66% 0 Encrypt Proc

3 248 196 1265 0.32% 0.18% 0.07% 0 Exec

4 4124 229 18008 0.32% 0.04% 0.00% 0 Check heaps

18 752 839 896 0.08% 0.01% 0.00% 0 Net Background

...

END

[awaheed]

Hi Sukawa,

With some of the IOS versions on the 3600 we have seen the problem with the High CPU, kindly make sure you notify the Cisco TAC with your "show tech" for them to recommend a proper course of action, or you can let me know what your version is and I can check if there is a known issue with that.

Regards,

Aamir Waheed,

Cisco Systems, Inc.

CCIE#8933

-=-=-

[tnatsukawa]

Aamir, I'm running 12.2(16b). Tatsuo

[dcayer]

Hi Sukawa,

How did you resolve your hi CPU problem?

I've been troubleshooting the same issue with a 3640.

I tried upgrading IOS to version 12.2(19a), but that did not resolve the issue. I'm now back to running version 12.1(3a)XI6 and the symptoms are still present.

Daniel Cayer

EDS Canada

[tnatsukawa]

Daniel, sorry, I was not able to get this problem fixed... I'll be most interested to know how you get it fiexed! Good luck. Tatsuo

[dcayer]

Environment:

-VPN tunnel between two Cisco 3640 router via the WAN (Transparent LAN servirce).

-Routers are both configured with 64Mbytes, IP-PLUS/3DES IOS version c3640-ik9s-mz.122-19a.bin.

-Router configuration:

!

crypto isakmp policy 1

encr 3des

hash md5

authentication pre-share

!

crypto isakmp key key_dan address 192.168.205.17

!

crypto ipsec transform-set WAN-DATA esp-3des esp-md5-hmac

!

crypto map WAN_Link1 10 ipsec-isakmp

set peer 192.168.205.17

set transform-set WAN-DATA

match address 101

!

interface FastEthernet0/0

description TLS connection to ROUTER2

bandwidth 10000

ip address 192.168.205.18 255.255.255.252

ip ospf message-digest-key 15 md5

speed 10

full-duplex

crypto map WAN_Link1

!

access-list 101 deny ip any 192.168.205.8 0.0.0.3

access-list 101 deny ip 192.168.205.8 0.0.0.3 any

access-list 101 deny ip any host 192.168.208.225

access-list 101 deny ip host 192.168.208.225 any

access-list 101 permit ip any any

!

Problem definition:

By generating approximately 600Kbps of throughput with an extended ping between the routers (using the loopback addresses) I can cause the CPU to hit 100% even when the VPN accelerator modules are enabled. This causes critical network connectivity problems for our users. When I use the router Ethernet IP's (which are denied in ACL 101) for the extended ping instead of the loopbacks I get no CPU hit because the packets do not get encrypted.

Solution:

To fix this problem I enabled CEF (which is disabled by default). I also enabled "ip route-cache flow" on all interfaces to further reduce CPU load. Now the CPU utilization stays relatively low when I repeat my extended ping test (using the loopback IPs) and I have a significant increase in throughput (approximately 1Mbps versus 600Kbps).

Conclusion:

During peak hours, the CPU utilization is still peaking at 95% for short period of times, but the users no longer complain. I am now starting to believe that our 3DES VPN routers are not suitable for traffic loads over 1Mbps.

Daniel Cayer

EDS Canada

[msheik]

Hi all,

Iam about to add VPN/MP module for one side peer 3620, running DES with another 3620 between two international locations.My case, voice is passing between the locations. Issues are:

1. CPU goes above 90% and users complain about the voice quality.I can see that Encry proce and IP Input taking about 30% CPU peak times.

IOS : 12.2(13b)

So is it going to solve my issue..?Please advice.

Thanks in advance

MS

[DNATA]

We have 3640 running for quite sometime, recently we have seen that the CPU utilization reached 99%/97% and was there for 5 hours and then later dropped, but we sniffed the network we could not see any virus attack or so. We have E3 ATM WAN Card where in the utilization during peak hours reaches to 24 Mbps.

Now the Utilization is around 35 to 57 %, we have CEF enabled and netflow on the interfaces.

The version we are running is 12.1(2)T. Why is the CPU utilization still high.