Hello
I'm writing regarding an issue of high CPU utilization that I have on a Cisco Router 2911. It appears when any traffic longer than few seconds is started. I have gone through many documentations and forums and for now I haven't found the right solution for this problem. In the attachment you can find an output od "show processes cpu history" and "show processes cpu sorted" (first few lines). One specific line is worth of noticing:
CPU utilization for five seconds: 99%/85%; one minute: 97%; five minutes: 93%
The documentation that explains something:
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html
There I have found this:
"The second number, 0% , shows the percentage of time at the interrupt level in the past 5 seconds. The interrupt percentage is the CPU time spent receiving packets from the switch hardware. The percentage of time at interrupt level is always less than or equal to the total CPU utilization."
So according to this explanation I understand that 85% of CPU utilization appeared because of errors on the router, like drops, overruns, etc., am I right? Or it is something else?
The output of the "show interfaces" of the interface connected to the Internet is also in the attachment. This is the output of last 6 days. There you can find this:
Input queue: 1/75/105/3117406 (size/max/drops/flushes); Total output drops: 1627
...
452330 input errors, 0 CRC, 0 frame, 452330 overrun, 0 ignored
I have temporarily increased the "hold-queue X in" once to 100, once to 1000 and once to 240000. It didn't help at all. There is turned on also inspection on all vlan interfaces. I tried to increase "ip inspect tcp reassembly... queue length (to 1024), timeout (to 60), memory limit (once to 4096 and once to 4194303). Also nothing. And I temporarily limited the inspected services only to tcp, udp, icmp and ftp services, and no improvement.
There are also running ACL, NAT, QoS (I temporarily turned it off, but it didn't work), EIGRP with load balancing (I temporarily turned off load balancing, but it also didn't work), WEB VPN (around 20 AnyConnect connections at the same time at rush hours, but normally 5-10) and 3 IPsec tunnels. There are 2 links to Internet, the main link 120Mbps and a backup link 20Mbps. With the high CPU issue we are not able to use even half of the speed when transmitting over the IPsec tunnels. But when transmitting only to or from Internet (no IPsec nor AnyConnect) it works with full speed (at least at night).
There are also some errors after displaying "show crypto ipsec sa". Each interface tunnel has received errors ( for example: #send errors 0, #recv errors 11737)
I've even made a simple test during working hours. I connected from the Internet to local network via AnyConnect and I started downloading a file from local server to my computer and the CPU utilization increased immediately to 99-100%. The output of "show processes cpu..." is from that test. It shows only 10% of SSL VPN using CPU. From what I wrote at the beginning according to the documentation the second number shows percent of interrupt level. So what is the problem?
And of course I made an upgrade of a firmware to the newest stable one:
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.7(3)M4, RELEASE SOFTWARE (fc2)
I included also the white papers in the attachment. Do you think that this high CPU utilization in the result of to weak hardware for all of those services, and that is why the router does not handle to process all packets and that's why there is so many input errors like overruns, and they are dropped?
Could you help, please? I would be very grateful.
