High CPU utilization on a Cisco Router 2911 / high percentage of time at the interrupt level
I'm writing regarding an issue of high CPU utilization that I have on a Cisco Router 2911. It appears when any traffic longer than few seconds is started. I have gone through many documentations and forums and for now I haven't found the right solution for this problem. In the attachment you can find an output od "show processes cpu history" and "show processes cpu sorted" (first few lines). One specific line is worth of noticing:
CPU utilization for five seconds: 99%/85%; one minute: 97%; five minutes: 93%
There I have found this: "The second number, 0% , shows the percentage of time at the interrupt level in the past 5 seconds. The interrupt percentage is the CPU time spent receiving packets from the switch hardware. The percentage of time at interrupt level is always less than or equal to the total CPU utilization."
So according to this explanation I understand that 85% of CPU utilization appeared because of errors on the router, like drops, overruns, etc., am I right? Or it is something else?
The output of the "show interfaces" of the interface connected to the Internet is also in the attachment. This is the output of last 6 days. There you can find this:
I have temporarily increased the "hold-queue X in" once to 100, once to 1000 and once to 240000. It didn't help at all. There is turned on also inspection on all vlan interfaces. I tried to increase "ip inspect tcp reassembly... queue length (to 1024), timeout (to 60), memory limit (once to 4096 and once to 4194303). Also nothing. And I temporarily limited the inspected services only to tcp, udp, icmp and ftp services, and no improvement.
There are also running ACL, NAT, QoS (I temporarily turned it off, but it didn't work), EIGRP with load balancing (I temporarily turned off load balancing, but it also didn't work), WEB VPN (around 20 AnyConnect connections at the same time at rush hours, but normally 5-10) and 3 IPsec tunnels. There are 2 links to Internet, the main link 120Mbps and a backup link 20Mbps. With the high CPU issue we are not able to use even half of the speed when transmitting over the IPsec tunnels. But when transmitting only to or from Internet (no IPsec nor AnyConnect) it works with full speed (at least at night).
There are also some errors after displaying "show crypto ipsec sa". Each interface tunnel has received errors ( for example: #send errors 0, #recv errors 11737)
I've even made a simple test during working hours. I connected from the Internet to local network via AnyConnect and I started downloading a file from local server to my computer and the CPU utilization increased immediately to 99-100%. The output of "show processes cpu..." is from that test. It shows only 10% of SSL VPN using CPU. From what I wrote at the beginning according to the documentation the second number shows percent of interrupt level. So what is the problem?
And of course I made an upgrade of a firmware to the newest stable one: Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.7(3)M4, RELEASE SOFTWARE (fc2)
I included also the white papers in the attachment. Do you think that this high CPU utilization in the result of to weak hardware for all of those services, and that is why the router does not handle to process all packets and that's why there is so many input errors like overruns, and they are dropped?
hmm i thought the 2911 was only recommended up to 50mbps even some posts Cisco state a conservative lower value than 50mb for wan circuits , if theres constant overruns and input errors your over utilizing the device and it cant cope with the flow and may require something faster, the buffer is struggling to cope on the interface , increasing the hold queue could cause you latency and is not really advised on WAN circuits , and ASR or ISR 4K could give you that full WAN throughput,
i think the cpu proc is not really showing the issue as the packets maybe are being punted to cpu and interrupt switched as its overloaded
Thank you for your suggestion. I also claim that this device is not proper to cope with these network requirements. However I was wondering if there is a possibility to lower the utilization of the CPU. So temporarily I reduced most of services on the router like:
- turning off load balancing - turning off QoS - turning off EIGRP and setting up static routes - minimalizing inspection of the services to tcp, udp and icmp - reducing the encryption strenght of IPsec tunnels - reducing the encryption strenght of AnyConnect connections
It also didn't help. After those changes again I made the test, this time at night, when almost no one works at that time. I connected via AnyConnect to the company network and started downloading 100MB file from the server. The download speed was just about 700-800Kbps (while QoS was turned) and the CPU utilization immediately increased almost up to 100%. Normally when we download or upload over Internet (without any VPN) the speed can reach 120Mbps (the maximum of the bandwidth).
So is really AnyConnect not recommended for ISR routers? Why then such possibility?
Do you use Cisco DNA Center? Have you used and are you willing to provide your feedback in using the Cisco DNA Center help and documentation?
If so, we’d like you to complete the survey linked below. Your feedback will help provide more effective and easi...
Listen: https://smarturl.it/CCRS9E18Follow us: https://twitter.com/CiscoChampion Reaching the height of your career is no simple feat. It often requires a combination of pursuing the right education, building the right professional network and being ...
In a typical production SD-WAN deployment, we would probably have many remote sites connected via many different Internet connections to a centralized data center or a regional hub. In most regions in the world, Internet providers will always use some typ...