High CPU on 2811 but no visible process that causes it

[kasper123]

We have a cisco 2811 that we use to terminate 2 VPN's and to get access to internet.

Several times a day there is high CPU on the router but I can't seem to be able to find what causes it.

Router#show processes cpu sorted

CPU utilization for five seconds: 98%/94%; one minute: 94%; five minutes: 88%

PID Runtime(uS)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

118  3747800704     3426462       2347  2.13%  2.71%  2.37%   0 IP Input        

126     3060000         175      17485  0.45%  0.03%  0.04% 514 SSH Process     

  22  1275008000     4030571        316  0.22%  0.24%  0.22%   0 ARP Input       

274   310512000      786648        394  0.22%  0.19%  0.18%   0 IP NAT Ager     

...

These are the top processes. The CPU is at 98% but there is no process using more then 3%. This processes usage is the same even when the CPU load is normal.

Any ideas how to troubleshoot this?

Regards.

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

The "missing" CPU is "interrupt" CPU, which generally is devoted to forwarding traffic.  What's the aggregate of all ingress interface bandwidth?  My experience, the 2811 will max out with about 40 to 50 Mbps, aggregate.

If your bandwidth consumption is much lower, I also noticed you mentioned VPN.  That will consume more CPU per packet forwarded expecially if there's any fragmentation being done.  (The latter can greatly add to the load of the router.)  Are you using tcp mss-adjust?

[kasper123]

Hi Joseph,

there are several WAN links terminating at the router. Two of them are dedicated for the VPN's and there is a third (an ADSL PPPoE connection) that is 50Mbps and that is the default route.

And yes we are using tcp adjust-mss on the LAN facing interface.

ip tcp adjust-mss 1452

Is there something we can do to mitigate the problem?

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Well the 1452 would adjust for the PPPoE overhead, but VPN tunnels have even more overhead.  Do you adjust for them too?

There other tunnel interface options that sometime can help a little too, but if you're dealing with any full size UDP packets (or similar traffic), they will need to be fragmented, and there's nothing you can do unless you adjust the MTU of the host.

BTW, a 50 Mbps ADSL, alone, is, again in my experience, enough to overwhelm a 2811.

If you have enough traffic, especially traffic that requires fragmentation, the only real solution is a "faster" router.