2811 Router with High CPU (processes are fine)

Fabio Bustamante · ‎09-25-2012

Hi. I have a 2811 Router that is experiencing high CPU utilization. I have already issued the "show cpu proc" command, and everything seems to be fine (it's not a process issue) . According to our traffic graphs, we've seen that CPU is related to high link utilization, since we have 2 Fast Ethernet interfaces (Internet 4 MB and MPLS 6 MB), and when those interfaces have to deal with a lot of traffic at some hours, the CPU Load increases in the Router.

I found an article where they say that 2811 ISR Routers can deal with up to 61,44 Mbps (120.000 pps), but our router is far away from reaching that limit.It usually manages up to 16 MBPS. This router manages some services such as NAT traslation, ACLs, voice traslation profiles, policy routing (QoS), firewall and DHCP.

What can be happening to the router?.. Is it normal that when it has to process a lot of traffic, its cpu must increase up to 95%, even if it is handling just 16 MB compared to the theorical 61 MB? Is there any command or tool that I could use to troubleshoot this issue? Is there any limit in terms of WAN traffic that it could handle? How much influence do those services have over CPU Utilization?

Given these incidents, which router series could suit better in my network to avoid high CPU issues?

Thank you very much.

Fabio.

Giuseppe Larosa · ‎09-26-2012

Hello Fabio,

performance data sheet are based on pure CEF switching of traffic without any feature applied.

NAT, ACLs, IOS Firewall, QoS policies, Voice Gateway all use CPU resources so the cpu can reach a high level even if total traffic volume is less then the rate reported in performance data.

Moving to a faster router like C2921 (you need also the VOICE package) can be a long term solution.

Meanwhile, you may deploy a second router and you can move some features over the new box

Hope to help

Giuseppe

Joseph W. Doherty · ‎09-26-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

My experience with 2811s, just routing/forwarding with QoS, has been they max out at about 20 Mbps (duplex). Since you've also noted you're doing FW, NAT, etc., maxing out at 16 Mbps might be about just right.

Since about everything on a 2800 is done by the main CPU, everything that can be done to reduce processing should allow more CPU for forwarding packets. That's why something as simple as sequencing ACLs by hit frequency, or combining discreet ACLs into aggregate ACLs, can be important. Unless you're done something such as disabling CEF, unfortunately, you might only be able to increase forwarding performance slightly with optimal tuning.

The "easisest" soluiton would be to, as Giuseppe suggests, obtaining a "faster" device.

Fabio Bustamante · ‎09-26-2012

Hi guys Thank you very much for your help!!!

Based on those things you've said, we are considering changing our router. Nevertheless, we need to justify technically the reason why we need to change it.

We need to demonstrate to our company that our router needs to be upgraded, since we have to support those costs.

What command or what document could we use to justify this change?. Know about any 'show command' at peak cpu hours? We know that CPU increases at some hours and that all those services have an impact, but how could we show them that our router is working at limit conditions? We also must justify why we are proposing a C2921 Router to solve this issue. We need to demonstrate this and we still haven't found anything that could help us.

I appreciate your help.

Thanks

Fabio.

Giuseppe Larosa · ‎09-26-2012

Hello Fabio,

to show high cpu usage you can use

show proc cpu history

Hope to help

Giuseppe

Joseph W. Doherty · ‎09-26-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

BTW, you may not need a 2921 as the 2900 ISRs are considerably faster than their 2800 series counterparts.

I've attached two references on Cisco router performances.

Rick Morris · ‎09-27-2012

Here is a link to a post about eem scripts for high cpu. This was for the 6500 but with some slight changes it should work for you too. This will create a file that you can look through later.

https://supportforums.cisco.com/docs/DOC-17985

Basically, the justification is in the slowness of the network. You can show that the slowness is being caused by the CPU by the sh proc cpu as you have done. As Giuseppe mentioned, sh proc cpu history will give you a snap shot over the past 72 hours, breaking it down in, I think, 3 different sections. When the issue happens you can also do a sh proc cpu sorted to see which proc is causing the issue. This may help pinpoint what is going on and why you need to move some processes and applications off the router or purchase a larger one.

gajanangavli · ‎09-27-2012

Hi,

Router performance sheet shows total traffic the router can handle when there are no features enabled.

61 Mbps ( total traffic on all interfaces) = 100 % CPU without any features.With enabling other feautes like QOS ,access-list CPU increses though you have less traffc.

Can you paste sh pro cpu ?

Fabio Bustamante · ‎10-01-2012

Hi everyone.

First of all, thank you so much for your help. I really appreciate it.

I'm going to paste the show process cpu command. However, as I said before, I already know that the problem is related to high traffic rates and many services running in our router, because we haven't seen high values at any process. Besides, I have some graphs where I can confirm that CPU increases as traffic does.

BOG-RT-01#show proc cpu sort | e 0.0

CPU utilization for five seconds: 97%/91%; one minute: 86%; five minutes: 77%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

142 66946920 43443874 1540 2.17% 2.02% 2.07% 0 IP Input

423 3011184 8611065 349 0.85% 0.27% 0.13% 0 EIGRP-IPv4

402 19215496 925291332 20 0.28% 1.22% 2.00% 0 IP SLAs XOS Even

395 7656064 237335757 32 0.28% 0.37% 0.54% 0 HSRP Common

30 12525048 18523058 676 0.28% 0.34% 0.41% 0 ARP Input

403 6815684 4456774 1529 0.18% 0.11% 0.12% 0 IP SNMP

114 9661192 481913475 20 0.18% 0.50% 0.82% 0 Ethernet Msec Ti

These days we were asked about the maximum traffic in Mbps that our 2811 router could handle, taking into account those services it currently includes. We know that sometimes it reaches up to 15 Mbps, but we have no way to demonstrate that 15 Mbps (plus all those services) is close to the limit and that an upgrade is needed due to CPU resources.

Thanks.

Fabio

Joseph W. Doherty · ‎10-01-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You already have your proof and you've found the performance limits of a 2811 specific to your traffic and your configuration, i.e. graph CPU with aggregate traffic transiting router.

Also show, from the referrence I previously provided, WAN bandwidth recommendations for current Cisco ISRs and contrast their PPS rating with 2811.

gajanangavli · ‎10-01-2012

97%/91% ------ in this figures it shows 91% Average utilization due to interrupts, during last five seconds.

only 6 % is used by other process .

Joseph W. Doherty · ‎10-01-2012

gajanangavli wrote:

97%/91% ------ in this figures it shows 91% Average utilization due to interrupts, during last five seconds.

only 6 % is used by other process .

Correct, and so?