CPU utilization hitting 100% on Cisco 2621XM

[hytron]

Hi everyone,

I am not sure if the issue I have is due to router's performance of my configuration. Currently I have 2621XM connected in my internal network where Fa0/1 is considered untrusted zone and Fa0/0 is considered trusted. It appears that traffic is limited to about 3.17MBps (Mega Bytes) between Fa0/0 and Fa0/1. There are ACLs on the Fa0/1 inbound and Fa0/0 inbound. I am also using ip inspect rules for the Fa0/1 outbound traffic. When I remove ip inspect policy from the Fa0/1 interface, the traffic speed bumps up to about 4.8MBps (\which is about 38Mbps - Mega bits).

Each time the CPU utilization is:

CPU utilization for five seconds: 99%/99%; one minute: 19%; five minutes: 4% PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

Access to the router during that time is veerrrryyyyyy slow! Does this mean I am hitting the maximum limit this router can handle?

Thanks!!!

[Leo Laohoo]

Could be an IOS bug too.

[hytron]

I am currently running..

IOS (tm) C2600 Software (C2600-ADVIPSERVICESK9-M), Version 12.3(26), RELEASE SOFTWARE (fc2)

But I have tried several different versions including c2600-advipservicesk9-mz.124-25d.bin and with the newer IOS it runs even slower!

I found some info that says this router can process 30kpps. Not sure how to convert that to Mbps?

[Rozsa Illes]

Hello Darko,

The output (99/99) indicates that the high cpu is under interrupt when the packets are processed by CEF switching. Very often this indicates a limitation of the platform. There are a few things we can check though:

- Are there any spurious memory errors in the logs? Those could indicate a software failure

- Do you see any pattern in the high cpu? Does it follow business hours? You can have a look at "show process cpu history"

- I found the same information about the limitation. The 2621XM should be able to handle 30,000 pps. This test was made with same size IP packets without any features configured though. I cannot simply be converted to Mbps but if you do a show interface, it will record the number of packets process in the last 5 minutes. For this test, I would recommend to set the load-interval under the interface configuration to 30 so we could get more accurate results (show interface would update the statistics every 30 seconds instead of the default 5 minutes).

- What has to be taken into account that extra features would decrease the throughput. Some CPU intensive features would be inspection (which you already noted), QoS, NBAR, large ACLs, netflow for example.

- Also if we have a large amount of small packets, it would decrease the throughput as we would need the same CPU cycles.

I hope it will help.

Warm Regards,

Rose

[Joseph W. Doherty]

Disclaimer

The       Author of this posting offers the information contained within  this      posting without consideration and with the reader's  understanding   that    there's no implied or expressed suitability or  fitness for any    purpose.   Information provided is for informational  purposes only  and   should not   be construed as rendering professional  advice of any  kind.   Usage of  this  posting's information is solely  at reader's own  risk.

Liability Disclaimer

In       no event shall Author be liable for any damages whatsoever     (including,   without limitation, damages for loss of use, data or     profit) arising  out  of the use or inability to use the posting's     information even if  Author  has been advised of the possibility of    such  damage.

Posting

I found some info that says this router can process 30kpps. Not sure how to convert that to Mbps?

Depends on the packet size (also router pps sometimes varies with packet size).

For minimum size Ethernet packets 30 Kpps equals about 20 Mbps.  For maximum (standard) Ethernet packets 30 Kpps equals about 370 Mbps.  For actual routers you also need to allow for duplex and other services.

[hytron]

Thank you for your replies.

I checked the log for any errors and did not find any.

Both interfaces are connected to a Cisco 2950 switch and are running at 100/full. I also checked on the switch and there  are no errors or collisions on the ports.

99% CPU utlization occurs when I transfer files across the router. Even though the interfaces are running at the 100/Full, the download or upload speed seems to be limited to about 3MBps which is about 24mbps.

During the transfer, the data is running at their maximum MTU (1500):

(DF) (ttl 64, id 15907, len 1500) - from tcpdump

Suggested by Rose, I configured the interfaces to report load every 30 seconds and here are my findings during the download/upload across the router:

CPU utilization for five seconds: 78%/75%; one minute: 42%; five minutes: 13%

Fa0/0:
30 second input rate 433000 bits/sec, 881 packets/sec
30 second output rate 20700000 bits/sec, 1718 packets/sec

Fa0/1:
30 second input rate 20555000 bits/sec, 1709 packets/sec
30 second output rate 391000 bits/sec, 876 packets/sec

These results were taken during the file transfer between two zones across the router. Please note that I limited the download speed on the client side otherwise I would not be able to issue any commands on the router, because the router becomes temporarily unresponsive during the time when CPU runs at 100%.

Appears that the speed is about 20mbps and the cpu is loaded at 75%.

I have tried a few different versions of IOS and appears that newer version (12.4 vs 12.3) run slower.

[Joseph W. Doherty]

Disclaimer

The        Author of this posting offers the information contained within   this      posting without consideration and with the reader's   understanding   that    there's no implied or expressed suitability or   fitness for any    purpose.   Information provided is for informational   purposes only  and   should not   be construed as rendering  professional  advice of any  kind.   Usage of  this  posting's  information is solely  at reader's own  risk.

Liability Disclaimer

In        no event shall Author be liable for any damages whatsoever      (including,   without limitation, damages for loss of use, data or      profit) arising  out  of the use or inability to use the posting's      information even if  Author  has been advised of the possibility of     such  damage.

Posting

From everything you've described, it would seem you're just hitting the capacity limits of the router.

About the only thing you might still try is minimize CPU processing per packet.  For instance are the ACLs as "compressed" (e.g. one ACL with larger mask vs several ACLs with small mask) and as "simple" (i.e. examine the least amount of necessary data) as possible, and are they ordered in hit probability?  Are there any other services you can live without?  Have you tried neflow caching?  Have you checked buffer stats, and tuned to avoid creates/trims?

[hytron]

Thank you very much for your replies. Yes it appears to be the case that I am hitting the hardware capacity of the router. When I disable ip inspect, I gain an extra 1-1.5MBps (Mega Bytes ps), so inspection alone takes up quite a bit of the cpu cycles. Also if I disable ACLs and ip inspect that speed increases even more.

To be able to handle interface speed of 80-90mbps using ACLs and inspection rules, which router would you recommend?

Another 26xx series or should I jump to 1841 or similar...?

Thanks!!!

[Joseph W. Doherty]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

In one of Cisco's more recent documents, they recommend a 2951 for WAN bandwidths up to 75 Mbps and 3925 for WAN bandwidths up to 100 Mbps.  Cisco recommendations are generally conservative insuring you can do almost anything on the router up to that bandwidth limit, but if your actual CPU processing is less you can go with a less powerful router.

I'm unsure any 26xx or 18xx, except maybe the 1861, would be sufficient.  I suggest you look at 2821 or better, and even any 28xx might be insufficient too.  You might get by with a low end 29xx, as even the 2901 would have about 11x the performance of your 2621XM.