Cisco 2801 High CPU

[huaingomez]

In a Cisco 2801 providing internet access to 50 users, working as Firewall (Zone Firewall) and being part of a site-to-site VPN in having hugh load in the CPU.

The CPU is configured as Router in a Stick with around 10 VLANS but as i said just 50 users.

Show Proc command return this information:

RT2801-IGLVEN-BARB#sh proc cpu hist

RT2801-IGLVEN-BARB   01:56:24 PM Friday Nov 6 2015 VEN


    999999999999999999999999999999999999999999999999999999999999
    999999999888886666655558888888888999999999999999999999999999
100 ************************************************************
 90 ************************************************************
 80 ************************************************************
 70 ************************************************************
 60 ************************************************************
 50 ************************************************************
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6
             0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)


    999999999999969799999999999999684745799644299899499919135683
    999999999998925199999999999991325455359400749392899386419308
100 ##########*** * **###*###**#*        **     * *  *#  *
 90 ##########**# * **#######**##*       **    ** ** *#* *
 80 ##########**# * *############* *     *#    ***** ##* *    *
 70 ###########*# ***############* * *  **#    *#*#* ##* *    *
 60 ###########*#****############*** * **##*   *#*#* ##* *  ***
 50 ###########*#***#############********##*   *#*#**##* *  ***
 40 #############*#################******##*** #####*### #  ****
 30 ################################****######*######### # ***#*
 20 ##################################**################*# *#*##
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6
             0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%


    999999938253424349999999999999954442425279999999999999492635244424999999
    999999983525839009999999999999935229880919999999999999598866960388999999
100 *******          **************          ************* *          ******
 90 *******          ****####******          ****#*#****** *          ******
 80 **##*** *        ****######****          ***####****** *          ******
 70 **##*** *        **########****         ***#########** * *        ******
 60 **##*** *        *##########***         *###########** * * *      ******
 50 **##*** * * * *  ###########*#***   * * *############*** * * *   *******
 40 #####**** *** * *###########*#***** * * *#############** *** *** ******#
 30 ######******* ***#############**********##############***************###
 20 ######***********#############**********##############***************###
 10 #######**********##############*********##############*************#####
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%


RT2801-IGLVEN-BARB#sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 96%/87%; one minute: 89%; five minutes: 93%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 104    92002644    17859960       5151  6.02%  6.81%  5.90%   0 IP Input

   2     1802304      290586       6202  1.56%  0.33%  0.44%   0 Load Meter

  40     8720128     1434769       6077  0.22%  0.25%  0.25%   0 Per-Second Jobs

 311     5095268    42275855        120  0.07%  0.08%  0.06%   0 PPP manager

 111         128          72       1777  0.07%  0.03%  0.01% 194 Virtual Exec

 312     1467176     2743301        534  0.07%  0.08%  0.08%   0 IP NAT Ager

 177     2530348     2761833        916  0.07%  0.08%  0.08%   0 FW DP Inspect pr

 314     1354092      234208       5781  0.07%  0.03%  0.01%   0 DHCPD Receive

Any idea what's the proble and how can fix it?

Thanks in advance.

[Mark Malone]

Hey

huge amount of interrupts 89% your process switching traffic killing the cpu , is cef enabled ? Have you tried changing the software version incase its a bug , are you logging acls , anything in the logs

You could try capture a bit more while its occurring it may help identify what exactly is causing it

couple of docs may point you to something regarding high cpus on routers and interrupts

http://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/15095-highcpu.html

http://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41120-highcpu-interrupts.html

You could run this script it will take some commands and send it to flash while its high , will give more of an idea whats happening

event manager applet High_CPU

    event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op ge entry-val "80" exit-time 10 poll-interval 5

    action 0.1 syslog msg "CPU Utilization is high"

    action 0.2 cli command "enable"

    action 0.4 cli command "show log | append flash:CPU_Profile.txt"

    action 0.5 cli command "show process cpu sorted | append flash:CPU_Profile.txt"

    action 0.6 cli command "show interfaces | append flash:CPU_Profile.txt"

    action 0.7 cli command " show ip cef switching stat | append flash:CPU_Profile.txt"

    action 0.8 cli command " show ip traffic | append flash:CPU_Profile.txt"

    action 0.9 cli command " show int switching | append flash:CPU_Profile.txt"

    action 1.0 cli command "end"

[huaingomez]

Hi

Thanks for your answer.

Yes cef is enabled.

Question: Running this script can not cause the router performance issue to worsen? or deplete flash memory?

[huaingomez]

One question:

Is it as Mr. Doherty suggest that this is too much load for this router?

Should i get a better or newer or bigger router?

[Mark Malone]

I agree with Joe it was something I missed when originally looking at it , 2800s only have fe ports you want to check see what type of load you are running when these high outputs are occurring, high interrupts can be caused by overloading the router with traffic just cant process it as quick as it comes in

The script I gave you will also collect that in the show interfaces section but you should easily be able to see what the interfaces are using during high outputs to gauge the issue , the script if your going to use it should be removed when a sample has been taken and sent to flash.

A 2900 series is probably more what you want gig link , you prob have  gig switch behind it ? if that's the case you could be bottlenecking traffic at the 2800 fe uplink

[huaingomez]

The 2801 router is connected to a 2960G Switch which is working as core switch (i think all this is too much for just 50 ot 60 users top).

It occurs to me that if i low the different interfaces of the switch to 100Mbps i could minimize the bottleneck.

What do you think?

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I suspect you're volume of traffic is just too much for your 2801.

Run your 2801's FE interfaces at 10 Mbps, and see how it performs then.

Even though the 2801 has FE interfaces, it's not a 100 Mbps capable router.

If you need to route 100 Mbps (or gig), you'll need a more powerful router.

[huaingomez]

Hi

Thanks for your answer.

Is there a way i can upgrade this router so it can handle the load that in your opinion is too high?

Memory upgrade is an option?

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Unfortunately, on that model, its CPU runs at a fixed rate, so the only way to improve (CPU) performance is to replace with a more "powerful" ISR variant.

I've attached a Cisco whitepaper that documents performance for the later ISRs.

[huaingomez]

This is weird, i have a squid proxy behind the router, i had implemented bandwidth control through Delay Pools, well, it occurred to me that it could be the problem, to be honest i was just blindly guessing, so i removed the Delay Pools instructions from my proxy's configuration, and magically everything went back to normal in the router and it is that way ever since.

I post this here because it may be usefull for some one somewhere.

[huaingomez]

This is weird, i have a squid proxy behind the router, i had implemented bandwidth control through Delay Pools, well, it occurred to me that it could be the problem, to be honest i was just blindly guessing, so i removed the Delay Pools instructions from my proxy's configuration, and magically everything went back to normal in the router and it is that way ever since.

I post this here because it may be usefull for some one somewhere.