High CPU Usage on C6509

[John Robert Picache]

Good day, can someone analyze this for me, in regards with high cpu usage.

2222222222222222222222222222222222333332222244444222222222

     8888555559999955555777773333344444000007777711111555559999

100                                                           

90                                                           

80                                                           

70                                                           

60                                                           

50                                                           

40                                              *****        

30  ************************          ************************

20  **********************************************************

10  **********************************************************

    0....5....1....1....2....2....3....3....4....4....5....5....

              0    5    0    5    0    5    0    5    0    5   

               CPU% per second (last 60 seconds)

     4443333334443433334333333333333333333333343333333333333333

     1006530591135079734601132535328062177773414263667168577916

100                                                           

90                                                           

80                                                           

70                                                           

60                                                           

50                                                           

40  *****  ********** **     * *  * *  ****  *  * *** ****** *

30  #############**##*####*#*##*###*#*#########*##*****###****

20  ##########################################################

10  ##########################################################

    0....5....1....1....2....2....3....3....4....4....5....5....

              0    5    0    5    0    5    0    5    0    5   

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

     4744444545444444444444444444333344444444444444444434444444433344344444

     4547425182436632379924402500899822115303104140406471591734499730802883

100                                                                       

90                                                                       

80   *                                                                   

70   *                                                                   

60   *                                                                   

50   * *  ****  **   ***     *          *           *   ** *           **

40  **********************************************************************

30  #*******###**********##**********#********************************####

20  ######################################################################

10  ######################################################################

    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

              0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

Local_Backbone#sh proc cpu sorted 5sec

CPU utilization for five seconds: 24%/16%; one minute: 27%; five minutes: 27%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

123  1380698152 938782622       1470  6.07%  6.66%  6.52%   0 IP Input        

   9    71590456 190912340        374  0.39%  0.48%  0.47%   0 ARP Input       

213    53552228 246488976        217  0.23%  0.42%  0.48%   0 SNMP ENGINE     

176     2804332  35409778         79  0.15%  0.02%  0.00%   0 OSPF Hello      

319      918144  11719407         78  0.15%  0.01%  0.00%   0 OSPF Router 100 

   4         236       195       1210  0.07%  0.00%  0.00%   1 Virtual Exec    

169     2374788   2035919       1166  0.07%  0.03%  0.00%   0 Adj Manager     

118     5885520  15709517        374  0.07%  0.07%  0.07%   0 CDP Protocol    

  36       69788   9934569          7  0.07%  0.00%  0.00%   0 TTY Background  

179    30669516 309965501         98  0.07%  0.23%  0.24%   0 IP SNMP

need a quick opinion as i have another backbone abroad that uses only 5-10% of cpu usage.

- same config

- same setup (2BB for Sec/firewall, 2BB for Integration, 6Distro SW)

- different number of users

thank you and god bless.

[stephen.stack]

Thats not really that high

And is most likley traffic related. You can read over this useful document which explains the IP Input process in detail and how to determine why is is rising higher than other process.

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml

Be careful with running some of the debugs on your switch.

==========================
http://www.rConfig.com 

A free, open source network device configuration management tool, customizable to your needs!

- Always vote on an answer if you found it helpful

[InayathUlla Sharieff]

I also agree with Stephan that this is not a concern that the cpu you are seeing it sohuld reduce down.

If in case still want to do the troubleshooting here is the procedure:

Here are some of the commands you can refer while you see the high cpu:

• •1-  Show proc cpu sorted | ex 0.00%  >> First check whether CPU is due to Interrupt or due to Process Switching.

Eg: CPU utilization for five seconds: 6%/2%; one minute: 2%; five minutes: 2%

6% is total CPU utilization and 2 % is due to Interrupt. You would see this on a higher bit when you have the high cpu. Total-interrupt will give you the value of Process.

More info: http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml

b) Show proc cpu history :---Check since how long is the CPU been affected.

2- Check if there is any interface drops are been seen:

show interface | include line|drops   ( Check if any vlan/interface are having input drops)

3- check if there are any STP issue:

show spanning tree detail

show spann det | in ieee|from|occur|is exec

show spann det | in executing | changes

show interface | include line|drops

show cef drop

show ip traffic   >> Check the traffic utilization.

4- There are some internal debugging which would help to find out which packets ( from Src,dst) is hitting the CPU:

debug netdr capture rx  <<<<( DEBUG OF NETDR IS VERY USEFULL TO CHECK WHAT PACKETSS ARE COMING TO CPU)

show netdr capture-packet  >> This will give you the captured outputs were you can see the Src and dst . Then you can take the action with those src ip address guys to check why they are punting the traffic.

HTH

Regards

Inayath

***Plz rate if this info is helpfull.

[ALEXIS N. FLORES]

Thanks Inayath, it was very helpfull !!!
I started with step 4, and the result was very complex, but finally show me too many flood packets to the RP.

Step 3 showed me the problem, good convination of filters.

Regards

Alexis

[InayathUlla Sharieff]

Glad the info was helpfull.

Below is the link as well.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/63992-6k-high-cpu.html

HTH

REgards

Inayath

***Plz rate if this info is helpfull.