cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2381
Views
10
Helpful
7
Replies

High CPU Utilization Cisco WS-C4507R+E

Hello,


I want to disscuss here cisco WS-C4507R+E with (cat4500e-UNIVERSAL-M), Version 03.02.00.XO RELEASE SOFTWARE
CPU utilization, the output of #show processes cpu  illustrated below:

#show processes cpu
Core 0: CPU utilization for five seconds: 7%; one minute: 6%; five minutes: 6%
Core 1: CPU utilization for five seconds: 44%; one minute: 47%; five minutes: 47%

So for Core1 i have high utilization of cpu as i think.
and as shown in my output the most recursive process is: "app_printf.sh".

and the detailed process for it is shown down as:

C4507#show processes cpu detailed process app_printf.sh
Core 0: CPU utilization for five seconds: 7%; one minute: 6%; five minutes: 6%
Core 1: CPU utilization for five seconds: 45%; one minute: 47%; five minutes: 47%
PID     T C  TID     Runtime(ms) Invoked   uSecs  5Sec   1Min 5Min TTY   Process
                                                  (%)    (%)  (%)               
4999    L            22          750       30480  0.00 A 0.00 0.00 0     app_printf.sh      

5271    L            22          772       29253  0.00 A 0.00 0.00 0     app_printf.sh      

7975    L            23          753       31462  0.00 A 0.00 0.00 0     app_printf.sh      

7995    L            22          776       28864  0.00 A 0.00 0.00 0     app_printf.sh      

7998    L            24          750       32222  0.00 A 0.00 0.00 0     app_printf.sh      

8006    L            22          769       29573  0.00 A 0.00 0.00 0     app_printf.sh      

8021    L            22          757       29614  0.00 A 0.00 0.00 0     app_printf.sh      

8026    L            23          758       30348  0.00 A 0.00 0.00 0     app_printf.sh      

8032    L            22          782       29157  0.00 A 0.00 0.00 0     app_printf.sh      

8036    L            22          764       29340  0.00 A 0.00 0.00 0     app_printf.sh      

8039    L            22          769       29344  0.00 A 0.00 0.00 0     app_printf.sh      

8063    L            22          781       29247  0.00 A 0.00 0.00 0     app_printf.sh      

8065    L            22          766       29866  0.00 A 0.00 0.00 0     app_printf.sh      

8102    L            22          774       29502  0.00 A 0.00 0.00 0     app_printf.sh      

9430    L            23          795       29353  0.00 A 0.00 0.00 0     app_printf.sh      

9445    L            24          790       30793  0.00 A 0.00 0.00 0     app_printf.sh      

9470    L            23          796       29193  0.00 A 0.00 0.00 0     app_printf.sh      

9480    L            22          792       29005  0.00 A 0.00 0.00 0     app_printf.sh      

9482    L            21          792       27071  0.00 A 0.00 0.00 0     app_printf.sh      

10060   L            22          819       27316  0.00 A 0.00 0.00 0     app_printf.sh      

10515   L            25          825       30506  0.00 A 0.00 0.00 0     app_printf.sh      

10825   L            22          826       27225  0.00 A 0.00 0.00 0     app_printf.sh      

10927   L            25          866       29509  0.00 A 0.00 0.00 0     app_printf.sh      

11376   L            22          830       27071  0.00 A 0.00 0.00 0     app_printf.sh      

11389   L            24          833       29818  0.00 A 0.00 0.00 0     app_printf.sh      

11915   L            21          850       25441  0.00 A 0.00 0.00 0     app_printf.sh 

Do i have an issue with my CPU or my IOS? 
i am trying to figure out what is the cause of this utilization because it makes my monitoring sys by stg for some interfaces does not work properly (so i think it could effect my working snmp protocol on this 4500 cisco device too).

Appreciate any response

7 Replies 7

brselzer
Cisco Employee
Cisco Employee

Hello,

 

A couple things:

 

1) 3.2.0 is pretty old code at this point (2011), it would probably be worth looking into upgrading your code even if it is not the cause of your high cpu.

 

2) I see there are multiple app_printf.sh processes but they are all 0%. Is that the highest process? Can you post the output of "show proc cpu sorted"? You probably only need to post the top 10-15 lines. As you can see from this post, having multiple of these processes is expected:

 

https://community.cisco.com/t5/switching/high-cpu-utilization-in-cisco-switch-4507/td-p/2563435

 

You would want to look at your highest process and not the one with the most processes. 

 

3) 45% CPU shouldn't keep SNMP from working. You still have plenty of CPU remaining. Does it burst higher than 45% at times? Can you also post the output of "show proc cpu history" and "show snmp status"?

 

Hope that helps!

-Bradley Selzer
CCIE# 60833

Hi brselzer,

At first i want to thank you for your response sir,

i'm going to upgrade the software as you said even its not the cause of this issue. But after determine what exactly cause this problem with SNMP.

the output that you required fo showing proc cpu sorted:

 

C4507#show proc cpu sorted
Core 0: CPU utilization for five seconds: 7%; one minute: 7%; five minutes: 6%
Core 1: CPU utilization for five seconds: 40%; one minute: 44%; five minutes: 45%
PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process
10130  2264298     32258605 0      27.59960 29.30468 29.46875 0     iosd               
5382   2853910     29469277 6      0.217773 0.223633 0.222656 0     ffm                
10934  830755      58083057 31     0.038086 0.039062 0.039062 0     cpumemd            
5389   924780      52208972 26     0.002930 0.002930 0.002930 0     plogd              
5458   831431      52200767 47     0.002930 0.002930 0.002930 0     pdsd               
8204   741496      58089182 47     0.002930 0.002930 0.002930 0     ngdumper_provid    
8      446567      18108133 231    0.001953 0.001953 0.001953 0     events/1           
5332   558584      46437478 5      0.001953 0.001953 0.001953 0     ha_mgr             
5338   568565      46307552 35     0.001953 0.001953 0.001953 0     oscore_p           
5346   704972      52200559 11     0.001953 0.001953 0.001953 0     hwcontrol          
5350   574515      46306824 70     0.001953 0.001953 0.001953 0     ns_oir_proxy       
5352   576386      46309476 18     0.001953 0.001953 0.001953 0     sysmgr             
5356   562995      46310802 7      0.001953 0.001953 0.001953 0     profiled           
5366   595983      46407043 70     0.001953 0.001953 0.001953 0     os_info_p          
5372   648444      47113797 89     0.001953 0.001953 0.001953 0     eicored

obviously the process that i mentioned before "app_printf.sh" is not exist. So as you said may we don't have to consider it because are all are all 0%.

 

 

 

The output of proc cpu history for System, Core 0 and Core 1 until last 72 hours are as shown below:

 

C4507#show proc cpu history

History information for system:

                                                                
    222222222222222222222222222222222222222222222222222222222222
    755555888888888899999555557777744444666664444477777444447777
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30 *******************************     *****     *****     ****
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)

                                                                
    233323222322222332232222222322222232222222322332232223333322
    700080888078988109827879787076887809989879099007827985101199
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                      *      
 30 ##########################*##*##############################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                                                            
    334333333333333333444434444433333343444433333443444333334333444333433343
    670275621746445475404143232274995229341147648108201866951677032460477505
100                                                                         
 90                                                                         
 80                                                                         
 70                                                                         
 60                                                                         
 50                                                                         
 40 *** ***  * *  * ****** ****** *** ****** ** ******************* * ******
 30 ########**#####****#############***#####################**##############
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%


History information for core 0:

                                                                
                                                                
    777777888886666666666555554444455555555555555555555555556666
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20                                                             
 10 **************************     *****************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)
          
                                                                
     1 11       1 111111   11  111    11  1  11 111      111 11 
    618189957887380011108873198030899700992771483019877982339229
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20     *                                                       
 10 #######*##############*###*#################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                                                            
    311111112121221111112122112111111221112211111211111111111412111111111111
    879699760716146975770500961886894835660168668296687789367363979446994557
100                                                                         
 90                                                                         
 80                                                                         
 70                                                                         
 60                                                                         
 50                                                                         
 40 *                                                        *              
 30 *                                *                       *              
 20 ******************************** ********************* ********  *** ***
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%


History information for core 1:

                                                                
    444444444444444455555444445555544444444444444444444444444444
    833333888889999922222555550000044444888884444499999333339999
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50 *     *************************     *****     *****     ****
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)

                                                                
    555545555555555555455545444544554455555544554554454555555554
    004084111100100342960191888175008932212099219209979219536219
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                    *                             *   ** *   
 50 ####*#############*#####**##**#############**###############
 40 ############################################################
 30 ############################################################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                                                            
    566565655656655566767657676655565566776656656566666655656666666555666565
    936606144381078701261772613997917523037161272913867558588011688993904779
100                                                                         
 90                                                                         
 80                                                                         
 70   *               **** *** *        ***         ****  * *   ***   *   * 
 60 *******  ******************************************************** ******
 50 ####********###****#########********################***#***####*#######*
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

and for SNMP status is:

C4507#show snmp 
284938989 SNMP packets input
    0 Bad SNMP version errors
    422 Unknown community name
    0 Illegal operation for community name supplied
    0 Encoding errors
    303609980 Number of requested variables
    0 Number of altered variables
    281279330 Get-request PDUs
    3658646 Get-next PDUs
    0 Set-request PDUs
    0 Input queue packet drops (Maximum queue size 1000)
284938567 SNMP packets output
    0 Too big errors (Maximum packet size 1500)
    0 No such name errors
    0 Bad values errors
    0 General errors
    284938567 Response PDUs
    0 Trap PDUs
SNMP global trap: disabled
SNMP agent enabled

SNMP logging: disabled

I will check the URL that you posted too.
If you have any suggestions after seeing the outputs please inform me about it.

Appreciate your help Thanks again.

 

Hello,

 

From a CPU standpoint, the process that is high is IOSd:

 

10130  2264298     32258605 0      27.59960 29.30468 29.46875 0     iosd  

 

Can you post the output of "show processes cpu detailed process iosd"  and "show platform cpu packet statistics". However, I do not believe this is the reason SNMP doesn't work as your CPU is pretty stable. Does SNMP work sometimes and not other times or is it always broken? I do see some bad community counters:

 

422 Unknown community name

 

However, the counter is pretty low and if it is not increasing, it is probably not your problem either. You might want to validate your SNMP configuration matches. Can you also post the output of "Show redundancy"?

 

Hope that helps!

-Bradley Selzer
CCIE# 60833

Hello,

thanks for your help again the output of "show processes cpu detailed process iosd" shown bellow:

 

C4507#show processes cpu detailed process iosd
Core 0: CPU utilization for five seconds: 5%; one minute: 5%; five minutes: 6%
Core 1: CPU utilization for five seconds: 44%; one minute: 47%; five minutes: 47%
PID     T C  TID     Runtime(ms) Invoked   uSecs  5Sec   1Min 5Min TTY   Process
                                                  (%)    (%)  (%)               
10130   L            1806577     323507834 0      28.0 A 29.4 29.6 0     iosd               
10130   L 1  10130   1075337     424293859 0      45.8 A 48.6 48.7 0     iosd               
10130   L 0  11479   726810      328709177 0      10.1 A 10.3 10.4 0     iosd               
10130   L 0  11480   4428        15272     0      0.00 A 0.00 0.00 0     iosd

 

and "show platform cpu packet statistics":

C4507#show platform cpu packet statistics
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport  TxQueue 0       TxQueue 1       TxQueue 2       TxQueue 3
------------ --------------- --------------- --------------- ---------------
           1               0            8032               0               0


RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0

Packets Dropped In Processing Overall

Total                5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
          1068984494        97       108        88         82

Packets Dropped In Processing by CPU event

Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss                     1068319306        97       108        88         82
Input Acl Fwd                        9         0         0         0          0
Input ACl Copy                  665179         0         0         0          0

Packets Dropped In Processing by Priority

Priority          Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown                     1066081147        96       107        88         81
Normal                          662133         0         0         0          0
Medium                         2239560         0         0         0          0
High                               149         0         0         0          0
Crucial                           1505         0         0         0          0

Packets Dropped In Processing by Reason

Reason             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
NoL2Vlan                     1066081147        96       107        88         81
STPDrop                            1242         0         0         0          0
Tx Mode Drop                    2902105         0         0         0          0

Total packet queues 64

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping)        12407162         0         0         0          0
Host Learning               1068246303       100       110        89         81
L2 Control                  1354360556       129       138       112        104
Input ACL log, unreach          665179         0         0         0          0
L3 Receive                   295526372         0        36        26         16
L2 router to CPU, 7           97160121         7         2         2          3

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning                      436         0         0         0          0
L2 Control 

About SNMP: i monitor my interfaces by stg program then sometimes looks like is just completely gone and sometimes working but unstable, like it goes UP and Down.


the "show redundancy" output shown down:

C4507#Show redundancy
Redundant System Information :

------------------------------
       Available system uptime = 0 minute
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

                 Hardware Mode = Simplex
    Configured Redundancy Mode = Stateful Switchover
     Operating Redundancy Mode = Stateful Switchover
              Maintenance Mode = Disabled
                Communications = Down      Reason: Simplex mode

Current Processor Information :
------------------------------
               Active Location = slot 3
        Current Software state = ACTIVE
       Uptime in current state = 136 years, 10 weeks, 6 hours, 26 minutes
                 Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSAL-M), Version 03.02.00.XO RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 22-Sep-11 19:53 by prod_r
        Configuration register = 0x2101

Peer (slot: 4) information is not available because it is in 'DISABLED' state

But i just want your advice too about which IOS it could be good for my situation to do upgrading process?

Appreciate your help...


 

Hello, 

 

You are punting some packets to your CPU for different reasons:

 

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping)        12407162         0         0         0          0
Host Learning               1068246303       100       110        89         81
L2 Control                  1354360556       129       138       112        104
Input ACL log, unreach          665179         0         0         0          0
L3 Receive                   295526372         0        36        26         16
L2 router to CPU, 7           97160121         7         2         2          3


 

 

This is probably expected behavior and 45% cpu really isn't that bad depending on scale of your network. 


For your SNMP issue, it would be tough to say without more in depth troubleshooting. It might be a connectivity issue in your network and not the 4500 specifically. 

 

I think 3.8.6 is the current recommended version and is a stable code version.

 

Hope that helps!

-Bradley Selzer
CCIE# 60833

 

Hello,

Ok then, at first i will run my IOS upgrading process. if the issue with snmp is still then i will deeply troubleshoot snmp issue with considering connectivity problem probability.

I will post the final results here.

Thanks for your efforts, it was helpful.

Good day.
 

Read the Release Notes carefully.
There are certain cases where you need to perform a ROMmon upgrade first before you can upgrade the IOS.