- Cisco Community
- Technology and Support
- Networking
- Switching
- High CPU Utilization Cisco WS-C4507R+E
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
High CPU Utilization Cisco WS-C4507R+E
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2018 11:45 PM - edited 03-08-2019 04:36 PM
Hello,
I want to disscuss here cisco WS-C4507R+E with (cat4500e-UNIVERSAL-M), Version 03.02.00.XO RELEASE SOFTWARE
CPU utilization, the output of #show processes cpu illustrated below:
#show processes cpu Core 0: CPU utilization for five seconds: 7%; one minute: 6%; five minutes: 6% Core 1: CPU utilization for five seconds: 44%; one minute: 47%; five minutes: 47%
So for Core1 i have high utilization of cpu as i think.
and as shown in my output the most recursive process is: "app_printf.sh".
and the detailed process for it is shown down as:
C4507#show processes cpu detailed process app_printf.sh
Core 0: CPU utilization for five seconds: 7%; one minute: 6%; five minutes: 6%
Core 1: CPU utilization for five seconds: 45%; one minute: 47%; five minutes: 47%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
4999 L 22 750 30480 0.00 A 0.00 0.00 0 app_printf.sh
5271 L 22 772 29253 0.00 A 0.00 0.00 0 app_printf.sh
7975 L 23 753 31462 0.00 A 0.00 0.00 0 app_printf.sh
7995 L 22 776 28864 0.00 A 0.00 0.00 0 app_printf.sh
7998 L 24 750 32222 0.00 A 0.00 0.00 0 app_printf.sh
8006 L 22 769 29573 0.00 A 0.00 0.00 0 app_printf.sh
8021 L 22 757 29614 0.00 A 0.00 0.00 0 app_printf.sh
8026 L 23 758 30348 0.00 A 0.00 0.00 0 app_printf.sh
8032 L 22 782 29157 0.00 A 0.00 0.00 0 app_printf.sh
8036 L 22 764 29340 0.00 A 0.00 0.00 0 app_printf.sh
8039 L 22 769 29344 0.00 A 0.00 0.00 0 app_printf.sh
8063 L 22 781 29247 0.00 A 0.00 0.00 0 app_printf.sh
8065 L 22 766 29866 0.00 A 0.00 0.00 0 app_printf.sh
8102 L 22 774 29502 0.00 A 0.00 0.00 0 app_printf.sh
9430 L 23 795 29353 0.00 A 0.00 0.00 0 app_printf.sh
9445 L 24 790 30793 0.00 A 0.00 0.00 0 app_printf.sh
9470 L 23 796 29193 0.00 A 0.00 0.00 0 app_printf.sh
9480 L 22 792 29005 0.00 A 0.00 0.00 0 app_printf.sh
9482 L 21 792 27071 0.00 A 0.00 0.00 0 app_printf.sh
10060 L 22 819 27316 0.00 A 0.00 0.00 0 app_printf.sh
10515 L 25 825 30506 0.00 A 0.00 0.00 0 app_printf.sh
10825 L 22 826 27225 0.00 A 0.00 0.00 0 app_printf.sh
10927 L 25 866 29509 0.00 A 0.00 0.00 0 app_printf.sh
11376 L 22 830 27071 0.00 A 0.00 0.00 0 app_printf.sh
11389 L 24 833 29818 0.00 A 0.00 0.00 0 app_printf.sh
11915 L 21 850 25441 0.00 A 0.00 0.00 0 app_printf.sh
Do i have an issue with my CPU or my IOS?
i am trying to figure out what is the cause of this utilization because it makes my monitoring sys by stg for some interfaces does not work properly (so i think it could effect my working snmp protocol on this 4500 cisco device too).
Appreciate any response
- Labels:
-
LAN Switching
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2018 12:03 AM
Hello,
A couple things:
1) 3.2.0 is pretty old code at this point (2011), it would probably be worth looking into upgrading your code even if it is not the cause of your high cpu.
2) I see there are multiple app_printf.sh processes but they are all 0%. Is that the highest process? Can you post the output of "show proc cpu sorted"? You probably only need to post the top 10-15 lines. As you can see from this post, having multiple of these processes is expected:
https://community.cisco.com/t5/switching/high-cpu-utilization-in-cisco-switch-4507/td-p/2563435
You would want to look at your highest process and not the one with the most processes.
3) 45% CPU shouldn't keep SNMP from working. You still have plenty of CPU remaining. Does it burst higher than 45% at times? Can you also post the output of "show proc cpu history" and "show snmp status"?
Hope that helps!
CCIE# 60833
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2018 12:50 AM
Hi brselzer,
At first i want to thank you for your response sir,
i'm going to upgrade the software as you said even its not the cause of this issue. But after determine what exactly cause this problem with SNMP.
the output that you required fo showing proc cpu sorted:
C4507#show proc cpu sorted Core 0: CPU utilization for five seconds: 7%; one minute: 7%; five minutes: 6% Core 1: CPU utilization for five seconds: 40%; one minute: 44%; five minutes: 45% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 10130 2264298 32258605 0 27.59960 29.30468 29.46875 0 iosd 5382 2853910 29469277 6 0.217773 0.223633 0.222656 0 ffm 10934 830755 58083057 31 0.038086 0.039062 0.039062 0 cpumemd 5389 924780 52208972 26 0.002930 0.002930 0.002930 0 plogd 5458 831431 52200767 47 0.002930 0.002930 0.002930 0 pdsd 8204 741496 58089182 47 0.002930 0.002930 0.002930 0 ngdumper_provid 8 446567 18108133 231 0.001953 0.001953 0.001953 0 events/1 5332 558584 46437478 5 0.001953 0.001953 0.001953 0 ha_mgr 5338 568565 46307552 35 0.001953 0.001953 0.001953 0 oscore_p 5346 704972 52200559 11 0.001953 0.001953 0.001953 0 hwcontrol 5350 574515 46306824 70 0.001953 0.001953 0.001953 0 ns_oir_proxy 5352 576386 46309476 18 0.001953 0.001953 0.001953 0 sysmgr 5356 562995 46310802 7 0.001953 0.001953 0.001953 0 profiled 5366 595983 46407043 70 0.001953 0.001953 0.001953 0 os_info_p 5372 648444 47113797 89 0.001953 0.001953 0.001953 0 eicored
obviously the process that i mentioned before "app_printf.sh" is not exist. So as you said may we don't have to consider it because are all are all 0%.
The output of proc cpu history for System, Core 0 and Core 1 until last 72 hours are as shown below:
C4507#show proc cpu history
History information for system:
222222222222222222222222222222222222222222222222222222222222
755555888888888899999555557777744444666664444477777444447777
100
90
80
70
60
50
40
30 ******************************* ***** ***** ****
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
233323222322222332232222222322222232222222322332232223333322
700080888078988109827879787076887809989879099007827985101199
100
90
80
70
60
50
40 *
30 ##########################*##*##############################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
334333333333333333444434444433333343444433333443444333334333444333433343
670275621746445475404143232274995229341147648108201866951677032460477505
100
90
80
70
60
50
40 *** *** * * * ****** ****** *** ****** ** ******************* * ******
30 ########**#####****#############***#####################**##############
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
History information for core 0:
777777888886666666666555554444455555555555555555555555556666
100
90
80
70
60
50
40
30
20
10 ************************** *****************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1 11 1 111111 11 111 11 1 11 111 111 11
618189957887380011108873198030899700992771483019877982339229
100
90
80
70
60
50
40
30
20 *
10 #######*##############*###*#################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
311111112121221111112122112111111221112211111211111111111412111111111111
879699760716146975770500961886894835660168668296687789367363979446994557
100
90
80
70
60
50
40 * *
30 * * *
20 ******************************** ********************* ******** *** ***
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
History information for core 1:
444444444444444455555444445555544444444444444444444444444444
833333888889999922222555550000044444888884444499999333339999
100
90
80
70
60
50 * ************************* ***** ***** ****
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
555545555555555555455545444544554455555544554554454555555554
004084111100100342960191888175008932212099219209979219536219
100
90
80
70
60 * * ** *
50 ####*#############*#####**##**#############**###############
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
566565655656655566767657676655565566776656656566666655656666666555666565
936606144381078701261772613997917523037161272913867558588011688993904779
100
90
80
70 * **** *** * *** **** * * *** * *
60 ******* ******************************************************** ******
50 ####********###****#########********################***#***####*#######*
40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
and for SNMP status is:
C4507#show snmp
284938989 SNMP packets input
0 Bad SNMP version errors
422 Unknown community name
0 Illegal operation for community name supplied
0 Encoding errors
303609980 Number of requested variables
0 Number of altered variables
281279330 Get-request PDUs
3658646 Get-next PDUs
0 Set-request PDUs
0 Input queue packet drops (Maximum queue size 1000)
284938567 SNMP packets output
0 Too big errors (Maximum packet size 1500)
0 No such name errors
0 Bad values errors
0 General errors
284938567 Response PDUs
0 Trap PDUs
SNMP global trap: disabled
SNMP agent enabled
SNMP logging: disabled
I will check the URL that you posted too.
If you have any suggestions after seeing the outputs please inform me about it.
Appreciate your help Thanks again.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2018 01:07 AM
Hello,
From a CPU standpoint, the process that is high is IOSd:
10130 2264298 32258605 0 27.59960 29.30468 29.46875 0 iosd
Can you post the output of "show processes cpu detailed process iosd" and "show platform cpu packet statistics". However, I do not believe this is the reason SNMP doesn't work as your CPU is pretty stable. Does SNMP work sometimes and not other times or is it always broken? I do see some bad community counters:
422 Unknown community name
However, the counter is pretty low and if it is not increasing, it is probably not your problem either. You might want to validate your SNMP configuration matches. Can you also post the output of "Show redundancy"?
Hope that helps!
CCIE# 60833
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2018 02:33 AM
Hello,
thanks for your help again the output of "show processes cpu detailed process iosd" shown bellow:
C4507#show processes cpu detailed process iosd
Core 0: CPU utilization for five seconds: 5%; one minute: 5%; five minutes: 6%
Core 1: CPU utilization for five seconds: 44%; one minute: 47%; five minutes: 47%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
10130 L 1806577 323507834 0 28.0 A 29.4 29.6 0 iosd
10130 L 1 10130 1075337 424293859 0 45.8 A 48.6 48.7 0 iosd
10130 L 0 11479 726810 328709177 0 10.1 A 10.3 10.4 0 iosd
10130 L 0 11480 4428 15272 0 0.00 A 0.00 0.00 0 iosd
and "show platform cpu packet statistics":
C4507#show platform cpu packet statistics
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)
CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3
------------ --------------- --------------- --------------- ---------------
1 0 8032 0 0
RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
1068984494 97 108 88 82
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss 1068319306 97 108 88 82
Input Acl Fwd 9 0 0 0 0
Input ACl Copy 665179 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown 1066081147 96 107 88 81
Normal 662133 0 0 0 0
Medium 2239560 0 0 0 0
High 149 0 0 0 0
Crucial 1505 0 0 0 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
NoL2Vlan 1066081147 96 107 88 81
STPDrop 1242 0 0 0 0
Tx Mode Drop 2902105 0 0 0 0
Total packet queues 64
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping) 12407162 0 0 0 0
Host Learning 1068246303 100 110 89 81
L2 Control 1354360556 129 138 112 104
Input ACL log, unreach 665179 0 0 0 0
L3 Receive 295526372 0 36 26 16
L2 router to CPU, 7 97160121 7 2 2 3
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning 436 0 0 0 0
L2 Control
About SNMP: i monitor my interfaces by stg program then sometimes looks like is just completely gone and sometimes working but unstable, like it goes UP and Down.
the "show redundancy" output shown down:
C4507#Show redundancy
Redundant System Information :
------------------------------
Available system uptime = 0 minute
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Simplex
Configured Redundancy Mode = Stateful Switchover
Operating Redundancy Mode = Stateful Switchover
Maintenance Mode = Disabled
Communications = Down Reason: Simplex mode
Current Processor Information :
------------------------------
Active Location = slot 3
Current Software state = ACTIVE
Uptime in current state = 136 years, 10 weeks, 6 hours, 26 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSAL-M), Version 03.02.00.XO RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 22-Sep-11 19:53 by prod_r
Configuration register = 0x2101
Peer (slot: 4) information is not available because it is in 'DISABLED' state
But i just want your advice too about which IOS it could be good for my situation to do upgrading process?
Appreciate your help...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2018 02:41 AM
Hello,
You are punting some packets to your CPU for different reasons:
Packets Received by Packet Queue Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg ---------------------- --------------- --------- --------- --------- ---------- Input ACL fwd(snooping) 12407162 0 0 0 0 Host Learning 1068246303 100 110 89 81 L2 Control 1354360556 129 138 112 104 Input ACL log, unreach 665179 0 0 0 0 L3 Receive 295526372 0 36 26 16 L2 router to CPU, 7 97160121 7 2 2 3
This is probably expected behavior and 45% cpu really isn't that bad depending on scale of your network.
For your SNMP issue, it would be tough to say without more in depth troubleshooting. It might be a connectivity issue in your network and not the 4500 specifically.
I think 3.8.6 is the current recommended version and is a stable code version.
Hope that helps!
CCIE# 60833
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2018 02:51 AM
Ok then, at first i will run my IOS upgrading process. if the issue with snmp is still then i will deeply troubleshoot snmp issue with considering connectivity problem probability.
I will post the final results here.
Thanks for your efforts, it was helpful.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-16-2018 02:21 AM
There are certain cases where you need to perform a ROMmon upgrade first before you can upgrade the IOS.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide