- Cisco Community
- Technology and Support
- Networking
- Switching
- Re: Cisco 3850 high cpu due to "iosd" process.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Cisco 3850 high cpu due to "iosd" process.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-13-2018 08:40 AM - edited 03-08-2019 03:53 PM
Hi Guys,
We are facing high cpu utilization issue for Cisco 3850 standalone switch due to "iosd" process.
Core 0: CPU utilization for five seconds: 98%; one minute: 74%; five minutes: 79%
Core 1: CPU utilization for five seconds: 2%; one minute: 12%; five minutes: 8%
Core 2: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 3%
Core 3: CPU utilization for five seconds: 3%; one minute: 6%; five minutes: 3%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
8900 969057 42130219 7607 24.47 21.65 21.58 0 iosd
6036 79740 23637088 74 0.93 1.18 1.20 1088 fed
#show processes cpu sorted | ex 0.00 his
DEL-DC-INCSW-01#show processes cpu history
History information for system:
222222222222222222222222222222222222222222222222222222222222
611111111113333366666222221111122222666661111122222111117777
100
90
80
70
60
50
40
30 * ***** ***** ****
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
222222222222222222222222222522222222222222222222222222222222
776555466677767666565566777277797664445556677666666556666677
--More-- 100
--More-- 90
--More-- 80
--More-- 70
--More-- 60
--More-- 50 *
--More-- 40 *
--More-- 30 ****** ********************#******* **********************
--More-- 20 ############################################################
--More-- 10 ############################################################
--More-- 0....5....1....1....2....2....3....3....4....4....5....5....
--More-- 0 5 0 5 0 5 0 5 0 5
--More-- CPU% per minute (last 60 minutes)
--More-- * = maximum CPU% # = average CPU%
--More--
--More--
--More-- 242323242222332323223322222322352222222322222332222322252222322322322222
--More-- 887193888888218483881179897077119998888177788227777188818878179578678977
--More-- 100
--More-- 90
--More-- 80
--More-- 70
--More-- 60
--More-- 50 * * * *
--More-- 40 * * * * * *
--More-- 30 ************************************************************************
--More-- 20 ########################################################################
--More-- 10 ########################################################################
--More-- 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
--More-- 0 5 0 5 0 5 0 5 0 5 0 5 0
--More-- CPU% per hour (last 72 hours)
--More-- * = maximum CPU% # = average CPU%
--More--
--More--
--More-- History information for core 0:
--More--
--More-- 1 11111 1111
--More-- 088888888888888800000888888888877777111118888888888888880000
--More-- 022222111115555500000111111111133333888881111133333222220000
--More-- 100 * ***** ****
--More-- 90 * ********** ****
--More-- 80 ******************************* *******************
--More-- 70 ************************************ *******************
--More-- 60 ************************************ *******************
--More-- 50 ************************************ *******************
--More-- 40 ************************************ *******************
--More-- 30 ************************************ *******************
--More-- 20 ************************************************************
--More-- 10 ************************************************************
--More-- 0....5....1....1....2....2....3....3....4....4....5....5....
--More-- 0 5 0 5 0 5 0 5 0 5
--More-- CPU% per second (last 60 seconds)
--More--
--More-- 11 1111111111 11111111 111111111 11111
--More-- 009999900000000009999999900000000999999990000000009999900000
--More-- 006855500000000009886447700000000663235670000000006436900000
--More-- 100 ********************* ************ ************* *******
--More-- 90 ***#**#***#####******#####**###***####**#**######*##****#***
80 *####*#########*##########**###*##########*#################
70 ############################################################
60 ############################################################
50 ############################################################
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
111111111111111111111111111111111111111111111111111111111111111111111111
000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000
100 ************************************************************************
90 ************************************************************************
80 ########################################################################
70 ########################################################################
60 ########################################################################
50 ########################################################################
--More-- 40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
History information for core 1:
1111188888
122222333333333322222222222222222222333332222233333444442222
100
90
80 *****
70 *****
60 *****
50 *****
40 *****
30 *****
--More-- 20 *****
10 **********
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1
88 88 506 6 77 19 8 6 69 47
327693440856646547555467657264818745555555465567775654845556
100 * * *
90 * * * * *
80 ** ** * * * * * *
70 ** ** ** * ** * * * * *
60 ** ** ** * ** * * * ** *
50 ** ** *** * ** * * * ** **
40 ** ** *** * ** * * * ** **
30 ** ** *** * ** * * * ** **
20 ** ** *** * ** * * * ** **
10 ##**## ###*** ** *#** ****##* **#* ****#*# **********## *##*
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
--More--
1111 1111 111 111111111 11 111111111111 11 111 1111 111 11 11 1 11 11
800008000090009000000000900900000000000090099900090000900090090090900900
500009000090009000000000900800000000000090090700050000900070040040800300
100 **** ************************************** **************** ** **** **
90 ************************************************************************
80 ************************************************************************
70 ************************************************************************
60 ************************************************************************
50 ************************************************************************
40 ************************************************************************
30 ************************************************************************
20 ************************************************************************
10 ##*##*##*#########*#**##*###*################*###**##*##########*######*
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
History information for core 2:
--More--
1 2222211111 11111 22222 111112222
100
90
80
70
60
50
40
30
20
10
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1 1
62 9 1 055 6 1170 9 8 7 8 5
823481642440645017614432036033492864225444833343214392453474
100 * * *
90 * * * * *
80 * * ** * * * *
70 * * * ** * * * *
--More-- 60 * * * * * ** * * * * *
50 * * *** * ** * * * * *
40 * * *** * ** * * * * *
30 * * *** * ** * * * * *
20 ** * #** * *# * * # * *
10 #* *#* ** *###*# **## # #* * # # # * *
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1 111 11 111111 1 111 1111 11 11 1 111 1111111 1111 1 1111 1
090009900900000099809000999990000900999009099000900000009000088099000090
090009600600000094106000899890000800634007099000000000008000085034000020
100 ***************** ****************** ********* ************ * **** *
90 ****************** *****************************************************
80 ************************************************************************
70 ************************************************************************
60 ************************************************************************
50 ************************************************************************
40 ************************************************************************
30 ************************************************************************
20 ************************************************************************
--More-- 10 ************************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
History information for core 3:
333333222223333311111555553333333333111112222222222 6666
100
90
80
70
60
50
40
30
20
10 ***** ****
0....5....1....1....2....2....3....3....4....4....5....5....
--More-- 0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1 1 1
9 9 2 0 6 0 0
666241232132423562423363735946305332450614579666255667403320
100 * * * *
90 * * * * *
80 * * * * *
70 * * * * *
60 * * * * * *
50 * * * * * *
40 * * * * * *
30 * * * * * * *
20 * * * * * * *
10 **# ** # * * ** * #* *#* ****** ***** # #
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
111 1111 1 1111 11 111 11 1 1 1111 111 1 111 11 11111111 11 11 1 111
000800009090000900900090090909000090009909800090099800000000900900909000
--More-- 000400004030000900500090040409000090006809300080070100000000900800909000
100 *** **** * ************** * ************** ******* ********************
90 *** ************************************** ******** ********************
80 ************************************************************************
70 ************************************************************************
60 ************************************************************************
50 ************************************************************************
40 ************************************************************************
30 ************************************************************************
20 ************************************************************************
10 ************************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
============================================
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 56 WS-C3850-48T 03.06.05E cat3k_caa-universalk9 INSTALL
================================================
Kindly let me know how can we fix it.
Regards,
Vijay
- Labels:
-
LAN Switching
-
Other Switches
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-13-2018 09:18 AM
Hello Vinay,
I hope you´re doing well.
Could you please provide the output of the following commands?
show processes cpu detailed process iosd sort | ex 0.0
show processes cpu detailed process fed sorted | ex 0.0
Regards,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-16-2018 05:47 AM
Hi andres,
We have not observed high cpu from last 2-3 days. I have attached one snapshot for the requested logs but I think we will not get any clue cs utilization is only 9%.
Regards,
Vijay
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-16-2018 07:37 AM
Hello Vinay,
Sorry for my delayed response.
Let's try to catch the event with a EEM script:
configure terminal
process cpu threshold type total rising 80 interval 90 switch 1
event manager applet HIGH_CPU
event syslog pattern "%CPUMEM-5-RISING_THRESHOLD" maxrun 300
action 1.0 syslog msg "High CPU detected"
action 2.0 cli command "enable"
action 2.1 cli command "terminal len 0"
action 2.5 cli command "show clock | append flash:CPUlog.txt"
action 3.0 cli command "show process cpu sorted | append flash:CPUlog.txt"
action 4.0 cli command "show processes cpu detailed process iosd sort | append flash:CPUlog.txt"
action 5.0 cli command "show processes cpu detailed process fed sort | append flash:CPUlog.txt"
action 6.0 cli command "show platform punt statistics port-asic 0 cpuq -1 direction rx | append flash:CPUlog.txt"
action 7.0 cli command "show platform punt statistics port-asic 1 cpuq -1 direction rx | append flash:CPUlog.txt"
action 8.0 cli command "show logging | append flash:CPUlog.txt"
action 8.5 cli command "terminal len 24"
action 9.0 syslog msg "HIGH_CPU EEM Script finished running"
action 9.5 cli command "end"
end
!
The previous will detect when the CPU goes over 80% during a certain period of time generating a log that will be used as the trigger of the EEM script.
Once the EEM script triggers it will generate the CPUlog.txt file in flash memory and will also show a message in the logs.
Please share the file for further analysis.
Regards,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-20-2018 10:42 PM
Hello Vinay,
Any updates regarding this particular issue? Did you try configuring the EEM script?
I look forward to hearing from you.
Regards,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-13-2018 03:28 PM
Is IPv6 present in the network? If yes, read THIS.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-16-2018 05:50 AM
Hi Leo,
There is no ipv6 in our environment and we have not observed high cpu from last 2-3 days.
REgards,
Vijay
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2020 03:36 AM
what's the root cause of the high utilization
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2020 03:54 AM
If the stack is running on 3.6.X or 3.7.X then it is most likely due to CSCuo14511 &/or CSCvc08452.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:
