07-26-2015 08:14 AM - last edited on 03-25-2019 04:35 PM by ciscomoderator
Hi all, need some help. Notice very high CPU utilisazation on 4500-x with VSS.
Notice two processes with the Show Platform Health command that was very high
K5Acl Input Action U 2.00 27.45 15 47 100 500 26 28 22 362:05
K5Acl Output Action 2.00 28.31 15 48 100 500 25 27 21 362:38
here some command that a i made.
show processes cpu sorted | ex 0.00
Core 0: CPU utilization for five seconds: 91%; one minute: 89%; five minutes: 89%
Core 1: CPU utilization for five seconds: 92%; one minute: 92%; five minutes: 93%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4659 1526014 43077493 422 68.85 68.20 68.28 0 iosd
4614 14570 27041511 635 22.78 21.96 22.05 0 ffm
4606 737721 37677752 19 0.28 0.41 0.38 0 pdsd
4595 400371 7210280 55 0.12 0.18 0.17 0 pdsvslifshim
21 81735 3155100 25 0.08 0.03 0.04 0 sirq-net-rx/1
4342 122678 1630942 75 0.04 0.02 0.01 0 system_mgr
4600 37517 77997 481 0.04 0.03 0.01 0 oom_poll.sh
4617 517149 1715848 301 0.04 0.02 0.02 0 cpumemd
show processes cpu detailed process iosd sorted | excl 0.00
Core 0: CPU utilization for five seconds: 86%; one minute: 90%; five minutes: 90%
Core 1: CPU utilization for five seconds: 95%; one minute: 94%; five minutes: 93%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
4659 L 591895 4366054 424 70.14 68.41 68.51 0 iosd
4659 L 1 4659 4014113 2527249 0 44.99 43.65 44.23 0 iosd
4659 L 0 6962 694180 1819274 0 24.99 24.61 24.14 0 iosd.fastpath
4659 L 0 6963 162427 1065253 0 0.16 0.14 0.13 0 CMI Thread
91 I 3069668 1197969 0 27.00 22.44 22.99 0 Cat4k Mgmt HiPri
183 I 3280780 2502126 0 18.66 20.66 20.77 0 IP ARP Adjacency
30 I 2324660 6996563 0 16.77 17.66 17.88 0 ARP Background
29 I 221184 6708043 0 8.77 8.00 8.11 0 ARP Input
185 I 211972 1446270 0 5.88 3.77 4.00 0 IP Input
21 I 2375836 1500127 0 5.44 5.33 5.33 0 CMI IOSd task
197 I 956712 796215 0 1.77 2.44 2.55 0 XDR mcast
92 I 1407722 1965092 0 1.66 1.99 1.99 0 Cat4k Mgmt LoPri
101 I 194932 990826 0 0.99 0.44 0.44 0 ARP HA
208 I 2747868 1198515 0 0.77 0.66 0.66 0 ADJ background
89 I 202592 1876129 0 0.66 0.33 0.33 0 cpf_process_tpQ
134 I 520264 1181414 0 0.44 0.44 0.44 0 Ethernet Msec Timer
194 I 805736 9545728 0 0.44 0.55 0.55 0 Spanning Tree
272 I 310316 1436951 0 0.22 0.11 0.11 0 DHCPD Receive
68 I 2154400 817063 0 0.11 0.11 0.11 0 IDB Work
148 I 212624 8170015 0 0.11 0.11 0.11 0 UDLD
show platform health | ex 0.00
Executing the command on VSS member switch role = VSS Standby, id = 2
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
VSI slot-01 6.00 0.07 6 39 100 500 0 0 0 39:33
VSI backplane 6.00 0.01 6 14 100 500 0 0 0 3:19
GalChassisVp 3.00 0.09 20 17 100 500 0 0 0 12:12
Lj-poll 1.00 0.01 2 19 100 500 0 0 0 3:07
StatValueMan Update 1.00 0.02 1 31 100 500 0 0 0 8:02
Ebm-host-review 1.00 0.44 8 23 100 500 0 0 0 3:44
EbmHostRedundancyMan 2.00 0.09 20 23 100 500 0 0 0 3:57
GalK5TatooineStatsMa 0.70 0.03 4 12 100 500 0 0 0 4:33
K5L3Unicast Adj Chan 2.00 0.49 15 29 100 500 0 0 0 6:57
K5L3Unicast Adj Tabl 2.00 0.98 15 28 100 500 0 0 0 13:05
K5FlcHitMan review 2.00 0.03 5 41 100 500 0 0 0 29:33
K5PortMan Regular Re 2.00 0.29 15 48 100 500 0 0 0 28:43
K5PortMan Ondemand L 6.00 0.40 30 32 100 500 0 0 0 37:55
K5PortMan Stats Revi 2.00 0.01 15 24 100 500 0 0 0 6:01
K5 L2 Aging Table Re 2.00 0.20 20 35 100 500 0 0 0 24:37
K5 L2 Hardware Addre 2.00 0.63 20 27 100 500 0 0 0 7:51
K5RetStatsMan Review 2.00 0.02 5 22 100 500 0 0 0 26:31
K5ForerunnerPacketMa 2.00 1.59 4 37 100 500 1 1 1 61:23
K5ForerunnerPacketMa 2.00 0.22 4 18 100 500 0 0 0 29:39
K5QosDhmMan Rate DBL 2.00 0.34 7 29 100 500 0 0 0 42:07
K5VlanStatsReview 2.00 1.08 10 31 100 500 1 1 0 129:02
K5RwAdjStatsMan Revi 2.00 0.08 10 35 100 500 0 0 0 22:29
K5Acl Input Action U 2.00 27.45 15 47 100 500 26 28 22 362:05
K5Acl Output Action 2.00 28.31 15 48 100 500 25 27 21 362:38
RkiosPortMan Port Re 2.00 0.22 12 24 100 500 0 0 0 18:39
Rkios Module State R 4.00 0.03 40 7 100 500 0 0 0 3:34
Rkios Online Diag Re 4.00 0.03 40 13 100 500 0 0 0 3:17
RkNovaL3Fib L3If Rev 2.00 0.03 10 12 100 500 0 0 0 3:22
PDS Rx Review 2.00 0.95 10 36 100 500 0 1 0 23:41
GalGlmLinecardVp(1) 5.00 0.36 20 1527 100 500 0 0 0 34:38
Temperature monitor 0.40 0.02 4 10 100 500 0 0 0 1:57
GalGlmPollerMan 3.00 0.90 20 28 100 500 0 0 0 79:37
Stub ondemand global 0.50 4.38 5 346 100 500 5 5 4 611:13
Xgstub Stats Review 0.50 1.06 5 42 100 500 1 1 0 125:32
Stub CTS Periodic Re 0.80 0.05 8 16 100 500 0 0 0 6:39
Stub CTS PeriodicSta 0.80 0.04 8 19 100 500 0 0 0 11:23
Stub CTS Periodic Re 0.80 0.04 8 15 100 500 0 0 0 6:49
Stub CTS PeriodicSta 0.80 0.04 8 24 100 500 0 0 0 11:02
Stub CTS Periodic Re 0.80 0.05 8 17 100 500 0 0 0 6:51
Stub CTS PeriodicSta 0.80 0.04 8 26 100 500 0 0 0 10:45
Stub CTS Periodic Re 0.80 0.05 8 17 100 500 0 0 0 6:40
Stub CTS PeriodicSta 0.80 0.04 8 33 100 500 0 0 0 11:13
EpmPortGroup(10:N) s 0.50 0.07 2 23 100 500 0 0 0 7:27
-------------
%CPU Totals 284.90 89.27
07-27-2015 02:09 AM
Hi
There is a bug on ios-xe for that process , your most likely hitting that , i would upgrade or open a TAC case get the specific bug id or check the release notes online against your version see if there's a matching open caveat
IOSd: This is the Cisco IOS daemon that runs on the Linux kernel. It is run as a software process within the kernel.
07-27-2015 05:06 AM
We open a TAC and find out that was coming from one vlan. After a "Shut" "no Shut", CPU drop down from 90% to 14%. The engineer told me that was a bug as you mention, but we don't know how we trigger it.
07-27-2015 05:16 AM
get the bug id off him and ask him what release its fixed in then upgrade
08-26-2015 05:40 AM
Anyone have the bug ID for this? I am running into the same issue but we are running the latest code.
08-26-2015 06:05 AM
TAC was unable to find reference to this internal bug and close the case. If it can help, for us we think that a subnet mask modification when the vlan was up trigger the problem. The vlan was set 255.255.255.240, but it was supposed to be 255.255.240.0. After that, problem begin. After we shut and no shut the vlan in question, everything come back to normal.
If you execute commands from my first post, what the result ?
08-26-2015 01:21 PM
Our issue turned out to be a bug. We downgraded the code and CPU normalized.
11-13-2015 06:34 AM
Do you have a bug ID for this ?
11-14-2016 05:49 AM
I know this post is a year old but I believe we are seeing this bug. on Release 3.7.3E. Can you tell me what version of code you used or are currently using.
11-14-2016 07:51 AM
Hi
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.03.02.SG RELEASE SOFTWARE (fc1)
11-14-2016 12:11 PM
So on the (cat4500e-UNIVERSALK9-M), Version 03.03.02.SG RELEASE SOFTWARE (fc1) you are not seeing the CPU issue correct? Just making sure before we give it a try.
11-15-2016 01:51 AM
that is correct, current stats ( taken at the start of the business day)
Core 0: CPU utilization for five seconds: 2%; one minute: 2%; five minutes: 2%
Core 1: CPU utilization for five seconds: 14%; one minute: 12%; five minutes: 12%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
8450 1787449 26973071 314 7.99 7.26 7.23 0 iosd
11781 3946781 79932243 96 0.20 0.19 0.20 0 ffm
5930 1241269 12578979 9 0.04 0.01 0.02 0 os_info_p
5934 1234568 12663252 9 0.04 0.10 0.17 0 eicored
5940 2098618 14159522 14 0.04 0.01 0.02 0 plogd
10750 2087421 12623037 16 0.04 0.00 0.00 0 licensed
11242 1553216 15748434 146 0.04 0.04 0.04 0 cpumemd
08-26-2015 01:21 PM
Our issue turned out to be a bug. We downgraded the code and CPU normalized.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide