I know this post is a year

jfk_web_cisco · ‎07-26-2015

Hi all, need some help. Notice very high CPU utilisazation on 4500-x with VSS.

Notice two processes with the Show Platform Health command that was very high

K5Acl Input Action U 2.00 27.45 15 47 100 500 26 28 22 362:05
K5Acl Output Action 2.00 28.31 15 48 100 500 25 27 21 362:38

here some command that a i made.

show processes cpu sorted | ex 0.00
Core 0: CPU utilization for five seconds: 91%; one minute: 89%; five minutes: 89%
Core 1: CPU utilization for five seconds: 92%; one minute: 92%; five minutes: 93%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4659 1526014 43077493 422 68.85 68.20 68.28 0 iosd
4614 14570 27041511 635 22.78 21.96 22.05 0 ffm
4606 737721 37677752 19 0.28 0.41 0.38 0 pdsd
4595 400371 7210280 55 0.12 0.18 0.17 0 pdsvslifshim
21 81735 3155100 25 0.08 0.03 0.04 0 sirq-net-rx/1
4342 122678 1630942 75 0.04 0.02 0.01 0 system_mgr
4600 37517 77997 481 0.04 0.03 0.01 0 oom_poll.sh
4617 517149 1715848 301 0.04 0.02 0.02 0 cpumemd

show processes cpu detailed process iosd sorted | excl 0.00
Core 0: CPU utilization for five seconds: 86%; one minute: 90%; five minutes: 90%
Core 1: CPU utilization for five seconds: 95%; one minute: 94%; five minutes: 93%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
4659 L 591895 4366054 424 70.14 68.41 68.51 0 iosd
4659 L 1 4659 4014113 2527249 0 44.99 43.65 44.23 0 iosd
4659 L 0 6962 694180 1819274 0 24.99 24.61 24.14 0 iosd.fastpath
4659 L 0 6963 162427 1065253 0 0.16 0.14 0.13 0 CMI Thread
91 I 3069668 1197969 0 27.00 22.44 22.99 0 Cat4k Mgmt HiPri
183 I 3280780 2502126 0 18.66 20.66 20.77 0 IP ARP Adjacency
30 I 2324660 6996563 0 16.77 17.66 17.88 0 ARP Background
29 I 221184 6708043 0 8.77 8.00 8.11 0 ARP Input
185 I 211972 1446270 0 5.88 3.77 4.00 0 IP Input
21 I 2375836 1500127 0 5.44 5.33 5.33 0 CMI IOSd task
197 I 956712 796215 0 1.77 2.44 2.55 0 XDR mcast
92 I 1407722 1965092 0 1.66 1.99 1.99 0 Cat4k Mgmt LoPri
101 I 194932 990826 0 0.99 0.44 0.44 0 ARP HA
208 I 2747868 1198515 0 0.77 0.66 0.66 0 ADJ background
89 I 202592 1876129 0 0.66 0.33 0.33 0 cpf_process_tpQ
134 I 520264 1181414 0 0.44 0.44 0.44 0 Ethernet Msec Timer
194 I 805736 9545728 0 0.44 0.55 0.55 0 Spanning Tree
272 I 310316 1436951 0 0.22 0.11 0.11 0 DHCPD Receive
68 I 2154400 817063 0 0.11 0.11 0.11 0 IDB Work
148 I 212624 8170015 0 0.11 0.11 0.11 0 UDLD

show platform health | ex 0.00

Executing the command on VSS member switch role = VSS Standby, id = 2

%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
VSI slot-01 6.00 0.07 6 39 100 500 0 0 0 39:33
VSI backplane 6.00 0.01 6 14 100 500 0 0 0 3:19
GalChassisVp 3.00 0.09 20 17 100 500 0 0 0 12:12
Lj-poll 1.00 0.01 2 19 100 500 0 0 0 3:07
StatValueMan Update 1.00 0.02 1 31 100 500 0 0 0 8:02
Ebm-host-review 1.00 0.44 8 23 100 500 0 0 0 3:44
EbmHostRedundancyMan 2.00 0.09 20 23 100 500 0 0 0 3:57
GalK5TatooineStatsMa 0.70 0.03 4 12 100 500 0 0 0 4:33
K5L3Unicast Adj Chan 2.00 0.49 15 29 100 500 0 0 0 6:57
K5L3Unicast Adj Tabl 2.00 0.98 15 28 100 500 0 0 0 13:05
K5FlcHitMan review 2.00 0.03 5 41 100 500 0 0 0 29:33
K5PortMan Regular Re 2.00 0.29 15 48 100 500 0 0 0 28:43
K5PortMan Ondemand L 6.00 0.40 30 32 100 500 0 0 0 37:55
K5PortMan Stats Revi 2.00 0.01 15 24 100 500 0 0 0 6:01
K5 L2 Aging Table Re 2.00 0.20 20 35 100 500 0 0 0 24:37
K5 L2 Hardware Addre 2.00 0.63 20 27 100 500 0 0 0 7:51
K5RetStatsMan Review 2.00 0.02 5 22 100 500 0 0 0 26:31
K5ForerunnerPacketMa 2.00 1.59 4 37 100 500 1 1 1 61:23
K5ForerunnerPacketMa 2.00 0.22 4 18 100 500 0 0 0 29:39
K5QosDhmMan Rate DBL 2.00 0.34 7 29 100 500 0 0 0 42:07
K5VlanStatsReview 2.00 1.08 10 31 100 500 1 1 0 129:02
K5RwAdjStatsMan Revi 2.00 0.08 10 35 100 500 0 0 0 22:29
K5Acl Input Action U 2.00 27.45 15 47 100 500 26 28 22 362:05
K5Acl Output Action 2.00 28.31 15 48 100 500 25 27 21 362:38
RkiosPortMan Port Re 2.00 0.22 12 24 100 500 0 0 0 18:39
Rkios Module State R 4.00 0.03 40 7 100 500 0 0 0 3:34
Rkios Online Diag Re 4.00 0.03 40 13 100 500 0 0 0 3:17
RkNovaL3Fib L3If Rev 2.00 0.03 10 12 100 500 0 0 0 3:22
PDS Rx Review 2.00 0.95 10 36 100 500 0 1 0 23:41
GalGlmLinecardVp(1) 5.00 0.36 20 1527 100 500 0 0 0 34:38
Temperature monitor 0.40 0.02 4 10 100 500 0 0 0 1:57
GalGlmPollerMan 3.00 0.90 20 28 100 500 0 0 0 79:37
Stub ondemand global 0.50 4.38 5 346 100 500 5 5 4 611:13
Xgstub Stats Review 0.50 1.06 5 42 100 500 1 1 0 125:32
Stub CTS Periodic Re 0.80 0.05 8 16 100 500 0 0 0 6:39
Stub CTS PeriodicSta 0.80 0.04 8 19 100 500 0 0 0 11:23
Stub CTS Periodic Re 0.80 0.04 8 15 100 500 0 0 0 6:49
Stub CTS PeriodicSta 0.80 0.04 8 24 100 500 0 0 0 11:02
Stub CTS Periodic Re 0.80 0.05 8 17 100 500 0 0 0 6:51
Stub CTS PeriodicSta 0.80 0.04 8 26 100 500 0 0 0 10:45
Stub CTS Periodic Re 0.80 0.05 8 17 100 500 0 0 0 6:40
Stub CTS PeriodicSta 0.80 0.04 8 33 100 500 0 0 0 11:13
EpmPortGroup(10:N) s 0.50 0.07 2 23 100 500 0 0 0 7:27
-------------
%CPU Totals 284.90 89.27

Mark Malone · ‎07-27-2015

Hi

There is a bug on ios-xe for that process , your most likely hitting that , i would upgrade or open a TAC case get the specific bug id or check the release notes online against your version see if there's a matching open caveat

IOSd: This is the Cisco IOS daemon that runs on the Linux kernel. It is run as a software process within the kernel.

jfk_web_cisco · ‎07-27-2015

We open a TAC and find out that was coming from one vlan. After a "Shut" "no Shut", CPU drop down from 90% to 14%. The engineer told me that was a bug as you mention, but we don't know how we trigger it.

Mark Malone · ‎07-27-2015

get the bug id off him and ask him what release its fixed in then upgrade

hmcgonagle · ‎08-26-2015

Anyone have the bug ID for this? I am running into the same issue but we are running the latest code.

jfk_web_cisco · ‎08-26-2015

TAC was unable to find reference to this internal bug and close the case. If it can help, for us we think that a subnet mask modification when the vlan was up trigger the problem. The vlan was set 255.255.255.240, but it was supposed to be 255.255.240.0. After that, problem begin. After we shut and no shut the vlan in question, everything come back to normal.

If you execute commands from my first post, what the result ?

hmcgonagle · ‎08-26-2015

Our issue turned out to be a bug. We downgraded the code and CPU normalized.

amcgrory · ‎11-13-2015

Do you have a bug ID for this ?

charliey_2000 · ‎11-14-2016

I know this post is a year old but I believe we are seeing this bug. on Release 3.7.3E. Can you tell me what version of code you used or are currently using.

amcgrory · ‎11-14-2016

Hi

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.03.02.SG RELEASE SOFTWARE (fc1)

charliey_2000 · ‎11-14-2016

So on the (cat4500e-UNIVERSALK9-M), Version 03.03.02.SG RELEASE SOFTWARE (fc1) you are not seeing the CPU issue correct? Just making sure before we give it a try.

amcgrory · ‎11-15-2016

that is correct, current stats ( taken at the start of the business day)

Core 0: CPU utilization for five seconds: 2%; one minute: 2%; five minutes: 2%
Core 1: CPU utilization for five seconds: 14%; one minute: 12%; five minutes: 12%
PID    Runtime(ms) Invoked uSecs 5Sec     1Min     5Min     TTY   Process
8450   1787449     26973071 314    7.99     7.26     7.23     0     iosd
11781 3946781     79932243 96     0.20     0.19     0.20     0     ffm
5930   1241269     12578979 9      0.04     0.01     0.02     0     os_info_p
5934   1234568     12663252 9      0.04     0.10     0.17     0     eicored
5940   2098618     14159522 14     0.04     0.01     0.02     0     plogd
10750 2087421     12623037 16     0.04     0.00     0.00     0     licensed
11242 1553216     15748434 146    0.04     0.04     0.04     0     cpumemd

hmcgonagle · ‎08-26-2015

Our issue turned out to be a bug. We downgraded the code and CPU normalized.

High CPU on 4500-X VSS - need help!