3560X high CPU/Interrupts because of ICMP queue?

[wowzie]

Hello,

We've a 3560X switch which is using a lot of CPU (around 80-90% CPU) and 25-30% interrupts.

Top 3 processes:

175 1060646641 3649393632 0 16.13% 14.93% 14.93% 0 Hulc LED Process
224 3956630454 3320689710 0 10.86% 10.97% 11.01% 0 IP Input
13 2497812332 3353714124 0 3.35% 3.77% 3.72% 0 ARP Input

I think it's caused by ICMP traffic flooding the CPU because "show controllers cpu-interface" shows me this output (I monitored it for 24 hours):

cpu-queue-frames, icmp: increasing with 5800 packets/sec

cpu-queue-frames, sw forwarding: increasing with 485 packets/sec

cpu-queue-frames, routing: increasing with 297 packets/sec

show buffers output:

RxQ11 buffers, 2040 bytes (total 16, permanent 16):
1 in free list (0 min, 16 max allowed)
2225720443 hits, 2500324642 misses

How can I find what is causing this? I found that I can use "debug platform cpu-queues icmp-q" but this will probably crash the switch which is not an option because the amount of traffic going through this switch.

Thanks,

Erik

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
sh logs
sh proc cpu sort | ex 0.00

[wowzie]

#sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 71%/24%; one minute: 76%; five minutes: 82%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
175 1061116087 3649447988 0 14.23% 15.15% 15.65% 0 Hulc LED Process
224 3956956216 3320891212 0 12.15% 11.19% 11.13% 0 IP Input
13 2497921404 3353822980 0 4.15% 3.65% 3.67% 0 ARP Input
85 2415659242 571532433 4226 2.23% 1.76% 1.80% 0 RedEarth Tx Mana
178 1022797110 107440483 9519 1.75% 1.11% 1.10% 0 HL3U bkgrd proce
14 1188472142 164323555 7232 1.59% 1.16% 1.12% 0 ARP Background
91 1448298093 58537854 24741 1.59% 1.33% 1.41% 0 Adjust Regions
212 371093820 224918596 1649 0.95% 0.64% 0.64% 0 CEF: IPv4 proces
84 784236347 789874155 992 0.63% 0.97% 0.95% 0 RedEarth I2C dri
129 707665518 147989673 4781 0.63% 0.78% 0.79% 0 hpm counter proc
242 227879339 539345236 422 0.31% 0.10% 0.12% 0 Spanning Tree
223 116344653 4149362332 0 0.31% 0.06% 0.05% 0 IP ARP Retry Age
56 100671811 147991845 680 0.15% 0.04% 0.02% 0 Per-Second Jobs
190 278277891 29473918 9441 0.15% 0.19% 0.20% 0 HQM Stack Proces
290 32244963 184700818 174 0.15% 0.14% 0.15% 0 TCP Protocols
420 207838851 1220917627 170 0.15% 0.11% 0.14% 0 HSRP IPv4
86 19754962 394937080 50 0.15% 0.04% 0.04% 0 RedEarth Rx Mana
126 288853092 2015114174 143 0.15% 0.18% 0.16% 0 hpm main process
107 483417410 4149037853 0 0.15% 0.10% 0.06% 0 HLFM address lea

#show version
Cisco IOS Software, C3560E Software (C3560E-UNIVERSALK9-M), Version 15.2(4)E10, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 31-Mar-20 21:44 by prod_rel_team

ROM: Bootstrap program is C3560E boot loader
BOOTLDR: C3560E Boot Loader (C3560X-HBOOT-M) Version 12.2(58r)SE1, RELEASE SOFTWARE (fc1)

License Level: ipservices
License Type: Permanent
Next reload license Level: ipservices

cisco WS-C3560X-24 (PowerPC405) processor (revision M0) with 262144K bytes of memory.
Processor board ID FDO1852F1J3
Last reset from power-on
7 Virtual Ethernet interfaces
1 FastEthernet interface
28 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 74:A2:E6:67:D0:80
Motherboard assembly number : 73-12554-12
Motherboard serial number : FDO185302LK
Model revision number : M0
Motherboard revision number : A0
Model number : WS-C3560X-24T-E
Daughterboard assembly number : 800-32786-02
Daughterboard serial number : FDO18520QZ6
System serial number : FDO1852F1J3
Top Assembly Part Number : 800-31331-09
Top Assembly Revision Number : F0
Version ID : V06
CLEI Code Number : CMMPW00DRA
Hardware Board Revision Number : 0x05

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 30 WS-C3560X-24 15.2(4)E10 C3560E-UNIVERSALK9-M

show log only shows rows of:

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success

Nothing else.

[balaji.bandi]

You already running latest IOS, is this CPU high you seeing after any recent upgrade?

[wowzie]

We upgraded last week from 15.0(2)SE7 to this version to see if this fixed the issue. We've this high CPU Problem for a few weeks now.

[Leo Laohoo]

No logs? That's strange.
Try this:

1. Disable (or shutdown) the ports to the downstream clients for about 2 minutes. 
2. See if the CPU drops when the downstream ports are disabled. 

[wowzie]

I'm looking for a way to find the cause without downtime..