4503E high cpu. (iosd,Cat4k Mgmt LoPri)

[inb]

The cpu has suddenly increased.
Core1 is very high. 90% keep.
The network speed is also very slow.
The following log is also displayed.
What action should be taken?

Show process cpu history
Show process cpu sort | ex 0.00
Show proc cpu detailed process iosd sorted
Show platform health
Show platform cpu packet statistics all
Show platform cpu packet buffered
Show platform hardware acl statistics utilization brief
Show ip traffic

log attach.
help me please.

OS: cat4500e-universalk9.SPA.03.04.00.SG.151-2.SG.bin

Log:

Dec 18 17:24:21: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware
Dec 18 17:24:21: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware
Dec 18 17:24:21: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware
Dec 18 17:24:21: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware
Dec 18 17:24:21: %C4K_COMMONHWACLMAN-4-ALLACLINHW: All configured ACLs now fully loaded in hardware - hardware switching / QoS restored.

#Show proc cpu detailed process iosd sorted
Core 0: CPU utilization for five seconds: 6%; one minute: 7%; five minutes: 8%
Core 1: CPU utilization for five seconds: 87%; one minute: 87%; five minutes: 87%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
8594 L 3182694 1089267 323 48.60 47.22 47.45 0 iosd
8594 L 1 8594 3732938 7074928 0 46.32 44.43 44.72 0 iosd
8594 L 0 12160 3721758 3809726 0 2.28 2.78 2.72 0 iosd
8594 L 0 12161 22964 801565 0 0.00 0.00 0.00 0 iosd
86 I 3854660 5678066 0 59.22 54.33 54.99 0 Cat4k Mgmt LoPri
183 I 645566 1885991 0 14.88 13.66 13.88 0 IP Input
191 I 2563957 1046791 0 9.88 11.77 11.88 0 Spanning Tree
85 I 2708499 1179767 0 4.44 4.22 4.22 0 Cat4k Mgmt HiPri
215 I 2049609 2852047 0 1.33 1.44 1.33 0 HSRP Common
24 I 3339942 2385189 0 0.11 0.00 0.00 0 ARP Input

Switch1#Show platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

K5CpuMan Review       30.00  65.26     30    252  100  500   79  73   58  51469:40

[Georg Pauwen]

Hello,

post the output of 'show tech' so we can run this through the CLI analyzer...

[inb]

what is CLI analyzer?
"K5CpuMan Review" process high.
If this process is high, is not it reasonable to judge that the cpu is high because there is a lot of simple traffic?

[Georg Pauwen]

Hello,

the CLI analyzer is a TAC tool that can provide hints as to what causes the high CPU (among other things).

Do you have a lot of access lists and QoS in your configuration ? Post at least the full config, that might provide a clue...

[inb]

Switch#Show platform hardware acl statistics utilization brief
CAM Utilization Statistics
--------------------------

Used Free Total
--------------------------------
Input Security (160) 325 (15 %) 1723 (85 %) 2048
Input Security (320) 36 (1 %) 2012 (99 %) 2048
Input Forwarding (160) 7 (0 %) 2041 (100%) 2048
Input Forwarding (320) 24 (1 %) 2024 (99 %) 2048
Input Unallocated (160) 0 (0 %) 24576 (100%) 24576

Output Security (160) 294 (14 %) 1754 (86 %) 2048
Output Security (320) 14 (0 %) 2034 (100%) 2048
Output Qos (160) 13 (0 %) 2035 (100%) 2048
Output Qos (320) 4 (0 %) 2044 (100%) 2048
Output Unallocated (160) 0 (0 %) 24576 (100%) 24576

Input Profiles (logical) : used 1 / 32
Input Profiles (physical): used 4 / 32

Output Profiles (logical) : used 1 / 32
Output Profiles (physical): used 4 / 32

ACLs have only telnet ACLs and ACLs for VACLs.
Usage rate is not as high as 15%.
What problems should we look at?
The service has expired and we are in the process of verifying Google serach and cisco community.

[inb]

Allocation ceiling Current allocation
------------------ ------------------
kbytes % in use kbytes % in use

Chassis 1 Linecard 1 5120.00 10% 522.28 100%
Chassis 1 Linecard 2 5120.00 7% 390.28 100%
Chassis 1 Linecard 3 5120.00 7% 390.28 100%


Current allocation is 100%...

[inb]

When adding / deleting acl, the following logs will be generated.

bssw.log:Dec 13 10:05:30 10.0.100.2 3087: Dec 13 10:05:29: %C4K_HWACLMAN-4-ACLHWPROGERR: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - hardware TCAM limit, some packet processing will be software switched.
bssw.log:Dec 13 10:05:30 10.0.100.2 3088: Dec 13 10:05:29: %C4K_HWACLMAN-4-ACLHWPROGERRREASON: Input(NQ:2/Unknown, Q:16382/NoPolicing) Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - hardware TCAM L4 operators/TCP flags usage capability exceeded or hardware unsupported TCP flags
bssw.log:Dec 13 10:05:31 10.0.100.2 3090: Dec 13 10:05:31: %C4K_COMMONHWACLMAN-4-ALLACLINHW: All configured ACLs now fully loaded in hardware - hardware switching / QoS restored.
bssw.log:Dec 13 10:05:33 10.0.100.2 3091: Dec 13 10:05:32: %SYS-5-CONFIG_I: Configured from console by jgkim on vty0 (10.0.100.122)
bssw.log:Dec 13 10:11:46 10.0.100.2 3092: Dec 13 10:11:45: %C4K_HWACLMAN-4-ACLHWPROGERR: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - hardware TCAM limit, some packet processing will be software switched.
bssw.log:Dec 13 10:11:46 10.0.100.2 3093: Dec 13 10:11:45: %C4K_HWACLMAN-4-ACLHWPROGERRREASON: Input(NQ:0/Unknown, Q:16382/NoPolicing) Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - hardware TCAM L4 operators/TCP flags usage capability exceeded or hardware unsupported TCP flags
bssw.log:Dec 13 10:11:48 10.0.100.2 3097: Dec 13 10:11:48: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware
bssw.log:Dec 13 10:11:48 10.0.100.2 3105: Dec 13 10:11:48: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware
bssw.log:Dec 13 10:11:48 10.0.100.2 3109: Dec 13 10:11:48: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware
bssw.log:Dec 13 10:11:48 10.0.100.2 3110: Dec 13 10:11:48: %C4K_COMMONHWACLMAN-4-HWPROGSUCCESS: Input Security: Permit NS/NS-DAD (Internal), Security: Permit Arps(Internal), Security: VACL - now fully loaded in hardware

[jkilleda · Accepted solution]

4503E high cpu. (iosd,Cat4k Mgmt LoPri)

One of the resons being that the Supervisor Engine maximizes the CPU cycles in order to perform all the necessary software packet forwarding and background jobs, however below are some of the resons of processed switched packets which cause high utilization.

Some of the common reasons for high CPU utilization due to process-switched packets are:

• A high number of spanning-tree port instances
• ICMP redirects; routing packets on the same interface
• Layer 2 forwarding loops