High CPU and Packet drops C9300L

[~Saj~]

Hi Community,

In a Cat9300L switch ( IOS XE 17.6.4 ), can see some high CPU and packet drops. Any leads to nail the actual root cause appreciated.

Switch-1#show controllers cpu-interface
queue retrieved dropped invalid hol-block
-------------------------------------------------------------------------
Routing Protocol 59650908 0 0 0
L2 Protocol 9874946 0 0 0
sw forwarding 11154871 145477 0 0
broadcast 24517725 0 0 0
icmp gen 0 0 0 0
icmp redirect 0 0 0 0
logging 119783 0 0 0
rpf-fail 52453 0 0 0
DOT1X authentication 1140832 0 0 0
Forus Traffic 354432720 0 0 0
Forus Resolution 790 0 0 0
Inter FED 0 0 0 0
L2 LVX control 1437010 0 0 0
EWLC control 0 0 0 0
EWLC data 0 0 0 0
L2 LVX data 1752987 0 0 0
Openflow 0 0 0 0
Topology control 14736 0 0 0
Proto snooping 70677371 0 0 0
DHCP snooping 1795541 0 0 0
Transit Traffic 0 0 0 0
Multi End station 16682051 0 0 0
Webauth 0 0 0 0
High rate app 2523074615 2523074615 0 0
Exception 0 0 0 0
System Critical 0 0 0 0
NFL sampled data 0 0 0 0
Low latency 276836529 0 0 0
EGR exception 126426 0 0 0
Stackwise Virtual OOB 0 0 0 0
Multicast data 368181 0 0 0
Gold packet 0 0 0 0

Switch-1#sho pl ha fed sw 1 qo qu st intern cpu pol

CPU Queue Statistics
============================================================================================
(default) (set) Queue Queue
QId PlcIdx Queue Name Enabled Rate Rate Drop(Bytes) Drop(Frames)
--------------------------------------------------------------------------------------------
0 11 DOT1X Auth Yes 1000 1000 0 0
1 1 L2 Control Yes 2000 2000 0 0
2 14 Forus traffic Yes 4000 4000 0 0
3 0 ICMP GEN Yes 600 600 0 0
4 2 Routing Control Yes 5400 5400 0 0
5 14 Forus Address resolution Yes 4000 4000 0 0
6 0 ICMP Redirect Yes 600 600 0 0
7 16 Inter FED Traffic Yes 2000 2000 0 0
8 4 L2 LVX Cont Pack Yes 1000 1000 0 0
9 19 EWLC Control Yes 13000 13000 0 0
10 16 EWLC Data Yes 2000 2000 0 0
11 13 L2 LVX Data Pack Yes 1000 1000 418268 5613
12 0 BROADCAST Yes 600 600 24688 184
13 10 Openflow Yes 200 200 0 0
14 13 Sw forwarding Yes 1000 1000 430213087 347100
15 8 Topology Control Yes 13000 13000 0 0
16 12 Proto Snooping Yes 2000 2000 0 0
17 6 DHCP Snooping Yes 400 400 0 0
18 13 Transit Traffic Yes 1000 1000 0 0
19 10 RPF Failed Yes 200 200 0 0
20 15 MCAST END STATION Yes 2000 2000 0 0
21 13 LOGGING Yes 1000 1000 1565761599 1863945
22 7 Punt Webauth Yes 1000 1000 0 0
23 18 High Rate App Yes 13000 13000 0 0
24 10 Exception Yes 200 200 0 0
25 3 System Critical Yes 1000 1000 0 0
26 10 NFL SAMPLED DATA Yes 200 200 0 0
27 2 Low Latency Yes 5400 5400 0 0
28 10 EGR Exception Yes 200 200 0 0
29 5 Stackwise Virtual OOB Yes 8000 8000 0 0
30 9 MCAST Data Yes 400 400 0 0
31 3 Gold Pkt Yes 1000 1000 0 0

* NOTE: CPU queue policer rates are configured to the closest hardware supported value

CPU Queue Policer Statistics
====================================================================
Policer Policer Accept Policer Accept Policer Drop Policer Drop
Index Bytes Frames Bytes Frames
-------------------------------------------------------------------
0 5753798835 37233248 429668207 351494
1 2823953192 9876032 0 0
2 26757768649 336523241 0 0
3 0 0 0 0
4 91994624 1437416 0 0
5 0 0 0 0
6 663969686 1795722 0 0
7 0 0 0 0
8 945679 14736 0 0
9 50896435 368268 0 0
10 23588074 178971 0 0
11 390986242 1141109 0 0
12 5151976177 70691463 0 0
13 126310074 318566 1566749435 1865348
14 89170715583 354478381 0 0
15 1127444252 16684165 0 0
16 0 0 0 0
17 0 0 0 0
18 536216829094 2523754489 0 0
19 0 0 0 0

Second Level Policer Statistics
====================================================================
20 29582667520 346414009 0 0
21 101887710296 482531587 0 0

Policer Index Mapping and Settings
--------------------------------------------------------------------
level-2 : level-1 (default) (set)
PlcIndex : PlcIndex rate rate
--------------------------------------------------------------------
20 : 1 2 8 13000 13000
21 : 0 4 7 9 10 11 12 13 14 15 6000 6000
====================================================================

Second Level Policer Config
====================================================================
level-1 level-2 level-2
QId PlcIdx PlcIdx Queue Name Enabled
--------------------------------------------------------------------
0 11 21 DOT1X Auth Yes
1 1 20 L2 Control Yes
2 14 21 Forus traffic Yes
3 0 21 ICMP GEN Yes
4 2 20 Routing Control Yes
5 14 21 Forus Address resolution Yes
6 0 21 ICMP Redirect Yes
7 16 - Inter FED Traffic No
8 4 21 L2 LVX Cont Pack Yes
9 19 - EWLC Control No
10 16 - EWLC Data No
11 13 21 L2 LVX Data Pack Yes
12 0 21 BROADCAST Yes
13 10 21 Openflow Yes
14 13 21 Sw forwarding Yes
15 8 20 Topology Control Yes
16 12 21 Proto Snooping Yes
17 6 - DHCP Snooping No
18 13 21 Transit Traffic Yes
19 10 21 RPF Failed Yes
20 15 21 MCAST END STATION Yes
21 13 21 LOGGING Yes
22 7 21 Punt Webauth Yes
23 18 - High Rate App No
24 10 21 Exception Yes
25 3 - System Critical No
26 10 21 NFL SAMPLED DATA Yes
27 2 20 Low Latency Yes
28 10 21 EGR Exception Yes
29 5 - Stackwise Virtual OOB No
30 9 21 MCAST Data Yes
31 3 - Gold Pkt No

CPP Classes to queue map
======================================================================================
PlcIdx CPP Class : Queues
--------------------------------------------------------------------------------------
0 system-cpp-police-data : ICMP GEN/ BROADCAST/ ICMP Redirect/
10 system-cpp-police-sys-data : Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/
13 system-cpp-police-sw-forward : Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Transit Traffic/
9 system-cpp-police-multicast : MCAST Data/
15 system-cpp-police-multicast-end-station : MCAST END STATION /
7 system-cpp-police-punt-webauth : Punt Webauth/
1 system-cpp-police-l2-control : L2 Control/
2 system-cpp-police-routing-control : Routing Control/ Low Latency/
3 system-cpp-police-system-critical : System Critical/ Gold Pkt/
4 system-cpp-police-l2lvx-control : L2 LVX Cont Pack/
8 system-cpp-police-topology-control : Topology Control/
11 system-cpp-police-dot1x-auth : DOT1X Auth/
12 system-cpp-police-protocol-snooping : Proto Snooping/
6 system-cpp-police-dhcp-snooping : DHCP Snooping/
14 system-cpp-police-forus : Forus Address resolution/ Forus traffic/
5 system-cpp-police-stackwise-virt-control : Stackwise Virtual OOB/
16 system-cpp-default : Inter FED Traffic/ EWLC Data/
18 system-cpp-police-high-rate-app : High Rate App/
19 system-cpp-police-ewlc-control : EWLC Control/
20 system-cpp-police-ios-routing : L2 Control/ Topology Control/ Routing Control/ Low Latency/
21 system-cpp-police-ios-feature : ICMP GEN/ BROADCAST/ ICMP Redirect/ L2 LVX Cont Pack/ Proto Snooping/ Punt Webauth/ MCAST Data/ Transit Traffic/ DOT1X Auth/ Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Forus traffic/ Forus Address resolution/ MCAST END STATION / Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/

[Cisco_Virtual_Engineer]

Based on the information provided, it appears that there are packet drops in the "sw forwarding" queue. High CPU usage and packet drops can be caused by various factors, such as traffic bursts, misconfigurations, or faulty hardware. To further troubleshoot the issue, you can follow these steps:

1. Check the CPU utilization history to see if the high CPU usage is consistent or occurs during specific intervals. You can use the command `show processes cpu history` to view the CPU usage history.

2. Identify the processes consuming the most CPU resources using the command `show processes cpu sorted`. This will help you determine if any specific process is causing the high CPU usage.

3. Verify the configuration of quality of service (QoS) and ensure that it is correctly configured for your network requirements. Misconfigured QoS settings can lead to packet drops and high CPU usage.

4. Analyze the traffic going through the switch using the command `show interfaces | include (line protocol|input rate|output rate)`. This can help you identify any sudden spikes in traffic or unusually high traffic rates that may be causing the high CPU usage and packet drops.

5. Check for any error messages in the logs using the command `show logging`. This can help you identify any issues related to hardware faults or misconfigurations.

6. Verify the switch's hardware health, such as power supplies, fans, and temperature, to ensure that there are no hardware issues causing the high CPU usage and packet drops.

7. If the issue persists, consider upgrading the switch's firmware to the latest version, as there might be a bug or issue in the current firmware that is causing the problem.

8. If none of the above steps help resolve the issue, you may need to contact Cisco Support for further assistance and investigation.