cisco 3850 3650 cpu punt queues

paul amaral · ‎04-08-2020

I have been trying to figure out 3850 / 3650 CPU queue traffic with not much success. I have a couple of 3850/3650 that are dropping traffic punted to the CPU. For example, on a 3650 I have traffic punted to the CPU from FED for SW forwarding queue and notice a lot of dropped packets. Are these dropped packets at the CPU itself? I’m assuming sw forwarding traffic, is traffic that can’t be processed in hardware therefore processed by the main cpu before being forwarded? If there a way to look at that RX CPU queue and look at the packets destined to it. I keep running into show buffer pool RxQ(id) but that command no longer works on IOS 16.9.x. I did setup a monitor capture to the control plane but nothing stood out to me.

looking at this closer I believe its cef packets being punted

sh platform software fed switch active punt cause sum
Statistics for all causes

Cause Cause Info Rcvd Dropped
------------------------------------------------------------------------------
2 IPv4 Options 975 0
7 ARP request or response 3191132 0
11 For-us data 27501786 0
21 RP<->QFP keepalive 974395 0
24 Glean adjacency 73749075 1362726
------------------------------------

#sh platform software fed switch active punt cpuq br

Punt CPU Q Statistics Brief

=====================================================================================

Q | Queue | Rx | Rx | Rx | Drop | Drop | Drop

====================================================================================

0 CPU_Q_DOT1X_AUTH 0 0 0 0 0 0

1 CPU_Q_L2_CONTROL 22726839 22775124 48285 0 0 0

2 CPU_Q_FORUS_TRAFFIC 27410316 27452377 42061 0 0 0

3 CPU_Q_ICMP_GEN 0 0 0 0 0 0

4 CPU_Q_ROUTING_CONTROL 472482 473460 978 0 0 0

5 CPU_Q_FORUS_ADDR_RESOLUTION 3230656 3236894 6238 0 0 0

6 CPU_Q_ICMP_REDIRECT 4405728 4409997 4269 0 0 0

7 CPU_Q_INTER_FED_TRAFFIC 0 0 0 0 0 0

8 CPU_Q_L2LVX_CONTROL_PKT 0 0 0 0 0 0

9 CPU_Q_EWLC_CONTROL 0 0 0 0 0 0

10 CPU_Q_EWLC_DATA 0 0 0 0 0 0

11 CPU_Q_L2LVX_DATA_PKT 0 0 0 0 0 0

12 CPU_Q_BROADCAST 892821 894644 1823 0 0 0

13 CPU_Q_LEARNING_CACHE_OVFL 0 0 0 0 0 0

14 CPU_Q_SW_FORWARDING 73568522 73696006 127484 1364361 1366266 1905

15 CPU_Q_TOPOLOGY_CONTROL 0 0 0 0 0 0

16 CPU_Q_PROTO_SNOOPING 0 0 0 0 0 0

17 CPU_Q_DHCP_SNOOPING 0 0 0 0 0 0

18 CPU_Q_TRANSIT_TRAFFIC 0 0 0 0 0 0

19 CPU_Q_RPF_FAILED 0 0 0 0 0 0

20 CPU_Q_MCAST_END_STATION_SERVICE 22484 22530 46 0 0 0

21 CPU_Q_LOGGING 0 0 0 0 0 0

22 CPU_Q_PUNT_WEBAUTH 0 0 0 0 0 0

23 CPU_Q_HIGH_RATE_APP 0 0 0 0 0 0

24 CPU_Q_EXCEPTION 0 0 0 0 0 0

25 CPU_Q_SYSTEM_CRITICAL 0 0 0 0 0 0

26 CPU_Q_NFL_SAMPLED_DATA 0 0 0 0 0 0

27 CPU_Q_LOW_LATENCY 40 40 0 0 0 0

28 CPU_Q_EGR_EXCEPTION 266288 267006 718 0 0 0

29 CPU_Q_FSS 0 0 0 0 0 0

30 CPU_Q_MCAST_DATA 0 0 0 0 0 0

31 CPU_Q_GOLD_PKT 0 0 0 0 0 0

I notice that many packets to SW forwarding queue have been dropped on the cpu queue itself but COPP policing doesn’t match this, while I see packets dropped in real-time with the command above, when looking at COPP policing I don’t see as many packets being dropped, so not sure how something dropped on the CPU queue is not necessarily dropped by the policer.

TIA, Paul

Policer stats

CPU Queue Statistics

============================================================================================

(default) (set) Queue

QId PlcIdx Queue Name Enabled Rate Rate Drop(Bytes)

-----------------------------------------------------------------------------

0 11 DOT1X Auth Yes 1000 1000 0

1 1 L2 Control Yes 2000 2000 0

2 14 Forus traffic Yes 4000 4000 0

3 0 ICMP GEN Yes 600 600 0

4 2 Routing Control Yes 5400 5400 0

5 14 Forus Address resolution Yes 4000 4000 0

6 0 ICMP Redirect Yes 600 600 2430232

7 16 Inter FED Traffic Yes 2000 2000 0

8 4 L2 LVX Cont Pack Yes 1000 1000 0

9 16 EWLC Control Yes 2000 2000 0

10 16 EWLC Data Yes 2000 2000 0

11 13 L2 LVX Data Pack Yes 1000 1000 0

12 0 BROADCAST Yes 600 600 954

13 10 Openflow Yes 100 100 0

14 13 Sw forwarding Yes 1000 1000 49251043

15 8 Topology Control Yes 13000 13000 0

16 12 Proto Snooping Yes 2000 2000 0

17 6 DHCP Snooping Yes 500 500 0

18 9 Transit Traffic Yes 500 500 0

19 10 RPF Failed Yes 100 100 0

20 15 MCAST END STATION Yes 2000 2000 0

21 13 LOGGING Yes 1000 1000 0

22 7 Punt Webauth Yes 1000 1000 0

23 10 High Rate App Yes 100 100 0

24 10 Exception Yes 100 100 0

25 3 System Critical Yes 1000 1000 0

26 10 NFL SAMPLED DATA Yes 100 100 0

27 2 Low Latency Yes 5400 5400 0

28 10 EGR Exception Yes 100 100 0

29 5 Stackwise Virtual OOB Yes 8000 8000 0

30 9 MCAST Data Yes 500 500 0

31 3 Gold Pkt Yes 1000 1000 0

* NOTE: CPU queue policer rates are configured to the closest hardware supported value

CPU Queue Policer Statistics

====================================================================

Policer Policer Accept Policer Drop

Index Bytes Bytes

------------------------------------------------

0 545864331 2431186

1 2665264654 0

2 61734583 0

3 0 0

4 0 0

5 0 0

6 0 0

7 0 0

8 0 0

9 0 0

10 20320974 0

11 0 0

12 0 0

13 5613065589 49251043

14 2931255243 0

15 2644260 0

16 0 0

17 0 0

CPP Classes to queue map

======================================================================================

PlcIdx CPP Class : Queues

--------------------------------------------------------------------------------------

0 system-cpp-police-data : ICMP GEN/BROADCAST/ICMP Redirect/

10 system-cpp-police-sys-data : Openflow/High Rate App/Exception/EGR Exception/NFL

SAMPLED DATA/RPF Failed/

13 system-cpp-police-sw-forward : Sw forwarding/LOGGING/L2 LVX Data Pack/

9 system-cpp-police-multicast : Transit Traffic/MCAST Data/

15 system-cpp-police-multicast-end-station : MCAST END STATION /

7 system-cpp-police-punt-webauth : Punt Webauth/

1 system-cpp-police-l2-control : L2 Control/

2 system-cpp-police-routing-control : Routing Control/Low Latency/

3 system-cpp-police-system-critical : System Critical/Gold Pkt/

4 system-cpp-police-l2lvx-control : L2 LVX Cont Pack/

8 system-cpp-police-topology-control : Topology Control/

11 system-cpp-police-dot1x-auth : DOT1X Auth/

12 system-cpp-police-protocol-snooping : Proto Snooping/

6 system-cpp-police-dhcp-snooping : DHCP Snooping/

14 system-cpp-police-forus : Forus Address resolution/Forus traffic/

5 system-cpp-police-stackwise-virt-control : Stackwise Virtual OOB/

16 system-cpp-default : Inter FED Traffic/EWLC Control/EWLC Data/

ACCESS-FRV-3600-Stack-SW#