3850 CPU Queues

[Peter P]

I'm troubleshooting high cpu on a 3850 stack using Cisco guidance from this link; https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html#anc4.

Can anyone explain what we are looking at in this output snippet? What specifically is of concern? Is anything above zero "IOSd failed count" a concern or is there an acceptable ratio of attempts to fails?

In the show platform punt statistics port-asic 0 cpuq -1 direction rx command, the -1 argument lists all of the queues. Therefore, this command lists all receive queues for Port-ASIC 0.

Now, you must identify which queue pushes a large number of packets at a high rate. In this example, an examination of the queues revealed this culprit:

<snip>
RX (ASIC2CPU) Stats (asic 0 qn 16 lqn 16):
RXQ 16: CPU_Q_PROTO_SNOOPING
----------------------------------------
Packets received from ASIC     : 79099152
Send to IOSd total attempts    : 79099152
Send to IOSd failed count      : 1240331
RX suspend count               : 1240331
RX unsuspend count             : 1240330
RX unsuspend send count        : 1240330
RX unsuspend send failed count : 0
RX dropped count               : 0
RX conversion failure dropped  : 0
RX pkt_hdr allocation failure  : 0
RX INTACK count                : 0
RX packets dq'd after intack   : 0
Active RxQ event               : 9906280
RX spurious interrupt          : 0
<snip>

[andresfr]

Hello pp1001,

I hope you're doing well.

In that same document you've to important definitions:

• Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst 3850 Series Switch and is responsible for all hardware programming/forwarding.
  
  
• IOSd: This is the Cisco IOS daemon that runs on the Linux kernel. It is run as a software process within the kernel.

The idea with switches is to process the  traffic at the hardware level (using the ASICs) when possible. Is that's not possible, then traffic is punted to the Control Plane/CPU so it can be processed . IOSd resides at the control plane. So, in the snippet below we have that the packets received from ASIC ( for RXQ 16) are being sent to the IOSd (CPU/Control plane). If the packets were not being sent to the CPU (because of those being processed in hardware depending on the type of traffic and the ASIC capabilities) then those counters should be at zero.

When FED cannot send the packets to IOSd, then the Send to IOSd failed count will increase and the switch will suspend sending those packets to IOSd for a while to then try to send those again (unsuspend). So, the value of these counters will depend on the particular scenario that you're troubleshooting regarding the type of traffic, the rate and the amount of traffic that is hitting the switch, and the capacity of the CPU to handle all that. It could be expected to see values different from zero but you will need to determine the source of the traffic and if the values of these counters are increasing or not (by checking the output multiple times) to then determine how to proceed.

RX (ASIC2CPU) Stats (asic 0 qn 16 lqn 16):
RXQ 16: CPU_Q_PROTO_SNOOPING
----------------------------------------
Packets received from ASIC     : 79099152  <-------
Send to IOSd total attempts    : 79099152  <-------
Send to IOSd failed count      : 1240331   <-------
RX suspend count               : 1240331   <-------
RX unsuspend count             : 1240330   <-------
RX unsuspend send count        : 1240330   <-------
RX unsuspend send failed count : 0
RX dropped count               : 0
RX conversion failure dropped  : 0
RX pkt_hdr allocation failure  : 0
RX INTACK count                : 0
RX packets dq'd after intack   : 0
Active RxQ event               : 9906280
RX spurious interrupt          : 0

I hope this helps.

Regards,