ACI troubleshoot level3 rx drops

[thomasnoppe]

Hi,

We have not configured anything special regarding QoS.
I do see on a lot of ports RX drops for the default qos class (level3).

We see it on server ports but also on the uplinks from a leaf towards the spines.
On the spine side there are no TX or RX drops for level3.

How can I see which traffic is being dropped or why traffic is dropped?

We run 5.2.(4e).

kind regards

[mukesh-gupta]

i am facing the same issue 

[AshSe]

Dear @thomasnoppe & @mukesh-gupta  

In Cisco ACI, when you observe RX drops for the default QoS class (level3) on various ports, including server ports and uplinks from leaf to spine, it is important to diagnose the root cause of these drops. Here are the steps you can take to identify which traffic is being dropped and why:

1. Check Interface Statistics and Errors

First, gather detailed statistics and error counters for the affected interfaces. This can help you understand the nature and extent of the drops.

Using the APIC GUI:

1. Navigate to Fabric > Inventory > Pod > Leaf/Spine > Interfaces > Physical Interfaces.
2. Select the affected interface and go to the Statistics (Operational > Ethernet Statistic Count) tab.
3. Review the RX and TX statistics, including error counters.

Using the CLI:

You can use the following commands on the leaf switches to check interface statistics:

2. Check QoS Policies

Review the QoS policies applied to the interfaces. Ensure that the QoS policies are correctly configured and that there is no misconfiguration causing drops.

Using the APIC GUI:

1. Navigate to Fabric > Access Policies > Policies > Global > QoS Class.
2. Review the QoS policies and ensure they are correctly applied to the interfaces.

3. Check for Congestion and Buffer Utilization

High buffer utilization and congestion can lead to packet drops. Check the buffer utilization on the affected interfaces.

Using the CLI:

4. Use SPAN or ERSPAN to Capture Traffic

To identify which traffic is being dropped, you can use SPAN (Switched Port Analyzer) or ERSPAN (Encapsulated Remote SPAN) to capture and analyze the traffic on the affected interfaces.

Configuring SPAN:

1. Create a SPAN session:

2. Capture and analyze the traffic using a network analyzer like Wireshark.

5. Check for Microbursts

Microbursts can cause short-term congestion and packet drops. Use tools like Cisco Tetration or third-party network performance monitoring tools to detect microbursts.

6. Review System Logs and Faults

Check the system logs and faults in the APIC to see if there are any related messages that can provide more context about the drops.

Using the APIC GUI:

1. Navigate to System > Faults.
2. Navigate to System > History > Event Records

7. Check for Software Bugs

Ensure that you are running the latest recommended software version. Sometimes, software bugs can cause unexpected behavior, including packet drops. Check the Cisco Bug Search Tool for any known issues related to your APIC version (5.2(4e)).

And, The Lender of the Last Resort

8. Contact Cisco TAC

If you are unable to determine the cause of the drops after performing the above steps, consider reaching out to Cisco Technical Assistance Center (TAC) for further assistance. They can provide more in-depth troubleshooting and support.