Gig Interface Total Output Drops

[escapethgrind]

Pls advise how to calculate the percent of Total Output Drops on this Layer2 interface (C9300-48T).   Am I correct in saying the interface below is dropping 27% of its packets?     My calculation =  Total output drops / (output packets + Total output drops) 

GigabitEthernet1/0/14 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 007e.9590.528e (bia 007e.9590.528e)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 18:14:34
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1586061
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 25000 bits/sec, 31 packets/sec
5 minute output rate 819000 bits/sec, 62 packets/sec
2199277 packets input, 201406641 bytes, 0 no buffer
Received 1029 broadcasts (819 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 819 multicast, 0 pause input
0 input packets with dribble condition detected
4422570 packets output, 6179956292 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
182 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Also, what is an acceptable drop percentage?  Thought I read some where it was just 1%. Pls confirm.

[Reza Sharifi]

Total output is drops: 1586061

output, 6179956292

1586061/6179956292 x 100=0.025

So, this is a lot less than 1% drop.

HTH 

[escapethgrind]

So the Total output drop is listed in bytes as opposed to packets?  Pls confirm.

Also, that appears to be 2.5%, correct?

[Reza Sharifi]

So the Total output drop is listed in bytes as opposed to packets?  Pls confirm. Correct both numbers are in bytes.

Also, that appears to be 2.5%, correct? No, it is 0.025 of 1%. 

HTH

[Georg Pauwen · Accepted solution]

Hello,

ideally, obviously, you don't want to see any output drops at all. Try and configure:

qos queue-softmax-multiplier 1200

and also post the output of:

sh platform hardware fed switch 1 qos queue stats interface gigabitEthernet 1/0/14

[escapethgrind]

Georg, here is the output you requested...

sh platform hardware fed switch 1 qos queue stats interface gi1/0/14

DATA Port:13 Enqueue Counters
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
0 0 9222883 1771977552 1432768641
1 0 0 0 4945120143
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0

DATA Port:13 Drop Counters
-------------------------------
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
----- ----------- ----------- ----------- ----------- -----------
0 0 0 0 0 0
1 0 0 186394755 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0

Note: Queuing stats are in bytes

Also, I'm seeing this interface flap quite often. Here is the latest from the logs...

006531: Nov 26 07:11:51.884 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to down
006532: Nov 26 07:11:53.978 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to up
006535: Nov 26 07:38:07.282 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to down
006536: Nov 26 07:38:11.328 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to up
006541: Nov 26 10:03:18.169 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to down
006542: Nov 26 10:03:22.388 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to up

In reference to physical layer, this is new CAT6 that's well within 100m.

Thanks for the feedback.

[Reza Sharifi]

What does port 1/0/14 connect to? Have you tried a different cable? What if you connect the same cable to a different (available) port?

[escapethgrind]

Increasing the Softmax buffers to 1200 has eliminated the output drops.

Thank you.

[Joseph W. Doherty]

"Am I correct in saying the interface below is dropping 27% of its packets? "

Yes and no. Your percentage is an average for the time since the counters were reset (i.e. "Last clearing of "show interface" counters 18:14:34"). Could be much, much higher during short time periods.

What's an acceptable drop percentage? That depends on the application. 1%, or less, is an "old" rule-of-thumb; mainly (I believe) based on providing a "typical" TCP flow, like FTP, a decent "goodput" rate.

What's interesting with your stats, you have a very, very high drop rate, yet a low utilization average for last five minutes. (Although a really high drop rate can really tank TCP transmission rates.)

Your later posting, showing additional stats, besides showing drops, also shows much being queued. Of course, if your interface keeps shutting down, that will backup your queues rapidly.

[paul driver]

Hello

---

@escapethgrind wrote:

Pls advise how to calculate the percent of Total Output Drops on this Layer2 interface (C9300-48T).   Am I correct in saying the interface below is dropping 27% of its packets?     My calculation =  Total output drops / (output packets + Total output drops) 
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1586061
4422570 packets output, 6179956292 bytes, 0 underruns

---

I have it as this
1586061/4422570 x 100 =35.8%
4422570/ 1586061 = 2.788 so a packet drop every 2.8 packets

[Joseph W. Doherty]

"I have it as this
1586061/4422570 x 100 =35.8%
4422570/ 1586061 = 2.788 so a packet drop every 2.8 packets"

I too used to believe drop percentage would be drops over "packet output", but re-reading documentation, "packets output" might only count packets actually transmitted (i.e. not all sent toward interface). If true, as OP noted, formula would be drops divided by ("packets output" plus drops).

Paul, do you have any documentation reference that clarifies which it is?

[JhonnyPacheco]

Hi Joseph and Paul,

I have read some articles and follow how ro calculate the percentage of the total output bytes transmitted on an interface that were dropped:

• Collect the total output drops count from the interface.
• Collect the total output bytes count from the interface.
• Calculate the percentage of output drops. Divide the total output drops count by the total output byte count and multiplying by 100.

This provides the percentage of output bytes that were dropped on the interface, which can help you determine if there is a congestion or buffer allocation issue that needs to be addressed or if the output drops were cause by transitory microbusrt traffic.

Use the showinterface <interface> command to collect the information.

Cat9k#show interfaces twentyFiveGigE 1/0/41               
TwentyFiveGigE1/0/41 is up, line protocol is up (connected)  
  Hardware is Twenty Five Gigabit Ethernet, address is dc77.4c8a.4289 (bia dc77.4c8a.4289)
  MTU 1500 bytes, BW 25000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 3/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-AOC1M
  input flow-control is on, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:06, output 00:00:10, output hang never
  Last clearing of "show interface" counters 6w1d  
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 299040207
  Queueing strategy: Class-based queueing
  Output queue: 0/40 (size/max)
  30 second input rate 767000 bits/sec, 155 packets/sec
  30 second output rate 14603000 bits/sec, 1819 packets/sec
     931864194 packets input, 572335285416 bytes, 0 no buffer
     Received 933005 broadcasts (933005 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     1067891106 packets output, 5930422327799 bytes, 0 underruns  
--snip--                     

Total output drops: 299040207
Total output bytes: 5930422327799

Percentage of output drops = 299040207/5930422327799 x 100 = 0.005%

In this example, the total output drops represent 0.005% of the total amount of bytes transmitted on this interface in the past six weeks (last clearing of counters 6w1d).

• The total number of packets versus number dropped is minor, and should not have any impact.

Font: https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9600-series-switches/220491-understand-output-drops-on-high-speed-in.html

[Joseph W. Doherty]

I've marked your reply helpful because it (indirectly) raises a change in how a 9K registers drops (described in your reference) which is:  "On Catalyst 9000 series switches, egress packet drops are shown in bytes by default instead of packets."  As OP is asking about a 9300, this change applies.

Using the OP stats, and the methodology provided in the reference we obtain:

1586061 / 6179956292  * 100 = 0.025%  (BTW, which is what @Reza Sharifi noted in his original reply and confirmed in his second reply.)

Whether counting drops in bytes or packets, there's still the question of what exactly is being counted for "output".  Does "output"  count just actually transmitted, or implicitly drops too.

For example if I send 100 packets (or bytes) to an egress interface and 25 are dropped, is "output" counter 100 (submitted to interface for transmission) or 75 (count actually transmitted)?  25 out of 100 is 25% but if output count is recorded as 75, without knowing you need to add back in drops, you could calculate drop rate as 33.3%, i.e. 25 out of 75.

That aside, the reference's example statement of:

"The total number of packets versus number dropped is minor, and should not have any impact."

Is interesting, since it mentions packets, when example is calculating drops using bytes, and that it also mentions (earlier) this is for a six week period, I think it a bit presumptuous to assume there's no possible impact (because it assumes, I believe, the drop rate is a nice low percentage all the time).

Also in the above reference, we have "This provides the percentage of output bytes that were dropped on the interface, which can help you determine if there is a congestion or buffer allocation issue that needs to be addressed or if the output drops were cause by transitory <sic> microbusrt traffic.", again, unsure how a drop percentage helps determines the causing issue.

Understand, different kinds of traffic can have very different tolerances for packet drops.  Further, don't overlook, such drops, to the network application, are the sum of all end-to-end.

Personally, I consider "managing" drops an important aspect of QoS, and with much other QoS, "one size" solutions are often far from optimal.  Likewise truly understanding the cause of interface drops, and their possible impact, can often be more complex than just computing some interface drop rate percentage.