Output drops

RyanJohnstone · ‎01-23-2019

We have a 3850-24T showing high packet drops on some interfaces, have read various post and recommendations around this but my suspicion is the switch may have been underspec'd, from day 1 and any buffer tweaking will yield little/no benefit.

Is there any way to validate the belief that the switch is under spec?

The switch is the main transit off an MPLS WAN and the interfaces and bandwidth consumption do not max out (avg 30-40% utilisation on a 1Gb interface)

Counters output below

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 215626136
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 250082000 bits/sec, 36528 packets/sec
5 minute output rate 242451000 bits/sec, 34556 packets/sec
     9523250086 packets input, 10786888966602 bytes, 0 no buffer
     Received 15963 broadcasts (15963 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 15963 multicast, 0 pause input
     0 input packets with dribble condition detected
     1116764247 packets output, 854195337634 bytes, 0 underruns

Joseph W. Doherty · ‎01-23-2019

"Is there any way to validate the belief that the switch is under spec?"

Well, first you would need to describe the spec which you believe the 3850 doesn't meet.

". . . bandwidth consumption do not max out (avg 30-40% utilisation on a 1Gb interface)"

You believe such utilization (measured over a 5 minute average) precludes interface queuing drops, and perhaps remediation by buffer tuning? If so, are you familiar with the term "micro bursts"? If not, you might start here: https://en.wikipedia.org/wiki/Micro-bursting_(networking)

I haven't worked with the 3850 series, but on the earlier 3750 series, I've seen interfaces that constantly drop multiple packets per second be reduced to only a few dropped packets per day after buffer tuning.

RyanJohnstone · ‎01-25-2019

Hi thanks for reply, is there any way we can identify the cause of these dropped packets as being a result of micro bursts? I will have a look at the buffer tweaking available but I am still dubious this is the switch for the job.

Joseph W. Doherty · ‎01-25-2019

About the best way to "see" micro bursts to with a network analyzer that can see what's on a wire.

Other "clues" that's what's happening, SNMP monitoring interface load and packet drops for much, much smaller time intervals. (Unfortunately, as micro bursts happen in the millisecond range, highly frequent SNMP polling makes for its own problems. However, as the polling interval decreases, what your looking for is more hills and valleys, i.e. your earlier 30-40% utilization becomes much more variable. [Possible less of a load on the platform, than SNMP polling, would by some kind of device interface self monitoring using an on-board script.])

You can also try some buffer tuning, and with increase buffer resources, if you see a marked decrease in drops, you're likely have also identified the issue.

Leo Laohoo · ‎01-25-2019

Thanks for the output.
Here's one way to determine if it's caused by microburst or not: Drop the speed to 100 Mbps.
If the output drops counter don't increment, it's micro- or sustained burst.

Can you also post the complete output to the command "sh controll e G 2/0/3". I want to see the Defer Frames counter.

RyanJohnstone · ‎01-28-2019

Hi there, please see output as requested, I can bump the speed down to 100mbps but would need to organise this as its a production link so this will have an impact...may be something I need to do though to validate. Thanks.

sho controllers ethernet-controller g2/0/3
Transmit                  GigabitEthernet2/0/3          Receive
7117546495142 Total bytes               21263695412300 Total bytes
   9013564447 Unicast frames              22981950582 Unicast frames
7116986670202 Unicast bytes             21263685944780 Unicast bytes
      2870736 Multicast frames                  73965 Multicast frames
    402116161 Multicast bytes                 9467520 Multicast bytes
      1587496 Broadcast frames                      0 Broadcast frames
    157708779 Broadcast bytes                       0 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames
            0 MacUnderrun frames                    0 MacOverrun frames
            0 Pause frames                          0 Pause frames
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames
            0 Oam frames                            0 OamProcessed frames
            0 Oam frames                            0 OamDropped frames
      6182525 Minimum size frames          3000890636 Minimum size frames
   2767754412 65 to 127 byte frames        2996018280 65 to 127 byte frames
   1311842951 128 to 255 byte frames       2456287058 128 to 255 byte frames
    529282345 256 to 511 byte frames        973288116 256 to 511 byte frames
    160355796 512 to 1023 byte frames       400809563 512 to 1023 byte frames
   2105609089 1024 to 1518 byte frames     2381664430 1024 to 1518 byte frames
   2136995564 1519 to 2047 byte frames    10773066464 1519 to 2047 byte frames
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames
            0 Late collision frames                 0 SymbolErr frames
21997054842 Excess Defer frames                   0 Collision fragments
            0 Good (1 coll) frames                  0 ValidUnderSize frames
            0 Good (>1 coll) frames                 0 InvalidOverSize frames
            0 Deferred frames                       0 ValidOverSize frames
            0 Gold frames dropped                   0 FcsErr frames
            0 Gold frames truncated
            0 Gold frames successful
            0 1 collision frames
            0 2 collision frames
            0 3 collision frames
            0 4 collision frames
            0 5 collision frames
            0 6 collision frames
            0 7 collision frames
            0 8 collision frames
            0 9 collision frames
            0 10 collision frames
            0 11 collision frames
            0 12 collision frames
            0 13 collision frames
            0 14 collision frames
            0 15 collision frames
            0 Excess collision frames

Leo Laohoo · ‎01-23-2019

Can you please post the complete output to the command "sh interface <PORT>"?

RyanJohnstone · ‎01-25-2019

hi, thanks for reply, please see output below

switch#sho int g2/0/3
GigabitEthernet2/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0059.dc34.8903 (bia 0059.dc34.8903)
Description: uplink
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 82/255, rxload 43/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:07, output never, output hang never
Last clearing of "show interface" counters 1w0d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 774385099
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 170685000 bits/sec, 33189 packets/sec
5 minute output rate 324852000 bits/sec, 37840 packets/sec
     10740687701 packets input, 11361166074444 bytes, 0 no buffer
     Received 21941 broadcasts (21941 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 21941 multicast, 0 pause input
     0 input packets with dribble condition detected
     5148426521 packets output, 3809604761401 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out