9500-40X output drops w/ no output errors

Joel.Benson · ‎10-30-2019

Folks, first off thanks for taking the time, I have an interesting issue,

I have constant burst output drops only on the 9500-40X in VSS, on multiple interfaces even when they are not being over utilized, ie a L-3 2G PO only using 2-10mbps, and mulitple 20G POs, and other 10g interfaces that are dropping, still with no output errors only discards, which makes me think output queue is dropping them but it still doesn't make sense to me why links that are using maybe 10-15% have output queue drops. TAC gave us the solution of this below, reduced drops but increased latency from 5ms to 400ms, Tac is still investigating but taking their time. any ideas?

Switch version 16.9.2

transceivers being used match on both sides of fiber path, no physical errors have been seen.

SFP-10G-LR-CURV

SSFP-10G-SR-S

1000_GLS-SX-MM

Once again thank you for the time.

________________________________

qos queue-softmax-multiplier 1200

class-map match-any TACTEST
match access-group name TACTEST

policy-map TACTEST
class class-default
bandwidth percent 100

__________________________________

Port-channel5 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0cd0.f8dc.42ec (bia 0cd0.f8dc.42ec)
Description: To mesh router_g0/0/4-g0/0/5
Internet address is XX
MTU 9198 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Te1/0/22 Te2/0/22
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:00, output hang never
Last clearing of "show interface" counters 02:13:15
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 14289760
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2603000 bits/sec, 1888 packets/sec
5 minute output rate 19504000 bits/sec, 2740 packets/sec
6395387 packets input, 1286950518 bytes, 0 no buffer
Received 8115 broadcasts (123 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 8115 multicast, 0 pause input
0 input packets with dribble condition detected
11475715 packets output, 10367451498 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Joseph W. Doherty · ‎10-30-2019

Sounds like transient microbursts. If you, or other readers are unfamiliar with microbursts, you might start by reading: https://www.ntop.org/ntop/what-is-a-microburst-and-how-to-detect-it/ and https://www.arista.com/assets/data/pdf/TechBulletins/AristaMicrobursts.pdf.

As to what might be done to mitigate, one common technique is to increase buffer capacity, as already suggested by Cisco. (As you've noticed, though, a common side effect is adding additional latency due to queuing.)

Beyond that, it gets more difficult, especially if only using a Cisco switch because their QoS support is much weaker than a Cisco router's.

Probably the "easiest" and most effective solution might be obtained by using a 3rd party traffic management appliance, which for traffic like TCP, can manage an individual flow's return ACK rate and/or manage a receiver's RWIN.

Can you do anything on the switch itself? Perhaps, but it's "hard core", and painful to implement and/or monitor. Again, you're very limited on a switch like yours (which, I believe, doesn't offer other higher end Cisco switch QoS features like microflow policing, DBL or FRED.)