3850 Output Drops

[garrybe]

I am currently investigating the cause of consistent output drops on certain interfaces in a port-channel. Only one of the two links is showing drops and the one that is having the issue is steadily increasing the OQD counter. Note that this is only affecting one interface on one switch. The connected switch is showing no errors. I have tried changing the LACP load-balancing method to src-dst-ip with little noticeable difference. I have also tried changing the physical switch port which caused the other interface in the port-channel to start exhibiting the output drops. From my research, I'm pretty sure this is due to link saturation but I'm not convinced simply adding more interfaces in the port-channel will resolve the issue as only one of the two interfaces seems to be exhibiting the problem as it is. There is no QoS. See output below.

Switch#sh int g 3/1/4
GigabitEthernet3/1/4 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is f09e.63de.bca4 (bia f09e.63de.bca4)
Description: Uplink
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 7/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:08, output 00:00:06, output hang never
Last clearing of "show interface" counters 15:52:28
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 40758850
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 14330000 bits/sec, 1541 packets/sec
5 minute output rate 28650000 bits/sec, 3001 packets/sec
68023289 packets input, 84530141025 bytes, 0 no buffer
Received 214579 broadcasts (102315 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 102315 multicast, 0 pause input
0 input packets with dribble condition detected
100475684 packets output, 115117741773 bytes, 0 underruns
Output 86137 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Switch#sh controllers e g 3/1/4
Transmit GigabitEthernet3/1/4 Receive
111041089260 Total bytes 86295237766 Total bytes
94644291 Unicast frames 69247620 Unicast frames
110370792500 Unicast bytes 86256672759 Unicast bytes
2771970 Multicast frames 106520 Multicast frames
642938784 Multicast bytes 28376139 Multicast bytes
231265 Broadcast frames 124374 Broadcast frames
27358582 Broadcast bytes 10188868 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
3150932 Minimum size frames 5471952 Minimum size frames
16648356 65 to 127 byte frames 1573814 65 to 127 byte frames
2657069 128 to 255 byte frames 3467814 128 to 255 byte frames
1007953 256 to 511 byte frames 2298823 256 to 511 byte frames
970900 512 to 1023 byte frames 1828666 512 to 1023 byte frames
73212012 1024 to 1518 byte frames 15659014 1024 to 1518 byte frames
304 1519 to 2047 byte frames 39178431 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
37083764 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 213 msecs AGO

[Joseph W. Doherty]

"There is no QoS."

I recall (?) the 3650/3850s "do" QoS by default, and you might not be unable to disable it.  If so, their "default" QoS often is far from ideal in many use cases.

A single gig flow will fill any gig port channel link.  Any additional traffic on that link may cause drops.  (Remember, Etherchannel doesn't split a flow across links nor pay any attention to how busy a link is for assigning flows to a link.)

So, yes, it's certainly possible occasionally one of your port channel links gets oversubscribed and begins to drop packets.  As you further describe it's always the same link, as flows to Etherchannel links mapping is deterministic, that may indicate a specific host's flow(s) is the root problem.

"QoS" tuning might help, because such tuning can reallocate buffer resource allocations.  Checking the QoS stats would be worthwhile.

Understand, though, buffer tuning can often mitigate transient congestion (e.g. microbursts), but not long term congestion.

Regarding whether additional Etherchannel links would help, possibly, if the issue is a single gig flow filling a gig link, you slightly increase the odds other traffic will be distributed onto other links, but that's not the same as insuring no additional traffic is directed to a fully saturated link.