WS-C3850-48U - High output drops

[darren-carr]

Hi community,

We are experiencing a high number of output drops on several port-channels connected to a stack of core switches at a branch site. Specifically connected to our wireless controllers (Aruba) which are the primary access medium for the site. 

We are running Denali 16.3.8. 

We are receiving complaints regarding intermittent performance issues of SaaS services (O365, etc). We have had our ISP and SaaS providers perform the usual checks on their side and they are stating there are no issues. I'd like to rule out this being the cause of the problem. Any idea on how to troubleshoot this issue? I've seen various previous threads regarding adjusting queue settings, etc but these appear related to older versions of code. Any suggestions are welcome.

Thanks

Port-channel2 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 2c86.d236.e003 (bia 2c86.d236.e003)
Description: xxxxxxx
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is
input flow-control is on, output flow-control is unsupported
Members in this channel: Gi1/0/3 Gi2/0/3
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/7 (size/max/drops/flushes); Total output drops: 68813807
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 338000 bits/sec, 114 packets/sec
5 minute output rate 335000 bits/sec, 123 packets/sec
2887976157 packets input, 2566644339760 bytes, 0 no buffer
Received 9176578 broadcasts (5726925 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5726925 multicast, 0 pause input
0 input packets with dribble condition detected
2986681209 packets output, 2514893673438 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Reza Sharifi]

Hi,

For po2, Are both physical interfaces showing output drops as well or just one of them?

HTH

[darren-carr]

Hi,

It is both interfaces. However, one is considerably more than the other. I suspect this is because of the hashing we are using to load-balance the traffic over the two interfaces.

xxxx#sh int gi 1/0/3 | inc drops
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 68329073
0 unknown protocol drops
xxxx#sh int gi 2/0/3 | inc drops
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 491974
0 unknown protocol drops

[Georg Pauwen]

Hello,

I have looked around, and apparently there was a similar case where TAC recommended applying the service policy below to the member interfaces of the port channel:

qos queue-softmax-multiplier 1200
!
class-map match-any OUTPUT_CLASS
match access-group name OUTPUT_ACL
!
policy-map OUTPUT_POLICY
class class-default
bandwidth percent 100
!
interface GigabitEthernet1/0/3
service-policy output OUTPUT_POLICY
!
interface GigabitEthernet2/0/3
service-policy output OUTPUT_POLICY
!
ip access-list extended OUTPUT_ACL
permit ip any any

Also, what are you using for load balancing (src-mac is the default) ?

You could also try to configure extended load balancing which gives you the option to specify multiple criteria, e.g.:

3850(config)# port-channel load-balance extended src-mac dst-mac

[Mark Malone]

Hi
if im calculating that right its nearly a 44% drop rate for traffic through that interface
did this only start after an upgrade ? what was the trigger that started this off ?
I have Aruba WLCs 2 of them but i have them split onto 4500x 10GBs about 50aps on site but i dont see anything like this , theres no 10GB module in this switch you could move them over to ?

have you ran the show tech yet through the cli analyzer to see of there are any specific issues with the software

[darren-carr]

Hi Mark,

I've ran the output of 'show tech' through the CLI analyser, nothing out of the ordinary came out of this analysis with regards to the software version we are using. 

Also, the Aruba controllers are 7030s which don't support 10Gbps. We only very rarely see throughput over 70Mbps.

Thanks

[Giuseppe Larosa]

Hello Mark,

duplicated post please ignore

Best Regards

Giuseppe

[Giuseppe Larosa]

Hello Mark,

the drop rate calculated on the first show interface in this thread is

0,02252132850476231966467066231994 = output drops / (output drops + output packets)

2,25 % high but not very high.

All your other considerations are wise and valid

Hope to help

Giuseppe

[Mark Malone]

Oh yep sorry my bad on the calculations, thanks Giuseppe

[Joseph W. Doherty]

Discounting bugs and/or less than optimal Etherchannel hashing, have you seen: https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html?

[darren-carr]

Hi,

Is this configuration still relevant in newer releases of code? we are running 16.3.8

We have also logged a TAC case with Cisco, waiting to hear back now

[Joseph W. Doherty]

"Is this configuration still relevant in newer releases of code? we are running 16.3.8"

TAC should be to better answer that, however sometimes tuning recommendations holds across multiple versions, and sometimes not, the latter depending what Cisco may have changed in a later version.