cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

367
Views
5
Helpful
11
Replies
Highlighted
Beginner

WS-C3850-48U - High output drops

Hi community,

 

We are experiencing a high number of output drops on several port-channels connected to a stack of core switches at a branch site. Specifically connected to our wireless controllers (Aruba) which are the primary access medium for the site. 

 

We are running Denali 16.3.8. 

 

We are receiving complaints regarding intermittent performance issues of SaaS services (O365, etc). We have had our ISP and SaaS providers perform the usual checks on their side and they are stating there are no issues. I'd like to rule out this being the cause of the problem. Any idea on how to troubleshoot this issue? I've seen various previous threads regarding adjusting queue settings, etc but these appear related to older versions of code. Any suggestions are welcome.

 

Thanks

 

Port-channel2 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 2c86.d236.e003 (bia 2c86.d236.e003)
Description: xxxxxxx
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is
input flow-control is on, output flow-control is unsupported
Members in this channel: Gi1/0/3 Gi2/0/3
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/7 (size/max/drops/flushes); Total output drops: 68813807
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 338000 bits/sec, 114 packets/sec
5 minute output rate 335000 bits/sec, 123 packets/sec
2887976157 packets input, 2566644339760 bytes, 0 no buffer
Received 9176578 broadcasts (5726925 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5726925 multicast, 0 pause input
0 input packets with dribble condition detected
2986681209 packets output, 2514893673438 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

11 REPLIES 11
Highlighted
Hall of Fame Expert

Re: WS-C3850-48U - High output drops

Hi,

For po2, Are both physical interfaces showing output drops as well or just one of them?

HTH

Highlighted
Beginner

Re: WS-C3850-48U - High output drops

Hi,

 

It is both interfaces. However, one is considerably more than the other. I suspect this is because of the hashing we are using to load-balance the traffic over the two interfaces.

 

xxxx#sh int gi 1/0/3 | inc drops
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 68329073
0 unknown protocol drops
xxxx#sh int gi 2/0/3 | inc drops
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 491974
0 unknown protocol drops

Highlighted
VIP Mentor

Re: WS-C3850-48U - High output drops

Hello,

 

I have looked around, and apparently there was a similar case where TAC recommended applying the service policy below to the member interfaces of the port channel:

 

qos queue-softmax-multiplier 1200
!
class-map match-any OUTPUT_CLASS
match access-group name OUTPUT_ACL
!
policy-map OUTPUT_POLICY
class class-default
bandwidth percent 100
!
interface GigabitEthernet1/0/3
service-policy output OUTPUT_POLICY
!
interface GigabitEthernet2/0/3
service-policy output OUTPUT_POLICY
!
ip access-list extended OUTPUT_ACL
permit ip any any

 

Also, what are you using for load balancing (src-mac is the default) ?

 

You could also try to configure extended load balancing which gives you the option to specify multiple criteria, e.g.:

 

3850(config)# port-channel load-balance extended src-mac dst-mac

Highlighted
VIP Mentor

Re: WS-C3850-48U - High output drops

Hi
if im calculating that right its nearly a 44% drop rate for traffic through that interface
did this only start after an upgrade ? what was the trigger that started this off ?
I have Aruba WLCs 2 of them but i have them split onto 4500x 10GBs about 50aps on site but i dont see anything like this , theres no 10GB module in this switch you could move them over to ?

have you ran the show tech yet through the cli analyzer to see of there are any specific issues with the software
Highlighted
Beginner

Re: WS-C3850-48U - High output drops

Hi Mark,

 

I've ran the output of 'show tech' through the CLI analyser, nothing out of the ordinary came out of this analysis with regards to the software version we are using. 

 

Also, the Aruba controllers are 7030s which don't support 10Gbps. We only very rarely see throughput over 70Mbps.

 

Thanks

Highlighted
Hall of Fame Master

Re: WS-C3850-48U - High output drops

Hello Mark,

duplicated post please ignore

 

Best Regards

Giuseppe

 

Highlighted
Hall of Fame Master

Re: WS-C3850-48U - High output drops

Hello Mark,

the drop rate calculated on the first show interface in this thread is

0,02252132850476231966467066231994 = output drops / (output drops + output packets)

2,25 % high but not very high.

All your other considerations are wise and valid

 

Hope to help

Giuseppe

 

Highlighted
VIP Mentor

Re: WS-C3850-48U - High output drops

Oh yep sorry my bad on the calculations, thanks Giuseppe
Highlighted
VIP Expert

Re: WS-C3850-48U - High output drops

Highlighted
Beginner

Re: WS-C3850-48U - High output drops

Hi,

 

Is this configuration still relevant in newer releases of code? we are running 16.3.8

 

We have also logged a TAC case with Cisco, waiting to hear back now

Highlighted
VIP Expert

Re: WS-C3850-48U - High output drops

"Is this configuration still relevant in newer releases of code? we are running 16.3.8"

TAC should be to better answer that, however sometimes tuning recommendations holds across multiple versions, and sometimes not, the latter depending what Cisco may have changed in a later version.
CreatePlease to create content
Content for Community-Ad