Re: Output drop

salemmahara · ‎01-20-2019

Hello everybody

We have a couple of 3850 (Stacked) and we use it in our aggregation layer.

ports on this switches are bundled in port-channels to downstream switches

More than half of port are free(not connected for future).

As you know, 3850 series are really powerful and meanwhile when you stack them, their capabilities will increase.

but here are an output of one of port-channel interfaces which looks really strange:

https://pitstop.manageengine.com/portal/community/topic/report-and-print-problems

Hardware is EtherChannel, address is ---------- (bia -------------)
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/0/24 Gi2/0/24
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:13:28, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 562303015
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 153000 bits/sec, 134 packets/sec
5 minute output rate 2330000 bits/sec, 325 packets/sec
     127201005 packets input, 44246365237 bytes, 0 no buffer
     Received 1266947 broadcasts (903818 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 903818 multicast, 0 pause input
     0 input packets with dribble condition detected
     380610888 packets output, 410883103982 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Its IOS EX is almost updated (3.6). no buffer failures, no error or ,,,, but Output drop. this interfaces are 1G but their utilization is at most 30% ant lower. some of them utilize even under 10% . Switches are stacked and most of their interfaces are not connected so there isn't any limitation in buffer or ...

Leo Laohoo · ‎01-20-2019

@salemmahara wrote:

Members in this channel: Gi1/0/24 Gi2/0/24

Post the complete output to the command "sh interface G1/0/24".

salemmahara · ‎01-21-2019

Hi @Leo Laohoo

Thanks for replying

I'll send the output as soon as possible . interface counters are already cleared.

MUHAMMAD TAYYAB MUNIR · ‎01-21-2019

Hi Salem,

Please share the below outputs including show version;

show platform qos queue config gigabitEthernet 1/0/x switch x
show platform qos queue stats gigabitEthernet 1/0/x switch x

Regards,

Tayyab - www.tayyabmunir.com

*** Please rate ***

*** Please rate all helpful responses and mark solutions***

balaji.bandi · ‎01-21-2019

adding to other posts.

below guide help you more troubleshotting.

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

salemmahara · ‎01-21-2019

Hello again @balaji.bandi @MUHAMMAD TAYYAB MUNIR @Leo Laohoo

Thanks for replying

Here is the output of another interface:

show int port 12

Port-channel12 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0056.2bf9.028b (bia 0056.2bf9.028b)
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/0/32 Gi2/0/32
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:10:04, output never, output hang never
Last clearing of "show interface" counters 01:20:06
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 6652936
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1185000 bits/sec, 301 packets/sec
5 minute output rate 2815000 bits/sec, 427 packets/sec
     2210810 packets input, 973025553 bytes, 0 no buffer
     Received 25082 broadcasts (9802 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 9802 multicast, 0 pause input
     0 input packets with dribble condition detected
     3296599 packets output, 3404563408 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

show int gig 1/0/32

GigabitEthernet1/0/32 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0056.2b2d.270b (bia 0056.2b2d.270b)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output never, output hang never
Last clearing of "show interface" counters 01:20:22
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 157000 bits/sec, 173 packets/sec
5 minute output rate 2586000 bits/sec, 339 packets/sec
     693131 packets input, 88077262 bytes, 0 no buffer
     Received 20543 broadcasts (6147 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 6147 multicast, 0 pause input
     0 input packets with dribble condition detected
     2524051 packets output, 2780826626 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

show int gig 2/0/32

GigabitEthernet2/0/11 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0056.2bf9.028b (bia 0056.2bf9.028b)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:21, output never, output hang never
Last clearing of "show interface" counters 01:20:25
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 6652936
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 993000 bits/sec, 124 packets/sec
5 minute output rate 159000 bits/sec, 72 packets/sec
     1521097 packets input, 885331485 bytes, 0 no buffer
     Received 4600 broadcasts (3678 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 3678 multicast, 0 pause input
     0 input packets with dribble condition detected
     778683 packets output, 627283514 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Note : We monitored interfaces . The most traffic passing them was 20-30M. They most interface utilization is 0.9% !

I can't understand it! An interface without any overflow, no heavy traffic , no .... and big output drop! If it's a 1G interface it's supposed to handle traffic without any drop till it get full load or above. 35Mb traffic on a Gig interface and millions of output drops :) .

MUHAMMAD TAYYAB MUNIR · ‎01-21-2019

Hi Salem,

If possible, please share following outputs as well!

show version

show platform qos queue config gigabitEthernet 1/0/32 switch 1
show platform qos queue stats gigabitEthernet 1/0/32 switch 1

!

show platform qos queue config gigabitEthernet 2/0/32 switch 2
show platform qos queue stats gigabitEthernet 2/0/32 switch 2

BR,

Tayyab

*** Please rate all helpful responses and mark solutions***

salemmahara · ‎01-21-2019

show platform qos queue config gig 1/0/32

DATA Port:19 GPN:626 AFD:Disabled QoSMap:0 HW Queues: 152 - 159
DrainFast:Disabled PortSoftStart:1 - 1080
----------------------------------------------------------
DTS Hardmax   Softmax PortSMin GlblSMin PortStEnd
--- -------- -------- -------- --------- ---------
0   1 5   120 6   480 6   320   0     0   3 1440
1   1 4     0 7   720 3   480   2   180   3 1440
2   1 4     0 5     0 5     0   0     0   3 1440
3   1 4     0 5     0 5     0   0     0   3 1440
4   1 4     0 5     0 5     0   0     0   3 1440
5   1 4     0 5     0 5     0   0     0   3 1440
6   1 4     0 5     0 5     0   0     0   3 1440
7   1 4     0 5     0 5     0   0     0   3 1440
Priority   Shaped/shared   weight shaping_step
--------   ------------   ------ ------------
0      0     Shared            50           0
1      0     Shared            75           0
2      0     Shared         10000          34
3      0     Shared         10000           0
4      0     Shared         10000           0
5      0     Shared         10000           0
6      0     Shared         10000         120
7      0     Shared         10000           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
   ------- ------- ------ ------ ------ ------ ------ ------ ------
0      0     478       0       0     534       0       0     600       0
1      0     573       0       0     641       0       0     720       0
2      0       0       0       0       0       0       0       0       0
3      0       0       0       0       0       0       0       0       0
4      0       0       0       0       0       0       0       0       0
5      0       0       0       0       0       0       0       0       0
6      0       0       0       0       0       0       0       0       0
7      0       0       0       0       0       0       0       0       0

show platform qos queue stats gig 1/0/32

DATA Port:19 Enqueue Counters
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
    0       0           0           0    53115037
    1       0           0           0 22135356198
    2       0           0           0           0
    3       0           0           0           0
    4       0           0           0           0
    5       0           0           0           0
    6       0           0           0           0
    7       0           0           0           0
DATA Port:19 Drop Counters
-------------------------------
Queue Drop-TH0    Drop-TH1    Drop-TH2    SBufDrop    QebDrop
----- ----------- ----------- ----------- ----------- -----------
    0           0           0           0           0           0
    1           0           0    43900746           0           0
    2           0           0           0           0           0
    3           0           0           0           0           0
    4           0           0           0           0           0
    5           0           0           0           0           0
    6           0           0           0           0           0
    7           0           0           0           0           0
AQM Broadcast Early WTD COUNTERS(In terms of Bytes)
--------------------------------------------------
PORT TYPE          ENQUEUE             DROP
--------------------------------------------------
UPLINK PORT-0        N/A               0
UPLINK PORT-1        N/A               0
UPLINK PORT-2        N/A               0
UPLINK PORT-3        N/A               0
NETWORK PORTS   488070962          943137028
RCP PORTS               0                  0
CPU PORT                0                  0
Note: Queuing stats are in bytes

show platform qos queue config gig 2/0/32

DATA Port:19 GPN:627 AFD:Disabled QoSMap:0 HW Queues: 152 - 159
DrainFast:Disabled PortSoftStart:1 - 1080
----------------------------------------------------------
DTS Hardmax   Softmax PortSMin GlblSMin PortStEnd
--- -------- -------- -------- --------- ---------
0   1 5   120 6   480 6   320   0     0   3 1440
1   1 4     0 7   720 3   480   2   180   3 1440
2   1 4     0 5     0 5     0   0     0   3 1440
3   1 4     0 5     0 5     0   0     0   3 1440
4   1 4     0 5     0 5     0   0     0   3 1440
5   1 4     0 5     0 5     0   0     0   3 1440
6   1 4     0 5     0 5     0   0     0   3 1440
7   1 4     0 5     0 5     0   0     0   3 1440
Priority   Shaped/shared   weight shaping_step
--------   ------------   ------ ------------
0      0     Shared            50           0
1      0     Shared            75           0
2      0     Shared         10000         148
3      0     Shared         10000           0
4      0     Shared         10000           0
5      0     Shared         10000           0
6      0     Shared         10000         120
7      0     Shared         10000           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
   ------- ------- ------ ------ ------ ------ ------ ------ ------
0      0     478       0       0     534       0       0     600       0
1      0     573       0       0     641       0       0     720       0
2      0       0       0       0       0       0       0       0       0
3      0       0       0       0       0       0       0       0       0
4      0       0       0       0       0       0       0       0       0
5      0       0       0       0       0       0       0       0       0
6      0       0       0       0       0       0       0       0       0
7      0       0       0       0       0       0       0       0       0

show platform qos queue stats gig 2/0/32

DATA Port:19 Enqueue Counters
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
    0       0           0           0 1405065616
    1       0           0           0 1547787037
    2       0           0           0           0
    3       0           0           0           0
    4       0           0           0           0
    5       0           0           0           0
    6       0           0           0           0
    7       0           0           0           0
DATA Port:19 Drop Counters
-------------------------------
Queue Drop-TH0    Drop-TH1    Drop-TH2    SBufDrop    QebDrop
----- ----------- ----------- ----------- ----------- -----------
    0           0           0           0           0           0
    1           0           0    87429447           0           0
    2           0           0           0           0           0
    3           0           0           0           0           0
    4           0           0           0           0           0
    5           0           0           0           0           0
    6           0           0           0           0           0
    7           0           0           0           0           0
AQM Broadcast Early WTD COUNTERS(In terms of Bytes)
--------------------------------------------------
PORT TYPE          ENQUEUE             DROP
--------------------------------------------------
UPLINK PORT-0        N/A               0
UPLINK PORT-1        N/A               0
UPLINK PORT-2        N/A               0
UPLINK PORT-3        N/A               0
NETWORK PORTS       82938              82938
RCP PORTS               0                  0
CPU PORT                0                  0
Note: Queuing stats are in bytes

Georg Pauwen · ‎01-21-2019

Hello,

try and configure:

qos queue-softmax-multiplier 1200

globally on the switch that has the output drops...

MUHAMMAD TAYYAB MUNIR · ‎01-21-2019

@@Georg Pauwen you guys are awesome :D providing the solutions without checking the queueing outputs and providing the correct commands which will not work unless reloading the switch.

But I don't propose any solution until full investigation. Carry on :P

BR

Tayyab

*** Please rate all helpful responses and mark solutions***

salemmahara · ‎01-21-2019

Do you think we need to increase buffer size for 0 and 1 queue ? If yes, Why there is not any buffer failure and ... in show interface output ?

It's great if you could analyze the output as a practical case . It would be helpful for those with the similar issue .

Leo Laohoo · ‎01-21-2019

CSCvb31906

salemmahara · ‎01-21-2019

Thank you Leo

We're using 3.6.8 so it should be solved in this release

MUHAMMAD TAYYAB MUNIR · ‎01-21-2019

Hi Salem,

Thank you for sharing the outputs.

1)

Short explanation: An Output Queue Drop is a packet that was about to leave the switch, but the latter detected temporary congestion on the outgoing link so it was stored in a buffer for a short amount of time. If the switch detected the link remained congested, the packet will be dropped and the Output Drop counter will increase. Output drops are generally a result of interface oversubscription caused by many to one or a 1Gig transfer and etc. Interface buffers are a limited resource and can only absorb a burst up to a point after which packets will drop.

2) Also note “Total output drops” counter on 3850/3650/9K switch is representing bytes, not packets.

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 562303015 --> Drops in bytes

380610888 packets output, 410883103982 bytes, 0 underruns --> 410883103982 total output bytes sent

So, the actual drop rate is around 1.xx*1, which is quite low, that these drops happened on a non-priority queue.

3) Also make sure that the softmax is configured to the following value:

conf t

qos queue-softmax-multiplier 600

(current value is 100)

Apply the workaround if the issue still exists, please upgrade the switch to 03.07.04

or

Description :	CAT3850/3650 UNIVERSAL
Release :	3.7.5E
Release Date :	15-Feb-2017
FileName :	cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin
Min Memory :	DRAM 4096 Flash 2048
Size :	313.39 MB ( 328618032 bytes)
MD5 Checksum :	d6f1f24a30dc67697a0c815661ba670c

Regards,

Tayyab - www.tayyabmunir.com

*** Rate if Response Helpful ***

*** Please rate all helpful responses and mark solutions***

salemmahara · ‎01-21-2019

Hello @MUHAMMAD TAYYAB MUNIR

Thanks a million for your detailed reply

Do you confirm below information ( What I'v understood ) ?

1) Output drop rate is : (562303015 / 410883103982) * 100 = 0.13% which means reasonable (Less than 1 percent) or (562303015 / 410883103982) * 1000 = 1.3 packet within 1000 .

2) By applying the command Qos queue-softmax-multiplier 600 we will increase the buffer size multiplied by 6 which means 300*6*256= 450KB for each interface. Right ? Does it effect on TenGig interfaces too? I have some TenGig interfaces which are working fine with their buffer size without any drop.

Does this command need reload?

3) When we analyze the graphs, no interfaces touch the maximum BW . So why a 1G interface fails to transfer 30-40Mb data with output drop? Only because of that we are using them as uplink( One to many ) ? But 3850 series are supposed to be used in aggregation layer ! Even at core in environments with lower data rate.

BTW , I want to share another thing with you . Our 3850s are connected to downstream switches which their uplink ports are SFP+ but we use GLC-T(1G) for converting Fibber to copper . But I think it doesn't effect the output drop. I have similar experience in other projects without any output drop. What is your idea ?