cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
578
Views
130
Helpful
31
Replies
Beginner

Output drop

Hello everybody

We have a couple of 3850 (Stacked) and we use it in our aggregation layer.

ports on this switches are bundled in port-channels to downstream switches

More than half of port are free(not connected for future).

As you know, 3850 series are really powerful and meanwhile when you stack them, their capabilities will increase.

but here are an output of one of port-channel interfaces which looks really strange:

https://pitstop.manageengine.com/portal/community/topic/report-and-print-problems

  Hardware is EtherChannel, address is ---------- (bia -------------)
  MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, link type is auto, media type is
  input flow-control is off, output flow-control is unsupported
  Members in this channel: Gi1/0/24 Gi2/0/24
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:13:28, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 562303015
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 153000 bits/sec, 134 packets/sec
  5 minute output rate 2330000 bits/sec, 325 packets/sec
     127201005 packets input, 44246365237 bytes, 0 no buffer
     Received 1266947 broadcasts (903818 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 903818 multicast, 0 pause input
     0 input packets with dribble condition detected
     380610888 packets output, 410883103982 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

Its IOS EX is almost updated (3.6). no buffer failures, no error or ,,,, but Output drop. this interfaces are 1G but their utilization is at most 30% ant lower. some of them utilize even under 10% . Switches are stacked and most of their interfaces are not connected so there isn't any limitation in buffer or ...

31 REPLIES 31
Hall of Fame Community Legend

Re: Output drop


@salemmahara wrote:

 Members in this channel: Gi1/0/24 Gi2/0/24


Post the complete output to the command "sh interface G1/0/24".

Beginner

Re: Output drop

 

Hi @Leo Laohoo

Thanks for replying

I'll send the output as soon as possible . interface counters are already cleared.

Beginner

Re: Output drop

Hi Salem,

 

Please share the below outputs including show version;

 

show platform qos queue config gigabitEthernet 1/0/x switch x
show platform qos queue stats gigabitEthernet 1/0/x switch x

 

Regards,

Tayyab - www.tayyabmunir.com

 

*** Please rate ***

*** Please rate all helpful responses and mark solutions***
VIP Advisor

Re: Output drop

adding to other posts. 

 

below guide help you more troubleshotting.

 

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html

BB
*** Rate All Helpful Responses ***
Beginner

Re: Output drop

Hello again @balaji.bandi @TAYYAB MUNIR @Leo Laohoo

Thanks for replying

Here is the output of another interface:

 

show int port 12

Port-channel12 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 0056.2bf9.028b (bia 0056.2bf9.028b)
  MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, link type is auto, media type is
  input flow-control is off, output flow-control is unsupported
  Members in this channel: Gi1/0/32 Gi2/0/32
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:10:04, output never, output hang never
  Last clearing of "show interface" counters 01:20:06
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 6652936
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1185000 bits/sec, 301 packets/sec
  5 minute output rate 2815000 bits/sec, 427 packets/sec
     2210810 packets input, 973025553 bytes, 0 no buffer
     Received 25082 broadcasts (9802 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 9802 multicast, 0 pause input
     0 input packets with dribble condition detected
     3296599 packets output, 3404563408 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

show int gig 1/0/32

 

GigabitEthernet1/0/32 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 0056.2b2d.270b (bia 0056.2b2d.270b)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:02, output never, output hang never
  Last clearing of "show interface" counters 01:20:22
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 157000 bits/sec, 173 packets/sec
  5 minute output rate 2586000 bits/sec, 339 packets/sec
     693131 packets input, 88077262 bytes, 0 no buffer
     Received 20543 broadcasts (6147 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 6147 multicast, 0 pause input
     0 input packets with dribble condition detected
     2524051 packets output, 2780826626 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

show int gig 2/0/32

 

GigabitEthernet2/0/11 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 0056.2bf9.028b (bia 0056.2bf9.028b)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:21, output never, output hang never
  Last clearing of "show interface" counters 01:20:25
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 6652936
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 993000 bits/sec, 124 packets/sec
  5 minute output rate 159000 bits/sec, 72 packets/sec
     1521097 packets input, 885331485 bytes, 0 no buffer
     Received 4600 broadcasts (3678 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 3678 multicast, 0 pause input
     0 input packets with dribble condition detected
     778683 packets output, 627283514 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

Note : We monitored interfaces . The most traffic passing them was 20-30M. They most interface utilization is 0.9% !

 

I can't understand it! An interface without any overflow, no heavy traffic , no .... and big output drop! If it's a 1G interface it's supposed to handle traffic without any drop till it get full load or above. 35Mb traffic on a Gig interface and millions of output drops :) .

 

 

 

 

Beginner

Re: Output drop

Hi Salem,

 

If possible, please share following outputs as well!

 

show version

show platform qos queue config gigabitEthernet 1/0/32 switch 1
show platform qos queue stats gigabitEthernet 1/0/32 switch 1

!

show platform qos queue config gigabitEthernet 2/0/32 switch 2
show platform qos queue stats gigabitEthernet 2/0/32 switch 2

 

BR,

Tayyab

*** Please rate all helpful responses and mark solutions***
Beginner

Re: Output drop

show platform qos queue config gig 1/0/32

 

DATA Port:19 GPN:626 AFD:Disabled QoSMap:0 HW Queues: 152 - 159
  DrainFast:Disabled PortSoftStart:1 - 1080
----------------------------------------------------------
  DTS Hardmax   Softmax  PortSMin GlblSMin  PortStEnd
  --- --------  -------- -------- --------- ---------
 0   1  5   120  6   480  6   320   0     0   3  1440
 1   1  4     0  7   720  3   480   2   180   3  1440
 2   1  4     0  5     0  5     0   0     0   3  1440
 3   1  4     0  5     0  5     0   0     0   3  1440
 4   1  4     0  5     0  5     0   0     0   3  1440
 5   1  4     0  5     0  5     0   0     0   3  1440
 6   1  4     0  5     0  5     0   0     0   3  1440
 7   1  4     0  5     0  5     0   0     0   3  1440
 Priority   Shaped/shared   weight  shaping_step
 --------   ------------   ------  ------------
 0      0     Shared            50           0
 1      0     Shared            75           0
 2      0     Shared         10000          34
 3      0     Shared         10000           0
 4      0     Shared         10000           0
 5      0     Shared         10000           0
 6      0     Shared         10000         120
 7      0     Shared         10000           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
   ------- -------  ------  ------  ------  ------  ------  ------ ------
 0      0     478       0       0     534       0       0     600       0
 1      0     573       0       0     641       0       0     720       0
 2      0       0       0       0       0       0       0       0       0
 3      0       0       0       0       0       0       0       0       0
 4      0       0       0       0       0       0       0       0       0
 5      0       0       0       0       0       0       0       0       0
 6      0       0       0       0       0       0       0       0       0
 7      0       0       0       0       0       0       0       0       0

 

show platform qos queue stats gig 1/0/32

 

DATA Port:19 Enqueue Counters             
-------------------------------             
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2             
----- ------- ----------- ----------- -----------
    0       0           0           0    53115037
    1       0           0           0 22135356198
    2       0           0           0           0
    3       0           0           0           0
    4       0           0           0           0
    5       0           0           0           0
    6       0           0           0           0
    7       0           0           0           0
DATA Port:19 Drop Counters              
-------------------------------              
Queue Drop-TH0    Drop-TH1    Drop-TH2    SBufDrop    QebDrop              
----- ----------- ----------- ----------- ----------- -----------
    0           0           0           0           0           0
    1           0           0    43900746           0           0
    2           0           0           0           0           0
    3           0           0           0           0           0
    4           0           0           0           0           0
    5           0           0           0           0           0
    6           0           0           0           0           0
    7           0           0           0           0           0
 AQM Broadcast Early WTD COUNTERS(In terms of Bytes)
--------------------------------------------------
  PORT TYPE          ENQUEUE             DROP
--------------------------------------------------
 UPLINK PORT-0        N/A               0
 UPLINK PORT-1        N/A               0
 UPLINK PORT-2        N/A               0
 UPLINK PORT-3        N/A               0
 NETWORK PORTS   488070962          943137028
 RCP PORTS               0                  0
 CPU PORT                0                  0
Note: Queuing stats are in bytes

show platform qos queue config gig 2/0/32

DATA Port:19 GPN:627 AFD:Disabled QoSMap:0 HW Queues: 152 - 159
  DrainFast:Disabled PortSoftStart:1 - 1080
----------------------------------------------------------
  DTS Hardmax   Softmax  PortSMin GlblSMin  PortStEnd
  --- --------  -------- -------- --------- ---------
 0   1  5   120  6   480  6   320   0     0   3  1440
 1   1  4     0  7   720  3   480   2   180   3  1440
 2   1  4     0  5     0  5     0   0     0   3  1440
 3   1  4     0  5     0  5     0   0     0   3  1440
 4   1  4     0  5     0  5     0   0     0   3  1440
 5   1  4     0  5     0  5     0   0     0   3  1440
 6   1  4     0  5     0  5     0   0     0   3  1440
 7   1  4     0  5     0  5     0   0     0   3  1440
 Priority   Shaped/shared   weight  shaping_step
 --------   ------------   ------  ------------
 0      0     Shared            50           0
 1      0     Shared            75           0
 2      0     Shared         10000         148
 3      0     Shared         10000           0
 4      0     Shared         10000           0
 5      0     Shared         10000           0
 6      0     Shared         10000         120
 7      0     Shared         10000           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
   ------- -------  ------  ------  ------  ------  ------  ------ ------
 0      0     478       0       0     534       0       0     600       0
 1      0     573       0       0     641       0       0     720       0
 2      0       0       0       0       0       0       0       0       0
 3      0       0       0       0       0       0       0       0       0
 4      0       0       0       0       0       0       0       0       0
 5      0       0       0       0       0       0       0       0       0
 6      0       0       0       0       0       0       0       0       0
 7      0       0       0       0       0       0       0       0       0

 

show platform qos queue stats gig 2/0/32

 

DATA Port:19 Enqueue Counters             
-------------------------------             
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2             
----- ------- ----------- ----------- -----------
    0       0           0           0  1405065616
    1       0           0           0  1547787037
    2       0           0           0           0
    3       0           0           0           0
    4       0           0           0           0
    5       0           0           0           0
    6       0           0           0           0
    7       0           0           0           0
DATA Port:19 Drop Counters              
-------------------------------              
Queue Drop-TH0    Drop-TH1    Drop-TH2    SBufDrop    QebDrop              
----- ----------- ----------- ----------- ----------- -----------
    0           0           0           0           0           0
    1           0           0    87429447           0           0
    2           0           0           0           0           0
    3           0           0           0           0           0
    4           0           0           0           0           0
    5           0           0           0           0           0
    6           0           0           0           0           0
    7           0           0           0           0           0
 AQM Broadcast Early WTD COUNTERS(In terms of Bytes)
--------------------------------------------------
  PORT TYPE          ENQUEUE             DROP
--------------------------------------------------
 UPLINK PORT-0        N/A               0
 UPLINK PORT-1        N/A               0
 UPLINK PORT-2        N/A               0
 UPLINK PORT-3        N/A               0
 NETWORK PORTS       82938              82938
 RCP PORTS               0                  0
 CPU PORT                0                  0
Note: Queuing stats are in bytes

VIP Mentor

Re: Output drop

Hello,

 

try and configure:

 

qos queue-softmax-multiplier 1200

 

globally on the switch that has the output drops...

 

Beginner

Re: Output drop

@@Georg Pauwen you guys are awesome :D providing the solutions without checking the queueing outputs and providing the correct commands which will not work unless reloading the switch.

But I don't propose any solution until full investigation. Carry on :P

 

BR

Tayyab

 

 

*** Please rate all helpful responses and mark solutions***
Beginner

Re: Output drop

Do you think we need to increase buffer size for 0 and 1 queue ? If yes, Why there is not any buffer failure and ... in show interface output ?

It's great if you could analyze the output as a practical case . It would be helpful for those with the similar issue .

Hall of Fame Community Legend

Re: Output drop

CSCvb31906

Beginner

Re: Output drop

Thank you Leo

We're using 3.6.8 so it should be solved in this release

Beginner

Re: Output drop

Hi Salem,

 

Thank you for sharing the outputs.

 

1)

Short explanation: An Output Queue Drop is a packet that was about to leave the switch, but the latter detected temporary congestion on the outgoing link so it was stored in a buffer for a short amount of time. If the switch detected the link remained congested, the packet will be dropped and the Output Drop counter will increase. Output drops are generally a result of interface oversubscription caused by many to one or a 1Gig transfer and etc. Interface buffers are a limited resource and can only absorb a burst up to a point after which packets will drop.

 

2) Also note “Total output drops” counter on 3850/3650/9K switch is representing bytes, not packets. 

 

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 562303015   -->   Drops in bytes

380610888 packets output, 410883103982 bytes, 0 underruns    --> 410883103982 total output bytes sent

 

So, the actual drop rate is around 1.xx*1, which is quite low, that these drops happened on a non-priority queue.

 

3) Also make sure that the softmax is configured to the following value:

conf t

qos queue-softmax-multiplier 600

(current value is 100)

 

Apply the workaround if the issue still exists,  please upgrade the switch  to 03.07.04

 

or 

 

Description :

CAT3850/3650 UNIVERSAL

Release :

3.7.5E

Release Date :

15-Feb-2017

FileName :

cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin

Min Memory :

DRAM 4096 Flash 2048

Size :

313.39 MB ( 328618032 bytes)

MD5 Checksum :

d6f1f24a30dc67697a0c815661ba670c

 

 

 

Regards,

Tayyab - www.tayyabmunir.com 

*** Rate if Response Helpful ***

*** Please rate all helpful responses and mark solutions***
Beginner

Re: Output drop

Hello @TAYYAB MUNIR

Thanks a million for your detailed reply

 Do you confirm below information ( What I'v understood ) ?

 

1) Output drop rate is : (562303015  / 410883103982) * 100 = 0.13% which means reasonable (Less than 1 percent) or (562303015  / 410883103982) * 1000 = 1.3 packet within 1000 .

2) By applying the command Qos queue-softmax-multiplier 600 we will increase the buffer size multiplied by 6 which means 300*6*256= 450KB for each interface. Right ? Does it effect on TenGig interfaces too? I have some TenGig interfaces which are working fine with their buffer size without any drop. 

Does this command need reload?

 

3) When we analyze the graphs, no interfaces touch the maximum BW . So why a 1G interface fails to transfer 30-40Mb data with output drop? Only because of that we are using them as uplink( One to many ) ? But 3850 series are supposed to be used in aggregation layer ! Even at core in environments with lower data rate. 

 

BTW , I want to share another thing with you . Our 3850s are connected to downstream switches which their uplink ports are SFP+ but we use GLC-T(1G) for converting Fibber to copper . But I think it doesn't effect the output drop. I have similar experience in other projects without any output drop. What is your idea ?

CreatePlease to create content
Content for Community-Ad
July's Community Spotlight Awards