Catalyst 3750 switch interface output drops

[jackson.ku]

Dear Sir,

One of my Catalyst 3750 switch have many out drops, I execute "sh mls qos int g2/0/3 statist" command, there are many output drops in queue3 threshold3.

Please help to advise me about this problem?

Best Regards,

Jackson Ku

""sh mls qos int g2/0/3 statist" 

his3750sw_srv_5#sh int g2/0/3
GigabitEthernet2/0/3 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 3cdf.1e0c.dc47 (bia 3cdf.1e0c.dc47)
  Internet address is 172.16.52.53/30
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 24/255, rxload 2/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:04, output hang never
  Last clearing of "show interface" counters 02:05:45
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 15097
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 839000 bits/sec, 627 packets/sec
  5 minute output rate 9798000 bits/sec, 1002 packets/sec
     4504251 packets input, 921192165 bytes, 0 no buffer
     Received 859 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 859 multicast, 0 pause input
     0 input packets with dribble condition detected
     6705426 packets output, 8193018075 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
    
his3750sw_srv_5#sh mls qos int g2/0/3 statist
GigabitEthernet2/0/3 (All statistics are in packets)

  dscp: incoming 
-------------------------------

  0 -  4 :   174305096            0            0            0          775 
  5 -  9 :           0            4            0            0            0 
10 - 14 :           0            0            0            0            0 
15 - 19 :           0            0            0            0            0 
20 - 24 :           0            0            0            0            0 
25 - 29 :           0            0            0            0            0 
30 - 34 :           0            0            0            0            0 
35 - 39 :           0            0            0            0            0 
40 - 44 :           0            0            0            0            0 
45 - 49 :           0            0            0       949674            0 
50 - 54 :           0            0            0            0            0 
55 - 59 :           0            0            0            0            0 
60 - 64 :           0            0            0            0 
  dscp: outgoing
-------------------------------

  0 -  4 :   250570018            0            0            0            0 
  5 -  9 :           0            0            0            0            0 
10 - 14 :           0            0            0            0            0 
15 - 19 :           0            0            0            0            0 
20 - 24 :           0            0            0            0            0 
25 - 29 :           0            0            0            0            0 
30 - 34 :           0            0            0            0            0 
35 - 39 :           0            0            0            0            0 
40 - 44 :           0            0            0            0            0 
45 - 49 :           0            0            0      1497337            0 
50 - 54 :           0            0            0            0            0 
55 - 59 :           0            0            0            0            0 
60 - 64 :           0            0            0            0 
  cos: incoming 
-------------------------------

  0 -  4 :   175261405            0            0            0            0 
  5 -  7 :           0            0            0 
  cos: outgoing
-------------------------------

  0 -  4 :   250570284            0            0            0            0 
  5 -  7 :           0      1497140         5851 
  output queues enqueued:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           2           0           0
queue 1:          14         516     3853875
queue 2:           0           0           0
queue 3:           0           0   251320914

  output queues dropped:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           0           0           0
queue 1:           0           0           0
queue 2:           0           0           0
queue 3:           0           0     1199828

Policer: Inprofile:            0 OutofProfile:            0

[InayathUlla Sharieff]

Hi Jackson,

The counter "Total output drops:" is incremented each time the output queue

was overloaded, and therefore some packets had to be dropped due to a lack

of

output buffers. If this counter for output drops was incrementing while you

are on the

router, and you did a "show interface", the output queue would most

definitely have been 40/40,

which means it was full. If the output queue is 0/40 like in the capture you

provided, the

output drops should not be incrementing at that particular time.

The output queue can become full during periods where the interface is

sending close to full capacity of traffic, or if there are short bursts of

high traffic rates that needs to go out on that interface.

1) If you clear the counters, how is the rate of the output drop

accumulation?

2) Verify that the cable is good by using a cable tester or replace it with

a known good cable.

3) Verify that the Network Interface Card (NIC) is compatible and working

properly. Refer to

< http://www.cisco.com/en/US/customer/products/hw/switches/ps700/products_tec

h_note09186a00800a7af0.shtml>

http://www.cisco.com/en/US/customer/products/hw/switches/ps700/products_tech

_note09186a00800a7af0.shtml

4) Verify that the flow control settings are the same on both sides of a

Ethernet link.

Action plan:

kindly apply the following:

- Change the load interval from 5 minute to 30 seconds >> load-interval 30

"under the interface mode "

- Clear the counters.

- disable qos( conf t #no mls qos) but this will disable the qos .

- After that if you would still see the discards increasing , please capture

the following commands 2-3 times (1 minute interval between each output)

#sh controllers

#sh interfaces

#show interface counter error

#show platform port-asic stats drop

#sh interface interface-ID | i output drop

#show mls qos int stat   ////////this for the interfaces seeing the

output drops

Regards

Inayath

*Plz rate the usefull posts.

[paolo bevilacqua]

You should disable QoS completely, Having it enable causes the drops, due to queue size becoming too small.

[jackson.ku]

Hi,

The output queue is always "0/40", but the output drop still incrementing!!

The other side of this switch port is another switch provided by ISP, the ISP provide a 100Mbps WAN link to connect to branch office , not a server. The duplex type / speed is ok.

The QoS is disabled :

sh mls qos

QoS is disabled

QoS ip packet dscp rewrite is enabled

Best Regards,

Jackson

[InayathUlla Sharieff]

HI Jackson,

Could you please clear the counters and provide the following outputs:

#sh controllers

#sh interfaces

#show interface counter error

#show platform port-asic stats drop

#sh interface interface-ID | i output drop

#show mls qos int stat   ////////this for the interfaces seeing the

output drops

Regards

Inayath

[Joseph W. Doherty · Accepted solution]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

The output queue is always "0/40", but the output drop still incrementing!!

Some "show" 3750 stats, I believe, are not always correctly updated by the hardware.

If you have transient congestion, or microbursts, you might be able to reduce drops by increasing buffer resources.  On a 3750, believe that's only possible by enabling QoS, which often introduces even more drops because by default it evenly distributes resources to four egress queues rather than one.  (This is why Paolo suggested disabling QoS, if enabled.)

Tuning 3750 QoS is possible, but it's somewhat complex.  There's a good document on theses forums, https://supportforums.cisco.com/docs/DOC-8093, discussing 3750 QoS buffer tuning.

When tuning for drop avoidance, most try increasing queuing resources, but sometimes decreasing queuing resources is the "better" solution.  The latter, though, is not well understood by most.  Also a LAN 3750's advanced QoS capabilities are few.

(BTW, huge buffers/queues may avoid drops but then you may have transient high latencies.)

The simplest solution might be to obtain increased bandwidth from your provider, e.g. another 100 Mbps link or a (fractional) gig link.

If you want to minimize drops using QoS features, you'll want to replace the 3750, on this link, with an ISR.  Again, though, drop reduction using QoS isn't trivial and you're probably going to have some drops even with "ideal" QoS.  QoS also is the "better" solution if you have traffic with different service requirements.

[jackson.ku]

Hi,

The output of the show command are A/F :

https://dl.dropbox.com/u/106896276/3750_log.rar

Best Regards,

Jackson

[jackson.ku]

Dear All,

Please help to mention me how to solve this problem.

Best Regards,

[paolo bevilacqua]

If you have disabled QoS there is not much else you can do.

Droprs are due to burst of traffic when goign from 1 Gb to 100 mbps, are expected and normal.