output drops on fastethernet not saturated.

[s.balon]

Hi,

We had a Cisco 3750 with a interface that encounter output drop .

We change the cable, we change the interface.

The traffic have the same bandwidth pattern since a long time but now we have output drop.

We try to increase the output hold queue but with the same result.

It's a 100Mbps Full duplex Lease line.

sh int gi 2/0/10
GigabitEthernet2/0/10 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is xxxx.xxxx.xxxx
  MTU 1528 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 139/255, rxload 56/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 9w4d, output 00:00:56, output hang never
  Last clearing of "show interface" counters 00:03:53
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 25824
  Queueing strategy: fifo
  Output queue: 0/100 (size/max)
  30 second input rate 22006000 bits/sec, 11820 packets/sec
  30 second output rate 65047000 bits/sec, 14305 packets/sec
     2506589 packets input, 573262613 bytes, 0 no buffer
     Received 496 broadcasts (495 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 495 multicast, 0 pause input
     0 input packets with dribble condition detected
     3096893 packets output, 1729579549 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Does someone have a idea ?

What can cause output drops except saturation.

Thanks for your help.

[sebastian.lemke]

your current output rate is ~65Mbps (65047000 bits/sec). This just shows a 30 seconds history. Indeed imaginable that there was a peak with output traffic >100Mbps.

[s.balon]

I don't think so.

We have the same amount of traffic since a long time...

and before it was just working fine...

See the graph below...

[francisco_1]

Is qos enable on the interface?. tx-ring filled up, buffer tail drops.

Post sh interfaces gi 2/0/10  counters errors

sh mls qos

sh run int gi 2/0/10

Francisco.

[s.balon]

Hi,

No QOS enable for the moment.

sh int gi 2/0/10 counters errors

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Gi2/0/10            0           0           0           0          0      1281502

Port      Single-Col  Multi-Col   Late-Col  Excess-Col  Carri-Sen      Runts     Giants
Gi2/0/10           0          0          0           0          0          0          0

#sh mls qos
QoS is enabled
QoS ip packet dscp rewrite is enabled

interface GigabitEthernet2/0/10
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 4,9,504
switchport mode trunk
load-interval 30
speed 100
duplex full
spanning-tree bpdufilter enable
hold-queue 100 out
end

Currently, we have 42 Mbps and stil output packet drops ???????

Very strange.

CPU below 10%

nothing in the log..

Steve

[jorge.calvo]

Hello,

Did this link always have the MTU set to 1528 bytes and the hold-queue set to a lenght of 100?

Does the end device of this link support a MTU of 1528 bytes?

Regards.

[s.balon]

Hi,

The mtu has this value since a long time because other link use Dot1q tunneling feature.

It's true that the device on the other end is configured with a System mtu of 1500 but the packet should not be discarded by this reason .... I guess.

The hold queue of 100 has been added yesterday evening to try to solve the issue.

This is the status of the interface in front of this one :

GigabitEthernet0/23 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is XXX

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 0/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:06, output 00:00:01, output hang never
  Last clearing of "show interface" counters 2d00h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1727377308 packets input, 924655050824 bytes, 0 no buffer
     Received 697007 broadcasts (696942 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 696942 multicast, 0 pause input
     0 input packets with dribble condition detected
     1368868183 packets output, 310843530505 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Have a nice day

[jorge.calvo]

Hello,

Could you please put the output of the "show controllers ethernet-controller Gix/x/x" command on both sides of the link?

Anyway, your number of pps seems to be too high. Setting both sides at 1000/FD should improve the situation.

Regards.

[s.balon]

There is the output where the drops happens :

     Transmit GigabitEthernet2/0/10           Receive
   2390085320 Bytes                        544355384 Bytes                   
   1640646975 Unicast frames              1295459996 Unicast frames          
       672394 Multicast frames                307366 Multicast frames        
           69 Broadcast frames                   154 Broadcast frames        
            0 Too old frames              3903595965 Unicast bytes           
            0 Deferred frames               47827631 Multicast bytes         
            0 MTU exceeded frames               9856 Broadcast bytes         
            0 1 collision frames                   0 Alignment errors        
            0 2 collision frames                   1 FCS errors              
            0 3 collision frames                   0 Oversize frames         
            0 4 collision frames                   0 Undersize frames        
            0 5 collision frames                   0 Collision fragments     
            0 6 collision frames      
            0 7 collision frames               22390 Minimum size frames     
            0 8 collision frames           920607110 65 to 127 byte frames   
            0 9 collision frames           159906176 128 to 255 byte frames  
            0 10 collision frames           68231318 256 to 511 byte frames  
            0 11 collision frames           62005676 512 to 1023 byte frames 
            0 12 collision frames           63208565 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames          
            0 14 collision frames                  0 Pause frames            
            0 15 collision frames     
            0 Excessive collisions                 1 Symbol error frames     
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames           21786282 Valid frames, too large 
            0 Excess defer frames                  1 Invalid frames, too small
        22401 64 byte frames                       0 Valid frames, too small 
    738522779 127 byte frames         
    282620700 255 byte frames                      0 Too old frames          
     79042280 511 byte frames                      0 Valid oversize frames   
     75507883 1023 byte frames                     0 System FCS error frames 
    352333126 1518 byte frames                     0 RxPortFifoFull drop frame
    113270268 Too large frames        
            0 Good (1 coll) frames    
            0 Good (>1 coll) frames  

The other site :

     Transmit GigabitEthernet0/23             Receive
   3869468015 Bytes                       4206641421 Bytes                   
    122686975 Unicast frames               135606046 Unicast frames          
     44026215 Multicast frames             110130204 Multicast frames        
         9068 Broadcast frames                516141 Broadcast frames        
            0 Too old frames              3419647358 Unicast bytes           
            0 Deferred frames             4018635552 Multicast bytes         
            0 MTU exceeded frames           91179920 Broadcast bytes         
            0 1 collision frames                   0 Alignment errors        
            0 2 collision frames                   4 FCS errors              
            0 3 collision frames                   1 Oversize frames         
            0 4 collision frames                   0 Undersize frames        
            0 5 collision frames                   0 Collision fragments     
            0 6 collision frames      
            0 7 collision frames             2798914 Minimum size frames     
            0 8 collision frames            55476478 65 to 127 byte frames   
            0 9 collision frames           164961283 128 to 255 byte frames  
            0 10 collision frames         1762888513 256 to 511 byte frames  
            0 11 collision frames          673329789 512 to 1023 byte frames 
            0 12 collision frames         3603526869 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames          
            0 14 collision frames                  0 Pause frames            
            0 15 collision frames     
            0 Excessive collisions                 1 Symbol error frames     
            0 Late collisions                      1 Invalid frames, too large
            0 VLAN discard frames         2573205141 Valid frames, too large 
            0 Excess defer frames                  1 Invalid frames, too small
      2799941 64 byte frames                       0 Valid frames, too small 
    640966060 127 byte frames         
    934639337 255 byte frames                      0 Too old frames          
   1116030662 511 byte frames                      0 Valid oversize frames   
    859567456 1023 byte frames                     0 System FCS error frames 
   1026005999 1518 byte frames                     0 RxPortFifoFull drop frame
   4176647395 Too large frames        
            0 Good (1 coll) frames    
            0 Good (>1 coll) frames  

The lease line between the 2 side are limited to 100mpbs, so we need to ask to upgrade the local endpoint but the line itslef will stay @100 Mbps

Have a nice day.

[Nagaraja Thanthry]

Hello,

In the output, we see that there are quite a few packets that are classified as "too large" packets. This means that the packet size exceeded 1500 bytes. Have you enabled support for Jumbo frames on the switch (it needs to be enabled throughout the path)? If you have not and you do not want to, then you need to identify the device that is generating packets that are bigger than the MTU of the switch and fragment the packets. The problem could be that when too many of these large packets show up on the interface at the same time, the interface may not be able to allocate buffer space for all the packets. This results in packet drops.

Regards,

NT

[s.balon]

Concerning the jumbo frame, I define the same amount for each type :

sh system mtu

System MTU size is 1528 bytes
System Jumbo MTU size is 1528 bytes
Routing MTU size is 1500 bytes

We have the same on each switch of the backbone but not on the remote end of this specific link. We still have default of 1500 for each.

For you info, we even loose small ping packets when testing the link.

Sending 10000, 100-byte ICMP Echos to Y.Y.Y.Y, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (1299/1301), round-trip min/avg/max = 16/24/218 ms

Regards.

[Nagaraja Thanthry]

Hello,

Can you post the output of "show buffers" here?

Regards,

NT

[jorge.calvo]

Hello,

I see no errors on your counters explaining the output drops. The giant frames caused by the MTU size are sent and received properly on both sides.

Do you have any other connection on any of the below ports transmitting at a high throughput?

Gi2/0/9
Gi2/0/11
Gi2/0/12

They are manged by the same ASIC and it could be congested. If that is the case, setting the leased line in a different ASIC should prevent the output drops.

Cheers.

[s.balon]

Hi,

Unfortunatly, the 3 ports are down down.. :-(...

And for your informations, we allready try to use another port (1/0/10), on the same stack but another physical switch.

And We try also with another cable between the provider of the link and the cisco Switch.

very strange ...

Kind regards.

Steve

[s.balon]

Here you are :-)...

sh buffers
Buffer elements:
     1160 in free list (500 max allowed)
     540764366 hits, 0 misses, 1024 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 116 @ 7w0d):
     48 in free list (20 min, 150 max allowed)
     220886073 hits, 22 misses, 66 trims, 66 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 61 @ 7w0d):
     24 in free list (10 min, 150 max allowed)
     877997 hits, 41 misses, 123 trims, 123 created
     0 failures (0 no memory)
Big buffers, 1536 bytes (total 67, permanent 50, peak 203 @ 7w0d):
     67 in free list (5 min, 150 max allowed)
     160609813 hits, 1618 misses, 4837 trims, 4854 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 16, permanent 10, peak 16 @ 7w0d):
     1 in free list (0 min, 100 max allowed)
     239 hits, 3 misses, 622 trims, 628 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
     0 in free list (0 min, 10 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 10 @ 1w2d):
     1 in free list (0 min, 4 max allowed)
     575 hits, 5 misses, 98 trims, 99 created
     0 failures (0 no memory)

Interface buffer pools:
Syslog ED Pool buffers, 600 bytes (total 132, permanent 132):
     100 in free list (132 min, 132 max allowed)
     1654 hits, 0 misses
RxQFB buffers, 2040 bytes (total 904, permanent 904):
     894 in free list (0 min, 904 max allowed)
     8660640 hits, 0 misses
RxQ0 buffers, 2040 bytes (total 1200, permanent 1200):
     700 in free list (0 min, 1200 max allowed)
     90324642 hits, 0 misses
RxQ1 buffers, 2040 bytes (total 128, permanent 128):
     7 in free list (0 min, 128 max allowed)
     125408109 hits, 5587482 fallbacks
RxQ2 buffers, 2040 bytes (total 128, permanent 128):
     1 in free list (0 min, 128 max allowed)
     8145498 hits, 63636 fallbacks, 0 trims, 0 created
     63636 failures (0 no memory)
RxQ3 buffers, 2040 bytes (total 128, permanent 128):
     1 in free list (0 min, 128 max allowed)
     94686074 hits, 1332356 fallbacks
RxQ4 buffers, 2040 bytes (total 128, permanent 128):
     0 in free list (0 min, 128 max allowed)
     509 hits, 2857855382 misses
RxQ5 buffers, 2040 bytes (total 128, permanent 128):
     64 in free list (0 min, 128 max allowed)
     64 hits, 0 misses
RxQ6 buffers, 2040 bytes (total 128, permanent 128):
     0 in free list (0 min, 128 max allowed)
     135164 hits, 134673 misses
RxQ7 buffers, 2040 bytes (total 192, permanent 192):
     56 in free list (0 min, 192 max allowed)
     97450763 hits, 0 misses
RxQ8 buffers, 2040 bytes (total 64, permanent 64):
     0 in free list (0 min, 64 max allowed)
     33435481 hits, 31008981 misses
RxQ9 buffers, 2040 bytes (total 1, permanent 1):
     0 in free list (0 min, 1 max allowed)
     1 hits, 0 misses
RxQ10 buffers, 2040 bytes (total 64, permanent 64):
     1 in free list (0 min, 64 max allowed)
     76615022 hits, 1197109 fallbacks
RxQ11 buffers, 2040 bytes (total 16, permanent 16):
     0 in free list (0 min, 16 max allowed)
     407573 hits, 412977 misses
RxQ12 buffers, 2040 bytes (total 96, permanent 96):
     0 in free list (0 min, 96 max allowed)
     96 hits, 0 misses
RxQ13 buffers, 2040 bytes (total 16, permanent 16):
     0 in free list (0 min, 16 max allowed)
     16 hits, 0 misses
RxQ15 buffers, 2040 bytes (total 4, permanent 4):
     0 in free list (0 min, 4 max allowed)
     118042123 hits, 118042119 misses
IPC buffers, 2048 bytes (total 300, permanent 300):
     288 in free list (150 min, 500 max allowed)
     9788608 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)

Header pools:

Regards.