09-14-2010 07:38 AM - edited 03-06-2019 12:58 PM
Hi,
We had a Cisco 3750 with a interface that encounter output drop .
We change the cable, we change the interface.
The traffic have the same bandwidth pattern since a long time but now we have output drop.
We try to increase the output hold queue but with the same result.
It's a 100Mbps Full duplex Lease line.
sh int gi 2/0/10
GigabitEthernet2/0/10 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is xxxx.xxxx.xxxx
MTU 1528 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 139/255, rxload 56/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 9w4d, output 00:00:56, output hang never
Last clearing of "show interface" counters 00:03:53
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 25824
Queueing strategy: fifo
Output queue: 0/100 (size/max)
30 second input rate 22006000 bits/sec, 11820 packets/sec
30 second output rate 65047000 bits/sec, 14305 packets/sec
2506589 packets input, 573262613 bytes, 0 no buffer
Received 496 broadcasts (495 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 495 multicast, 0 pause input
0 input packets with dribble condition detected
3096893 packets output, 1729579549 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Does someone have a idea ?
What can cause output drops except saturation.
Thanks for your help.
09-14-2010 07:49 AM
your current output rate is ~65Mbps (65047000 bits/sec). This just shows a 30 seconds history. Indeed imaginable that there was a peak with output traffic >100Mbps.
09-14-2010 08:02 AM
09-14-2010 08:21 AM
Is qos enable on the interface?. tx-ring filled up, buffer tail drops.
Post sh interfaces gi 2/0/10 counters errors
sh mls qos
sh run int gi 2/0/10
Francisco.
09-15-2010 12:44 AM
Hi,
No QOS enable for the moment.
sh int gi 2/0/10 counters errors
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Gi2/0/10 0 0 0 0 0 1281502
Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants
Gi2/0/10 0 0 0 0 0 0 0
#sh mls qos
QoS is enabled
QoS ip packet dscp rewrite is enabled
interface GigabitEthernet2/0/10
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 4,9,504
switchport mode trunk
load-interval 30
speed 100
duplex full
spanning-tree bpdufilter enable
hold-queue 100 out
end
Currently, we have 42 Mbps and stil output packet drops ???????
Very strange.
CPU below 10%
nothing in the log..
Steve
09-15-2010 03:10 AM
Hello,
Did this link always have the MTU set to 1528 bytes and the hold-queue set to a lenght of 100?
Does the end device of this link support a MTU of 1528 bytes?
Regards.
09-15-2010 04:44 AM
Hi,
The mtu has this value since a long time because other link use Dot1q tunneling feature.
It's true that the device on the other end is configured with a System mtu of 1500 but the packet should not be discarded by this reason .... I guess.
The hold queue of 100 has been added yesterday evening to try to solve the issue.
This is the status of the interface in front of this one :
GigabitEthernet0/23 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is XXX
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 0/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:06, output 00:00:01, output hang never
Last clearing of "show interface" counters 2d00h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1727377308 packets input, 924655050824 bytes, 0 no buffer
Received 697007 broadcasts (696942 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 696942 multicast, 0 pause input
0 input packets with dribble condition detected
1368868183 packets output, 310843530505 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Have a nice day
09-15-2010 05:15 AM
Hello,
Could you please put the output of the "show controllers ethernet-controller Gix/x/x" command on both sides of the link?
Anyway, your number of pps seems to be too high. Setting both sides at 1000/FD should improve the situation.
Regards.
09-15-2010 06:16 AM
There is the output where the drops happens :
Transmit GigabitEthernet2/0/10 Receive
2390085320 Bytes 544355384 Bytes
1640646975 Unicast frames 1295459996 Unicast frames
672394 Multicast frames 307366 Multicast frames
69 Broadcast frames 154 Broadcast frames
0 Too old frames 3903595965 Unicast bytes
0 Deferred frames 47827631 Multicast bytes
0 MTU exceeded frames 9856 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 1 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 22390 Minimum size frames
0 8 collision frames 920607110 65 to 127 byte frames
0 9 collision frames 159906176 128 to 255 byte frames
0 10 collision frames 68231318 256 to 511 byte frames
0 11 collision frames 62005676 512 to 1023 byte frames
0 12 collision frames 63208565 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 1 Symbol error frames
0 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 21786282 Valid frames, too large
0 Excess defer frames 1 Invalid frames, too small
22401 64 byte frames 0 Valid frames, too small
738522779 127 byte frames
282620700 255 byte frames 0 Too old frames
79042280 511 byte frames 0 Valid oversize frames
75507883 1023 byte frames 0 System FCS error frames
352333126 1518 byte frames 0 RxPortFifoFull drop frame
113270268 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames
The other site :
Transmit GigabitEthernet0/23 Receive
3869468015 Bytes 4206641421 Bytes
122686975 Unicast frames 135606046 Unicast frames
44026215 Multicast frames 110130204 Multicast frames
9068 Broadcast frames 516141 Broadcast frames
0 Too old frames 3419647358 Unicast bytes
0 Deferred frames 4018635552 Multicast bytes
0 MTU exceeded frames 91179920 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 4 FCS errors
0 3 collision frames 1 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 2798914 Minimum size frames
0 8 collision frames 55476478 65 to 127 byte frames
0 9 collision frames 164961283 128 to 255 byte frames
0 10 collision frames 1762888513 256 to 511 byte frames
0 11 collision frames 673329789 512 to 1023 byte frames
0 12 collision frames 3603526869 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 1 Symbol error frames
0 Late collisions 1 Invalid frames, too large
0 VLAN discard frames 2573205141 Valid frames, too large
0 Excess defer frames 1 Invalid frames, too small
2799941 64 byte frames 0 Valid frames, too small
640966060 127 byte frames
934639337 255 byte frames 0 Too old frames
1116030662 511 byte frames 0 Valid oversize frames
859567456 1023 byte frames 0 System FCS error frames
1026005999 1518 byte frames 0 RxPortFifoFull drop frame
4176647395 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames
The lease line between the 2 side are limited to 100mpbs, so we need to ask to upgrade the local endpoint but the line itslef will stay @100 Mbps
Have a nice day.
09-15-2010 07:02 AM
Hello,
In the output, we see that there are quite a few packets that are classified as "too large" packets. This means that the packet size exceeded 1500 bytes. Have you enabled support for Jumbo frames on the switch (it needs to be enabled throughout the path)? If you have not and you do not want to, then you need to identify the device that is generating packets that are bigger than the MTU of the switch and fragment the packets. The problem could be that when too many of these large packets show up on the interface at the same time, the interface may not be able to allocate buffer space for all the packets. This results in packet drops.
Regards,
NT
09-15-2010 07:26 AM
Concerning the jumbo frame, I define the same amount for each type :
sh system mtu
System MTU size is 1528 bytes
System Jumbo MTU size is 1528 bytes
Routing MTU size is 1500 bytes
We have the same on each switch of the backbone but not on the remote end of this specific link. We still have default of 1500 for each.
For you info, we even loose small ping packets when testing the link.
Sending 10000, 100-byte ICMP Echos to Y.Y.Y.Y, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (1299/1301), round-trip min/avg/max = 16/24/218 ms
Regards.
09-15-2010 02:56 PM
Hello,
Can you post the output of "show buffers" here?
Regards,
NT
09-15-2010 07:03 AM
Hello,
I see no errors on your counters explaining the output drops. The giant frames caused by the MTU size are sent and received properly on both sides.
Do you have any other connection on any of the below ports transmitting at a high throughput?
Gi2/0/9
Gi2/0/11
Gi2/0/12
They are manged by the same ASIC and it could be congested. If that is the case, setting the leased line in a different ASIC should prevent the output drops.
Cheers.
09-15-2010 07:15 AM
Hi,
Unfortunatly, the 3 ports are down down.. :-(...
And for your informations, we allready try to use another port (1/0/10), on the same stack but another physical switch.
And We try also with another cable between the provider of the link and the cisco Switch.
very strange ...
Kind regards.
Steve
09-16-2010 12:33 AM
Here you are :-)...
sh buffers
Buffer elements:
1160 in free list (500 max allowed)
540764366 hits, 0 misses, 1024 created
Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 116 @ 7w0d):
48 in free list (20 min, 150 max allowed)
220886073 hits, 22 misses, 66 trims, 66 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 61 @ 7w0d):
24 in free list (10 min, 150 max allowed)
877997 hits, 41 misses, 123 trims, 123 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 67, permanent 50, peak 203 @ 7w0d):
67 in free list (5 min, 150 max allowed)
160609813 hits, 1618 misses, 4837 trims, 4854 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 16, permanent 10, peak 16 @ 7w0d):
1 in free list (0 min, 100 max allowed)
239 hits, 3 misses, 622 trims, 628 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 10 @ 1w2d):
1 in free list (0 min, 4 max allowed)
575 hits, 5 misses, 98 trims, 99 created
0 failures (0 no memory)
Interface buffer pools:
Syslog ED Pool buffers, 600 bytes (total 132, permanent 132):
100 in free list (132 min, 132 max allowed)
1654 hits, 0 misses
RxQFB buffers, 2040 bytes (total 904, permanent 904):
894 in free list (0 min, 904 max allowed)
8660640 hits, 0 misses
RxQ0 buffers, 2040 bytes (total 1200, permanent 1200):
700 in free list (0 min, 1200 max allowed)
90324642 hits, 0 misses
RxQ1 buffers, 2040 bytes (total 128, permanent 128):
7 in free list (0 min, 128 max allowed)
125408109 hits, 5587482 fallbacks
RxQ2 buffers, 2040 bytes (total 128, permanent 128):
1 in free list (0 min, 128 max allowed)
8145498 hits, 63636 fallbacks, 0 trims, 0 created
63636 failures (0 no memory)
RxQ3 buffers, 2040 bytes (total 128, permanent 128):
1 in free list (0 min, 128 max allowed)
94686074 hits, 1332356 fallbacks
RxQ4 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
509 hits, 2857855382 misses
RxQ5 buffers, 2040 bytes (total 128, permanent 128):
64 in free list (0 min, 128 max allowed)
64 hits, 0 misses
RxQ6 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
135164 hits, 134673 misses
RxQ7 buffers, 2040 bytes (total 192, permanent 192):
56 in free list (0 min, 192 max allowed)
97450763 hits, 0 misses
RxQ8 buffers, 2040 bytes (total 64, permanent 64):
0 in free list (0 min, 64 max allowed)
33435481 hits, 31008981 misses
RxQ9 buffers, 2040 bytes (total 1, permanent 1):
0 in free list (0 min, 1 max allowed)
1 hits, 0 misses
RxQ10 buffers, 2040 bytes (total 64, permanent 64):
1 in free list (0 min, 64 max allowed)
76615022 hits, 1197109 fallbacks
RxQ11 buffers, 2040 bytes (total 16, permanent 16):
0 in free list (0 min, 16 max allowed)
407573 hits, 412977 misses
RxQ12 buffers, 2040 bytes (total 96, permanent 96):
0 in free list (0 min, 96 max allowed)
96 hits, 0 misses
RxQ13 buffers, 2040 bytes (total 16, permanent 16):
0 in free list (0 min, 16 max allowed)
16 hits, 0 misses
RxQ15 buffers, 2040 bytes (total 4, permanent 4):
0 in free list (0 min, 4 max allowed)
118042123 hits, 118042119 misses
IPC buffers, 2048 bytes (total 300, permanent 300):
288 in free list (150 min, 500 max allowed)
9788608 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Header pools:
Regards.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide