Output drops on 100mbit interface

Joris Deprouw · ‎10-15-2013

Hi All,

We have a 3750x stack with connections which are negotiated on 1000mbit and other on 100mbit. We can see only on the 100Mb interfaces drops, not on the 1000Mbit interfaces.

When we connect a laptop on the switch and fix the speed to 100Mbit/full duplex, we also see these drops. When we change the port settings to auto, the port is negiated to 1000Mbit/ full and the output drops disapear.

What could be the reason of these output drops? The interface itself is not heaviliy used.

Any ideas?

Best Regards,

Joris

switch#sh int gi 3/0/8

GigabitEthernet3/0/8 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 2894.0f6d.7988 (bia 2894.0f6d.7988)

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 1y1w, output 00:00:10, output hang never

Last clearing of "show interface" counters 00:36:54

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4859

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 729000 bits/sec, 174 packets/sec

5511 packets input, 430130 bytes, 0 no buffer

Received 255 broadcasts (220 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 220 multicast, 0 pause input

0 input packets with dribble condition detected

383231 packets output, 164567523 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

glen.grant · ‎10-15-2013

If you are hardcoding your pc to 100/full then you must hardcode the switch also , otherwise the switch will negotiate to 100/half which is a duplex mismatch and will cause drops and errors. Works the other way also , if you hardcode the switchport you "must" hardcode the pc nic otherwise you have a duplex imsmatch that way also . Unless there is a specific problem everything should be auto anyway .

Joseph W. Doherty · ‎10-15-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

An egress queue filling and dropping packets doesn't always show with high link utilization. A large burst can cause it. What's the possible ingress rate tot the switch's g3/0/8? (Not g3/0/8 ingress, but for other ingress ports that might send data to g3/0/8 for egress.)

If drops are seen at 100 but not at 1,000, the latter may handle the same burst better than at 100, so that could account for the difference.

Joris Deprouw · ‎10-15-2013

Hi,

I did some more research.

The output drops only happen on interfaces in a specific vlan10. Only on interfaces which are negotiated to 100mbit and 10mbit.

Other 100mbit ports in another vlan don't have this issue.

The vlan 10 is however a very large vlan, with probably a lot of broadcast traffic. Can this be an issue?

Thanks,

Joris

switch#sh int status

Port      Name               Status       Vlan       Duplex Speed Type
Fa0/1                        connected    11         a-full a-100 10/100BaseTX
Fa0/2                        connected    10         a-full a-100 10/100BaseTX
Fa0/3                        notconnect   10           auto   auto 10/100BaseTX
Fa0/4                        connected    11         a-full   a-10 10/100BaseTX
Fa0/5                        connected    10         a-half   a-10 10/100BaseTX
Fa0/6                        connected    10         a-full a-100 10/100BaseTX
Fa0/7                        connected    10         a-half   a-10 10/100BaseTX
Fa0/8                        connected    10         a-full a-100 10/100BaseTX
Fa0/9                        notconnect   1            auto   auto 10/100BaseTX
Fa0/10                       notconnect   1            auto   auto 10/100BaseTX

switch#sh int counters errors

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err UnderSize OutDiscards
Fa0/1               0           0           0           0          0            0
Fa0/2               0           0           0           0          0           62
Fa0/3               0           0           0           0          0            0
Fa0/4               0           0           0           0          0            0
Fa0/5               0           0           0           0          0          183
Fa0/6               0           0           0           0          0           63
Fa0/7               0           0           0           0          0          183
Fa0/8               0           0           0           0          0           62
Fa0/9               0           0           0           0          0            0
Fa0/10              0           0           0           0          0            0
Fa0/11              0           0           0           0          0            0

Joseph W. Doherty · ‎10-15-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The vlan 10 is however a very large vlan, with probably a lot of broadcast traffic. Can this be an issue?

Sure, but unusual unless you're having broadcast storms. How large is very large?

However, what's interesting in your stats is each 100 Mbps interface has about the same drop count as does each 10 Mbps interface in VLAN 10. (Higher drop rate on 10 likely because of the lower bandwidth and half duplex.)

Other possibilities for similar drops counts might be mutlticast flood/prune and/or unicast flooding.

Leo Laohoo · ‎10-15-2013

Post the following output:

1. sh controll util

2. sh controll e G 3/0/8

3. sh version

Joris Deprouw · ‎10-15-2013

Hi,

The vlan 10 has a subnet mask of /16

Below you can find the output:

We see this behaviour on several switches, from different types and different IOSses.

Thanks,

Joris

switch#sh interfaces counters errors

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err UnderSize OutDiscards
Fa0/1               0           0           0           0          0            0
Fa0/2               0           0           0           0          0          626
Fa0/3               0           0           0           0          0            0
Fa0/4               0           0           0           0          0            0
Fa0/5               0           0           0           0          0         6874
Fa0/6               0           0           0           0          0          627
Fa0/7               0           0           0           0          0         6874
Fa0/8               0           0           0           0          0          627
Fa0/9               0           0           0           0          0            0
Fa0/10              0           0           0           0          0            0
Fa0/11              0           0           0           0          0            0

switch#sh controllers utilization
Port       Receive Utilization Transmit Utilization
Fa0/1              0                    0
Fa0/2              0                    0
Fa0/3              0                    0
Fa0/4              0                    0
Fa0/5              0                    2
Fa0/6              0                    0
Fa0/7              0                    2
Fa0/8              0                    0
Fa0/9              0                    0
Fa0/10             0                    0
Fa0/11             0                    0
Fa0/12             0                    0
Fa0/13             0                    1
Fa0/14             0                    0
Fa0/15             0                    0
Fa0/16             0                    0
Fa0/17             0                    0
Fa0/18             0                    0
Fa0/19             0                    0
Fa0/20             0                    0
Fa0/21             0                    0
Fa0/22             0                    0
Fa0/23             0                    0
Fa0/24             0                    0
Fa0/25             0                    0
Fa0/26             0                    0
Fa0/27             0                    0
Fa0/28             0                    0
Fa0/29             0                    0
Fa0/30             0                    0
Fa0/31             0                    0
Fa0/32             0                    0
Fa0/33             0                    0
Fa0/34             0                    0
Fa0/35             0                    0
Fa0/36             0                    0
Fa0/37             0                    0
Fa0/38             0                    0
Fa0/39             0                    0
Fa0/40             0                    0
Fa0/41             0                    0
Fa0/42             0                    0
Fa0/43             0                    0
Fa0/44             0                    0
Fa0/45             0                    0
Fa0/46             0                    0
Fa0/47             0                    0
Fa0/48             0                    0
Gi0/1              0                    0
Gi0/2              0                    0
Gi0/3              0                    0
Gi0/4              0                    0

Total Ports : 52
Switch Receive Bandwidth Percentage Utilization : 0
Switch Transmit Bandwidth Percentage Utilization : 0

Switch Fabric Percentage Utilization : 0

switch# sh controllers ethernet-controller fa 0/5

     Transmit FastEthernet0/5                 Receive
    679313800 Bytes                        188200809 Bytes
     37463214 Unicast frames                 1644727 Unicast frames
    303918381 Multicast frames                     0 Multicast frames
   1090537886 Broadcast frames                  1865 Broadcast frames
            0 Too old frames               187999865 Unicast bytes
        21358 Deferred frames                      0 Multicast bytes
            0 MTU exceeded frames             119360 Broadcast bytes
         9505 1 collision frames                   0 Alignment errors
         3858 2 collision frames                   0 FCS errors
          931 3 collision frames                   0 Oversize frames
           93 4 collision frames                   0 Undersize frames
            2 5 collision frames               20396 Collision fragments
            0 6 collision frames
            0 7 collision frames              730977 Minimum size frames
            0 8 collision frames              302726 65 to 127 byte frames
            0 9 collision frames              589830 128 to 255 byte frames
            0 10 collision frames                 63 256 to 511 byte frames
            0 11 collision frames                114 512 to 1023 byte frames
            0 12 collision frames              22882 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large
           90 Excess defer frames              20396 Invalid frames, too small
   1061104678 64 byte frames                       0 Valid frames, too small
    221033391 127 byte frames
     26321161 255 byte frames                      0 Too old frames
     26283369 511 byte frames                      0 Valid oversize frames
      8143392 1023 byte frames                     0 System FCS error frames
     89033490 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames
         9505 Good (1 coll) frames
         4884 Good (>1 coll) frames

switch#

sh ver

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 52    WS-C3560-48PS      12.2(55)SE            C3560-IPBASEK9-M

Configuration register is 0xF

Wantser1981_2 · ‎10-16-2013

Looks like this is down to duplex mismatching and/or load.

Your switch does seem to be receiving a substantial amount of broadcast traffic on this port also.

Invalid frames too small and collision fragments are common with duplex issues. If you are fixing the PC end you need to fix the Switch end as previously mentioned. I have seen that in some cases although software is fixing a particular end node (PC or whatever) the actual PHY still runs in auto. Thus a mismatch can still occur with some hardwares. To be sure as to what you have actually running on the device PHY, you will need some NIC specific management software as there is nowhere in windows to see what the actual NIC is running, only what you are telling it to be.

That said, the amount of broadcast traffic is high. I would maybe look at setting up a SPAN and monitoring the port to see whats going on. But if VLAN10 is large (how large) I would maybe look at introducing some storm-control on vlan10 ports, or even attempting to utilize more VLANs. If you do a SPAN you may fine a few hosts that are causing the majority of your issues.

The fact that you have repeated attempts to send a packet (1,2,3,4 collision frames TX and collision fragments RX) suggests that either the delivery rate of frames is exceeded for the port (broadcast issue) or the same wire pair is being used for transmit and recieve (duplex issue).

HTH

Joris Deprouw · ‎10-16-2013

Thanks,

This is very usefull information.

We have already considered to put the devices in a separate vlan to limit broadcast effects.

I will check which speed and duplex settings are used on the devices.

the size of the vlan is a /16.

best Regards,

Joris

Wantser1981_2 · ‎10-16-2013

No Problem.

/16 is only a theoretical size. Potentially 65535 hosts. Just wondering what you think the usage is out of that /16?

Ie, have you used almost all address space in the /16?

Cheers

Wantser

Leo Laohoo · ‎10-16-2013

Your output doesn't say much other than you had SOME collisions due to speed and duplex mismatch.

If you notice your output drops incrementing sharply, run the two commands again.

1. The "sh controll util" is a real-time snapshot of your utilization (in percentage) the very momemt you run the command.

2. The "sh controll e " is a very good counter which will show you what specific line errors are being recorded. The reason why I like this command is EVERYONE knows how to clear the counters of an interface. Not everyone knows that another set of counters exist and not a lot of people know how to clear the counters of this command.

Also, your IOS is pretty old. Try 12.2(55)SE8.

Joris Deprouw · ‎02-20-2014

Hi All,

We have found a solution in the meantime by putting the devices on the ports with the high droprate in another vlan.

We assume they had to much interference of other traffic.

We also did a wireshark capture on a affected port and saw that each time we had drops there was a burst of NLB traffic in that vlan coming from a server with NLB configured.

We are planning to deactivate all NLB and use a proper load balancer.

Best Regards,

Joris