Output drops on 100mbit interface

[Joris Deprouw]

Hi All,

We have a 3750x stack with connections which are negotiated on 1000mbit and other on 100mbit. We can see only on the 100Mb interfaces drops, not on the 1000Mbit interfaces.

When we connect a laptop on the switch and fix the speed to 100Mbit/full duplex, we also see these drops. When we change the port settings to auto, the port is negiated to 1000Mbit/ full and the output drops disapear.

What could be the reason of these output drops? The interface itself is not heaviliy used.

Any ideas?

Best Regards,

Joris

switch#sh int gi 3/0/8

GigabitEthernet3/0/8 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 2894.0f6d.7988 (bia 2894.0f6d.7988)

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 1y1w, output 00:00:10, output hang never

  Last clearing of "show interface" counters 00:36:54

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4859

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 729000 bits/sec, 174 packets/sec

     5511 packets input, 430130 bytes, 0 no buffer

     Received 255 broadcasts (220 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 220 multicast, 0 pause input

     0 input packets with dribble condition detected

     383231 packets output, 164567523 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

[glen.grant]

  If you are hardcoding your pc to 100/full then you must hardcode the switch  also , otherwise the switch will negotiate to 100/half which is a duplex mismatch and will cause drops and errors.  Works the other way also , if you hardcode the switchport you "must" hardcode the pc nic otherwise you have a duplex imsmatch that way also . Unless there is a specific problem everything should be auto anyway .

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

An egress queue filling and dropping packets doesn't always show with high link utilization.  A large burst can cause it.  What's the possible ingress rate tot the switch's g3/0/8?  (Not g3/0/8 ingress, but for other ingress ports that might send data to g3/0/8 for egress.)

If drops are seen at 100 but not at 1,000, the latter may handle the same burst better than at 100, so that could account for the difference.

[Joris Deprouw]

Hi,

I did some more research.

The output drops only happen on interfaces in a specific vlan10. Only on interfaces which are negotiated to 100mbit and 10mbit.

Other 100mbit ports in another vlan don't have this issue.

The vlan 10 is however a very large vlan, with probably a lot of broadcast traffic. Can this be an issue?

Thanks,

Joris

switch#sh int status

Port      Name               Status       Vlan       Duplex  Speed Type
Fa0/1                        connected    11         a-full  a-100 10/100BaseTX
Fa0/2                        connected    10         a-full  a-100 10/100BaseTX
Fa0/3                        notconnect   10           auto   auto 10/100BaseTX
Fa0/4                        connected    11         a-full   a-10 10/100BaseTX
Fa0/5                        connected    10         a-half   a-10 10/100BaseTX
Fa0/6                        connected    10         a-full  a-100 10/100BaseTX
Fa0/7                        connected    10         a-half   a-10 10/100BaseTX
Fa0/8                        connected    10         a-full  a-100 10/100BaseTX
Fa0/9                        notconnect   1            auto   auto 10/100BaseTX
Fa0/10                       notconnect   1            auto   auto 10/100BaseTX

switch#sh int counters errors

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Fa0/1               0           0           0           0          0            0
Fa0/2               0           0           0           0          0           62
Fa0/3               0           0           0           0          0            0
Fa0/4               0           0           0           0          0            0
Fa0/5               0           0           0           0          0          183
Fa0/6               0           0           0           0          0           63
Fa0/7               0           0           0           0          0          183
Fa0/8               0           0           0           0          0           62
Fa0/9               0           0           0           0          0            0
Fa0/10              0           0           0           0          0            0
Fa0/11              0           0           0           0          0            0

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The vlan 10 is however a very large vlan, with probably a lot of broadcast traffic. Can this be an issue?

Sure, but unusual unless you're having broadcast storms.  How large is very large?

However, what's interesting in your stats is each 100 Mbps interface has about the same drop count as does each 10 Mbps interface in VLAN 10.  (Higher drop rate on 10 likely because of the lower bandwidth and half duplex.)

Other possibilities for similar drops counts might be mutlticast flood/prune and/or unicast flooding.

[Leo Laohoo]

Post the following output:

1.  sh controll util

2.  sh controll e G 3/0/8

3.  sh version

[Joris Deprouw]

Hi,

The vlan 10 has a subnet mask of /16

Below you can find the output:

We see this behaviour on several switches, from different types and different IOSses.

Thanks,

Joris

switch#sh interfaces counters errors

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Fa0/1               0           0           0           0          0            0
Fa0/2               0           0           0           0          0          626
Fa0/3               0           0           0           0          0            0
Fa0/4               0           0           0           0          0            0
Fa0/5               0           0           0           0          0         6874
Fa0/6               0           0           0           0          0          627
Fa0/7               0           0           0           0          0         6874
Fa0/8               0           0           0           0          0          627
Fa0/9               0           0           0           0          0            0
Fa0/10              0           0           0           0          0            0
Fa0/11              0           0           0           0          0            0

switch#sh controllers utilization
Port       Receive Utilization  Transmit Utilization
Fa0/1              0                    0
Fa0/2              0                    0
Fa0/3              0                    0
Fa0/4              0                    0
Fa0/5              0                    2
Fa0/6              0                    0
Fa0/7              0                    2
Fa0/8              0                    0
Fa0/9              0                    0
Fa0/10             0                    0
Fa0/11             0                    0
Fa0/12             0                    0
Fa0/13             0                    1
Fa0/14             0                    0
Fa0/15             0                    0
Fa0/16             0                    0
Fa0/17             0                    0
Fa0/18             0                    0
Fa0/19             0                    0
Fa0/20             0                    0
Fa0/21             0                    0
Fa0/22             0                    0
Fa0/23             0                    0
Fa0/24             0                    0
Fa0/25             0                    0
Fa0/26             0                    0
Fa0/27             0                    0
Fa0/28             0                    0
Fa0/29             0                    0
Fa0/30             0                    0
Fa0/31             0                    0
Fa0/32             0                    0
Fa0/33             0                    0
Fa0/34             0                    0
Fa0/35             0                    0
Fa0/36             0                    0
Fa0/37             0                    0
Fa0/38             0                    0
Fa0/39             0                    0
Fa0/40             0                    0
Fa0/41             0                    0
Fa0/42             0                    0
Fa0/43             0                    0
Fa0/44             0                    0
Fa0/45             0                    0
Fa0/46             0                    0
Fa0/47             0                    0
Fa0/48             0                    0
Gi0/1              0                    0
Gi0/2              0                    0
Gi0/3              0                    0
Gi0/4              0                    0

Total Ports : 52
Switch Receive Bandwidth Percentage Utilization  : 0
Switch Transmit Bandwidth Percentage Utilization : 0

Switch Fabric Percentage Utilization : 0

switch# sh controllers ethernet-controller fa 0/5

     Transmit FastEthernet0/5                 Receive
    679313800 Bytes                        188200809 Bytes                   
     37463214 Unicast frames                 1644727 Unicast frames          
    303918381 Multicast frames                     0 Multicast frames        
   1090537886 Broadcast frames                  1865 Broadcast frames        
            0 Too old frames               187999865 Unicast bytes           
        21358 Deferred frames                      0 Multicast bytes         
            0 MTU exceeded frames             119360 Broadcast bytes         
         9505 1 collision frames                   0 Alignment errors        
         3858 2 collision frames                   0 FCS errors              
          931 3 collision frames                   0 Oversize frames         
           93 4 collision frames                   0 Undersize frames        
            2 5 collision frames               20396 Collision fragments     
            0 6 collision frames      
            0 7 collision frames              730977 Minimum size frames     
            0 8 collision frames              302726 65 to 127 byte frames   
            0 9 collision frames              589830 128 to 255 byte frames  
            0 10 collision frames                 63 256 to 511 byte frames  
            0 11 collision frames                114 512 to 1023 byte frames 
            0 12 collision frames              22882 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames          
            0 14 collision frames                  0 Pause frames            
            0 15 collision frames     
            0 Excessive collisions                 0 Symbol error frames     
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large 
           90 Excess defer frames              20396 Invalid frames, too small
   1061104678 64 byte frames                       0 Valid frames, too small 
    221033391 127 byte frames         
     26321161 255 byte frames                      0 Too old frames          
     26283369 511 byte frames                      0 Valid oversize frames   
      8143392 1023 byte frames                     0 System FCS error frames 
     89033490 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames        
         9505 Good (1 coll) frames    
         4884 Good (>1 coll) frames   

switch#

sh ver

Switch Ports Model              SW Version            SW Image                
------ ----- -----              ----------            ----------              
*    1 52    WS-C3560-48PS      12.2(55)SE            C3560-IPBASEK9-M        

Configuration register is 0xF

[Wantser1981_2]

Looks like this is down to duplex mismatching and/or load.

Your switch does seem to be receiving a substantial amount of broadcast traffic on this port also.

Invalid frames too small and collision fragments are common with duplex issues. If you are fixing the PC end you need to fix the Switch end as previously mentioned. I have seen that in some cases although software is fixing a particular end node (PC or whatever) the actual PHY still runs in auto. Thus a mismatch can still occur with some hardwares. To be sure as to what you have actually running on the device PHY, you will need some NIC specific management software as there is nowhere in windows to see what the actual NIC is running, only what you are telling it to be.

That said, the amount of broadcast traffic is high. I would maybe look at setting up a SPAN and monitoring the port to see whats going on. But if VLAN10 is large (how large) I would maybe look at introducing some storm-control on vlan10 ports, or even attempting to utilize more VLANs. If you do a SPAN you may fine a few hosts that are causing the majority of your issues.

The fact that you have repeated attempts to send a packet (1,2,3,4 collision frames TX and collision fragments RX) suggests that either the delivery rate of frames is exceeded for the port (broadcast issue) or the same wire pair is being used for transmit and recieve (duplex issue).

HTH

[Joris Deprouw]

Thanks,

This is very usefull information.

We have already considered to put the devices in a separate vlan to limit broadcast effects.

I will check which speed and duplex settings are used on the devices.

the size of the vlan is a /16.

best Regards,

Joris

[Wantser1981_2]

No Problem.

/16 is only a theoretical size. Potentially 65535 hosts. Just wondering what you think the usage is out of that /16?

Ie, have you used almost all address space in the /16?

Cheers

Wantser

[Leo Laohoo]

Your output doesn't say much other than you had SOME collisions due to speed and duplex mismatch. 

If you notice your output drops incrementing sharply, run the two commands again.

1.  The "sh controll util" is a real-time snapshot of your utilization (in percentage) the very momemt you run the command.

2.  The "sh controll e " is a very good counter which will show you what specific line errors are being recorded.  The reason why I like this command is EVERYONE knows how to clear the counters of an interface.  Not everyone knows that another set of counters exist and not a lot of people know how to clear the counters of this command. 

Also, your IOS is pretty old.  Try 12.2(55)SE8.

[Joris Deprouw]

Hi All,

We have found a solution in the meantime by putting the devices on the ports with the high droprate in another vlan.

We assume they had to much interference of other traffic.

We also did a wireshark capture on a affected port and saw that each time we had drops there was a burst of NLB traffic in that vlan coming from a server with NLB configured.

We are planning to deactivate all NLB and use a proper load balancer.

Best Regards,

Joris