Total output drops on WS-C2960X-48TS-L

[reaper8055]

I see a lot of total output drops on my 2960X, not on all interfaces but on some specific interfaces. I have two of WS-C2960X-48TS-L. The switches are connected with to other using LACP on port 45,46. Each of the switches have 7 nodes (servers) running cent os 7 with bonded interfaces as follows:

DEVICE=bond0

ONBOOT=yes

TYPE=Bond

BONDING_MASTER=yes

BONDING_OPTS="miimon=100 updelay=0 downdelay=0 mode=802.3ad"

BOOTPROTO=none

IPADDR=192.168.1.10

NETMASK=255.255.255.0

GATEWAY=192.168.1.1

DNS1=8.8.8.8

On the switch side the LACP mode is set to active for every server underneath. I don't know why there are these packet drops but they keep happening every time. Here's a sample output

-------------------------------------------------------------------------------

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 23727

GigabitEthernet1/0/46 is up, line protocol is up (connected)

Port-channel1 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 23824

----------------------------------------------------------------------------------------

GigabitEthernet1/0/27 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 843

Port-channel7 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 927

----------------------------------------------------------------------------------------

GigabitEthernet1/0/32 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 211

GigabitEthernet1/0/34 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 65

Port-channel6 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 276

----------------------------------------------------------------------------------------

sw# sh lacp internal

Channel group 1

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/45  SA      bndl      32768         0x1       0x1     0x12E       0x3D 

Gi1/0/46  SA      bndl      32768         0x1       0x1     0x12F       0x3D 

Channel group 2

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/26  SA      bndl      32768         0x2       0x2     0x11B       0x3D 

Gi1/0/36  SA      bndl      32768         0x2       0x2     0x125       0x3D 

Channel group 3

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/29  SA      bndl      32768         0x3       0x3     0x11E       0x3D 

Gi1/0/39  SA      bndl      32768         0x3       0x3     0x128       0x3D 

Channel group 4

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/28  SA      bndl      32768         0x4       0x4     0x11D       0x3D 

Gi1/0/38  SA      bndl      32768         0x4       0x4     0x127       0x3D 

Channel group 5

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/31  SA      bndl      32768         0x5       0x5     0x120       0x3D 

Gi1/0/35  SA      bndl      32768         0x5       0x5     0x124       0x3D 

Channel group 6

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/32  SA      bndl      32768         0x6       0x6     0x121       0x3D 

Gi1/0/34  SA      bndl      32768         0x6       0x6     0x123       0x3D 

Channel group 7

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/27  SA      bndl      32768         0x7       0x7     0x11C       0x3D 

Gi1/0/33  SA      bndl      32768         0x7       0x7     0x122       0x3D 

Channel group 8

                            LACP port     Admin     Oper    Port        Port

Port      Flags   State     Priority      Key       Key     Number      State

Gi1/0/25  SA      bndl      32768         0x8       0x8     0x11A       0x3D 

Gi1/0/37  SA      bndl      32768         0x8       0x8     0x126       0x3D 

---------------------------------------------------------------------------------------

sw# sh int g1/0/45 count err

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards

Gi1/0/45            0           0           0           0          0        23727

Port      Single-Col  Multi-Col   Late-Col  Excess-Col  Carri-Sen      Runts     Giants

Gi1/0/45           0          0          0           0          0          0          0

----------------------------------------------------------------------------------------

sw# sh int g1/0/32 count err

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards

Gi1/0/32            0           0           0           0          0          211

Port      Single-Col  Multi-Col   Late-Col  Excess-Col  Carri-Sen      Runts     Giants

Gi1/0/32           0          0          0           0          0          0          0

I am not sure what is causing this. Also I don't know exactly to start debugging the issue. Any help is highly appreciated.

[Leo Laohoo]

Post the complete output to the command "sh int <PORT> controll".

[reaper8055]

Here's the output: sh int g1/0/45 controll

GigabitEthernet1/0/45 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 00f8.2ca6.2b2d (bia 00f8.2ca6.2b2d)

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 2/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:01, output 00:00:07, output hang never

  Last clearing of "show interface" counters 2w3d

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 23727

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 2156000 bits/sec, 945 packets/sec

  5 minute output rate 8048000 bits/sec, 1087 packets/sec

     1546978462 packets input, 480157112357 bytes, 0 no buffer

     Received 164758 broadcasts (129962 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 129962 multicast, 0 pause input

     0 input packets with dribble condition detected

     1230069485 packets output, 998157393652 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

     Transmit GigabitEthernet1/0/45           Receive

      3283669 Bytes                       3596153380 Bytes                    

   3567847126 Unicast frames              2720254033 Unicast frames           

      1959968 Multicast frames                158459 Multicast frames         

       311192 Broadcast frames                 92557 Broadcast frames         

            0 Too old frames              3561135311 Unicast bytes            

            0 Deferred frames               28960368 Multicast bytes          

            0 MTU exceeded frames            5923661 Broadcast bytes          

            0 1 collision frames                   0 Alignment errors         

            0 2 collision frames                   0 FCS errors               

            0 3 collision frames                   0 Oversize frames          

            0 4 collision frames                   0 Undersize frames         

            0 5 collision frames                   0 Collision fragments      

            0 6 collision frames       

            0 7 collision frames             1880269 Minimum size frames      

            0 8 collision frames           704003026 65 to 127 byte frames    

            0 9 collision frames          1181079199 128 to 255 byte frames   

            0 10 collision frames           22855388 256 to 511 byte frames   

            0 11 collision frames            2735429 512 to 1023 byte frames 

            0 12 collision frames          807951738 1024 to 1518 byte frames

            0 13 collision frames                  0 Overrun frames           

            0 14 collision frames                  0 Pause frames             

            0 15 collision frames      

            0 Excessive collisions                 0 Symbol error frames      

            0 Late collisions                      0 Invalid frames, too large

            0 VLAN discard frames                  0 Valid frames, too large 

            0 Excess defer frames                  0 Invalid frames, too small

      1655254 64 byte frames                       0 Valid frames, too small 

    272271472 127 byte frames          

   1180343648 255 byte frames                      0 Too old frames           

     40434406 511 byte frames                      0 Valid oversize frames    

     39454569 1023 byte frames                     0 System FCS error frames 

   2035958937 1518 byte frames                     0 RxPortFifoFull drop frame

            0 Too large frames         

            0 Good (1 coll) frames     

            0 Good (>1 coll) frames    

[Leo Laohoo]

Wow, That is a bursty client. Look at the iput/output counter after 2 weeks.
Is QoS enabled?

[reaper8055]

QoS is not configured. Although I checked if there's something out of the box with 

show running-config policy-map and show running-config class-map and I didn't see any config. This is the output for both the commands it gave:

----------------------------------------------------------------------------------------

Building configuration...

Current configuration : 6 bytes

end

-----------------------------------------------------------------------------------------

[reaper8055]

"That is a bursty client"
Can you please explain this a bit. I am kinda new to switching and I am not able figure out everything on the output of "sh int g1/0/45 controll". Thanks in advance.

[Joseph W. Doherty]

A common cause of egress drops is more traffic sent to the egress interface than can be transmitted. The excess traffic is queued, but queues have a finite capacity. When they're exceeded, you have egress drops.

BTW, the 2960 series generally doesn't have a lot of buffer memory, so with busy hosts, often servers, you're more likely to encounter this. Sometimes configuring buffer settings can minimize the drop frequency.

Also BTW, although you might think zero drops is ideal, it's not always, as TCP uses drops to manage its bandwidth usage. (NB: later TCP implementations also use a just in RTT to manage bandwidth usage too.) What you want to avoid is an excessive amount of drops. Traditionally that's if drops are more than 1%. (What's your percentage?)

PS:

Oh, I also wanted to mention, the 2K switches are not really designed for very heavy port usage, such as supporting busy servers.  Such hosts are generally better supported by using Cisco's "better" switches, which have better/more internal hardware resources.

[reaper8055]

Thanks that was really informative. Where can I read more about the TCP bandwidth management. Also, please let me know how do I calculate the percentage drop. 

We are running K8s cluster on these servers which server heavy traffic, can you please suggest any specific switch that is recommended for this. Also we have two of 3850X as well. Those were connected before but were removed because the output drops on the cross-connected switch interfaces were above 95Million. 

[Joseph W. Doherty]

"Where can I read more about the TCP bandwidth management. Also, please let me know how do I calculate the percentage drop. "

Have you tried to search the Internet on the subject?

"We are running K8s cluster on these servers which server heavy traffic, can you please suggest any specific switch that is recommended for this."

The Catalyst 4K and Catalyst 6K switches, both chassis and the "pizza box" models. Possibly some of the new Catalyst 9K switches too.

[reaper8055]

I will look for more information online. Thanks for the information, once again.

Basically, 2k series switches are not ideal for the current setup. Is there anything else that I can do with the hardware (we have two 24 port 3850X as well) we have to minimise the issue.