Catalyst 3850 high Total output drops and output errors

shayan_2k@hotmail.com · ‎10-16-2018

Hi All

I m running output drops issue on cisco 3850 stack wise

I have created ether-channel and there are 4 ports bundled in it

actually we are getting alert from Op Manager ( monitoring Tool ) that we reach 87% discard rate

this port-channel are connected to our Server 1 and every night when Server 1 backup and copy image to server 2 then we receive discard error

i think there must be some micro burst issue

Both server are connected to 3850

we have not configured any Qos or Storm control settings

i dont see any duplex mismatch errors

Cables are find we have tried this on different interfaces but same so cant be bad port

here are some show commands output

show interfaces po12 counters errors

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Po12 0 0 0 0 0 35357406

Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts
Po12 0 0 0 0 0 0

#show interfaces po12
Port-channel12 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0045.1d05.e10e (bia 0045.1d05.e10e)
Description: Link Aggregation Channel on (VRTX A) Master and Slave on Port 1/0/14,16, 2/0/14,16 (HOSTA02)
MTU 1500 bytes, BW 4000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/0/14 Gi1/0/16 Gi2/0/14 Gi2/0/16
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 20:15:13
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 35357406
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 1342000 bits/sec, 230 packets/sec
5 minute output rate 1233000 bits/sec, 334 packets/sec
94100365 packets input, 117696352460 bytes, 0 no buffer
Received 22226 broadcasts (9186 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 9186 multicast, 0 pause input
0 input packets with dribble condition detected
62931876 packets output, 50389460222 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Any help ????

brselzer · ‎10-16-2018

Hello,

Can you post the output of:

show int gig1/0/14 controllers

show int gig1/0/16 controllers

show int gig2/0/14 controllers

show int gig2/0/16 controllers

Thanks!

-Bradley Selzer
CCIE# 60833

Richard Burts · ‎10-16-2018

Am I correct in understanding that you have an EtherChannel with 4 ports and correct in understanding that you are having the problem when Server1 is sending large amounts of data to Server2? In that situation I believe that EtherChannel is doing providing much benefit. In the EtherChannel when most of the traffic is from a single host to a single host then the hashing on the EtherChannel will send all of that traffic over a single port. So you probably have one very over used port and three very lightly used ports.

HTH

Rick

HTH

Rick

shayan_2k@hotmail.com · ‎10-16-2018

on server side we created LAG as well and 4 ports are connected so in my understand when we create etherchannel it also do load balancing
and also if u see the drops we get drops on each interface as well

Joseph W. Doherty · ‎10-16-2018

Yes, it's possible from server to switch, the server will load balance across the LAG group. However, from switch to server, it's the switch that decides how to balance across it's LAG ports, and as Rick notes, from one host to another host, commonly only one link is used for all that host to host traffic. (I notice in your later post, with interface stats, the drops counts appears somewhat "skewed" across the LAG ports, correct?)

The next issue, you might be bumping into, Catalyst 2K and 3K switches aren't generally designed to handle really heavy "bursty" traffic, as they often don't have a lot of buffer RAM. That said, I have found, sometimes you can greatly diminish interface output drops by QoS configuration tuning by how the device allocates its buffers.

shayan_2k@hotmail.com · ‎10-16-2018

show int gig1/0/14 controller

GigabitEthernet1/0/14 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0045.1d05.e10e (bia 0045.1d05.e10e)
Description: VRTXA INT Port 2A (HOSTA02-HVLAG)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 35575954
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 152000 bits/sec, 36 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec
     1079756793 packets input, 1334468418537 bytes, 0 no buffer
     Received 382494 broadcasts (195254 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 195254 multicast, 0 pause input
     0 input packets with dribble condition detected
     1739859259 packets output, 1691202610997 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Transmit                  GigabitEthernet1/0/14                 Receive
1691202610997 Total bytes               1334468418537 Total bytes
   1730518243 Unicast frames               1079374299 Unicast frames
1690221709903 Unicast bytes             1334428526954 Unicast bytes
      5921010 Multicast frames                 195254 Multicast frames
    742715612 Multicast bytes                24664459 Multicast bytes
      3420006 Broadcast frames                 187240 Broadcast frames
    238185482 Broadcast bytes                15227124 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames
            0 MacUnderrun frames                    0 MacOverrun frames
            0 Pause frames                          0 Pause frames
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames
            0 Oam frames                            0 OamProcessed frames
            0 Oam frames                            0 OamDropped frames
    337419058 Minimum size frames           167904655 Minimum size frames
     38459719 65 to 127 byte frames          13728389 65 to 127 byte frames
     97612504 128 to 255 byte frames         11874001 128 to 255 byte frames
    138477468 256 to 511 byte frames         14979922 256 to 511 byte frames
    144682956 512 to 1023 byte frames         9598473 512 to 1023 byte frames
     11960518 1024 to 1518 byte frames       42973750 1024 to 1518 byte frames
    971247047 1519 to 2047 byte frames      818697790 1519 to 2047 byte frames
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames
            0 Late collision frames                 0 SymbolErr frames
     35575954 Excess Defer frames                   0 Collision fragments
            0 Good (1 coll) frames                  0 ValidUnderSize frames
            0 Good (>1 coll) frames                 0 InvalidOverSize frames
            0 Deferred frames                       0 ValidOverSize frames
            0 Gold frames dropped                   0 FcsErr frames
            0 Gold frames truncated
            0 Gold frames successful
            0 1 collision frames
            0 2 collision frames
            0 3 collision frames
            0 4 collision frames
            0 5 collision frames
            0 6 collision frames
            0 7 collision frames
            0 8 collision frames
            0 9 collision frames
            0 10 collision frames
            0 11 collision frames
            0 12 collision frames
            0 13 collision frames
            0 14 collision frames
            0 15 collision frames
            0 Excess collision frames

show int gig1/0/16 controller
GigabitEthernet1/0/16 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0045.1d05.e110 (bia 0045.1d05.e110)
Description: VRTXA EXT Slot 3 Port 1 (HOSTA02-HVLAG)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 144000 bits/sec, 37 packets/sec
5 minute output rate 34000 bits/sec, 27 packets/sec
     1085074451 packets input, 1337552566403 bytes, 0 no buffer
     Received 431567 broadcasts (195566 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 195566 multicast, 0 pause input
     0 input packets with dribble condition detected
     137715105 packets output, 19199404865 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Transmit                  GigabitEthernet1/0/16                 Receive
19199404865 Total bytes               1337552566403 Total bytes
     97299486 Unicast frames               1084642884 Unicast frames
15965056010 Unicast bytes             1337508243411 Unicast bytes
     39858836 Multicast frames                 195566 Multicast frames
   3188401860 Multicast bytes                24677983 Multicast bytes
       556783 Broadcast frames                 236001 Broadcast frames
     45946995 Broadcast bytes                19645009 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames
            0 MacUnderrun frames                    0 MacOverrun frames
            0 Pause frames                          0 Pause frames
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames
            0 Oam frames                            0 OamProcessed frames
            0 Oam frames                            0 OamDropped frames
      5289753 Minimum size frames           167427805 Minimum size frames
    100896847 65 to 127 byte frames          18151961 65 to 127 byte frames
     22775007 128 to 255 byte frames         11536142 128 to 255 byte frames
      4307374 256 to 511 byte frames         14824122 256 to 511 byte frames
      1290142 512 to 1023 byte frames         9547408 512 to 1023 byte frames
      1239601 1024 to 1518 byte frames       43016020 1024 to 1518 byte frames
      1916381 1519 to 2047 byte frames      820570993 1519 to 2047 byte frames
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames
            0 Late collision frames                 0 SymbolErr frames
            0 Excess Defer frames                   0 Collision fragments
            0 Good (1 coll) frames                  0 ValidUnderSize frames
            0 Good (>1 coll) frames                 0 InvalidOverSize frames
            0 Deferred frames                       0 ValidOverSize frames
            0 Gold frames dropped                   0 FcsErr frames
            0 Gold frames truncated
            0 Gold frames successful
            0 1 collision frames
            0 2 collision frames
            0 3 collision frames
            0 4 collision frames
            0 5 collision frames
            0 6 collision frames
            0 7 collision frames
            0 8 collision frames
            0 9 collision frames
            0 10 collision frames
            0 11 collision frames
            0 12 collision frames
            0 13 collision frames
            0 14 collision frames
            0 15 collision frames
            0 Excess collision frames

show int gig2/0/16 controller
GigabitEthernet2/0/16 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0045.1d98.3e10 (bia 0045.1d98.3e10)
Description: VRTXA EXT Slot 3 Port 3 (HOSTA02-HVLAG)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2215890
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 148000 bits/sec, 35 packets/sec
5 minute output rate 766000 bits/sec, 142 packets/sec
     1084990486 packets input, 1343304046738 bytes, 0 no buffer
     Received 391505 broadcasts (186969 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 186969 multicast, 0 pause input
     0 input packets with dribble condition detected
     327601217 packets output, 53046629235 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Transmit                  GigabitEthernet2/0/16                 Receive
53046629235 Total bytes               1343304046738 Total bytes
    320281361 Unicast frames               1084598981 Unicast frames
52178026101 Unicast bytes             1343263560030 Unicast bytes
      6855736 Multicast frames                 186969 Multicast frames
    827332637 Multicast bytes                24137509 Multicast bytes
       464120 Broadcast frames                 204536 Broadcast frames
     41270497 Broadcast bytes                16349199 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames
            0 MacUnderrun frames                    0 MacOverrun frames
            0 Pause frames                          0 Pause frames
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames
            0 Oam frames                            0 OamProcessed frames
            0 Oam frames                            0 OamDropped frames
      5239987 Minimum size frames           167252360 Minimum size frames
    288184675 65 to 127 byte frames          13772815 65 to 127 byte frames
     10576494 128 to 255 byte frames         11848187 128 to 255 byte frames
      2996040 256 to 511 byte frames         14990593 256 to 511 byte frames
      1837493 512 to 1023 byte frames         9620043 512 to 1023 byte frames
      3827839 1024 to 1518 byte frames       43174155 1024 to 1518 byte frames
     14938689 1519 to 2047 byte frames      824332333 1519 to 2047 byte frames
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames
            0 Late collision frames                 0 SymbolErr frames
      2215890 Excess Defer frames                   0 Collision fragments
            0 Good (1 coll) frames                  0 ValidUnderSize frames
            0 Good (>1 coll) frames                 0 InvalidOverSize frames
            0 Deferred frames                       0 ValidOverSize frames
            0 Gold frames dropped                   0 FcsErr frames
            0 Gold frames truncated
            0 Gold frames successful
            0 1 collision frames
            0 2 collision frames
            0 3 collision frames
            0 4 collision frames
            0 5 collision frames
            0 6 collision frames
            0 7 collision frames
            0 8 collision frames
            0 9 collision frames
            0 10 collision frames
            0 11 collision frames
            0 12 collision frames
            0 13 collision frames
            0 14 collision frames
            0 15 collision frames
            0 Excess collision frames

show int gig2/0/14 controller
GigabitEthernet2/0/14 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0045.1d98.3e0e (bia 0045.1d98.3e0e)
Description: VRTXA INT Port 2B (HOSTA02-HVLAG)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 600638533
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 146000 bits/sec, 36 packets/sec
5 minute output rate 17000 bits/sec, 18 packets/sec
     1050846285 packets input, 1297989362811 bytes, 0 no buffer
     Received 494772 broadcasts (184054 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 184054 multicast, 0 pause input
     0 input packets with dribble condition detected
     457230513 packets output, 667246087870 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Transmit                  GigabitEthernet2/0/14                 Receive
667246087870 Total bytes               1297989457619 Total bytes
    449930267 Unicast frames               1050351696 Unicast frames
666384981497 Unicast bytes             1297942548904 Unicast bytes
      6195734 Multicast frames                 184054 Multicast frames
    768098119 Multicast bytes                23798278 Multicast bytes
      1104512 Broadcast frames                 310718 Broadcast frames
     93008254 Broadcast bytes                23110437 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames
            0 MacUnderrun frames                    0 MacOverrun frames
            0 Pause frames                          0 Pause frames
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames
            0 Oam frames                            0 OamProcessed frames
            0 Oam frames                            0 OamDropped frames
      8206031 Minimum size frames           164217455 Minimum size frames
      2865766 65 to 127 byte frames          13280273 65 to 127 byte frames
      7272261 128 to 255 byte frames         11332337 128 to 255 byte frames
       846319 256 to 511 byte frames         14571000 256 to 511 byte frames
      1319456 512 to 1023 byte frames         9323974 512 to 1023 byte frames
      2931564 1024 to 1518 byte frames       42008000 1024 to 1518 byte frames
    433789116 1519 to 2047 byte frames      796113429 1519 to 2047 byte frames
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames
            0 Late collision frames                 0 SymbolErr frames
    600638533 Excess Defer frames                   0 Collision fragments
            0 Good (1 coll) frames                  0 ValidUnderSize frames
            0 Good (>1 coll) frames                 0 InvalidOverSize frames
            0 Deferred frames                       0 ValidOverSize frames
            0 Gold frames dropped                   0 FcsErr frames
            0 Gold frames truncated
            0 Gold frames successful
            0 1 collision frames
            0 2 collision frames
            0 3 collision frames
            0 4 collision frames
            0 5 collision frames
            0 6 collision frames
            0 7 collision frames
            0 8 collision frames
            0 9 collision frames
            0 10 collision frames
            0 11 collision frames
            0 12 collision frames
            0 13 collision frames
            0 14 collision frames
            0 15 collision frames
            0 Excess collision frames

brselzer · ‎10-17-2018

Hello,

As others have mentioned, you are bursting over 1Gbps on these links:

35575954 Excess Defer frames <----Means you have more frames than the link can handle

You will want to investigate moving buffers around or deploying a qos configuration. It is also possible that if the drops are happening late at night and not causing an impact, you don't need to do anything. If the backups are not failing or going too slow, this might just be expected behavior. If you have some drops, TCP will re-converge and continue the backup.

Hope that helps!

-Bradley Selzer
CCIE# 60833

Richard Burts · ‎10-17-2018

Bradley

Thank you for the explanation pointing out the meaning of excess defer frames. Very helpful. +5

HTH

Rick

HTH

Rick

shayan_2k@hotmail.com · ‎10-19-2018

yes right i think its overnight and not impacting any production network so we can leave it for now

Leo Laohoo · ‎10-16-2018

shayan_2k@hotmail.com wrote:

this port-channel are connected to our Server 1 and every night when Server 1 backup and copy image to server 2 then we receive discard error

we have not configured any Qos or Storm control settings

Yup. This is a QoS issue.