Re: 3750 output drops at exact multiples of 78

BrettCurtis5442 · ‎05-23-2019

I've got a 3750 switch that's being flagged in solarwinds for discards. This is a distribution switch with gig ports, and its single uplink port to a Riverbed and satellite wan link is the one recording drops. It's average utilization is a pretty consistent 300K. There's no QoS configured, just a basic trunk configuration.

First question, what is the time interval on the sh int output drops stat? Every time I display it, it's a multiple of exactly 78.

Second question: It's weird that it's exactly 78 bytes or an exact multiple of 78. Anyone have any ideas why that is?

sh int gi1/0/1
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 7c69.f6f1.9981 (bia 7c69.f6f1.9981)
Description:
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:31, output 00:00:35, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 78
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 307000 bits/sec, 87 packets/sec
5 minute output rate 275000 bits/sec, 77 packets/sec
5288829618 packets input, 2283416132780 bytes, 0 no buffer
Received 126878804 broadcasts (54084452 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 54084452 multicast, 0 pause input
0 input packets with dribble condition detected
8516447191 packets output, 9289365956118 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Hitting the show interface several times in a row:

Total output drops: 156

Total output drops: 234

Total output drops: 78

Total output drops: 156

etc, all day

BrettCurtis5442 · ‎05-23-2019

Actually it's happening on ten interfaces.

5 minute input rate 2290000 bits/sec, 315 packets/sec
5 minute output rate 889000 bits/sec, 292 packets/sec

Total output drops: 6215

Total output drops: 12430

Total output drops: 18645

5 minute input rate 21000 bits/sec, 3 packets/sec
5 minute output rate 29000 bits/sec, 32 packets/sec

Total output drops: 7297

Total output drops: 14594 (@0/18000b/s in/out)

5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 25000 bits/sec, 25 packets/sec

Total output drops: 7304

Total output drops: 14608

Joseph W. Doherty · ‎05-23-2019

What IOS are you running?

The 3750 is a bit infamous for drops due to it only having 4 MB RAM per 24 copper ports or its uplink ports. If QoS is enabled, and running defaults, often you see higher drops than with QoS totally disabled. However, often the least number of drops is with QoS enabled and configured for your needs.

That said, 78 drops out of 8,516,447,191 is usually not a cause for concern. Traditionally, you need to be concerned only when drop rate exceeds 1%.

BrettCurtis5442 · ‎05-23-2019

It's running 12.2. No qos. It's 3 x 12 ports stacked.

Now that I think about this maybe this will cut to the chase. Isn't the drop counter supposed to be a total since counters were last cleared? It's been a while since I looked closely at that counter. I'm definitely not getting a total. In my first example, it's going 78, 156, 78, 0, 234, etc, exact multiples randomly.

I didn't see this one earlier, but look at this one. On this interface, the drops are either 0, 457, 7304, or 4294959992, mostly cycling back and forth between 7304 and 4.3B. Randomly hit the show interface all day, and you get exactly one of those three numbers. It's dropping 4.3B packets over and over on a link that's running 20Kbps? It looks like some sort of bug to me. I don't believe these numbers are real. I want to re-boot it but I don't have an outage window.

GigabitEthernet1/0/5 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 7c69.f6f1.9985 (bia 7c69.f6f1.9985)
Description:
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:07, output hang never
Last clearing of "show interface" counters 02:51:32
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4294959992 :)
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 20000 bits/sec, 25 packets/sec
48368 packets input, 6431363 bytes, 0 no buffer
Received 10641 broadcasts (3629 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3629 multicast, 0 pause input
0 input packets with dribble condition detected
223881 packets output, 60474484 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

BrettCurtis5442 · ‎05-23-2019

Never mind. It's a bug https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtq86186

I hate being somewhere new looking at old software. Now that this has messed with my mind, someone confirm total output drops is a total counter since counters were last cleared?