Underruns, Overruns, Output Drops, Input Drops

[Peter.D.Brown]

Hi there,

I'm a little confused as to what some of the fields on the output of a 'show interfaces' for an ethernet interface. Am I right in thinking that:

'overruns' would show frames dropped that are coming in from a server (or other device) physically connected to the switch interface ( and are coming in too fast (i.e. the interface is being overrun with traffic)

'underruns' are similar but opposite, i.e. traffic is coming into the switch from other interfaces or VLANs and is queued up to be output from the interface in question and there's too much traffic for the buffer on the interface to handle so some of it gets dropped before it can be output to the connected server

If this is the case then what exactly is 'output drops'? I've pasted the output from a 'show interface' command below. There's no underruns or overruns but there's 111157 'Total output drops'. This seems strange in itself but I've also noticed that there's no corresponding 'Total input drops'. How come?

Can anyone shed any light on this for me please?

Switch#sh int fa3/7

FastEthernet3/7 is up, line protocol is up (connected)

Hardware is C6k 100Mb 802.3, address is 000b.bfa7.f40a (bia 000b.bfa7.f40a)

Description: Link to Derbyshire House

Internet address is 10.162.254.77/30

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 66/255, rxload 63/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 4d22h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 111157

Queueing strategy: fair queuing

Output queue: 0/40 (size/max)

5 minute input rate 24755000 bits/sec, 5104 packets/sec

5 minute output rate 26219000 bits/sec, 5337 packets/sec

L2 Switched: ucast: 704014 pkt, 63492639 bytes - mcast: 185652 pkt, 11172437 bytes

L3 in Switched: ucast: 914707971 pkt, 714070112021 bytes - mcast: 1 pkt, 60 bytes mcast

L3 out Switched: ucast: 875392728 pkt, 401015416078 bytes

915138941 packets input, 731553146961 bytes, 0 no buffer

Received 99928 broadcasts (8041349 IP multicast)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

875708975 packets output, 418861756988 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Switch#

Thanks

Pete.

[lgijssel]

Your definition of "underruns" was new to me and I think it does not really apply here. Please check Wikipedia on this: http://en.wikipedia.org/wiki/Buffer_underrun

Output drops are occuring when an output buffer cannot hold the data long enough to be transmitted or when it is not able to accept more packets (i.e. buffer=full)

Another very common cause for buffer drops is qos. Lower priority traffic gets dropped while higher priority traffic is forwarded. This is not necessarily an indication of a problem but merely a mechanism for the device to regulate traffic flows.

regards,

Leo

[garytayl]

Pete,

This is a confusing one but let me see if I can help you out.

Output drops are the Count of packets dropped on the output interface.

When packets are processed, they are sent to the output queue of the outgoing interface. Output drops are caused by a congested interface. For example, the traffic rate on the outgoing interface cannot accept all packets that should be sent out.

By definition:

overrun - The number of times the receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver's ability to handle the data.

underruns - The number of times the transmitter has been running faster than the router can handle.

As you can see the Output drops is oriented to the switch/router internal process of handling the packets while the overruns and the underruns is showing the number of times one side was faster that the other.

Hope it helps...

[Peter.D.Brown]

Thanks both for the replies. The thing is I've read the Cisco definition of overrun and underrun but they're not very clear, and the definition of underrun seems to conflict with what Leo sent in the Wikipedia link. In fact reading them again the Cisco definitions just create confusion. The definitions for overrun and underrun actually appear to say the same thing but with different words. And what's the receiver hardware? Would that be a server physically connected to the switch interface in question? It does seem that Cisco like to create confusion :o) Thanks.

[Joseph W. Doherty]

With underruns and overruns, you're dealing with the hardware at low level, similar to CRC errors.

Only Cisco could document what would make for either a overrun or underrun error on a particular piece of hardware, however in general . . .

As an Ethernet interface receives the frame's bits, it constructs the received Ethernet frame. It needs to store the frame while it is being received and constructed. Usually the received frame is stored in some special location within the hardware of the Ethernet interface or memory reserved to it. Either is usually a very finite and limited resource.

When the whole frame is received, the Ethernet hardware usually signals the rest of the system to take the received frame so that the receivign resource can be reused. Since additional frames may continue to be received by the Ethernet inteface, if the already received frame is not extracted in a timely manner, releasing the resource being used, a later received frame will "overrun" it. Such received frames that could not be received, due to lack of Ethenet receiving resources, are lost.

"Underruns" normally indicate hardware that requires data within certain time constraints, but did not receive it when it needed it.

Of the two, both might be caused by an overly busy device, but underruns, for sending data, are often especially uncommon even then because normally the effective transmission rates just falls (e.g. increased time between frames).

[Ryan Carretta]

I see you are posing this question given the output of a Cat6k - so I'm going to try to tailor my response to that platform.

Output drops happen when a link is congested. Imagine you have two 100MB server connections, and a 100MB uplink. (200MB -> 100MB). The 100M connection just simply won't be able to push the data if the servers are operating at capacity, and will be forced to drop whatever packets the interface is unable to serialize on the wire. These will manifest as output drops.

Underruns are exceedingly rare on hardware-based platforms. The output is a throwback from IOS, which was developed originally for SW platforms. I've yet to see a customer run into underruns on the 6k. That being said, Joseph's explanation of them is accurate. Underruns will occur when the low level transceiver tries to send data, but the buffer is still empty because the device hasn't serviced the packets yet.

Overruns often happen when an internal link between ASICs on board a module becomes congested. For example, on a WS-X6148-GE-TX, each sequential 8 ports share a 1GB pipe up to an ASIC. If two (or more) interfaces on this link are pushing an aggregate 1GB+, the system will start dropping these packets. This will manifest as overruns on the interfaces in question.

[Peter.D.Brown]

Thanks all for the input (slight pun intended there :o)

I'm now a lot clearer on what these things mean.

Cheers

Pete.

[Jackong]

Hi everyone,

I have the same issue with our network where ping drops.  We have firewall module in our Core switches as Active/Standby mode and servers are connected to Core 1 and Core 2 in port G2/43.  Our server has 2x10GB interface which is connected to Core 1 and Core 2 and our Core switches interface has Gigabit only.

Regards,

Jason

[Joseph W. Doherty]

OP was output drops, yours overruns, so not the same issue.  Possibly some similar hardware, a Catalyst 6500(?), but all the hardware isn't described in your or OP.

I would suspect your overruns are due to hardware being unable to keep up with offered load, which can happen on a 6k depending on sups and line cards (and DFCs), especially if 10g is being used in chassis.

As a percentage of drops, the overruns wouldn't seem to account for the frequency of ping drops, but you don't detail the ping topology plus ping drops are not always due to lost packets.

You also mention a FW service module, which have their own set of performance issues.

Recommend you start a new conversation.

[Jackong]

Hi Joseph,

Thank you for your response.

Yes, my Core switches are 6500 Series. 

My Line cards are all Gigabit interfaces and Servers connected on that particular port is 10GB. 

Actually, when ping drops constantly, it includes disconnection of our all application thru the databases.

I have included from the attachment with regards to the hardware inventory of my core switch 1 which is identical to my Core 2.

Regards,

Jason

[Joseph W. Doherty]

It appears the problematic port is on a 6148 line card.  If so, see @Ryan Carretta reply where he mentions you can see overrun errors on that card, and why.

What he didn't describe is one possible mitigation approach which is to carefully distribute what ports are used based on bandwidth needs and importance.  I.e. each bank of 8 ports has but just one gig to share.

BTW, I recall that card was for users not servers, as the former usually isn't as bandwidth demanding.

Also, BTW, the performance capacity of the 6500 series varies much, mostly based on what cards are installed in it.  (Fully understanding how to build out a 6500, to obtain various performance levels, was almost a subspecialty.)

[Jackong]

Hi Joseph,

Thank you for your clear and information with regards to my query.

Regards,

Jason