cause of buffer misses

[WILLIAM STEGMAN]

I have exessive buffer misses and am trying to nail down the cause. As you can see there are many huge buffer misses resulting in dropped packets. I've upgraded my IOS to make sure it wasn't a buffer leak, currently running

12.4(19b) on a 3825. My hunch is there are too many interfaces enabled, but before I start ripping out networks does anyone have any experience with this issue?

thank you,

Bill

hbgwan-t3#sh ip int bri

Interface IP- Status Protocol

GigabitEthernet0/0

x.x.x.x

GigabitEthernet0/1

x.x.x.x

Serial0/0/0 up up

Serial0/1/0 up up

FastEthernet1/0

x.x.x.x

FastEthernet1/1 unassigned

administratively down down

Serial2/0

x.x.x.x

Multilink1

x.x.x.x

hbgwan-t3#sh buffers

Buffer elements:

1118 in free list (1119 max allowed)

904056661 hits, 0 misses, 619 created

Public buffer pools:

Small buffers, 104 bytes (total 61, permanent 50, peak 121 @ 7w0d):

58 in free list (20 min, 250 max allowed)

2207053560 hits, 17449 misses, 22580 trims, 22591 created

22 failures (0 no memory)

Middle buffers, 600 bytes (total 38, permanent 25, peak 85 @ 7w0d):

34 in free list (25 min, 250 max allowed)

122196492 hits, 4799 misses, 8967 trims, 8980 created

20 failures (0 no memory)

Big buffers, 1536 bytes (total 67, permanent 50, peak 99 @ 7w0d):

65 in free list (25 min, 250 max allowed)

2186922488 hits, 12649 misses, 13786 trims, 13803 created

903 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 12 @ 7w0d):

9 in free list (0 min, 100 max allowed)

2514572 hits, 351 misses, 29 trims, 29 created

351 failures (0 no memory)

Large buffers, 5024 bytes (total 1, permanent 0, peak 3 @ 7w0d):

1 in free list (0 min, 30 max allowed)

37 hits, 314 misses, 2131 trims, 2132 created

314 failures (0 no memory)

Huge buffers, 18024 bytes (total 34, permanent 25, peak 130 @ 3w0d):

34 in free list (10 min, 30 max allowed)

1788827957 hits, 1411969 misses, 4235239 trims, 4235248 created

0 failures (0 no memory)

[Joseph W. Doherty]

Your stats aren't really that bad, since a "miss" isn't as bad as a "failure". If you haven't already, you might read http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a7b80.shtml.

If your IOS is a version that supports buffer auto tunning (e.g. 12.3[14]T or later), you could activate it ("buffers tune automatic").

[guruprasadr]

HI,

Please find the attached Useful document.

Understanding Buffer Misses & Failures.

Best Regards,

Guru Prasad R.

[WILLIAM STEGMAN]

I've followed that doc before and did tune buffers, increasing a few, but still get all these misses. What's concerning to me is the huge buffer misses. This translates to dropped packets. When I put the output of the sh buffers command into the cisco's output interpreter it seems to indicate an issue.

ERROR: Since it's last reload, this router has created or maintained a relatively

large number of 'Syslog ED Pool buffers' yet still has very few free buffers.

The above symptoms suggest that a buffer leak has occurred.

Not Enough Shared Memory for the Interfaces.

NOTE:

(1)Some of the Public Buffer pools should be abnormally large with few free buffers. After a reload, you may see that the number of free buffers never gets close to the number of total buffers.

(2)You should check the buffers on a regular basis. Some leaks are slow but others are very fast.

(3)If you configure or access the router through telnet,you need to check the buffers on a regular basis via remote access (telnet) before the router hang to see in which pool is the leak. Once you see that for one pool the total number is increasing and the free number is low (the faulty pool), you need to capture a 'show buffer pool dump'. But if you don't have any memory available on the box, it's too late

to collect the information . You have to collect the information before the hang.

TRY THIS:

Router is running low on shared memory, even after a reload, physically removing

interfaces solves the problem.

This could be a Cisco IOS software bug. Upgrade to the latest version in your release

train to fix known buffer leak bugs. For example, if you are running Cisco IOS

Software Release 11.2(14), upgrade to the latest 11.2(x).

[WILLIAM STEGMAN]

I guess the refresher helped. It's not misses, but failures that cause the packet drops. While I have a lot of misses and failures, there are no failures in the huge buffer pool, which indicates no dropped packets. So, is it accuarate to say that these misses are to be expected, and that I should only be concerned if there are failures in the huge packet buffer?

[Joseph W. Doherty]

"So, is it accuarate to say that these misses are to be expected, and that I should only be concerned if there are failures in the huge packet buffer?"

I think it most accurate to be most concerned about failures since this causes packet drops. But even then, you also need to consider percentages. If the drop percentage is very small, which it appears to be even for your Big Buffers, it shouldn't be a major concern.

As to "misses", I believe it makes the router perform a bit better if you can decrease this, which should also be true for "trims" and "created". All these impose some additional processing overhead.

In later IOSs, buffer stats show a bit more information now that "peak" is reported. Notice in your Huge Buffers, the peak appears to have the highest ratio to its permanent and max allowed settings, which likely accounts for all the misses/trims/created activity.

Since you're running 12.4(19b), you might go ahead and try the buffer auto tunning. BTW, you can "show" the adjustments it has made.

[WILLIAM STEGMAN]

Thanks Joseph. I ran the auto tuning a couple hours ago, but I dont' see much improvement. For example, the last hour we're had about 1900 huge buffer misses. I took a look at the interface stats and looked for any failures related to buffers, but don't see any. I'll continue to keep an eye on them and determine if the percentage of drops compared to total buffers is significant. If at some point that does happen, besides tuning, is there another means to reduce the misses and failures? Must you simply remove load from the router by installing a new router to take over the load of an existing interface and network?

[Joseph W. Doherty]

I haven't had to use buffer auto tunning extensively. However, the little I've used it, it seems to take some time for it to adjust your buffers for what it considers optimal settings. In other words, a day or two might be much better than a couple of hours.

As long as the memory is available, manual tuning of buffers, I would think, should preclude/minimize misses and failures. In other words, hopefully you shouldn't need to reduce your number of interfaces.

Considering where you're seeing most of the misses, I'm wondering if you're running Jumbo Ethernet. If so, that might be much more than buffer defaults expect.