Buffer errors on 7201 interface

faisal bargoob · ‎10-14-2013

Hi Experts,

I am having an issue and need your advise, basically 2 days ago my cisco 7201 box unresponsive and getting these messages... this happen once and logs as below

Oct 13 14:28:42.075: %IP_VFR-3-SYSTEM_ERROR: GigabitEthernet0/3: IP VFR System failure - Unable to coalesce fragments

.Oct 13 14:28:42.075: %IP_VFR-3-SYSTEM_ERROR: GigabitEthernet0/3: IP VFR System failure - Unable to coalesce fragments

.Oct 13 14:28:42.079: %IP_VFR-3-SYSTEM_ERROR: GigabitEthernet0/3: IP VFR System failure - Unable to coalesce fragments

After the above message CPU goes as high as 90% for almost 10 mins and the unable to access the box due to high CPU usage.

Please see the version, memory, cpu and buffer as below

VERSION:

Cisco IOS Software, 7200 Software (C7200P-ADVENTERPRISEK9-M), Version 12.4(22)T, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 10-Oct-08 11:19 by prod_rel_team

Cisco 7201 (c7201) processor (revision B) with 917504K/65536K bytes of memory.

Processor board ID 78003287

MPC7448 CPU at 1666Mhz, Implementation 0, Rev 2.2

1 slot midplane, Version 2.0

MEMORY:

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 67F29A0 813748608 97108312 716640296 449114188 554759020

I/O 38000000 67108864 6952028 60156836 57029728 58238044

Transient 37000000 16777216 29644 16747572 7760324 16714272

CPU:

CPU utilization for five seconds: 15%/11%; one minute: 20%; five minutes: 19%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

BUFFER:

Buffer elements:

1117 in free list (1119 max allowed)

558574750 hits, 0 misses, 619 created

Public buffer pools:

Small buffers, 104 bytes (total 251, permanent 50, peak 363 @ 1d17h):

99 in free list (20 min, 150 max allowed)

499472993 hits, 122234 misses, 103273 trims, 103474 created

24956 failures (0 no memory)

Middle buffers, 600 bytes (total 44, permanent 25, peak 535 @ 1d16h):

43 in free list (10 min, 150 max allowed)

1249422606 hits, 29578 misses, 48013 trims, 48032 created

6943 failures (0 no memory)

Big buffers, 1536 bytes (total 50, permanent 50, peak 342 @ 1d17h):

49 in free list (5 min, 150 max allowed)

3346351967 hits, 4992 misses, 5350 trims, 5350 created

903 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 16, permanent 10, peak 307 @ 1d17h):

15 in free list (0 min, 100 max allowed)

2356926850 hits, 89310 misses, 4075 trims, 4081 created

88517 failures (0 no memory)

Large buffers, 5024 bytes (total 1, permanent 0, peak 85 @ 1d16h):

1 in free list (0 min, 10 max allowed)

6036 hits, 82481 misses, 4563 trims, 4564 created

82481 failures (0 no memory)

Huge buffers, 18024 bytes (total 1, permanent 0, peak 50 @ 1d16h):

1 in free list (0 min, 4 max allowed)

2527 hits, 79959 misses, 4313 trims, 4314 created

79957 failures (0 no memory)

Interface buffer pools:

Syslog ED Pool buffers, 600 bytes (total 282, permanent 282):

250 in free list (282 min, 282 max allowed)

18540 hits, 0 misses

IPC buffers, 4096 bytes (total 2, permanent 2):

2 in free list (1 min, 8 max allowed)

0 hits, 0 fallbacks, 0 trims, 0 created

0 failures (0 no memory)

Header pools:

Header buffers, 0 bytes (total 511, permanent 256, peak 511 @ 7w0d):

255 in free list (256 min, 1024 max allowed)

171 hits, 85 misses, 0 trims, 255 created

0 failures (0 no memory)

256 max cache size, 256 in cache

12150946 hits in cache, 0 misses in cache

Particle Clones:

1024 clones, 8901583 hits, 0 misses

Public particle pools:

F/S buffers, 128 bytes (total 512, permanent 512):

0 in free list (0 min, 512 max allowed)

512 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

512 max cache size, 512 in cache

1 hits in cache, 0 misses in cache

Normal buffers, 512 bytes (total 2048, permanent 2048):

2048 in free list (1024 min, 4096 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Private particle pools:

HQF buffers, 0 bytes (total 2000, permanent 2000):

2000 in free list (0 min, 2000 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

GigabitEthernet0/0 buffers, 512 bytes (total 1000, permanent 1000):

0 in free list (0 min, 1000 max allowed)

1000 hits, 0 fallbacks

1000 max cache size, 866 in cache

2689206466 hits in cache, 0 misses in cache

14 buffer threshold, 0 threshold transitions

GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):

0 in free list (0 min, 1000 max allowed)

1000 hits, 0 fallbacks

1000 max cache size, 871 in cache

2284574763 hits in cache, 0 misses in cache

14 buffer threshold, 0 threshold transitions

GigabitEthernet0/2 buffers, 512 bytes (total 1000, permanent 1000):

0 in free list (0 min, 1000 max allowed)

1000 hits, 0 fallbacks

1000 max cache size, 871 in cache

2799321140 hits in cache, 0 misses in cache

14 buffer threshold, 0 threshold transitions

GigabitEthernet0/3 buffers, 2304 bytes (total 529, permanent 400):

129 in free list (128 min, 1200 max allowed)

357 hits, 43 misses, 6953 trims, 7082 created

0 failures (0 no memory)

400 max cache size, 271 in cache

1081763930 hits in cache, 0 misses in cache

14 buffer threshold, 0 threshold transitions

FastEthernet0/0 buffers, 2304 bytes (total 529, permanent 400):

129 in free list (128 min, 1200 max allowed)

357 hits, 43 misses, 6953 trims, 7082 created

0 failures (0 no memory)

400 max cache size, 272 in cache

482978334 hits in cache, 0 misses in cache

14 buffer threshold, 0 threshold transitions

I have seen post regarding buffer issues for 12.4(24) is one of the affected versions but my version is 12.4(22)T as you can see from sh verison:

Cisco link as below:

https://supportforums.cisco.com/thread/2110260

Your immediate response will be highly appericated

FAZ

Joseph W. Doherty · ‎10-14-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

If there's a later version of 12.4(22)T#, you might try it. The 12.4(#)T series, often introduced features with each release, and often the first iteration has some "defects".

Cisco IOS doesn't often deal with excessive memory fragmentation. Because of this, you might need to add RAM or try to find ways to avoid memory churn. (If memory fragmentation is an issue, you'll need to reload to start with a "fresh slate".)

You might try auto buffer tuning, buffers tune automatic, if supported.