Solved: Huge Buffers + MALLOC Errors

LuvMacheen · ‎03-09-2005

I have a 2600 router that supports a 250 user base at a remote site. Soon to be 800 users. Just yesterday I noticed the following Mallocfail error. Then (I will post error, sho ver and sho buff) I noticed the error is associated with Huge Buffer failures. I've read the associated cisco docs. How do I know if it would help me to modify the number of buffers allowed? And will it help me or is my best bet to find out who's trying to send the 5025+ byte packets and 'stop' them. I thought the router had plenty of memory when I installed it... recommendations?

Mar 9 10:11:34.043: %SYS-2-MALLOCFAIL: Memory allocation of 18196 bytes failed from 0x8041543C, alignment 32

Pool: I/O Free: 19520 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 35

-Traceback= 80472A40 80473678 80415440 804158BC 80415FB0 805809D0 805806B4 8057DBF8 8057DDDC 8057DE90 8057DFE4 8049831C 8049B9C8

sho version

Cisco Internetwork Operating System Software

IOS (tm) C2600 Software (C2600-BIN-M), Version 12.3(13), RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 10-Feb-05 02:12 by ssearch

Image text-base: 0x80008098, data-base: 0x80DBEC20

ROM: System Bootstrap, Version 12.1(3r)T2, RELEASE SOFTWARE (fc1)

ROM: C2600 Software (C2600-BIN-M), Version 12.3(13), RELEASE SOFTWARE (fc2)

router uptime is 5 days, 1 hour, 22 minutes

System returned to ROM by reload

System restarted at 09:10:41 UTC Fri Mar 4 2005

System image file is "flash:c2600-bin-mz.123-13.bin"

cisco 2621 (MPC860) processor (revision 0x600) with 59392K/6144K bytes of memory.

Processor board ID JAD05380NY7 (453869456)

M860 processor: part number 0, mask 49

Bridging software.

X.25 software, Version 3.0.0.

Primary Rate ISDN software, Version 1.1.

2 FastEthernet/IEEE 802.3 interface(s)

48 Serial network interface(s)

2 Channelized T1/PRI port(s)

32K bytes of non-volatile configuration memory.

16384K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

sho buffer

Buffer elements:

500 in free list (500 max allowed)

11424936 hits, 0 misses, 0 created

Public buffer pools:

Small buffers, 104 bytes (total 53, permanent 50, peak 62 @ 1d19h):

52 in free list (20 min, 150 max allowed)

5226107 hits, 112 misses, 53 trims, 56 created

23 failures (0 no memory)

Middle buffers, 600 bytes (total 25, permanent 25, peak 44 @ 5d01h):

25 in free list (10 min, 150 max allowed)

3105696 hits, 196 misses, 19 trims, 19 created

6 failures (0 no memory)

Big buffers, 1536 bytes (total 50, permanent 50):

50 in free list (5 min, 150 max allowed)

2132611 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 10, permanent 10):

10 in free list (0 min, 20 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Large buffers, 5024 bytes (total 0, permanent 0):

0 in free list (0 min, 10 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Huge buffers, 18024 bytes (total 4, permanent 0, peak 26 @ 4d15h):

4 in free list (0 min, 4 max allowed)

1812440 hits, 299700 misses, 572825 trims, 572829 created

9951 failures (22344 no memory)

mhussein · ‎03-09-2005

Your best bet is to open a case with Cisco TAC to rule out any bugs related to the traceback.

If you decide to tune the buffers, I'd suggest first increasing the iomem:

"cisco 2621 (MPC860) processor (revision 0x600) with 59392K/6144K bytes of memory"

to say 12% ~7.6MB (this will require rebooting the router):

router(config)#memory-size iomem 12

Then "show buffer" and see if the number of allocated buffers is changed or not.

Next, check the output of "show interfaces" and look for "no buffers" errors which indicate a process switching issue. This can be confirmed if the cpu utilization consistently shoots up to 100%.

Reference:

http://www.cisco.com/warp/public/63/buffertuning.html

Regards,

Mustafa

View solution in original post

don.fortes · ‎03-09-2005

Cisco Output interpreter results.

%SYS-2-MALLOCFAIL (x1): Memory allocation of [dec] bytes failed from [hex], alignment

[dec] Pool: [chars] Free: [dec] Cause: [chars] Alternate Pool: [chars] Free:

[dec] Cause: [chars]

Explanation: The requested memory allocation is not available from the specified memory pool. The current system configuration, network environment, or possibly a software error may have exhausted or fragmented the router's memory.

Recommended Action: If you suspect a software error, copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical support representative, and provide the representative with the information you have gathered.Otherwise, you may need to add more memory or reduce the number of features you are trying to support.

mhussein · ‎03-09-2005

Your best bet is to open a case with Cisco TAC to rule out any bugs related to the traceback.

If you decide to tune the buffers, I'd suggest first increasing the iomem:

"cisco 2621 (MPC860) processor (revision 0x600) with 59392K/6144K bytes of memory"

to say 12% ~7.6MB (this will require rebooting the router):

router(config)#memory-size iomem 12

Then "show buffer" and see if the number of allocated buffers is changed or not.

Next, check the output of "show interfaces" and look for "no buffers" errors which indicate a process switching issue. This can be confirmed if the cpu utilization consistently shoots up to 100%.

Reference:

http://www.cisco.com/warp/public/63/buffertuning.html

Regards,

Mustafa

LuvMacheen · ‎03-10-2005

Thanks Mustafa... I already had a TAC case and they made the same recommendation you did. Now that I understand the way the router uses memory a little better it all makes sense. I made the changes last night and since then no more errors or buffer failures.