Core issue
This is similar to how the full log message looks:
%SYS-2-MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x601617A4,
pool Processor, alignment 0 -Process= "IP Input", ipl= 2, pid= 21
These messages indicate that a process was unable to find a large enough contiguous block of memory. In this example, the IP input process attempted to get 1028 bytes from the Processor pool of memory.
The memory is divided into two different sections, Processor and I/O. Processor memory is used for the processes on the router that run periodically and need memory at certian times. I/O memory is reserved for interrupt processes. Very often, malloc fail messages are only a symptom of the problem that happened at that
time in the network due to excessive traffic. In some cases, this is not seen after a reload or hard resetting the switch.
These are the causes of these messages:
- Insufficient memory in the Route Switch Module (RSM) to support the Cisco IOS Software or the size of the routing table
- A memory leak occurs
- Fragmented memory
- Excessive number of Address Resolution Protocol (ARP) requests. If a default route points to a broadcast interface, the switch has to resolve ARP for all traffic that hit this default route. This in turn raises the memory utilization of ARP input.
A Catalyst 5000 Series Switch with an RSM can also display the %SYS-2-MALLOCFAIL messages.
Resolution
Complete these steps in order to resolve this issue:
- Verify that the Catalyst Switch or RSM has enough DRAM to support the Cisco IOS Software. Issue the show version command.
This is a sample command output: ------------------ show version ------------------
Cisco Internetwork Operating System Software
IOS (tm) C5RSM Software (C5RSM-DSV-M), Version 11.2(13)P, RELEASE
!--- Version
SOFTWARE (fc1)
Copyright (c) 1986-1998 by cisco Systems, Inc.
Compiled Wed 08-Apr-98 01:11 by dschwart
Image text-base: 0x60010900, data-base: 0x6085C000
ROM: System Bootstrap, Version 11.2(17523) [mohsen 102], INTERIM
SOFTWARE
BOOTFLASH: C5RSM Software (C5RSM-BOOT-M), Version 11.2(12a.P1)P1,
MAINTENANCE IN
TERIM SOFTWARE
RSM1 uptime is 17 hours, 21 minutes
System restarted by reload
System image file is "slot0:c5rsm-dsv-mz.112-13.P", booted via slot0
cisco RSP2 (R4700) processor with 32768K/2072K bytes of memory.
!--- DRAM
R4700 processor, Implementation 33, Revision 1.0
Last reset from power-on
G.703/E1 software, Version 1.0.
Bridging software.
X.25 software, Version 2.0, NET2, BFE and GOSIP compliant.
1 C5IP controller (35 VLAN).
35 Virtual Ethernet/IEEE 802.3 interface(s)
123K bytes of non-volatile configuration memory.
16384K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
8192K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x2102
In this example, RSM is running Cisco IOS Software Release 11.2(13)P with the DSV feature set (IP/ Internetwork Packet Exchange [IPX]/AT/DEC/Versatile Interface Processor [VIP]).
RSM has 32 MB of DRAM.
Verify the minimum memory requirements for the Cisco IOS Software (platform, train and feature set).
Verify that at least the minimum recommended amount of DRAM is installed in the RSM.
Note: A large routing table requires more than the minimum DRAM requirements. If the amount of memory installed is lower than the recommended amount, either add memory or go to a Cisco IOS Software code supported with the current amount of memory.
If the RSM has sufficient memory, but the memory allocation failures continue, a memory leak or memory fragmentation may be occurring. In order to analyze output of the show memory command captured over several minutes, refer to the Output Interpreter.
Compare the Largest and Free fields in order to determine if fragmentation occurs. Issue the show memory summary command.
If the number under Largest is much smaller than the number under Free, fragmentation occurs. This is because Largest indicates the largest contiguous free memory block. Normally, it should be close to the free memory, as shown in this example:
2503-R5#show memory summary
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor C0E48 13885880 1615712 12270168 12077808 12077808 I/O
E00000 2097152 398396 1698756 1698756 1698588
This is a brief description of each field:
Total is the total memory allocated to processor or I/O memory. This does not include the amount of memory taken up by the Cisco IOS Software.
Used is the amount of memory being used at the time the show memory summary command is issued.
Free is the amount of free memory available at the time the show memory summary command is issued.
Lowest is the lowest amount of memory available since the last reload.
Largest is the largest amount of free contiguous memory at the time the show memory summary command is issued. This should normally be close to the free memory. A small number at this point, compared to the free memory, indicates fragmentation.
In order to determine if a memory leak occurs, capture several show memory summary command outputs at regular intervals. The intervals depend on the length of time it takes for the memory allocation failures to appear. For example, if the RSM begins to display the errors after four days, one or two captures a day are sufficient to establish a pattern.
If the free memory steadily decreases, a memory leak is possible. A memory leak occurs when a process takes and uses memory but does not release it back to the system. In order to determine the offending process, issue the show processes memory command.
In order to determine which process does not free memory back to the system, the show processes memory command needs to be captured several times at regular intervals. The two counters used for this are Freed and Holding. If the Holding counter for a process increases, yet the Freed counter does not increase, this process can be the cause of the memory leak.
Once the process is identified, search for a memory leak issue related to the process affecting the Cisco IOS Software currently installed on the RSM. Refer to the Bug Toolkit.
If the error is due to the ARP input process, then configure the default route point to a specific next hop IP address.
Refer to Troubleshooting High CPU Utilization due to Processes for more information.
