11-16-2013 08:33 AM - edited 03-07-2019 04:38 PM
hi all,
i was updating the IOS on a batch of 1841 routers and ran into a problem on this particular device.
it showed me an error as below. i would appreciate if anyone could advise why it only happened on this device and if i need to add RAM in order to run this IOS code. TIA!
<RELOAD COMMAND>
Readonly ROMMON initialized
program load complete, entry point: 0x8000f000, size: 0xcb80
program load complete, entry point: 0x8000f000, size: 0xcb80
program load complete, entry point: 0x8000f000, size: 0x26a6e08
Self decompressing the image : ################################################################################################################################################################################################################ [OK]
IOMEM set to: 25
25 percent IO memory configuration is too large. Decreasing to 24117248 bytes.
PMem allocated: 110100480 bytes
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)
Technical Support: http://splashurl.com/n2a8b77
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 19-Oct-09 16:11 by prod_rel_team
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
If you require further assistance please contact us by sending email to
Installed image archive
Cisco 1841 (revision 7.0) with 107520K/23552K bytes of memory.
Processor board ID FHK112717FB
2 FastEthernet interfaces
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
62720K bytes of ATA CompactFlash (Read/Write)
SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!
%Software-forced reload
15:27:29 UTC Sat Nov 16 2013: Breakpoint exception, CPU signal 23, PC = 0x60673B88
--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------
-Traceback= 0x6066C2C8z 0x6066A794z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z
$0 : 00000000, AT : 64B50000, v0 : 00000000, v1 : 00000000
a0 : 00000000, a1 : 635C860C, a2 : 00000000, a3 : 635D0000
t0 : 64C83DE0, t1 : 64E60000, t2 : 60679488, t3 : 64B49184
t4 : 60679488, t5 : 6545CD48, t6 : 3401FF01, t7 : 3401FF00
s0 : 00000000, s1 : 00000000, s2 : 646D0000, s3 : 64790000
s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 6346E6DC
t8 : 00000000, t9 : 64E60000, k0 : 3041E801, k1 : 00100000
gp : 64B50F20, sp : 6545C788, s8 : 654971E4, ra : 60672054
EPC : 60673B88, ErrorEPC : BFC05FDC, SREG : 3401FF03
MDLO : 0000002E, MDHI : 00000000, BadVaddr : FE804254
TEXT_START : 0x600171C0
DATA_START : 0x632078C0
Cause 00000024 (Code 0x9): Breakpoint exception
Writing crashinfo to flash:crashinfo_20131116-152729
15:27:29 UTC Sat Nov 16 2013: Breakpoint exception, CPU signal 23, PC = 0x60673B88
--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------
-Traceback= 0x6066C2C8z 0x6066A794z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z
$0 : 00000000, AT : 64B50000, v0 : 00000000, v1 : 00000000
a0 : 00000000, a1 : 635C860C, a2 : 00000000, a3 : 635D0000
t0 : 64C83DE0, t1 : 64E60000, t2 : 60679488, t3 : 64B49184
t4 : 60679488, t5 : 6545CD48, t6 : 3401FF01, t7 : 3401FF00
s0 : 00000000, s1 : 00000000, s2 : 646D0000, s3 : 64790000
s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 6346E6DC
t8 : 00000000, t9 : 64E60000, k0 : 3041E801, k1 : 00100000
gp : 64B50F20, sp : 6545C788, s8 : 654971E4, ra : 60672054
EPC : 60673B88, ErrorEPC : BFC05FDC, SREG : 3401FF03
MDLO : 0000002E, MDHI : 00000000, BadVaddr : FE804254
TEXT_START : 0x600171C0
DATA_START : 0x632078C0
Cause 00000024 (Code 0x9): Breakpoint exception
-Traceback= 0x6066C2C8z 0x6066A794z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z
=== Flushing messages (15:27:30 UTC Sat Nov 16 2013) ===
Queued messages:
*Nov 16 15:27:30.699: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
*Nov 16 15:27:25.591: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Initialized
*Nov 16 15:27:25.595: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Enabled
*Nov 16 15:27:28.223: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Nov 16 15:27:28.823: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Nov 16 15:27:28.875: %SYS-2-MALLOCFAIL: Memory allocation of 32768 bytes failed from 0x602D9A60, alignment 0
Pool: Processor Free: 34708 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Init", ipl= 0, pid= 3, -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z 0x60F4AEC4z
*Nov 16 15:27:30.059: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
No warm reboot Storage
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x64e60db0
PC = 0x606882ec, Cause = 0x20, Status Reg = 0x3401c102
<HARD REBOOT>
ü
System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
Technical Support: http://splashurl.com/n2a8b77
Copyright (c) 2006 by cisco Systems, Inc.
PLD version 0x10
GIO ASIC version 0x127
c1841 platform with 131072 Kbytes of main memory
Main memory is configured to 64 bit mode with parity disabled
Readonly ROMMON initialized
program load complete, entry point: 0x8000f000, size: 0xcb80
program load complete, entry point: 0x8000f000, size: 0xcb80
program load complete, entry point: 0x8000f000, size: 0x26a6e08
Self decompressing the image : ################################################################################################################################################################################################################ [OK]
IOMEM set to: 25
25 percent IO memory configuration is too large. Decreasing to 24117248 bytes.
PMem allocated: 110100480 bytes
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)
Technical Support: http://splashurl.com/n2a8b77
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 19-Oct-09 16:11 by prod_rel_team
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
If you require further assistance please contact us by sending email to
Installed image archive
Cisco 1841 (revision 7.0) with 107520K/23552K bytes of memory.
Processor board ID FHK112717FB
2 FastEthernet interfaces
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
62720K bytes of ATA CompactFlash (Read/Write)
Warning: Attempting a memory percentage that does not provide enough Processor
memory for the currently running image. If you save your running-config now,
this version of software may not be able to run.
Press RETURN to get started!
*Nov 16 15:30:30.163: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Initialized
*Nov 16 15:30:30.167: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Enabled
*Nov 16 15:30:32.799: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Nov 16 15:30:33.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Nov 16 15:30:34.279: %SYS-5-CONFIG_I: Configured from memory by console
*Nov 16 15:30:35.215: %SYS-5-RESTART: System restarted --
Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)
Technical Support: http://splashurl.com/n2a8b77
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 19-Oct-09 16:11 by prod_rel_team
*Nov 16 15:30:35.219: %SNMP-5-COLDSTART: SNMP agent on host Router is undergoing a cold start
*Nov 16 15:30:35.271: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602D3B38, alignment 0
Pool: Processor Free: 181080 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "TPLUS", ipl= 0, pid= 179, -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D2F60z 0x6029C1E4z 0x602B4370z 0x631534E4z 0x6181A044z 0x6181C774z 0x6256E5E0z 0x625BB1BCz 0x625BB1A0z
*Nov 16 15:30:35.427: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 16 15:30:35.427: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 16 15:30:35.427: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 16 15:30:35.427: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 16 15:30:35.731: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Nov 16 15:30:36.147: %LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
*Nov 16 15:30:36.503: %SYS-6-BOOTTIME: Time taken to reboot after reload = 179 seconds
*Nov 16 15:30:49.359: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory
----
Router(config)#hostname ROUTER
ROUTER(config)#end
ROUTER#wr
Building configuration...
*Nov 16 15:43:14.887: %SYS-5-CONFIG_I: Configured from console by console[OK]
ROUTER#
ROUTER#
*Nov 16 15:43:15.783: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602DE440, alignment 0
Pool: Processor Free: 147748 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Exec", ipl= 0, pid= 3, -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D21A8z 0x61B53664z 0x61B56F1Cz 0x61B875D8z 0x6041EF08z 0x60F86868z 0x60F4B878z 0x60F4C188z 0x60FAD35Cz 0x60F8B194z 0x60FB1214z 0x625BB1BCz 0x625BB1A0z
ROUTER#sh run | i host
hostname ROUTER
ROUTER#
*Nov 16 15:43:49.655: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602DE440, alignment 0
Pool: Processor Free: 153604 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Exec", ipl= 0, pid= 3, -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D21A8z 0x61B56948z 0x61B875C0z 0x6041EF08z 0x60F86868z 0x60F4B878z 0x60F4BD14z 0x60FACC94z 0x60F8B194z 0x60FB1214z 0x625BB1BCz 0x625BB1A0z
11-16-2013 09:02 AM
Pretty clearly there is a problem with the size of available memory on this router for this version of code. The primary indicator is this
SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!
and the other indicators are the numerous messages about failure to allocate memory such as
%SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602D3B38, alignment 0
As for why it happens on this router and not on others I do not yet have enough information to answer that. Is it possible that this router just has less memory installed than the others? The other alternative might be that there is some problem with memory on this router. Some further investigation is called for. Is this router under a maintenance contract? If so a case with Cisco TAC would be a good idea.
HTH
Rick
11-16-2013 09:09 AM
hi rick,
thanks for the reply! is there a command to check the DRAM size so that i could compare it with the other routers (or am i missing it in the show version)?
i swapped from 64 MB CF to 32 MB and it works well with the IP BASE IOS.
also, do you have an idea what the part number for RAM to order for an 1841 router?
p.s. - these are my lab routers so TAC is out of the question here.
11-16-2013 10:02 AM
Hi John,
What is the image you are trying to install and how much Memory and flash is required for it?
If this image works with a 32 MB CF and not with a 64 MB, than the issue is the CF. If this is the case try formatting the 64MB CF and reuse it again, if not just get a new CF. The error message you posted is usually related to memory leak which is caused by a buggy IOS.
HTH
11-16-2013 10:11 AM
OK lab router = no TAC
Generally the amount of memory is one of the fields shown in show version like this
Cisco 1841 (revision 6.0) with 237568K/24576K bytes of memory.
And when I look at your original post a bit more carefully look what I find
Cisco 1841 (revision 7.0) with 107520K/23552K bytes of memory.
So it looks like this 1841 has 128 meg of memory. compare that to others.
My guess is that Advanced Enterprise requires more memory that this router has and that IP BASE is just fine.
HTH
Rick
11-16-2013 10:35 AM
hi rick,
thanks for confirming the RAM size! i also re-learned to add both numbers and divide by 1024.
i would need 256 MB RAM and 64 MB flash to run the IOS code. the errors isn't seen when using IP Base IOS on a 32 MB CF card.
not sure why i got away and not having the same problem with my other routers. will check again as i've already called it a night.
reza,
thank you also for replying but rick's assumptions is the closest with regards to my problem.
11-17-2013 07:41 AM
hi rick and all,
just an update, i've checked all my 1841s are using 128 MB DRAM.
i did some research (didn't sleep much yesterday) and tried the command default memory-size iomem moments ago and the unit didn't report any traceback error and loop in loading its IOS afterwards.
i'm not sure why this device triggered creating IOMEM 25 in the first place while the others didn't.
IOMEM set to: 25
25 percent IO memory configuration is too large. Decreasing to 24117248 bytes.
PMem allocated: 110100480 bytes
----
Router(config)#default memory-size iomem
Smart-init will be enabled upon reload.
Router(config)#do reload
System configuration has been modified. Save? [yes/no]: y
Building configuration...
*Nov 17 14:49:23.743: %SYS-2-MALLOCFAIL: Memory allocation of 32768 bytes failed from 0x602DA974, alignment 0
Pool: Processor Free: 160464 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Exec", ipl= 0, pid= 3, -Traceback= 0x6029C4C8z 0x602B9CF8z 0x602D217Cz 0x60F86AFCz 0x60F86B74z 0x60F86C0Cz 0x60416178z 0x60F8EBA0z 0x60420CE8z 0x60F86868z 0x60F4B878z 0x60F4C188z 0x60FAD35Cz 0x60FB2D5Cz 0x60F8B194z 0x60F8BDD0z[OK]
Proceed with reload? [confirm]
*Nov 17 14:49:29.827: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command
Router#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#host ROUTER
ROUTER(config)#^Z
ROUTER#
*Nov 17 14:58:17.211: %SYS-5-CONFIG_I: Configured from console by console
ROUTER#sh run | i host
hostname ROUTER
11-17-2013 05:19 PM
I checked the software download page from Cisco looking for 12.4(24)T2. It reports that Advanced Enterprise version of the software (which seems to be what the original post was trying to do) requires 192 meg of memory (and 64 of flash). I would be very surprised if you got any 1841 with 128 to load and run that version.
It also reports that for IP BASE it requires only 128 meg with 32 of flash. So it is easy to see that any of your 1841s woudl run fine on this version.
HTH
Rick
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide