1841 IOS/Traceback Error

[johnlloyd_13]

hi all,

i was updating the IOS on a batch of 1841 routers and ran into a problem on this particular device.

it showed me an error as below. i would appreciate if anyone could advise why it only happened on this device and if i need to add RAM in order to run this IOS code. TIA!

<RELOAD COMMAND>

Readonly ROMMON initialized

program load complete, entry point: 0x8000f000, size: 0xcb80

program load complete, entry point: 0x8000f000, size: 0xcb80

program load complete, entry point: 0x8000f000, size: 0x26a6e08

Self decompressing the image : ################################################################################################################################################################################################################ [OK]

IOMEM set to: 25

25 percent IO memory configuration is too large. Decreasing to 24117248 bytes.

PMem allocated: 110100480 bytes

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

(c) of the Commercial Computer Software - Restricted

Rights clause at FAR sec. 52.227-19 and subparagraph

(c) (1) (ii) of the Rights in Technical Data and Computer

Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.

           170 West Tasman Drive

           San Jose, California 95134-1706

Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)

Technical Support: http://splashurl.com/n2a8b77

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Mon 19-Oct-09 16:11 by prod_rel_team

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://splashurl.com/m7srtd6

If you require further assistance please contact us by sending email to

export@cisco.com.

Installed image archive

Cisco 1841 (revision 7.0) with 107520K/23552K bytes of memory.

Processor board ID FHK112717FB

2 FastEthernet interfaces

1 Virtual Private Network (VPN) Module

DRAM configuration is 64 bits wide with parity disabled.

191K bytes of NVRAM.

62720K bytes of ATA CompactFlash (Read/Write)

SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!

%Software-forced reload

15:27:29 UTC Sat Nov 16 2013: Breakpoint exception, CPU signal 23, PC = 0x60673B88

--------------------------------------------------------------------

   Possible software fault. Upon reccurence,  please collect

   crashinfo, "show tech" and contact Cisco Technical Support.

--------------------------------------------------------------------

-Traceback= 0x6066C2C8z 0x6066A794z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z

$0 : 00000000, AT : 64B50000, v0 : 00000000, v1 : 00000000

a0 : 00000000, a1 : 635C860C, a2 : 00000000, a3 : 635D0000

t0 : 64C83DE0, t1 : 64E60000, t2 : 60679488, t3 : 64B49184

t4 : 60679488, t5 : 6545CD48, t6 : 3401FF01, t7 : 3401FF00

s0 : 00000000, s1 : 00000000, s2 : 646D0000, s3 : 64790000

s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 6346E6DC

t8 : 00000000, t9 : 64E60000, k0 : 3041E801, k1 : 00100000

gp : 64B50F20, sp : 6545C788, s8 : 654971E4, ra : 60672054

EPC  : 60673B88, ErrorEPC : BFC05FDC, SREG     : 3401FF03

MDLO : 0000002E, MDHI     : 00000000, BadVaddr : FE804254

TEXT_START : 0x600171C0

DATA_START : 0x632078C0

Cause 00000024 (Code 0x9): Breakpoint exception

Writing crashinfo to flash:crashinfo_20131116-152729

15:27:29 UTC Sat Nov 16 2013: Breakpoint exception, CPU signal 23, PC = 0x60673B88

--------------------------------------------------------------------

   Possible software fault. Upon reccurence,  please collect

   crashinfo, "show tech" and contact Cisco Technical Support.

--------------------------------------------------------------------

-Traceback= 0x6066C2C8z 0x6066A794z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z

$0 : 00000000, AT : 64B50000, v0 : 00000000, v1 : 00000000

a0 : 00000000, a1 : 635C860C, a2 : 00000000, a3 : 635D0000

t0 : 64C83DE0, t1 : 64E60000, t2 : 60679488, t3 : 64B49184

t4 : 60679488, t5 : 6545CD48, t6 : 3401FF01, t7 : 3401FF00

s0 : 00000000, s1 : 00000000, s2 : 646D0000, s3 : 64790000

s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 6346E6DC

t8 : 00000000, t9 : 64E60000, k0 : 3041E801, k1 : 00100000

gp : 64B50F20, sp : 6545C788, s8 : 654971E4, ra : 60672054

EPC  : 60673B88, ErrorEPC : BFC05FDC, SREG     : 3401FF03

MDLO : 0000002E, MDHI     : 00000000, BadVaddr : FE804254

TEXT_START : 0x600171C0

DATA_START : 0x632078C0

Cause 00000024 (Code 0x9): Breakpoint exception

-Traceback= 0x6066C2C8z 0x6066A794z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z

=== Flushing messages (15:27:30 UTC Sat Nov 16 2013) ===

Queued messages:

*Nov 16 15:27:30.699: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 16 15:27:25.591: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Initialized

*Nov 16 15:27:25.595: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Enabled

*Nov 16 15:27:28.223: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down

*Nov 16 15:27:28.823: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

*Nov 16 15:27:28.875: %SYS-2-MALLOCFAIL: Memory allocation of 32768 bytes failed from 0x602D9A60, alignment 0

Pool: Processor  Free: 34708  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Init", ipl= 0, pid= 3,  -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D21A8z 0x60F88B94z 0x60FAF288z 0x60FAF7C4z 0x60FAF8D0z 0x60FAF97Cz 0x614C1C98z 0x614BBD8Cz 0x614B3D44z 0x614BEF40z 0x614BF0D0z 0x614BF158z 0x60F4AE78z 0x60F4AEC4z

*Nov 16 15:27:30.059: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up

No warm reboot Storage

*** System received a Software forced crash ***

signal= 0x17, code= 0x24, context= 0x64e60db0

PC = 0x606882ec, Cause = 0x20, Status Reg = 0x3401c102

<HARD REBOOT>

ü

System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Technical Support: http://splashurl.com/n2a8b77

Copyright (c) 2006 by cisco Systems, Inc.

PLD version 0x10

GIO ASIC version 0x127

c1841 platform with 131072 Kbytes of main memory

Main memory is configured to 64 bit mode with parity disabled

Readonly ROMMON initialized

program load complete, entry point: 0x8000f000, size: 0xcb80

program load complete, entry point: 0x8000f000, size: 0xcb80

program load complete, entry point: 0x8000f000, size: 0x26a6e08

Self decompressing the image : ################################################################################################################################################################################################################ [OK]

IOMEM set to: 25

25 percent IO memory configuration is too large. Decreasing to 24117248 bytes.

PMem allocated: 110100480 bytes

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

(c) of the Commercial Computer Software - Restricted

Rights clause at FAR sec. 52.227-19 and subparagraph

(c) (1) (ii) of the Rights in Technical Data and Computer

Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.

           170 West Tasman Drive

           San Jose, California 95134-1706

Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)

Technical Support: http://splashurl.com/n2a8b77

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Mon 19-Oct-09 16:11 by prod_rel_team

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://splashurl.com/m7srtd6

If you require further assistance please contact us by sending email to

export@cisco.com.

Installed image archive

Cisco 1841 (revision 7.0) with 107520K/23552K bytes of memory.

Processor board ID FHK112717FB

2 FastEthernet interfaces

1 Virtual Private Network (VPN) Module

DRAM configuration is 64 bits wide with parity disabled.

191K bytes of NVRAM.

62720K bytes of ATA CompactFlash (Read/Write)

Warning:  Attempting a memory percentage that does not provide enough Processor

memory for the currently running image.  If you save your running-config now,

this version of software may not be able to run.

Press RETURN to get started!

*Nov 16 15:30:30.163: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Initialized

*Nov 16 15:30:30.167: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Enabled

*Nov 16 15:30:32.799: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down

*Nov 16 15:30:33.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

*Nov 16 15:30:34.279: %SYS-5-CONFIG_I: Configured from memory by console

*Nov 16 15:30:35.215: %SYS-5-RESTART: System restarted --

Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)

Technical Support: http://splashurl.com/n2a8b77

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Mon 19-Oct-09 16:11 by prod_rel_team

*Nov 16 15:30:35.219: %SNMP-5-COLDSTART: SNMP agent on host Router is undergoing a cold start

*Nov 16 15:30:35.271: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602D3B38, alignment 0

Pool: Processor  Free: 181080  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "TPLUS", ipl= 0, pid= 179,  -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D2F60z 0x6029C1E4z 0x602B4370z 0x631534E4z 0x6181A044z 0x6181C774z 0x6256E5E0z 0x625BB1BCz 0x625BB1A0z

*Nov 16 15:30:35.427: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Nov 16 15:30:35.427: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

*Nov 16 15:30:35.427: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Nov 16 15:30:35.427: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

*Nov 16 15:30:35.731: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down

*Nov 16 15:30:36.147: %LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down

*Nov 16 15:30:36.503: %SYS-6-BOOTTIME: Time taken to reboot after reload =  179 seconds

*Nov 16 15:30:49.359: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory

----

Router(config)#hostname ROUTER

ROUTER(config)#end

ROUTER#wr

Building configuration...

*Nov 16 15:43:14.887: %SYS-5-CONFIG_I: Configured from console by console[OK]

ROUTER#

ROUTER#

*Nov 16 15:43:15.783: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602DE440, alignment 0

Pool: Processor  Free: 147748  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Exec", ipl= 0, pid= 3,  -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D21A8z 0x61B53664z 0x61B56F1Cz 0x61B875D8z 0x6041EF08z 0x60F86868z 0x60F4B878z 0x60F4C188z 0x60FAD35Cz 0x60F8B194z 0x60FB1214z 0x625BB1BCz 0x625BB1A0z

ROUTER#sh run | i host

hostname ROUTER

ROUTER#

*Nov 16 15:43:49.655: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602DE440, alignment 0

Pool: Processor  Free: 153604  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Exec", ipl= 0, pid= 3,  -Traceback= 0x6029C4C8z 0x602B310Cz 0x602D21A8z 0x61B56948z 0x61B875C0z 0x6041EF08z 0x60F86868z 0x60F4B878z 0x60F4BD14z 0x60FACC94z 0x60F8B194z 0x60FB1214z 0x625BB1BCz 0x625BB1A0z

[Richard Burts]

Pretty clearly there is a problem with the size of available memory on this router for this version of code. The primary indicator is this

SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!

and the other indicators are the numerous messages about failure to allocate memory such as

%SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x602D3B38, alignment 0

As for why it happens on this router and not on others I do not yet have enough information to answer that. Is it possible that this router just has less memory installed than the others? The other alternative might be that there is some problem with memory on this router. Some further investigation is called for. Is this router under a maintenance contract? If so a case with Cisco TAC would be a good idea.

HTH

Rick

[johnlloyd_13]

hi rick,

thanks for the reply! is there a command to check the DRAM size so that i could compare it with the other routers (or am i missing it in the show version)?

i swapped from 64 MB CF to 32 MB and it works well with the IP BASE IOS.

also, do you have an idea what the part number for RAM to order for an 1841 router?

p.s. - these are my lab routers so TAC is out of the question here.

[Reza Sharifi]

Hi John,

What is the image you are trying to install and how much Memory and flash is required for it?

If this image works with a 32 MB CF and not with a 64 MB, than the issue is the CF.  If this is the case try formatting the 64MB CF and reuse it again, if not just get a new CF. The error message you posted is usually related to memory leak which is caused by a buggy IOS.

HTH

[Richard Burts]

OK lab router = no TAC

Generally the amount of memory is one of the fields shown in show version like this

Cisco 1841 (revision 6.0) with 237568K/24576K bytes of memory.

And when I look at your original post a bit more carefully look what I find

Cisco 1841 (revision 7.0) with 107520K/23552K bytes of memory.

So it looks like this 1841 has 128 meg of memory. compare that to others.

My guess is that Advanced Enterprise requires more memory that this router has and that IP BASE is just fine.

HTH

Rick

[johnlloyd_13]

hi rick,

thanks for confirming the RAM size! i also re-learned to add both numbers and divide by 1024.

i would need 256 MB RAM and 64 MB flash to run the IOS code. the errors isn't seen when using IP Base IOS on a 32 MB CF card.

not sure why i got away and not having the same problem with my other routers. will check again as i've already called it a night.

reza,

thank you also for replying but rick's assumptions is the closest with regards to my problem.

[johnlloyd_13]

hi rick and all,

just an update, i've checked all my 1841s are using 128 MB DRAM.

i did some research (didn't sleep much yesterday) and tried the command default memory-size iomem moments ago and the unit didn't report any traceback error and loop in loading its IOS afterwards.

i'm not sure why this device triggered creating IOMEM 25 in the first place while the others didn't.

IOMEM set to: 25

25 percent IO memory configuration is too large. Decreasing to 24117248 bytes.

PMem allocated: 110100480 bytes

----

Router(config)#default memory-size iomem

Smart-init will be enabled upon reload.

Router(config)#do reload

System configuration has been modified. Save? [yes/no]: y

Building configuration...

*Nov 17 14:49:23.743: %SYS-2-MALLOCFAIL: Memory allocation of 32768 bytes failed from 0x602DA974, alignment 0

Pool: Processor  Free: 160464  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Exec", ipl= 0, pid= 3,  -Traceback= 0x6029C4C8z 0x602B9CF8z 0x602D217Cz 0x60F86AFCz 0x60F86B74z 0x60F86C0Cz 0x60416178z 0x60F8EBA0z 0x60420CE8z 0x60F86868z 0x60F4B878z 0x60F4C188z 0x60FAD35Cz 0x60FB2D5Cz 0x60F8B194z 0x60F8BDD0z[OK]

Proceed with reload? [confirm]

*Nov 17 14:49:29.827: %SYS-5-RELOAD: Reload requested  by console. Reload Reason: Reload Command

Router#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Router(config)#host ROUTER

ROUTER(config)#^Z

ROUTER#

*Nov 17 14:58:17.211: %SYS-5-CONFIG_I: Configured from console by console

ROUTER#sh run | i host

hostname ROUTER

[Richard Burts]

I checked the software download page from Cisco looking for  12.4(24)T2. It reports that Advanced Enterprise version of the software  (which seems to be what the original post was trying to do) requires 192  meg of memory (and 64 of flash). I would be very surprised if you got  any 1841 with 128 to load and run that version.

It also reports that for IP BASE it requires only 128 meg with 32 of flash. So it is easy to see that any of your 1841s woudl run fine on this version.

HTH

Rick