memory allocation error - snmp - WS-C3750G-48TS-S

[lkadlik]

Hi,

I have a stack of WS-C3750G-48TS-S - 2.  When we tried to add snmp to it and then have is walked we got the following:

Trying to figure out if this is software or hardware related.  Also, how do you tell which switch in the stack was the issue?

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:28:34 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:29:04 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:29:35 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:30:05 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:30:36 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:31:06 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:31:36 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:32:06 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:32:37 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:33:07 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:33:38 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:34:08 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:34:38 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:35:09 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:35:39 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:36:09 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:36:39 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:37:09 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:37:40 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:38:10 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:38:41 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:39:12 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:39:42 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:40:12 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:40:42 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:41:12 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:41:29 edt: %SEC-6-IPACCESSLOGS: list 12 permitted 192.168.13.128 2 packets

Sep 29 18:41:43 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:42:13 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:42:43 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:43:14 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:43:44 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 6384  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:43:51 edt: %SYS-5-CONFIG_I: Configured from console by lynne.beauregard on vty0 (192.168.13.128)

Sep 29 18:44:15 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 4632  Cause: Not enough free memory

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:44:45 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 4632  Cause: Not enough free memory

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:45:15 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 4632  Cause: Not enough free memory

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:45:45 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 4632  Cause: Not enough free memory

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:46:15 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 4632  Cause: Not enough free memory

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP SNMP", ipl= 0, pid= 217

-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C

Sep 29 18:46:46 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8

Pool: I/O  Free: 4632  Cause: Not enough free memory

Alternate Pool: None  Free: 0  Cause: No Alternate pool

when we stopped trying to reach it this way the message stopped.  About a week later someone else tried and the stack crashed.  We were thinking of replacing the switch but didn't know which switch was the issue or if it is both.  Is there anyway to tell which switch is the issue?

Does anyone know if this is a hardware or software issue?   IOS is 12.2(25)SEE2            C3750-IPBASE-M.

 sh mem free
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    1DC3A08    90789368    26921612    63867756    62151284    61547696
      I/O    7400000    12574720     8424904     4149816     3734608     3867416

Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Fri 28-Jul-06 08:46 by yenanh
Image text-base: 0x00003000, data-base: 0x00EE3E54

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SEE4, RELEASE SOFTWARE (fc1)

uptime is 5 days, 6 hours, 5 minutes
System returned to ROM by error - Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!, PC 0x330A84
System restarted at 12:28:26 edt Thu Oct 6 2016
System image file is "flash:c3750-ipbase-mz.122-25.SEE2/c3750-ipbase-mz.122-25.SEE2.bin"

cisco WS-C3750G-48TS (PowerPC405) processor (revision D0) with 118784K/12280K bytes of memory.
Processor board ID FOC1041ZE6A
Last reset from power-on
3 Virtual Ethernet interfaces
104 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:19:AA:E7:61:80
Motherboard assembly number     : 73-10218-08
Power supply part number        : 341-0107-01
Motherboard serial number       : FOC10411WHQ
Power supply serial number      : AZS1028063G
Model revision number           : D0
Motherboard revision number     : B0
Model number                    : WS-C3750G-48TS-S
System serial number            : FOC1041ZE6A
Top Assembly Part Number        : 800-26857-01
Top Assembly Revision Number    : A0
Version ID                      : V03
CLEI Code Number                : CNMWU00ARC
Hardware Board Revision Number  : 0x09

Switch   Ports  Model              SW Version              SW Image
------   -----  -----              ----------              ----------
*    1   52     WS-C3750G-48TS     12.2(25)SEE2            C3750-IPBASE-M
     2   52     WS-C3750G-48TS     12.2(25)SEE2            C3750-IPBASE-M

Switch 02
---------
Switch Uptime                   : 5 days, 6 hours, 5 minutes
Base ethernet MAC Address       : 00:19:AA:E7:63:80
Motherboard assembly number     : 73-10218-08
Power supply part number        : 341-0107-01
Motherboard serial number       : FOC10411WBY
Power supply serial number      : AZS1028062Y
Model revision number           : D0
Motherboard revision number     : B0
Model number                    : WS-C3750G-48TS-S
System serial number            : FOC1041ZE6K
Top assembly part number        : 800-26857-01
Top assembly revision number    : A0
Version ID                      : V03
CLEI Code Number                : CNMWU00ARC

I was also wondering if this bug may apply.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsc93144

If we are not snmp walking the switch it is fine.

Any help is appreciated.

Thank you.

[Leo Laohoo]

Post the complete output to the command "dir flash1:" and "dir flash2:". 

The IOS of the switch is very, very old.  Someone will need to make a decision to upgrade to a later IOS.  

[lkadlik]

dir flash1:
Directory of flash:/

    2  -rwx        1048   Oct 6 2016 12:38:34 -04:00  multiple-fs
    3  -rwx          27   Oct 6 2016 12:38:34 -04:00  private-config.text
    4  -rwx        1396  Feb 28 1993 19:02:46 -05:00  vlan.dat
    5  drwx         192  Feb 28 1993 19:09:21 -05:00  c3750-ipbase-mz.122-25.SEE2
  458  drwx          64   Oct 6 2016 12:27:43 -04:00  crashinfo_ext
  461  drwx          64  Feb 28 1993 19:02:42 -05:00  crashinfo
  463  -rwx       21268   Oct 6 2016 12:38:34 -04:00  config.text

32514048 bytes total (23449088 bytes free)

#dir flash2:
Directory of flash2:/

    2  -rwx          27   Oct 6 2016 12:38:35 -04:00  private-config.text
    3  -rwx        1048  Aug 25 2016 09:22:36 -04:00  multiple-fs
    4  -rwx        1396  Feb 28 1993 19:02:52 -05:00  vlan.dat
    5  drwx         192  Feb 28 1993 19:09:26 -05:00  c3750-ipbase-mz.122-25.SEE2
  458  drwx          64   Oct 6 2016 12:24:38 -04:00  crashinfo_ext
  459  -rwx       21268   Oct 6 2016 12:38:35 -04:00  config.text
  461  drwx          64   Oct 6 2016 12:27:40 -04:00  crashinfo

[Leo Laohoo]

Ok, thanks for the output. 

Can you post further output to the command "dir flash2:crashinfo_ext"?  

I want to see what caused the stack to crash on 06 October 2016.  I am suspecting there's a memory leak.  

[lkadlik]

***************************************************
******* Information of Last System Crash **********
***************************************************

Using flash:/crashinfo/crashinfo_1.
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Fri 28-Jul-06 08:46 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x00E692C0 SRR1 = 0x00029210 SRR2 = 0x0056A1EC SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000 PC = 0x00330A84 MSR = 0x00029210 CR = 0x50000009
LR = 0x0021CD04 CTR = 0x00330A3C XER = 0xE000003F
R0 = 0x036126C8 R1 = 0x02EFD608 R2 = 0x00000000 R3 = 0x00000000
R4 = 0x00000004 R5 = 0x02EFD658 R6 = 0x02EFD660 R7 = 0x02EFD668
R8 = 0x00000000 R9 = 0x00000000 R10 = 0x00000000 R11 = 0x00330A3B
R12 = 0x00000000 R13 = 0x00110000 R14 = 0x00214414 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x02EFD658 R23 = 0x02EFD660
R24 = 0x02EFD668 R25 = 0x01650000 R26 = 0x00000000 R27 = 0x02EFD660
R28 = 0x02EFD668 R29 = 0x02EFD658 R30 = 0x00000007 R31 = 0x0345BC88

Stack trace:
PC = 0x00330A84, SP = 0x02EFD608
Frame 00: SP = 0x02EFD628 PC = 0x0345BB18
Frame 01: SP = 0x02EFD698 PC = 0x0021CD04
Frame 02: SP = 0x02EFD6D8 PC = 0x002147B8
Frame 03: SP = 0x02EFD6F0 PC = 0x002135B0
Frame 04: SP = 0x02EFD730 PC = 0x00214524
Frame 05: SP = 0x02EFD738 PC = 0x00908064
Frame 06: SP = 0x00000000 PC = 0x008FE62C

[Leo Laohoo]

Upgrade the IOS of the stack.  It's a memory leak.