10-11-2016 03:39 PM - edited 03-08-2019 07:46 AM
Hi,
I have a stack of WS-C3750G-48TS-S - 2. When we tried to add snmp to it and then have is walked we got the following:
Trying to figure out if this is software or hardware related. Also, how do you tell which switch in the stack was the issue?
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:28:34 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:29:04 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:29:35 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:30:05 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:30:36 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:31:06 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:31:36 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:32:06 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:32:37 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:33:07 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:33:38 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:34:08 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:34:38 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:35:09 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:35:39 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:36:09 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:36:39 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:37:09 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:37:40 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:38:10 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:38:41 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:39:12 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:39:42 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:40:12 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:40:42 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:41:12 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:41:29 edt: %SEC-6-IPACCESSLOGS: list 12 permitted 192.168.13.128 2 packets
Sep 29 18:41:43 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:42:13 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:42:43 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:43:14 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:43:44 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 6384 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:43:51 edt: %SYS-5-CONFIG_I: Configured from console by lynne.beauregard on vty0 (192.168.13.128)
Sep 29 18:44:15 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 4632 Cause: Not enough free memory
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:44:45 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 4632 Cause: Not enough free memory
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:45:15 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 4632 Cause: Not enough free memory
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:45:45 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 4632 Cause: Not enough free memory
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:46:15 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 4632 Cause: Not enough free memory
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP SNMP", ipl= 0, pid= 217
-Traceback= AFC730 E58410 E5C3EC 296108 296530 298EF0 2953FC CCABDC FF0A0 FFD80 908064 8FE62C
Sep 29 18:46:46 edt: %SYS-2-MALLOCFAIL: Memory allocation of 4680 bytes failed from 0x296104, alignment 8
Pool: I/O Free: 4632 Cause: Not enough free memory
Alternate Pool: None Free: 0 Cause: No Alternate pool
when we stopped trying to reach it this way the message stopped. About a week later someone else tried and the stack crashed. We were thinking of replacing the switch but didn't know which switch was the issue or if it is both. Is there anyway to tell which switch is the issue?
Does anyone know if this is a hardware or software issue? IOS is 12.2(25)SEE2 C3750-IPBASE-M.
sh mem free
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 1DC3A08 90789368 26921612 63867756 62151284 61547696
I/O 7400000 12574720 8424904 4149816 3734608 3867416
I was also wondering if this bug may apply.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsc93144
If we are not snmp walking the switch it is fine.
Any help is appreciated.
Thank you.
10-11-2016 04:57 PM
Post the complete output to the command "dir flash1:" and "dir flash2:".
The IOS of the switch is very, very old. Someone will need to make a decision to upgrade to a later IOS.
10-18-2016 06:16 PM
dir flash1:
Directory of flash:/
2 -rwx 1048 Oct 6 2016 12:38:34 -04:00 multiple-fs
3 -rwx 27 Oct 6 2016 12:38:34 -04:00 private-config.text
4 -rwx 1396 Feb 28 1993 19:02:46 -05:00 vlan.dat
5 drwx 192 Feb 28 1993 19:09:21 -05:00 c3750-ipbase-mz.122-25.SEE2
458 drwx 64 Oct 6 2016 12:27:43 -04:00 crashinfo_ext
461 drwx 64 Feb 28 1993 19:02:42 -05:00 crashinfo
463 -rwx 21268 Oct 6 2016 12:38:34 -04:00 config.text
32514048 bytes total (23449088 bytes free)
#dir flash2:
Directory of flash2:/
2 -rwx 27 Oct 6 2016 12:38:35 -04:00 private-config.text
3 -rwx 1048 Aug 25 2016 09:22:36 -04:00 multiple-fs
4 -rwx 1396 Feb 28 1993 19:02:52 -05:00 vlan.dat
5 drwx 192 Feb 28 1993 19:09:26 -05:00 c3750-ipbase-mz.122-25.SEE2
458 drwx 64 Oct 6 2016 12:24:38 -04:00 crashinfo_ext
459 -rwx 21268 Oct 6 2016 12:38:35 -04:00 config.text
461 drwx 64 Oct 6 2016 12:27:40 -04:00 crashinfo
10-18-2016 09:32 PM
Ok, thanks for the output.
Can you post further output to the command "dir flash2:crashinfo_ext"?
I want to see what caused the stack to crash on 06 October 2016. I am suspecting there's a memory leak.
10-19-2016 06:43 AM
***************************************************
******* Information of Last System Crash **********
***************************************************
Using flash:/crashinfo/crashinfo_1.
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Fri 28-Jul-06 08:46 by yenanh
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x00E692C0 SRR1 = 0x00029210 SRR2 = 0x0056A1EC SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x01000000
CPU Register Context:
Vector = 0x00002000 PC = 0x00330A84 MSR = 0x00029210 CR = 0x50000009
LR = 0x0021CD04 CTR = 0x00330A3C XER = 0xE000003F
R0 = 0x036126C8 R1 = 0x02EFD608 R2 = 0x00000000 R3 = 0x00000000
R4 = 0x00000004 R5 = 0x02EFD658 R6 = 0x02EFD660 R7 = 0x02EFD668
R8 = 0x00000000 R9 = 0x00000000 R10 = 0x00000000 R11 = 0x00330A3B
R12 = 0x00000000 R13 = 0x00110000 R14 = 0x00214414 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x02EFD658 R23 = 0x02EFD660
R24 = 0x02EFD668 R25 = 0x01650000 R26 = 0x00000000 R27 = 0x02EFD660
R28 = 0x02EFD668 R29 = 0x02EFD658 R30 = 0x00000007 R31 = 0x0345BC88
Stack trace:
PC = 0x00330A84, SP = 0x02EFD608
Frame 00: SP = 0x02EFD628 PC = 0x0345BB18
Frame 01: SP = 0x02EFD698 PC = 0x0021CD04
Frame 02: SP = 0x02EFD6D8 PC = 0x002147B8
Frame 03: SP = 0x02EFD6F0 PC = 0x002135B0
Frame 04: SP = 0x02EFD730 PC = 0x00214524
Frame 05: SP = 0x02EFD738 PC = 0x00908064
Frame 06: SP = 0x00000000 PC = 0x008FE62C
10-19-2016 01:03 PM
Upgrade the IOS of the stack. It's a memory leak.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide