Hey all! I have a problem. I have a 3750 switch that is running out of memory and causing an outage. This switch is the main switch of a complex hosting residents in a workers camp. In each wing there's a 3750 which is connected to the main switch via fiber. The main switch then has 2 fiber connections to 6509s in our data room. The entire network is Layer 3, most interfaces have IPs associated with them. I'm more familar with layer 2 networks. I was thinking about maybe making the main switch and leaf switches layer 2 and use trunk ports instead.
The main switch had EIGRP running on it, I removed that and just made everything use static routes for now hoping it would fix the memory issue, it didn't. Please check out the errors and config below. Hopefully someone has some recommondations.
Thanks
Chris
%% Low on memory; try again later
%% Low on memory; try again later
Apr 29 18:20:39.015: %ADJ-3-ALLOCATEFAIL: Failed to allocate an adjacency
-Traceback= 1D96700 1D96C5C 244A8FC 244A96C 2282BB0 22833B8 22845E4 1DAA290 1BB9 928 1BB03A0
Apr 29 18:20:40.114: %SW_VLAN-3-VLAN_PM_NOTIFICATION_FAILURE: VLAN Manager synchronization failure with Port Manager over port mode change
-Traceback= 112F74C 197DD64 11080F0 19838D0 192449C 1763230 17634AC 185F918 185F D1C 12D2974 18599D4 175FA3C 175FACC 175F67C 1BB9928 1BB03A0
Apr 29 18:20:40.383: %SYS-2-MALLOCFAIL: Memory allocation of 38992 bytes failed from 0x1A137B0, alignment 0
Pool: Processor Free: 53220 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "HQM Stack Process", ipl= 0, pid= 152
-Traceback= 2914DAC 29172CC 2917530 2B981C0 1A137B4 19E7918 1BB9928 1BB03A0
Apr 29 18:21:10.431: %SYS-2-MALLOCFAIL: Memory allocation of 38992 bytes failed from 0x1A137B0, alignment 0
Pool: Processor Free: 53296 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "HQM Stack Process", ipl= 0, pid= 152
-Traceback= 2914DAC 29172CC 2917530 2B981C0 1A137B4 19E7918 1BB9928 1BB03A0
%% Low on memory; try again later
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 05-May-11 16:29 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02D00000
Initializing flashfs...
flashfs[1]: 80 files, 6 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 15998976
flashfs[1]: Bytes used: 12639232
flashfs[1]: Bytes available: 3359744
flashfs[1]: flashfs fsck took 1 seconds.
flashfs[1]: Initialization complete....done Initializing flashfs.
Checking for Bootloader upgrade.. not needed
POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed
POST: PortASIC Memory Tests : Begin
POST: PortASIC Memory Tests : End, Status Passed
POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed
POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed
Waiting for Stack Master Election...
POST: PortASIC CAM Subsystem Tests : Begin
POST: PortASIC CAM Subsystem Tests : End, Status Passed
POST: No Cable found on stack port 1
POST: No Cable found on stack port 2
POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed
Election Complete
Switch 1 booting as Master
Waiting for Port download...Complete
cisco WS-C3750G-12S (PowerPC405) processor (revision R0) with 131072K bytes of memory.
Processor board ID FDO1216Z1ML
Last reset from power-on
1 Virtual Ethernet interface
12 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:21:55:A8:88:80
Motherboard assembly number : 73-9678-07
Power supply part number : 341-0048-03
Motherboard serial number : FDO121604WN
Power supply serial number : LIT12070RV8
Model revision number : R0
Motherboard revision number : B0
Model number : WS-C3750G-12S-S
System serial number : FDO1216Z1ML
Top Assembly Part Number : 800-25856-04
Top Assembly Revision Number : A0
Version ID : V06
CLEI Code Number : CNM81V0GRB
Hardware Board Revision Number : 0x06
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 12 WS-C3750G-12S 12.2(55)SE3 C3750-IPBASEK9-M
NewAnnex-SW01#sh run
Building configuration...
Current configuration : 6750 bytes
!
! No configuration change since last restart
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname NewAnnex-SW01
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$7dYR$xEKicV2IJAeWBKh/vKgNp0
enable password 7 1062294C1344152B59
!
username admin privilege 15 password 7 04772B5319724B6E5C
username inland privilege 15 password 7 013F26514D58012F74
!
!
no aaa new-model
clock timezone MST -7
clock summer-time MDT recurring
switch 1 provision ws-c3750g-12s
system mtu routing 1500
ip routing
no ip domain-lookup
ip domain-name AlbianVillage.Local
!
!
!
!
crypto pki trustpoint TP-self-signed-1437108352
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-1437108352
revocation-check none
rsakeypair TP-self-signed-1437108352
!
!
crypto pki certificate chain TP-self-signed-1437108352
certificate self-signed 01
3082025A 308201C3 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 31343337 31303833 3532301E 170D3132 30313039 31353530
31305A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 34333731
30383335 3230819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100AC9C 40728B39 977B2DA1 B4A2849C E9370293 F0294EEC F8F12E2B 0659B7BF
4838CB52 B68AC32F E911FC0B D42CAE8F 197DF1F7 37B0D97C EF879097 C1681DCF
90E1EBFC AAF965A5 7AADA684 F3568D9C 0E53A5DE 1C44D79A 7E580552 8C80F03C
58C3ED01 A96CD2BF DC926CA2 8AEF8C73 EBF3FB27 180C9AA0 4864F355 266AD795
B6ED0203 010001A3 8181307F 300F0603 551D1301 01FF0405 30030101 FF302C06
03551D11 04253023 82214E65 77416E6E 65782D53 5730312E 416C6269 616E5669
6C6C6167 652E4C6F 63616C30 1F060355 1D230418 30168014 4A2574F6 A12C548F
942CC33D AD31A23E A47D50DE 301D0603 551D0E04 1604144A 2574F6A1 2C548F94
2CC33DAD 31A23EA4 7D50DE30 0D06092A 864886F7 0D010104 05000381 810017F4
00A638EC 5BD0B03A 88E231A3 51305C24 45E4C3D2 71D5AA2E 4A39C9CE 099ACF3C
4DE0C576 BBC876C6 ABFAFEAD CCD3AEF4 4F4F85B3 2D7FB314 29C86DD1 25B02CF9
E7084CC7 84C387DC 1ABD951A 4B58FB7C A5534E34 80333D13 F22E1906 747B9607
29AF2D01 240D6978 14A9DEAD 68627464 BB0C268F BBBDFA14 AED6A615 B007
quit
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
interface GigabitEthernet1/0/1
description Connectio to Albian Village Data Centre R1 Port GI2/10
no switchport
ip address 10.3.0.6 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/2
description Connectio to NewAnnex-A-SW01 GI0/1
no switchport
ip address 10.3.0.13 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/3
description Connectio to NewAnnex-B-SW01 GI0/1
no switchport
ip address 10.3.0.17 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/4
description Connectio to NewAnnex-C-SW01 GI0/1
no switchport
ip address 10.3.0.21 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/5
description Connectio to NewAnnex-D-SW01 GI0/1
no switchport
ip address 10.3.0.25 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/6
description Connectio to NewAnnex-E-SW01 GI0/1
no switchport
no ip address
shutdown
speed nonegotiate
!
interface GigabitEthernet1/0/7
description Connectio to NewAnnex-F-SW01 GI0/1
no switchport
ip address 10.3.0.33 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/8
description Connectio to NewAnnex-G-SW01 GI0/1
no switchport
ip address 10.3.0.37 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/9
description Connectio to NewAnnex-H-SW01 GI0/1
no switchport
ip address 10.3.0.41 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/10
no switchport
ip address 10.3.0.29 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
description Connectio to Albian Village Data Centre R2 Port GI2/10
no switchport
ip address 10.3.0.10 255.255.255.252
speed nonegotiate
!
interface Vlan1
no ip address
shutdown
!
ip classless
ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/1
ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/12 2
ip route 10.3.0.50 255.255.255.255 GigabitEthernet1/0/5
ip route 10.3.0.54 255.255.255.255 GigabitEthernet1/0/5
ip route 10.40.4.0 255.255.255.128 GigabitEthernet1/0/2
ip route 10.40.4.128 255.255.255.128 GigabitEthernet1/0/3
ip route 10.40.5.0 255.255.255.128 GigabitEthernet1/0/4
ip route 10.40.5.128 255.255.255.128 GigabitEthernet1/0/5
ip route 10.40.6.0 255.255.255.128 GigabitEthernet1/0/10
ip route 10.40.6.128 255.255.255.128 GigabitEthernet1/0/7
ip route 10.40.7.0 255.255.255.128 GigabitEthernet1/0/8
ip route 10.40.7.128 255.255.255.128 GigabitEthernet1/0/9
ip route 10.40.8.0 255.255.255.0 GigabitEthernet1/0/5
ip route 10.40.9.0 255.255.255.248 10.3.0.14
ip route 10.40.9.24 255.255.255.248 10.3.0.26
ip http server
ip http secure-server
!
ip sla enable reaction-alerts
logging trap warnings
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps tty
snmp-server enable traps cluster
snmp-server enable traps entity
snmp-server enable traps cpu threshold
snmp-server enable traps vtp
snmp-server enable traps vlancreate
snmp-server enable traps vlandelete
snmp-server enable traps flash insertion removal
snmp-server enable traps port-security
snmp-server enable traps envmon fan shutdown supply temperature status
snmp-server enable traps config-copy
snmp-server enable traps config
snmp-server enable traps hsrp
snmp-server enable traps bridge newroot topologychange
snmp-server enable traps stpx inconsistency root-inconsistency loop-inconsistency
snmp-server enable traps syslog
snmp-server enable traps mac-notification change move threshold
snmp-server enable traps vlan-membership
!
!
line con 0
login local
line vty 0 4
login local
transport input ssh
line vty 5 15
login local
transport input ssh
!
ntp clock-period 36029124
ntp server 10.3.0.5
end
