Solved: Memory problem with 3750 Fiber switch, what can I do?

cshannahan · ‎04-29-2012

Hey all! I have a problem. I have a 3750 switch that is running out of memory and causing an outage. This switch is the main switch of a complex hosting residents in a workers camp. In each wing there's a 3750 which is connected to the main switch via fiber. The main switch then has 2 fiber connections to 6509s in our data room. The entire network is Layer 3, most interfaces have IPs associated with them. I'm more familar with layer 2 networks. I was thinking about maybe making the main switch and leaf switches layer 2 and use trunk ports instead.

The main switch had EIGRP running on it, I removed that and just made everything use static routes for now hoping it would fix the memory issue, it didn't. Please check out the errors and config below. Hopefully someone has some recommondations.

Thanks

Chris

%% Low on memory; try again later

Apr 29 18:20:39.015: %ADJ-3-ALLOCATEFAIL: Failed to allocate an adjacency

-Traceback= 1D96700 1D96C5C 244A8FC 244A96C 2282BB0 22833B8 22845E4 1DAA290 1BB9 928 1BB03A0

Apr 29 18:20:40.114: %SW_VLAN-3-VLAN_PM_NOTIFICATION_FAILURE: VLAN Manager synchronization failure with Port Manager over port mode change

-Traceback= 112F74C 197DD64 11080F0 19838D0 192449C 1763230 17634AC 185F918 185F D1C 12D2974 18599D4 175FA3C 175FACC 175F67C 1BB9928 1BB03A0

Apr 29 18:20:40.383: %SYS-2-MALLOCFAIL: Memory allocation of 38992 bytes failed from 0x1A137B0, alignment 0

Pool: Processor Free: 53220 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "HQM Stack Process", ipl= 0, pid= 152

-Traceback= 2914DAC 29172CC 2917530 2B981C0 1A137B4 19E7918 1BB9928 1BB03A0

Apr 29 18:21:10.431: %SYS-2-MALLOCFAIL: Memory allocation of 38992 bytes failed from 0x1A137B0, alignment 0

Pool: Processor Free: 53296 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "HQM Stack Process", ipl= 0, pid= 152

-Traceback= 2914DAC 29172CC 2917530 2B981C0 1A137B4 19E7918 1BB9928 1BB03A0

%% Low on memory; try again later

Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE3, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 05-May-11 16:29 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02D00000

Initializing flashfs...

flashfs[1]: 80 files, 6 directories

flashfs[1]: 0 orphaned files, 0 orphaned directories

flashfs[1]: Total bytes: 15998976

flashfs[1]: Bytes used: 12639232

flashfs[1]: Bytes available: 3359744

flashfs[1]: flashfs fsck took 1 seconds.

flashfs[1]: Initialization complete....done Initializing flashfs.

Checking for Bootloader upgrade.. not needed

POST: CPU MIC register Tests : Begin

POST: CPU MIC register Tests : End, Status Passed

POST: PortASIC Memory Tests : Begin

POST: PortASIC Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin

POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin

POST: PortASIC RingLoopback Tests : End, Status Passed

Waiting for Stack Master Election...

POST: PortASIC CAM Subsystem Tests : Begin

POST: PortASIC CAM Subsystem Tests : End, Status Passed

POST: No Cable found on stack port 1

POST: No Cable found on stack port 2

POST: PortASIC Stack Port Loopback Tests : Begin

POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin

POST: PortASIC Port Loopback Tests : End, Status Passed

Election Complete

Switch 1 booting as Master

Waiting for Port download...Complete

cisco WS-C3750G-12S (PowerPC405) processor (revision R0) with 131072K bytes of memory.

Processor board ID FDO1216Z1ML

Last reset from power-on

1 Virtual Ethernet interface

12 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 00:21:55:A8:88:80

Motherboard assembly number : 73-9678-07

Power supply part number : 341-0048-03

Motherboard serial number : FDO121604WN

Power supply serial number : LIT12070RV8

Model revision number : R0

Motherboard revision number : B0

Model number : WS-C3750G-12S-S

System serial number : FDO1216Z1ML

Top Assembly Part Number : 800-25856-04

Top Assembly Revision Number : A0

Version ID : V06

CLEI Code Number : CNM81V0GRB

Hardware Board Revision Number : 0x06

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 12 WS-C3750G-12S 12.2(55)SE3 C3750-IPBASEK9-M

NewAnnex-SW01#sh run

Building configuration...

Current configuration : 6750 bytes

!

! No configuration change since last restart

!

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname NewAnnex-SW01

!

boot-start-marker

boot-end-marker

!

enable secret 5 $1$7dYR$xEKicV2IJAeWBKh/vKgNp0

enable password 7 1062294C1344152B59

!

username admin privilege 15 password 7 04772B5319724B6E5C

username inland privilege 15 password 7 013F26514D58012F74

!

no aaa new-model

clock timezone MST -7

clock summer-time MDT recurring

switch 1 provision ws-c3750g-12s

system mtu routing 1500

ip routing

no ip domain-lookup

ip domain-name AlbianVillage.Local

!

crypto pki trustpoint TP-self-signed-1437108352

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-1437108352

revocation-check none

rsakeypair TP-self-signed-1437108352

!

crypto pki certificate chain TP-self-signed-1437108352

certificate self-signed 01

3082025A 308201C3 A0030201 02020101 300D0609 2A864886 F70D0101 04050030

31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274

69666963 6174652D 31343337 31303833 3532301E 170D3132 30313039 31353530

31305A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649

4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 34333731

30383335 3230819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281

8100AC9C 40728B39 977B2DA1 B4A2849C E9370293 F0294EEC F8F12E2B 0659B7BF

4838CB52 B68AC32F E911FC0B D42CAE8F 197DF1F7 37B0D97C EF879097 C1681DCF

90E1EBFC AAF965A5 7AADA684 F3568D9C 0E53A5DE 1C44D79A 7E580552 8C80F03C

58C3ED01 A96CD2BF DC926CA2 8AEF8C73 EBF3FB27 180C9AA0 4864F355 266AD795

B6ED0203 010001A3 8181307F 300F0603 551D1301 01FF0405 30030101 FF302C06

03551D11 04253023 82214E65 77416E6E 65782D53 5730312E 416C6269 616E5669

6C6C6167 652E4C6F 63616C30 1F060355 1D230418 30168014 4A2574F6 A12C548F

942CC33D AD31A23E A47D50DE 301D0603 551D0E04 1604144A 2574F6A1 2C548F94

2CC33DAD 31A23EA4 7D50DE30 0D06092A 864886F7 0D010104 05000381 810017F4

00A638EC 5BD0B03A 88E231A3 51305C24 45E4C3D2 71D5AA2E 4A39C9CE 099ACF3C

4DE0C576 BBC876C6 ABFAFEAD CCD3AEF4 4F4F85B3 2D7FB314 29C86DD1 25B02CF9

E7084CC7 84C387DC 1ABD951A 4B58FB7C A5534E34 80333D13 F22E1906 747B9607

29AF2D01 240D6978 14A9DEAD 68627464 BB0C268F BBBDFA14 AED6A615 B007

quit

!

spanning-tree mode pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

interface GigabitEthernet1/0/1

description Connectio to Albian Village Data Centre R1 Port GI2/10

no switchport

ip address 10.3.0.6 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/2

description Connectio to NewAnnex-A-SW01 GI0/1

no switchport

ip address 10.3.0.13 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/3

description Connectio to NewAnnex-B-SW01 GI0/1

no switchport

ip address 10.3.0.17 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/4

description Connectio to NewAnnex-C-SW01 GI0/1

no switchport

ip address 10.3.0.21 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/5

description Connectio to NewAnnex-D-SW01 GI0/1

no switchport

ip address 10.3.0.25 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/6

description Connectio to NewAnnex-E-SW01 GI0/1

no switchport

no ip address

shutdown

speed nonegotiate

!

interface GigabitEthernet1/0/7

description Connectio to NewAnnex-F-SW01 GI0/1

no switchport

ip address 10.3.0.33 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/8

description Connectio to NewAnnex-G-SW01 GI0/1

no switchport

ip address 10.3.0.37 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/9

description Connectio to NewAnnex-H-SW01 GI0/1

no switchport

ip address 10.3.0.41 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/10

no switchport

ip address 10.3.0.29 255.255.255.252

speed nonegotiate

!

interface GigabitEthernet1/0/11

!

interface GigabitEthernet1/0/12

description Connectio to Albian Village Data Centre R2 Port GI2/10

no switchport

ip address 10.3.0.10 255.255.255.252

speed nonegotiate

!

interface Vlan1

no ip address

shutdown

!

ip classless

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/1

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/12 2

ip route 10.3.0.50 255.255.255.255 GigabitEthernet1/0/5

ip route 10.3.0.54 255.255.255.255 GigabitEthernet1/0/5

ip route 10.40.4.0 255.255.255.128 GigabitEthernet1/0/2

ip route 10.40.4.128 255.255.255.128 GigabitEthernet1/0/3

ip route 10.40.5.0 255.255.255.128 GigabitEthernet1/0/4

ip route 10.40.5.128 255.255.255.128 GigabitEthernet1/0/5

ip route 10.40.6.0 255.255.255.128 GigabitEthernet1/0/10

ip route 10.40.6.128 255.255.255.128 GigabitEthernet1/0/7

ip route 10.40.7.0 255.255.255.128 GigabitEthernet1/0/8

ip route 10.40.7.128 255.255.255.128 GigabitEthernet1/0/9

ip route 10.40.8.0 255.255.255.0 GigabitEthernet1/0/5

ip route 10.40.9.0 255.255.255.248 10.3.0.14

ip route 10.40.9.24 255.255.255.248 10.3.0.26

ip http server

ip http secure-server

!

ip sla enable reaction-alerts

logging trap warnings

snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart

snmp-server enable traps tty

snmp-server enable traps cluster

snmp-server enable traps entity

snmp-server enable traps cpu threshold

snmp-server enable traps vtp

snmp-server enable traps vlancreate

snmp-server enable traps vlandelete

snmp-server enable traps flash insertion removal

snmp-server enable traps port-security

snmp-server enable traps envmon fan shutdown supply temperature status

snmp-server enable traps config-copy

snmp-server enable traps config

snmp-server enable traps hsrp

snmp-server enable traps bridge newroot topologychange

snmp-server enable traps stpx inconsistency root-inconsistency loop-inconsistency

snmp-server enable traps syslog

snmp-server enable traps mac-notification change move threshold

snmp-server enable traps vlan-membership

!

line con 0

login local

line vty 0 4

login local

transport input ssh

line vty 5 15

login local

transport input ssh

!

ntp clock-period 36029124

ntp server 10.3.0.5

end

Matthew Blanshard · ‎05-17-2012

You have the following in your configuration:

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/1

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/12 2

ip route 10.3.0.50 255.255.255.255 GigabitEthernet1/0/5

ip route 10.3.0.54 255.255.255.255 GigabitEthernet1/0/5

ip route 10.40.4.0 255.255.255.128 GigabitEthernet1/0/2

ip route 10.40.4.128 255.255.255.128 GigabitEthernet1/0/3

ip route 10.40.5.0 255.255.255.128 GigabitEthernet1/0/4

ip route 10.40.5.128 255.255.255.128 GigabitEthernet1/0/5

ip route 10.40.6.0 255.255.255.128 GigabitEthernet1/0/10

ip route 10.40.6.128 255.255.255.128 GigabitEthernet1/0/7

ip route 10.40.7.0 255.255.255.128 GigabitEthernet1/0/8

ip route 10.40.7.128 255.255.255.128 GigabitEthernet1/0/9

ip route 10.40.8.0 255.255.255.0 GigabitEthernet1/0/5

What happens here is that the connected router will proxy arp for every ip that you need to talk to across those links. This is especially bad with the default routes because you will install a /32 route for every internet route. Since the 3750 can only hold around 8k routes in the tcam (depending on the SDM template) you will start software switching and fill all the available memory with the /32 routes.

If you change the routes to be the next hop IP instead of the interface you should fix this issue.

-Matt

View solution in original post

Matthew Blanshard · ‎05-17-2012

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/1

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/12 2

ip route 10.3.0.50 255.255.255.255 GigabitEthernet1/0/5

ip route 10.40.8.0 255.255.255.0 GigabitEthernet1/0/5

These routes are all going to have a better metric then anything learned through EIGRP so the static route will always be prefered. I suspect that this was your initial problem and causing you to run out of memory. If these users have internet access you can run out of routes super fast on a 3750 like this.

-Matt

View solution in original post

lgijssel · ‎04-29-2012

You are hitting a bug:

•CSCth87458

A memory leak occurs in the SSH process, and user authentication is required.

The workaround is to allow SSH connections only from trusted hosts.

As described in:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_55_se/release/notes/OL23054.html#wp1101160

Either use the suggested workaround or perform an upgrade to a newer version.

regards,

Leo

cshannahan · ‎04-29-2012

Thanks, so I can do this..

Router(config)#access-list 23 permit 10.10.10.0 0.0.0.255
Router(config)#line vty 5 15
Router(config-line)#transport input ssh
Router(config-line)#access-class 23 in
Router(config-line)#exit

Or upgrade to let's say 12.2(55)SE4

lgijssel · ‎04-29-2012

I would recommend configuring line vty 0 15 instead of 5 15.

Trusted hosts means "no authentication required" so you would also need to configure "no authentication login"

Essentially, upgrading is the preferred solution.

regards,

Leo

cshannahan · ‎04-29-2012

Thanks, I did do the 0 15 actually. I put it on all of the switches over there, all of them are running the same IOS and I noticed one of the leaf switches was complaining about the same thing. I will monitor it, if they still have issues I will upgrade. Will let you know.

Thanks

Chris

cshannahan · ‎04-29-2012

Also, what do you mean by this ?

Trusted hosts means "no authentication required" so you would also need to configure "no authentication login"

Leo Laohoo · ‎04-29-2012

Upgrade the IOS to either 12.2(55)SE4 pr 12.2(55)SE5.

cshannahan · ‎04-30-2012

Added the ACL didn't work, I will try upgrading today.

cshannahan · ‎05-01-2012

Ok I updated to 12.2 (55)SE5 and I'm still having the same issues. I'm actually having the issue with 2 few switches, one is a 3750 and one is a 3560. I have others running the same hardware and code there as well that are not having the issue.

Could I have 2 bad switches?

Did anyone look at the config? I don't see why it's running out of memory!!!

Leo Laohoo · ‎05-01-2012

I does not sound like faulty hardware.

You are getting Tracebacks. This means either a faulty hardware or an IOS bug. Next you are getting "low memory" and this means it's an IOS bug.

Can you please post the output to the command "sh version" and "sh logs"?

Andrew Cink · ‎05-01-2012

Can you do a show process cpu and show memory and attach the output?

Thanks,

Andy

cshannahan · ‎05-16-2012

Thanks for the replies, I will get the info and post it. I did try a second fiber switch, running some older code and I got the exact same thing.

glen.grant · ‎05-17-2012

The 3750 line has always had an issue with memory problems for some reason. After extended running they get low on memory and it will show up as errors in the logs or the inability to telnet or ssh into the box . If this happens the only real fix is to reload the box. I don't know if they have fixed this issue in newer codes or not but this keeps popping up occasionally in this forum and I have seen the issue myself .

cshannahan · ‎05-17-2012

We are running the same switch in other places but we are running it as a switch, not a router. I was thinking about making it all layer 2 and using trunks rather than routes, etc.

Matthew Blanshard · ‎05-17-2012

You have the following in your configuration:

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/1

ip route 0.0.0.0 0.0.0.0 GigabitEthernet1/0/12 2

ip route 10.3.0.50 255.255.255.255 GigabitEthernet1/0/5

ip route 10.3.0.54 255.255.255.255 GigabitEthernet1/0/5

ip route 10.40.4.0 255.255.255.128 GigabitEthernet1/0/2

ip route 10.40.4.128 255.255.255.128 GigabitEthernet1/0/3

ip route 10.40.5.0 255.255.255.128 GigabitEthernet1/0/4

ip route 10.40.5.128 255.255.255.128 GigabitEthernet1/0/5

ip route 10.40.6.0 255.255.255.128 GigabitEthernet1/0/10

ip route 10.40.6.128 255.255.255.128 GigabitEthernet1/0/7

ip route 10.40.7.0 255.255.255.128 GigabitEthernet1/0/8

ip route 10.40.7.128 255.255.255.128 GigabitEthernet1/0/9

ip route 10.40.8.0 255.255.255.0 GigabitEthernet1/0/5

What happens here is that the connected router will proxy arp for every ip that you need to talk to across those links. This is especially bad with the default routes because you will install a /32 route for every internet route. Since the 3750 can only hold around 8k routes in the tcam (depending on the SDM template) you will start software switching and fill all the available memory with the /32 routes.

If you change the routes to be the next hop IP instead of the interface you should fix this issue.

-Matt