Serious C3750 memory leaking problems

[vladakoci]

This is fyi.

We have a serious kind of memory leaking troubles on our C3750 stacks. We have hundreds of such stacks, but the problem appears only on those that have many features enabled - like many VLANs, many subnets, HSRP, STP root bridge etc.

The symptom is that first we lose SSH, an error message like this can be found in log

Feb 13 13:36:29.336: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory

then we lose telnet ( we normally do not use telnet but enabled on those )

then we are not able to log in through console, getting errors like these on console

%% Low on memory; try again later

and then the switch loses it L3 and L2 functionalities and needs to be restarted.

The whole process takes some time, like two weeks, it develops slowly, it is not a suden strike.

We reported this to Cisco about a month ago, they tried to match to these bugs

CSCtt96255

CSCuc03649

but it looks like it is a new bug.

We tried various IOS v15 versions, but did not help. Provided a lot of info from runing switches, Cisco development team is involved, but the root cause not known yet. Cisco's advice was to get te switch rebooted regurarly.

Internaly we made a decision to downgrade IOS on one of these switches, and went to 122-58.SE2 on 23 Feb and since that we have not had any issues. Of course there is no guarantee we will not have, but so far we are happy and going to downgrade the other ones as well.

[james.doukas]

I did the "no macro auto monitor" and the errors stopped. Didn't have to do any of the disabling of the lldp or cdp as mentioned in the bug notice. The one command seemed to have done the trick.

[vladakoci]

Let me just add that we downgraded to 12.2(58)SE2 on all our 'problematic' switches months ago and have not had any issues since. 

12.2(58)SE2 is feature-wise okay with us.

[Mark Mattix]

I'm currently having this problem using IOS version: 15.0(2)SE2

Today a Cisco TAC gave me the following suggestion of commands to stop the problems. He said they are not guranteed but may work and if they don't to upgrade the switches to version:

Description:        IP SERVICES

Release:               15.0.2-SE5

Release Date:    06/Nov/2013

File Name:           c3750-ipservicesk9-mz.150-2.SE5.bin

Min Memory:    DRAM 128 MB Flash 32 MB

Size:       17.42 MB (18262656 bytes)

MD5 Checksum:               cc18212e2be7838d946f645f4cb16fc7

My question is about the commands he gave me, he said to expect some downtime when entering them but I would like to know more first on what they're going to do:

Workaround:

Configure the following:

no macro auto monitor

device-sensor filter-spec dhcp exclude all

device-sensor filter-spec lldp exclude all

device-sensor filter-spec cdp exclude all

If additional leaks are seen in "DHCPD Receive" process,  disable the built-in DHCP server as well: no service dhcp

My switch stack configuration is pretty basic, I'm not sure what would cause downtime and don't know for how long.. Any help would be appreciated! I might e-mail the TAC back in hope of getting a more detailed description of when and how the downtime will occur.

[james.doukas]

I did this one command on my 3750 stack.

no macro auto monitor

There was no down time and the memory leak stopped. This was back in November of last year. Haven't seen the error since.

[Mark Mattix]

Thanks for the advice James, I justed entered the command so we'll see how it does. Do you remember if your memory usage went down after entering the command or did it just stop going up? Thanks again

[Leo Laohoo]

I'm currently having this problem using IOS version: 15.0(2)SE2

Do NOT use 15.0(2)SE2.

Do NOT use 15.0(2)SE5.

If you don't need feature found in 15.X, try 12.2(55)SE8.  Otherwise, if you really need 15.X then try 15.0(2)SE4.

[Mark Mattix]

Thank you Leo, I wish I would have known this last year before putting them in production. I know now to research the IOS version I plan to use before putting a device in production!

[Leo Laohoo]

Don't feel bad, Mark.

I had to do a lot of tests before we were able to identify a good IOS.

[james.doukas]

It just stopped increasing. To free up the memory, you have to do a reload.