after upgrading about 35 Catalyst 2960 and Catalyst 2960S to IOS 15.0(2)SE2, we experience a memory leak on several switches. After some days / weeks the switches are not accessible via Console/Telnet/SSH/Web any more. Only SNMP seems to work properly.
Attached users do not experience any decrease in service.
Trying to connect to the console, we get following error message:
"% Low on memory; try again later"
The only (temporary) solution is to reboot the switch. The behavior is similar to Bug CSCts52797.
With regards to the Bug notes this bug should only affect Catalyst 2960 with 64MB of RAM and should already be solved with IOS 15.0(2)SE2.
We experience the erroneous behavior with
-WS-C2960-48TC-S running IOS 15.0(2)SE2
-WS-C2960S-48LPS-L running IOS 15.0(2)SE2
See also discussion:
Could anyone provide an advice to which stable IOS release we could downgrade or if this bug will definetly be fixed in any specific future release?
Are these switches covered under a maintenance contract? If so then I would suggest that the best thing to do would be to open a case with Cisco TAC. They are the best source of advice about when a bug is really going to be fixed. Also they would be able to investigate and determine whether you are running into 52797 or whether you might be running into a different bug. Or even that it might be a new bug which they have not seen before.
Sadly they are not - if they were I had already opened a service request.
Surprisingly all symptoms of Bug CSCts52797 match; but the bug should already be solved for a long time....
Hence I assume we are dealing with a "new memory leak" and the symptoms we experience are the normal way the switch handles memory leaks in general.
Hmmmmm ... I've got fleets of 3560C and 2960/2960S running 15.0(2)SE2, since January 2013, and I have NOT seen this bug.
Can I ask if you remove all your SNMP traps and see if it helps?
While you are contemplating doing that, get ready to downgrade your IOS to 12.2(55)SE7 as this is considered, in my opinion, to be one of the most stable IOS around right now. ALL of my 3750/G/E/X and 3560/G/E/X are using this version.
Removing the SNMP- traps sounds interesting.
Sadly the affected switches are random. No switches that are "always" affected and others that are not. Also there is no specific time frame when the problem arises.
Hence I would have to remove the SNMP- Traps from all switches to see if it works - leaving my network management blind for a couple of days / weeks.....
Hopefully someone can give me a hint about the actual memory leak and version this new bug will be fixed.
Otherwise I will consider removing the SNMP traps....
Symptom: A Catalyst 2960 with 64Mb of DRAM might display Low memory on the console after upgrading to 12.2(58)SE or later Conditions: Workaround: Limit the memory used by different features on the switch if this release are required. Minimizing the number of trunk ports and vlans in use on the switch will reduce memory usage.
Recommendation as of now: To downgrade to the privious software version which you were running . (12.2(55) SE7 or SE4).
If you required more troubleshooting I request you to kindly open the TAC case and we shall provide you the more info on the same.
the advice to "reduce all unnecessary features, trunks, VLANs" was already discussed in the thread I referenced in my initial posting.
As Richard and leolaohoo we are also using a very clean and basic setup with 2 trunks (the uplinks) and a total of 4 VLANs on the switches. A little bit of snmp, loop-guard and bpduguard - no exotic features like 802.1x, DHCP- snooping or similar.....
Could you please provide any information, if Cisco already raised a new Bug-ID for "our" problem?
Searching around I found a pretty number of postings showing that other guys have also memory leaks with 15.0(x)SEyy versions on different platforms.
Hence I would be glad to hear the issue is already addressed and will be fixed in version 15.something.
Could you please provide any information, if Cisco already raised a new Bug-ID for "our" problem?Searching around I found a pretty number of postings showing that other guys have also memory leaks with 15.0(x)SEyy versions on different platforms
CPU bug and memory leaks on 15.0(1)SE and 15.0(2)SE are very well discussed in the CSC forums. A lot of people have hit it. I for one hit it even though my 2960-48PST did not have any config.
I have no idea how many INTERNAL bug IDs are there available, but everyone hinges to just one publically-known bug ID number which you've posted already.
My question to you is this: What is your operational requirement(s) to stick to 15.0(1)SE or 15.0(2)SE? If the features you seek are found in 12.2(55)SE7, then use this. All of my fleet of 3560/3750 are running 12.2(55)SE7.
All my fleet of 2960/2960S, however, are running 15.0(2)SE2 and I'm still testing. I am, unfortunately, not confident to personally recommend 15.0(2)SE2 as a "stable" IOS because I am not yet finish with our testing.
I was researching this issue myself, and I found that one of our 2960G switches had a difference on it that the others didn't, and that was I had an IP SLA configured on it. Monitoring the I/O memory showed a constant decline to a point where you couldn't remotely access the switch, and you had to powercycle it to recover, only to have it repeat after about three months. Onoce I remvoed the IP SLA statement, the I/O memory utilization has stabilized. Not sure if anyone else has something that is CPU intensive on their switches defined in their configs, but the best case is to strip the switch back to what it was made to do and see if that helps.
Hello to all,
to close this discussion I want to provide an update concerning this issue:
In the meanwhile I opened a Service Request at Cisco which sadly did not come to a solution.
TAC engineer found a non-public Bug-ID which he thought should match our problem. Sadly all further debugging did show a 100% match to this Bug-ID. So we spent day after day to collect additional information that did not exactly match the Bug description.
After several weeks of investigating with Cisco TAC with no result I decided to downgrade all Cat 2960 and 2960S to 12.2(55)SE7 what was recommended somewhere in this discussion (and also agreed by Cisco TAC engineer).
Since then we did not experinece above problem again....
By the way - as mentioned before the switches had no exotic or CPU- intensive config or features at all.
Thank you for providing this update about the problem. I am glad that you were able to open a Service Request with Cisco and that you found that downgrading the code made the problem go away. Hopefully Cisco will fix that problem in the next release of 15.0.
My last response is old.
Here's my update:
1. All of my 3750/G/E/X switches are running 12.2(55)SE8. The "55" is considered as one of the most well-behaved IOS I've ever come across;
2. All of my 3560E/X are running 15.0(2)SE4;
3. All of my 3560CG (compact 8-port) are running 15.0(2)SE4;
4. All of my 2960/G/S fleet are running 15.0(2)SE4.
I'm currently testing the new 15.2(1)E on a single 3560CG and so far, no issues.
Just as a forewarning to others, 15.2(1)E1 also has this problem. I was running 15.2(1) without any issues, running 15.2(1)E1 gave me the "%% Low on memory; try again later" message on the console and remote access is not possible. If I reboot the switch it still comes up this way. I have to unplug everything for it to comeup so I can have access to it. After the first reload to get the new image, the sitch CPU was at 80%, now it is back to around 20% which is where it normally is but I had to manually pull the plug on it and let it boot. So, I will be going back to 15.2(1) as I had no issues with it. I will open a TAC case later but just wanted to warn others.
Thanks for the warning, however, can I make a request? Can you make a new thread? If you do so, we can LINK your thread to any new threads opened up with 15.2(1)E1.