12.2(58)SE2 on a 2960 cause memory leaks

[pasztor.richard]

Hi all,

I had a very disappointed experience today  during the planned upgrade on my 2960 to IOS version 12.2(58)SE2 after business hours. We have a small office with 3x 2960 and a couple of 2950. There is a single 2960 called as the "central" as its the STP root, and most other switches are directly connected to the central. The remaining switches are conected to one of the switches conneting to the central.

I tested the aforementioned IOS on 2x 2960 las t week, but did not upgrade the central one. The 2x switches ran fine for a week, so I decided to upgrade the central one: basically it was a disaster. After the restart, the central became immediately unavailable via SSH (though admin IP pingable). I connected to the console, and realized that even console does not work, as I received the error message: "Low memory", and the console session could not be started. I restarted the switch once again (had to cut the power) and this time I managed to connect via the serial: the free memory was approx 2 MB, though its was over 5 MB if running 12.2(55)SE1. After 1-2 mins, free memory decreased down to 1.8 megs. SSH was still not functioning. At this time, fearing of a sudden memory leak during the night, I had to rollback to the previous version.

My config is quite simple: RSTP root, DHCP snooping, 1 LACP etherchannel, only 5x VLANs with VTP server, MOTD and BANNER (yeah I know its a risky feature since the previous IOS) I wonder what can be the problem here?

I have attached the running config of the central 2960 (removed some sensitive entries)

[Somasundaram Jayaraman]

Hi,

There is a bug raised for this issue and it is in assigned to a Development engineer now.

As of now i cannot share the bug id with you, since it is internal to cisco.

Symptomps:

===========

When booting a 2960 with 64Mb of DRAM a low memory error can be observed under certain 

circumstances when running 12.2(58)SE or later

This will prevent acess to the console/ssh/telnet of the switch.

++ As a workaround we need to downgrade the IOS until the bug is fixed.

Hope this helps.

Cheers

Somu

Rate helpful posts 

[Cedric DELAUNAY]

Hi,

Is there any news about this bug ?

Same problem with our 2960s.

We have more than 180 and some of them can't be managed with telnet.

Which release can we downgrade and be sure do not have any problems ?

12.2.58SE1 ? 12.2.55SE4 ?

Thanks

[Alfredo Bosca Bataller]

I have the same issue. And my current IOS is: 12.2(35)SE5. What advice would give me? I can to upgrade the IOS to the last stable: c2960-lanbasek9-mz.150-2.SE7.bin

show memory debug leaks summary

Adding blocks for GD...

                 Processor memory

Alloc PC        Size     Blocks      Bytes    What

0x3D1370  0000000040 0000000001 0000000040    hulc_flash init

0x3D13E4  0000001512 0000000001 0000001512    hulc_flash init

0x3D159C  0000000040 0000000001 0000000040    hulc_flash init

0x669FE0  0000000024 0000000014 0000000336    Init

0x669FE0  0000000056 0000000004 0000000224    Init

0x669FE0  0000000080 0000000002 0000000160    Init

0x669FE0  0000000120 0000000002 0000000240    Init

0x7380B0  0000000024 0000000001 0000000024    Init

0x7380D8  0000000040 0000000001 0000000040    Init

0x7380F4  0000000024 0000000001 0000000024    Init

0x73811C  0000000040 0000000001 0000000040    Init

0x96C878  0000000024 0000000001 0000000024    Init

0x96F0C8  0000000024 0000000001 0000000024    Init

0x96F0C8  0000000200 0000000001 0000000200    Init

Thanks,

[Leo Laohoo]

I've got the same issue with 15.0(1)SE.  Mostly affecting the WS-C2960-48-PST.  Currently got a TAC Case.  Don't know where it is now. 

[pasztor.richard]

Thanks for the update. I think I'll wait one more release to check this. Strange though, all the 3x 2960 are the same model 2960-24TC-L, and the 2 non-central ones are not experiencing such issue.

[Leo Laohoo]

Not happy with the 15.0 IOS.  I have good runs using 12.2(58)SE1 IOS.  No issues so far.

[pasztor.richard]

Its the 1st release of a new major version: I wasnt brave enough to install it on any production equipment.

[Leo Laohoo]

Good on you.  The 15.0 IOS has a major memory leak even before the config gets loaded.  This is happening across the board and not just on the 2960.

I recommend anyone reading this post to stay away from the 15.0 like a plague.

[ankugarg]

HI Leolaohoo,

I think its not that bad ...I was of the opinion that memory leak is specific to 2960s...Can you please share the memory leak..I am kinda sure that 3ks won't have memory leak without any config

[Leo Laohoo]

I value second opinions, Ankur.  Maybe you can make some sense ...

The TAC engineer successfully recreated the issue when loaded 15.0(1) IOS into a 2960 in his first attempt.

As soon as I upgraded to 15.01(SE) (c2960-lanbasek9-mz.150-1.SE) the switch memory went very low, this is without applying the configuration as per your network. Below is the output taken from my lab switch. I would suggest to downgrade the IOS Version image. We are getting in touch with the development team regarding this issue. 

Switch#sh processes memory sorted 

Processor Pool Total:   21341260 Used:   18412492 Free:    2928768 (Hardly 3MB left, in your case it was 2MB)

 I/O Pool Total:    4194304 Used:    2484948 Free:    1709356
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
0   0   24228980    6113380   15373936          0          0 *Init*
0   0   17229304   16069812    1172040    5380179    2002472 *Dead*

According to the TAC engineer, he has seen some cases of 3750's coming their desk.

Cisco Bug ID:  CSCts52797

[aranyushkin]

I'm also getting the same problem with WS-C2960-48PST-L. It even can't start CLI and gives a message "

%% Low on memory; try again later" for any keybord input.

[glen.grant]

Cisco seems to have a real issue with this . We saw this even on earlier versions of the 3750 where you could not telnet to the box at times . Reload was the only fix.

[smogra]

The bug has been added as a open caveat in release notes:

CSCts52797 (Catalyst 2960)

A Catalyst 2960 with 64Mb of DRAM might display low memory on the  console after you upgrade the switch to 12.2(58)SE or later.

The workaround is to limit the memory that is used by different features  on the switch if this release is required. You can reduce memory usage  by minimizing the number of trunk ports and VLANs in use on the switch.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/15.0_1_se/release/notes/OL25301.html#wp1044235

--- Sweta

Please rate helpful posts.

[Leo Laohoo]

The workaround is to limit the memory that is used by different features  on the switch if this release is required. You can reduce memory usage  by minimizing the number of trunk ports and VLANs in use on the switch.

Hi Sweta,

This ain't going to work.  All my switch has only ONE (1) trunk port and 5 VLANs allowed on the trunk port.

I'll say it once and I'll say it again, the 15.0(1)SE IOS is extremely buggy.  I would recommend this version to be pulled out from the website as well as recommend anyone to treat this version like a plague.