High memory utilization on the 5508 (Approx 87%)

[anthony.wathen1]

Hello everyone. So firstly, I'd like to say this was a recent issue, and this memory usage increased from about 70% from a couple of weeks ago. We upgraded our code to 7.4.140.0 a couple of months ago, but we hadn't seen any increases until a few weeks ago. The first controller has approximately 2000 users on it at roughly 3-5% CPU utilization at most times. This controller has had no changes to it within the last couple of months. There are approximately, on the primary controller, 435 APs, however our secondary controller is only running 300 clients and around 165 APs, and it is still running at about 82%, which was down to about 65% a couple of weeks ago.

We have approximately 6 SSIDs on this controller, with one of them being anchored to our anchor controller for guest access. All controllers, including the anchor, are currently running 7.4.140.0 and I couldn't find any bug reports on this version. Also, I am getting some  new logs I didn't see a couple of weeks ago that look like this:

*sisfSwitcherTask: Feb 12 08:03:42.701: #SISF-3-INTERNAL: sisf_shim_utils.c:316 Internal error, Cannot create binding entry
-Traceback:  0x11030444 0x10ffd718 0x10ffedbc 0x10fffe2c 0x115f1514 0x115f2210 0x11032e0c 0x10b18b10 0x1203b860 0x1209c9fc
*sisfSwitcherTask: Feb 12 08:03:42.700: #SISF-3-INTERNAL: sisf_shim_utils.c:316 Internal error, Alloc bt_chunk failed
-Traceback:  0x11030444 0x10fed704 0x10fee22c 0x10ffd6ec 0x10ffedbc 0x10fffe2c 0x115f1514 0x115f2210 0x11032e0c 0x10b18b10 0x1203b860 0x1209c9fc
*sisfSwitcherTask: Feb 12 08:03:42.700: #SISF-3-INTERNAL: sisf_shim_utils.c:316 Internal error, SISF chunk malloc failed: system running out of memory (13 % free mem left, threshold is 20 %)
-Traceback:  0x11030444 0x11031dc8 0x10fed6d8 0x10fee22c 0x10ffd6ec 0x10ffedbc 0x10fffe2c 0x115f1514 0x115f2210 0x11032e0c 0x10b18b10 0x1203b860 0x1209c9fc

One other thing to bring up, if it is relevant (Even though they've been running on these cores for roughly 6 months, and no code upgrade was done to them) is that we have these controllers connected to 2 Avaya VSP 9000s running an SMLT (Which is like a vPC for Avaya). Again, there were no changes done for these controllers nor any switch/router upgrades in the last couple of months. Please advise.

[Leo Laohoo]

*sisfSwitcherTask: Feb 12 08:03:42.700: #SISF-3-INTERNAL: sisf_shim_utils.c:316 Internal error, SISF chunk malloc failed: system running out of memory (13 % free mem left, threshold is 20 %)

Post the complete output to the command "sh memory stati" from the WLC of concern.

[anthony.wathen1]

Here is the primary one with the most APs/clients associated I had mentioned:

System Memory Statistics:
Total System Memory............: 1027293184 bytes
Used System Memory.............: 898105344 bytes
Free System Memory.............: 129187840 bytes
Bytes allocated from RTOS......: 373822272 bytes
Chunks Free....................: 944 bytes
Number of mmapped regions......: 59
Total space in mmapped regions.: 311787520 bytes
Total allocated space..........: 370408360 bytes
Total non-inuse space..........: 3413912 bytes
Top-most releasable space......: 322584 bytes
Total allocated (incl mmap)....: 685609792 bytes
Total used (incl mmap).........: 682195880 bytes
Total free (incl mmap).........: 3413912 bytes

[Leo Laohoo · Accepted solution]

Free System Memory.............: 129187840 bytes

I believe 5508 has a 512mb memory.  The output above shows 129mb.  Keep an eye on it on a daily basis.  Run the command "sh memory statis" and when this value falls below 99 mb the appliance may need a reboot. 

There's a memory leak somewhere (and only TAC can decipher that).  My recommendation is to ditch 7.4.140.X and go straight to 8.1.131.0 if all the APs are supported.

[Chawki Ozeir]

Hello Leo,

I am having a similar issue with HA WLC running 8.0.120.0. It keep on failing every several weeks from primary to secondary. it is at 83% as of today. Please advise

System Memory Statistics:
Total System Memory............: 1026019328 bytes (978.55 MB)
Used System Memory.............: 841527296 bytes (802.60 MB)
Free System Memory.............: 184492032 bytes (175.95 MB)
Bytes allocated from RTOS......: 293516092 bytes (279.93 MB)
Chunks Free....................: 328 bytes
Number of mmapped regions......: 54
Total space in mmapped regions.: 304967680 bytes (290.86 MB)
Total allocated space..........: 255906868 bytes (244.06 MB)
Total non-inuse space..........: 37609224 bytes (35.86 MB)
Top-most releasable space......: 1048568 bytes (1023.99 KB)
Total allocated (incl mmap)....: 598483772 bytes (570.79 MB)
Total used (incl mmap).........: 560874548 bytes (534.93 MB)
Total free (incl mmap).........: 37609224 bytes (35.86 MB)

[anthony.wathen1]

Hi Chawki,

I just wanted to say I got a response from TAC today that says this version (Version 8.0.132.0, Released April 1st) is supposed to have the fix for my particular case, however I'm not sure if your memory leak is related. They claim mine had something to do with the Anchor configuration, and advised disabling IPv6 globally. That didn't work, so we are going to implement the recommended WLC version (8.0.132.0). I'll check back here after we do the upgrade to see if that does the trick.

[Leo Laohoo]

They claim mine had something to do with the Anchor

That and a lot of stuff.  8.0.120.0 is a very buggy version with a lot of memory leak issues.  Cisco, rightfully, have pulled this version. 

Anyone else running 8.0.120.0 (and reading this post/thread) should seriously consider upgrading to a different version.

[Leo Laohoo]

I am having a similar issue with HA WLC running 8.0.120.0.

I can confidently say 8.0.120.0 has a memory leak.  I ran this firmware on four pairs of WiSM2 and I was getting crashes very often. 

8.0.132.0 was officially released on 01 April 2016.  8.0.132.0 Release Notes can be found HERE.

Alternatively, you can also use 8.1.131.0 which I find quite stable.  8.1.131.0 Release Notes can be found HERE.