Cisco 5508 Memory High Usage

[jamancill]

Hi all

I have a Issue WLC 5508 ,  the memory is kept at 80%, while the CPU is at 1%.

8.5.131.0 code ,    with 90 AP

Question : How can I check what is draining the memory of the WLC?

thanks

[Rasika Nayanajith]

"show memory summary" command should give some indication

(WLC) >show memory ?

allocations Query Memory pool allocations
history     Display system memory usage history statistics.
mbuf        Display mbuf information.
monitor     Display detailed memory monitor information.
pools       Query Memory pool per task allocations
statistics  Display system memory usage statistics.
summary     Display system memory usage statistics.

HTH

Rasika

*** Pls rate all useful responses ***

[jamancill]

Thanks Rasika.

what I really need is to know, specifically, what is increasing the memory in the WLC to 80% and CPU maintains to 1%

[Leo Laohoo]

SO the controllers are running in HA. Post the complete output to the command "sh memory mbuf summary".

[jamancill]

Hi Leo

wlc>show memory mbuf summary

System Up Time................................... 224 days 23 hrs 41 mins 8 secs
Maximum number of Mbufs.......................... 4608
Number of Mbufs Free............................. 4592
Number of Mbufs In Use........................... 16
                                                    Top 10 Mbuf Usage:
802.11 Mgmt...................................... 65522
DOT1X............................................ 65479
Fast Path Control................................ 65244
BONJOUR.......................................... 64912
IPv6............................................. 1521
IAPP............................................. 109
802.11 Probe..................................... 1
CAPWAP........................................... 1
DHCP............................................. 1
CDP.............................................. 1

[Leo Laohoo]

MBuf looks good.

[jamancill]

thanks Leo..

some other idea because so much memory is being used on WLC ..

[Leo Laohoo]

Disable HA and see if the memory improves.
I wouldn't be using 8.5.131.0. I believe this is the exact version where the hot standby will crash all the time and it won't give any error message or crash logs.

[jamancill]

my controller version is just this one.
Do you have news of a BUG for this version?
unfortunately I do not have a window of time, for now, to disable the HA


System Information
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.131.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.95.16
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014

[patoberli]

I suggest you upgrade to 8.5.140.0 which fixes a HA SSO issue. The reboot will also clear the memory for a time.

[jamancill]

thanks petoberli
the problem is not the HA SSO, but it is with high memory consumption (80%) and it does not go down from there, while the CPU is at 1%

[patoberli]

I know, but I'm pretty sure the reboot will (temporarily) fix the high memory load.
I would need to read the release notes of the newer 8.5.140.0 version to see if memory leaks were fixed.

[jamancill]

thanks so much Patoberli

I'm going to request a maintenance window, to restart the controller.

for now ....., how can I know which processes are occupying memory? or is it not so trivial to find out?

thanks again

Javier Mancilla

[patoberli]

show memory summary

shows you the memory usage and the processes using it.

In my case:

(Cisco Controller) >show memory summary 
-------------------------- System Memory Summary -------------------------
System Name:wlc-5520-1 Primary SW Ver:8.5.140.0
Current Time:Wed Apr 10 08:52:02 2019 System UP Time:75 days 9 hrs 30 mins 0 secs
NAME: "Chassis"    , DESCR: "Cisco 5520 Wireless Controller"
PID: AIR-CT5520-K9,  VID: V01,  SN: XXXXXX
Total System Memory.............................. (32380884   KB) 31621 MB
Total System Free Memory......................... (29339956   KB) 28652 MB (90 %)
Total Memory in Buffers.......................... (86580      KB)
Total Memory in Cache............................ (327892     KB) 320 MB
Total Active Memory.............................. (2527436    KB) 2468 MB
Total InActive Memory............................ (334916     KB) 327 MB
Total Memory in Anon Pages....................... (2447884    KB) 2390 MB
Total Memory in Slab............................. (32812      KB) 32 MB
Total Memory in Page Tables...................... (7816       KB) 7 MB
Effective Free Memory............................ (29395064   KB) 28706 MB (90 %)
WLC Peak Memory.................................. (6147056    KB) 6002 MB
WLC Virtual Memory Size.......................... (5838768    KB) 5701 MB
WLC Resident Memory.............................. (2350408    KB) 2295 MB
WLC Data Segment Memory.......................... (5728420    KB) 5594 MB
Total Heap Including Mapped Pages................ (1784002    KB) 1742 MB
Total Memory in Pmalloc Pools.................... (1713789    KB) 1673 MB
Total Used Memory in Pmalloc Pools............... (1614790    KB) 1576 MB

--More-- or (q)uit
Total Free Memory in Pmalloc Pools............... (73022      KB) 71 MB
------------------------- Pmalloc Pools Information --------------------
Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB
0     16        283800         279261        33257   /33186   /70      
1     64        160000         68442         26250   /20527   /5722    
2     128       108700         42849         24627   /16395   /8231    
3     256       21500          13476         7558    /5552    /2006    
4     384       28000          3390          13343   /4115    /9228    
5     512       63000          13943         37898   /13369   /24528   
6     1024      152700         148505        168208  /164013  /4195    
7     2048      10000          3727          21015   /8469    /12546   
8     4096      26350          24726         108076  /101580  /6496    
9     Raw-Pool  0              2421          1273311 /1273311 /0
-------------------------  MBUF Information ----------------------------
Maximum number of Mbufs.......................... 24576
Number of Mbufs Free............................. 24560
Number of Mbufs In Use........................... 16
(Cisco Controller) >

 The thing with the WLC is, to free memory, you have to reboot it (I don't know of another way). While you are doing that, I suggest that you also right upgrade to 8.5.140.0, as that update fixes some critical SSO HA bugs. Unless you use AP702 models, then don't upgrade to that version.

See here for some more details: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc9

And here the release notes: http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn85mr4.html#resolved-caveats_85mr4

There were at least three memory leaks fixed: CSCuw22659, CSCvm00214, CSCvm78368

[jamancill]

I'm going to restart the controller and see if the memory usage levels are maintained.
otherwise I will open a case to TAC

in my case

wlc >show memory summary
-------------------------- System Memory Summary -------------------------
Primary SW Ver:8.5.131.0
NAME: "Chassis"    , DESCR: "Cisco 5500 Series Wireless LAN Controller"
PID: AIR-CT5508-K9,  VID: V04,  S/N: xxxxx
Total System Memory.............................. (1002612    KB) 979 MB
Total System Free Memory......................... (27720      KB) 27 MB (2 %)
Total Memory in Buffers.......................... (5568       KB)
Total Memory in Cache............................ (151492     KB) 147 MB
Total Active Memory.............................. (876452     KB) 855 MB
Total InActive Memory............................ (33748      KB) 32 MB
Total Memory in Anon Pages....................... (753164     KB) 735 MB
Total Memory in Slab............................. (48276      KB) 47 MB
Total Memory in Page Tables...................... (3056       KB) 2 MB
Effective Free Memory............................ (31464      KB) 30 MB (3 %)
WLC Peak Memory.................................. (1298188    KB) 1267 MB
WLC Virtual Memory Size.......................... (1297224    KB) 1266 MB
WLC Resident Memory.............................. (799436     KB) 780 MB
WLC Data Segment Memory.......................... (1200112    KB) 1171 MB
Total Heap Including Mapped Pages................ (466820     KB) 455 MB
Total Memory in Pmalloc Pools.................... (616775     KB) 602 MB
Total Used Memory in Pmalloc Pools............... (599845     KB) 585 MB

--More or (q)uit current module or <ctrl-z> to abort
Total Free Memory in Pmalloc Pools............... (15279      KB) 14 MB
------------------------- Pmalloc Pools Information --------------------
Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB
0     16        50000          40741         3320    /3175    /144
1     64        56600          50499         6411    /6030    /381
2     128       20000          17629         3515    /3219    /296
3     256       976700         976614        293773  /293751  /21
4     384       6000           964           2554    /666     /1888
5     512       27500          23009         15146   /12900   /2245
6     1024      6000           4694          6304    /4998    /1306
7     2048      5500           2369          11279   /5017    /6262
8     4096      1000           316           4050    /1314    /2736
9     Raw-Pool  0              374           270404  /270404  /0
-------------------------  MBUF Information ----------------------------
Maximum number of Mbufs.......................... 4608
Number of Mbufs Free............................. 4592
Number of Mbufs In Use........................... 16