topic Re: Cisco 5508 Memory High Usage in Wireless

Cisco 5508 Memory High Usage

jamancill — Mon, 05 Jul 2021 17:12:11 GMT

Hi all

I have a Issue WLC 5508 , the memory is kept at 80%, while the CPU is at 1%.

8.5.131.0 code , with 90 AP

Question : How can I check what is draining the memory of the WLC?

thanks

Re: Cisco 5508 Memory High Usage

Rasika Nayanajith — Thu, 04 Apr 2019 19:51:08 GMT

"show memory summary" command should give some indication

(WLC) >show memory ?

allocations Query Memory pool allocations
history Display system memory usage history statistics.
mbuf Display mbuf information.
monitor Display detailed memory monitor information.
pools Query Memory pool per task allocations
statistics Display system memory usage statistics.
summary Display system memory usage statistics.

HTH

Rasika

*** Pls rate all useful responses ***

Re: Cisco 5508 Memory High Usage

Leo Laohoo — Fri, 05 Apr 2019 06:55:23 GMT

Is this WLC running on HA SSO?

Re: Cisco 5508 Memory High Usage

jamancill — Fri, 05 Apr 2019 11:18:52 GMT

yes , WLC is in HA SSO

>showredundancy summary
            Redundancy Mode = SSO ENABLED
                Local State = ACTIVE
                 Peer State = STANDBY HOT
                       Unit = Secondary (Inherited AP License Count = 250)
           Redundancy State = SSO
               Redundancy Port = UP
            BulkSync Status = Complete
Average Redundancy Peer Reachability Latency = 456 Micro Seconds
Average Management Gateway Reachability Latency = 11752 Micro Seconds

Re: Cisco 5508 Memory High Usage

jamancill — Fri, 05 Apr 2019 11:58:08 GMT

Thanks Rasika.

what I really need is to know, specifically, what is increasing the memory in the WLC to 80% and CPU maintains to 1%

Re: Cisco 5508 Memory High Usage

Leo Laohoo — Fri, 05 Apr 2019 21:29:36 GMT

SO the controllers are running in HA. Post the complete output to the command "sh memory mbuf summary".

Re: Cisco 5508 Memory High Usage

jamancill — Fri, 05 Apr 2019 23:53:55 GMT

Hi Leo

wlc>show memory mbuf summary

System Up Time................................... 224 days 23 hrs 41 mins 8 secs
Maximum number of Mbufs.......................... 4608
Number of Mbufs Free............................. 4592
Number of Mbufs In Use........................... 16
Top 10 Mbuf Usage:
802.11 Mgmt...................................... 65522
DOT1X............................................ 65479
Fast Path Control................................ 65244
BONJOUR.......................................... 64912
IPv6............................................. 1521
IAPP............................................. 109
802.11 Probe..................................... 1
CAPWAP........................................... 1
DHCP............................................. 1
CDP.............................................. 1

Re: Cisco 5508 Memory High Usage

Leo Laohoo — Sat, 06 Apr 2019 03:43:53 GMT

MBuf looks good.

Re: Cisco 5508 Memory High Usage

jamancill — Sat, 06 Apr 2019 14:10:20 GMT

thanks Leo..

some other idea because so much memory is being used on WLC ..

Re: Cisco 5508 Memory High Usage

Leo Laohoo — Sat, 06 Apr 2019 22:39:20 GMT

Disable HA and see if the memory improves.
I wouldn't be using 8.5.131.0. I believe this is the exact version where the hot standby will crash all the time and it won't give any error message or crash logs.

Re: Cisco 5508 Memory High Usage

jamancill — Sun, 07 Apr 2019 19:37:27 GMT

my controller version is just this one.
Do you have news of a BUG for this version?
unfortunately I do not have a window of time, for now, to disable the HA

System Information
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.131.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.95.16
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014

Re: Cisco 5508 Memory High Usage

patoberli — Mon, 08 Apr 2019 13:52:17 GMT

I suggest you upgrade to 8.5.140.0 which fixes a HA SSO issue. The reboot will also clear the memory for a time.

Re: Cisco 5508 Memory High Usage

jamancill — Mon, 08 Apr 2019 18:34:46 GMT

thanks petoberli
the problem is not the HA SSO, but it is with high memory consumption (80%) and it does not go down from there, while the CPU is at 1%

Re: Cisco 5508 Memory High Usage

patoberli — Mon, 08 Apr 2019 20:06:33 GMT

I know, but I'm pretty sure the reboot will (temporarily) fix the high memory load.
I would need to read the release notes of the newer 8.5.140.0 version to see if memory leaks were fixed.

Re: Cisco 5508 Memory High Usage

jamancill — Tue, 09 Apr 2019 23:20:17 GMT

thanks so much Patoberli

I'm going to request a maintenance window, to restart the controller.

for now ....., how can I know which processes are occupying memory? or is it not so trivial to find out?

thanks again

Javier Mancilla

Re: Cisco 5508 Memory High Usage

patoberli — Wed, 10 Apr 2019 06:57:33 GMT

show memory summary

shows you the memory usage and the processes using it.

In my case:

(Cisco Controller) >show memory summary 
-------------------------- System Memory Summary -------------------------
System Name:wlc-5520-1 Primary SW Ver:8.5.140.0
Current Time:Wed Apr 10 08:52:02 2019 System UP Time:75 days 9 hrs 30 mins 0 secs
NAME: "Chassis"    , DESCR: "Cisco 5520 Wireless Controller"
PID: AIR-CT5520-K9,  VID: V01,  SN: XXXXXX
Total System Memory.............................. (32380884   KB) 31621 MB
Total System Free Memory......................... (29339956   KB) 28652 MB (90 %)
Total Memory in Buffers.......................... (86580      KB)
Total Memory in Cache............................ (327892     KB) 320 MB
Total Active Memory.............................. (2527436    KB) 2468 MB
Total InActive Memory............................ (334916     KB) 327 MB
Total Memory in Anon Pages....................... (2447884    KB) 2390 MB
Total Memory in Slab............................. (32812      KB) 32 MB
Total Memory in Page Tables...................... (7816       KB) 7 MB
Effective Free Memory............................ (29395064   KB) 28706 MB (90 %)
WLC Peak Memory.................................. (6147056    KB) 6002 MB
WLC Virtual Memory Size.......................... (5838768    KB) 5701 MB
WLC Resident Memory.............................. (2350408    KB) 2295 MB
WLC Data Segment Memory.......................... (5728420    KB) 5594 MB
Total Heap Including Mapped Pages................ (1784002    KB) 1742 MB
Total Memory in Pmalloc Pools.................... (1713789    KB) 1673 MB
Total Used Memory in Pmalloc Pools............... (1614790    KB) 1576 MB

--More-- or (q)uit
Total Free Memory in Pmalloc Pools............... (73022      KB) 71 MB
------------------------- Pmalloc Pools Information --------------------
Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB
0     16        283800         279261        33257   /33186   /70      
1     64        160000         68442         26250   /20527   /5722    
2     128       108700         42849         24627   /16395   /8231    
3     256       21500          13476         7558    /5552    /2006    
4     384       28000          3390          13343   /4115    /9228    
5     512       63000          13943         37898   /13369   /24528   
6     1024      152700         148505        168208  /164013  /4195    
7     2048      10000          3727          21015   /8469    /12546   
8     4096      26350          24726         108076  /101580  /6496    
9     Raw-Pool  0              2421          1273311 /1273311 /0
-------------------------  MBUF Information ----------------------------
Maximum number of Mbufs.......................... 24576
Number of Mbufs Free............................. 24560
Number of Mbufs In Use........................... 16
(Cisco Controller) >

The thing with the WLC is, to free memory, you have to reboot it (I don't know of another way). While you are doing that, I suggest that you also right upgrade to 8.5.140.0, as that update fixes some critical SSO HA bugs. Unless you use AP702 models, then don't upgrade to that version.

See here for some more details: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc9

And here the release notes: http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn85mr4.html#resolved-caveats_85mr4

There were at least three memory leaks fixed: CSCuw22659, CSCvm00214, CSCvm78368

Re: Cisco 5508 Memory High Usage

jamancill — Thu, 11 Apr 2019 12:29:45 GMT

I'm going to restart the controller and see if the memory usage levels are maintained.
otherwise I will open a case to TAC

in my case

wlc >show memory summary
-------------------------- System Memory Summary -------------------------
Primary SW Ver:8.5.131.0
NAME: "Chassis"    , DESCR: "Cisco 5500 Series Wireless LAN Controller"
PID: AIR-CT5508-K9, VID: V04, S/N: xxxxx
Total System Memory.............................. (1002612    KB) 979 MB
Total System Free Memory......................... (27720      KB) 27 MB (2 %)
Total Memory in Buffers.......................... (5568       KB)
Total Memory in Cache............................ (151492     KB) 147 MB
Total Active Memory.............................. (876452     KB) 855 MB
Total InActive Memory............................ (33748      KB) 32 MB
Total Memory in Anon Pages....................... (753164     KB) 735 MB
Total Memory in Slab............................. (48276      KB) 47 MB
Total Memory in Page Tables...................... (3056       KB) 2 MB
Effective Free Memory............................ (31464      KB) 30 MB (3 %)
WLC Peak Memory.................................. (1298188    KB) 1267 MB
WLC Virtual Memory Size.......................... (1297224    KB) 1266 MB
WLC Resident Memory.............................. (799436     KB) 780 MB
WLC Data Segment Memory.......................... (1200112    KB) 1171 MB
Total Heap Including Mapped Pages................ (466820     KB) 455 MB
Total Memory in Pmalloc Pools.................... (616775     KB) 602 MB
Total Used Memory in Pmalloc Pools............... (599845     KB) 585 MB

--More or (q)uit current module or <ctrl-z> to abort
Total Free Memory in Pmalloc Pools............... (15279      KB) 14 MB
------------------------- Pmalloc Pools Information --------------------
Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB
0     16        50000          40741         3320    /3175    /144
1     64        56600          50499         6411    /6030    /381
2     128       20000          17629         3515    /3219    /296
3     256       976700         976614        293773 /293751 /21
4     384       6000           964           2554    /666     /1888
5     512       27500          23009         15146   /12900   /2245
6     1024      6000           4694          6304    /4998    /1306
7     2048      5500           2369          11279   /5017    /6262
8     4096      1000           316           4050    /1314    /2736
9     Raw-Pool 0              374           270404 /270404 /0
------------------------- MBUF Information ----------------------------
Maximum number of Mbufs.......................... 4608
Number of Mbufs Free............................. 4592
Number of Mbufs In Use........................... 16

Re: Cisco 5508 Memory High Usage

jamancill — Mon, 08 Jul 2019 19:33:28 GMT

finally, I restarted the WLC and solved the problem of high memory usage

thanks a lot

Re: Cisco 5508 Memory High Usage

tanner.zaitt — Mon, 18 Jul 2022 06:01:23 GMT

Hi,

I received an alarm "Free memory too low" from Zabbix for WLC 5508.

The product version is:

Product Version.................................. 8.0.152.0

(Cisco Controller) >show memory summary
-------------------------- System Memory Summary -------------------------
System Name:WC1 Primary SW Ver:8.0.152.0
Current Time:Mon Jul 18 08:58:00 2022 System UP Time:483 days 0 hrs 6 mins 23 secs
NAME: "Chassis" , DESCR: "Cisco 5500 Series Wireless LAN Controller"
PID: AIR-CT5508-K9, VID: V02, SN: F**5****B2
Total System Memory.............................. (1003000 KB) 979 MB
Total System Free Memory......................... (12104 KB) 11 MB (1 %)
Total Memory in Buffers.......................... (380 KB)
Total Memory in Cache............................ (119312 KB) 116 MB
Total Active Memory.............................. (879444 KB) 858 MB
Total InActive Memory............................ (38612 KB) 37 MB
Total Memory in Anon Pages....................... (798364 KB) 779 MB
Total Memory in Slab............................. (44600 KB) 43 MB
Total Memory in Page Tables...................... (2656 KB) 2 MB
WLC Peak Memory.................................. (1249364 KB) 1220 MB
WLC Virtual Memory Size.......................... (1247256 KB) 1218 MB
WLC Resident Memory.............................. (831004 KB) 811 MB
WLC Data Segment Memory.......................... (1179348 KB) 1151 MB
Total Heap Including Mapped Pages................ (263492 KB) 257 MB
Total Memory in Pmalloc Pools.................... (634935 KB) 620 MB
Total Used Memory in Pmalloc Pools............... (613843 KB) 599 MB
Total Free Memory in Pmalloc Pools............... (21089 KB) 20 MB

--More-- or (q)uit
------------------------- Pmalloc Pools Information --------------------
Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB
0 16 50000 23645 3320 /2908 /411
1 64 50300 50012 5698 /5680 /18
2 128 22300 22097 3919 /3894 /25
3 256 1357200 1357126 408220 /408201 /18
4 384 6000 333 2554 /429 /2125
5 512 24900 20017 13714 /11272 /2441
6 1024 3500 591 3677 /768 /2909
7 2048 7100 2223 14560 /4806 /9754
8 4096 1000 153 4050 /662 /3388
9 Raw-Pool 0 401 175203 /175203 /0

(Cisco Controller) >show system meminfo
MemTotal: 1003000 kB
MemFree: 11692 kB
Buffers: 380 kB
Cached: 119312 kB
SwapCached: 0 kB
Active: 878648 kB
Inactive: 38612 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 797568 kB
Mapped: 38596 kB
Slab: 44668 kB
SReclaimable: 3436 kB
SUnreclaim: 41232 kB
PageTables: 2656 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 501500 kB
Committed_AS: 1218312 kB

(Cisco Controller) >show memory allocations

SlNo Memory Magic0 Magic1 PoolSz Length PID Module Loc Time Metadata
-------------------------------------------------------------------------------------------------------------------------
1 0x2b88c974 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731169 0x00002305 0x0c93bd89
2 0x2b88f36c 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731169 0x00000c0e 0x0c93bd88
3 0x2b88fba8 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731168 0x000023f2 0x0c93bd75
4 0x2b89541c 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731161 0x000015dd 0x0c93bce8
5 0x2b8924d4 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731155 0x00001219 0x0c93bc68
6 0x2b88f9cc 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731150 0x00000df2 0x0c93bbf9
7 0x2b9633b8 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731145 0x00001bdd 0x0c93bba3
8 0x2b8901c4 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731141 0x00001c6b 0x0c93bb66
9 0x2b96719c 0xbe90be90 0xbe91be91 16 11 1349 k_mib_bsnwras 2964 41731119 0x2b9671d4 0x00000003
10 0x2b8932a4 0xbe90be90 0xbe91be91 16 8 1349 k_mib_bsnwras 2947 41731119 0x2b8932dc 0x00000000
11 0x2b8c5af0 0xbe90be90 0xbe91be91 16 14 1349 k_mib_bsnwras 2889 41731119 0x2b8c5b28 0x00000006
12 0x2b88db84 0xbe90be90 0xbe91be91 16 14 1349 k_mib_bsnwras 2883 41731119 0x2b88dbbc 0x00000006
13 0x2b898e8c 0xbe90be90 0xbe91be91 16 14 1349 k_mib_bsnwras 3300 41731119 0x2b898ec4 0x00000006
14 0x2b8c9f34 0xbe90be90 0xbe91be91 16 16 1532 ew_buffer 387 41731116 0x2b8caa04 0x2b890e2c
15 0x2b890dfc 0xbe90be90 0xbe91be91 16 8 1532 ew_buffer 458 41731116 0x2b902f48 0x00000100
16 0x2b8916c0 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731108 0x000021a9 0x0c93b923
17 0x2b8957d4 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731106 0x000023e0 0x0c93b8fe
18 0x2b890318 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731106 0x0000166e 0x0c93b8fd
19 0x2b89772c 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731097 0x00001a88 0x0c93b86f

--More-- or (q)uit
20 0x2b89354c 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731070 0x00001496 0x0c93b681
21 0x2b977cd4 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731065 0x00001a0c 0x0c93b646
22 0x2b8da560 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731058 0x000023aa 0x0c93b5b4
23 0x2b8c63b4 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731056 0x00001e58 0x0c93b582
24 0x2b88f3b0 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731053 0x000023f1 0x0c93b56a
25 0x2b8b510c 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 838 41731051 0x2b8b5144 0x00000004
26 0x2b8943a4 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 829 41731051 0x2b8943dc 0x00000004
27 0x2b890428 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 820 41731051 0x2b890460 0x00000004
28 0x2b893508 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 811 41731051 0x2b893540 0x00000004
29 0x2b88f25c 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 802 41731051 0x2b88f294 0x00000004
30 0x2b8940b8 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 793 41731051 0x2b8940f0 0x00000004
31 0x2b890d30 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 733 41731051 0x2b890d68 0x00000004
32 0x2b88f504 0xbe90be90 0xbe91be91 16 8 1349 k_mib_bsnwras 682 41731051 0x2b88f53c 0x00000000
33 0x2b88cfd4 0xbe90be90 0xbe91be91 16 8 1349 k_mib_bsnwras 671 41731051 0x2b88d00c 0x00000000
34 0x2b8b4eec 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 616 41731051 0x2b8b4f24 0x00000004
35 0x2b88de2c 0xbe90be90 0xbe91be91 16 12 1349 k_mib_bsnwras 557 41731051 0x2b88de64 0x00000004
36 0x2b892490 0xbe90be90 0xbe91be91 16 14 1349 k_mib_bsnwras 8661 41731051 0x2b8924c8 0x00000006
37 0x2b8e03ac 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731051 0x00001870 0x0c93b534
38 0x2b898188 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731050 0x00000d37 0x0c93b521
39 0x2b88e888 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731040 0x00000d35 0x0c93b470
40 0x2b890648 0xbe90be90 0xbe91be91 16 8 1514 sisf_shim_utils 641 41731039 0x000013c4 0x0c93b451

Fetched total 40 entries. Skipped 0 entries.

(Cisco Controller) >show memory mbuf summary

Maximum number of Mbufs.......................... 4608
Number of Mbufs Free............................. 4593
Number of Mbufs In Use........................... 15

(Cisco Controller) >show memory monitor

Memory Leak Monitor Status:
low_threshold(10000), high_threshold(30000), current status(disabled)

-------------------------------------------

Memory Error Monitor Status:
Crash-on-error flag currently set to (enabled)
No memory error detected.

(Cisco Controller) >show memory statistics

System Memory Statistics:
Total System Memory............: 1027072000 bytes (979.56 MB)
Used System Memory.............: 888586240 bytes (847.48 MB)
Free System Memory.............: 138485760 bytes (132.07 MB)
Bytes allocated from RTOS......: 75318076 bytes (71.83 MB)
Chunks Free....................: 30 bytes
Number of mmapped regions......: 52
Total space in mmapped regions.: 194498560 bytes (185.50 MB)
Total allocated space..........: 74181188 bytes (70.74 MB)
Total non-inuse space..........: 1136888 bytes (1.08 MB)
Top-most releasable space......: 866824 bytes (846.50 KB)
Total allocated (incl mmap)....: 269816636 bytes (257.33 MB)
Total used (incl mmap).........: 268679748 bytes (256.25 MB)
Total free (incl mmap).........: 1136888 bytes (1.08 MB)

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.152.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... WC1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 192.168.1.99
IPv6 Address..................................... ::
Last Reset....................................... Software reset
System Up Time................................... 483 days 0 hrs 7 mins 58 secs
System Timezone Location......................... (GMT +2:00) Jerusalem
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... BG - Bulgaria

--More-- or (q)uit
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +37 C
External Temperature............................. +23 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 7
Number of Active Clients......................... 100

Burned-in MAC Address............................ 30:E4:DB:1B:BB:60
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1/SHA2

Do you have an idea how to handle with high memory usage?

Should I plan reboot of the WLC?

Thank you in advance.
Best Regards.

Re: Cisco 5508 Memory High Usage

Leo Laohoo — Mon, 18 Jul 2022 06:22:54 GMT

@tanner.zaitt wrote:

Total System Free Memory......................... (12104 KB) 11 MB (1 %)

Reboot that WLC. Now.