ASA 5515 - Critical memory usage

Group IT · ‎03-29-2018

Hi All,

Recently we have been alerted by our monitoring systems that our ASA 5515 has reached a critical memory alert. Its exact alert is "CRITICAL: Memory 'MEMPOOL_GLOBAL_SHARED' Total: 123.08 MB Used: 117.56 MB (95.51%) Free: 5.52 MB (4.49%)"

This firewall is our HQ's primary security device so we expect it to be doing a lot but we weren't expecting it to reach a critical level so quickly.

Is there any way to see exactly what is causing the high memory usage? Or is this normal for such a device to use up all of its resources?

When I log into the device itself, the Total memory usage is reporting as 947MB. Is this reporting on a different memory? Should I be concerned?

Thanks in advance.

Cisco Adaptive Security Appliance Software Version 9.4(4)5 
Device Manager Version 7.6(2)150

Compiled on Thu 30-Mar-17 21:52 PDT by builders
System image file is "disk0:/asa944-5-smp-k8.bin"
Config file at boot was "startup-config"

fw-h002934 up 3 days 21 hours

Hardware:   ASA5515, 8192 MB RAM, CPU Clarkdale 3058 MHz, 1 CPU (4 cores)
            ASA: 4096 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB
BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB

Bogdan Nita · ‎03-29-2018

Memory issues are tricky on the asa and usually require TAC.

In order to get a reading on the free memory you should look at free memory system in "show memory detail".

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuu11733/?rfs=iqvred

To try to find the process using the memory you can use the following guide:

https://itsecworks.com/2010/11/23/troubleshooting-asa-high-memory-issues/

HTH

Bogdan

Group IT · ‎03-29-2018

Hi Bogdan,

Thanks for the information. What you are saying to check does tie up with whats being reported back from the ASA and our monitoring system.

The general memory of the appliance seems to be fine but I think from what I see, the MEMPOOL_GLOBAL_SHARED POOL is high.

Free memory heap:                    4704320 bytes ( 0%)
Free memory system:               3119279964 bytes (73%)
Used memory:
     Allocated memory in use:      358910992 bytes ( 8%)
     Reserved memory (DMA):        351547556 bytes ( 8%)
     Memory overhead:              460524464 bytes (11%)
-----------------------------   ------------------
Total memory:                     4294967296 bytes (100%)

Least free memory:       3583586876 bytes (83%)
Most used memory:         711380420 bytes (17%)
MEMPOOL_DMA POOL STATS:

Non-mmapped bytes allocated =    168824832
Number of free chunks       =          145
Number of mmapped regions   =            0
Mmapped bytes allocated     =            0
Max memory footprint        =    168824832
Keepcost                    =     47695264
Max contiguous free mem     =     47695264
Allocated memory in use     =    121036144
Free memory                 =     47788688

----- fragmented memory statistics -----

 fragment size       count          total
    (bytes)                        (bytes)
----------------  ----------  --------------
           48             6            288
           64             2            128
           96             1             96**
          112             1            112
          256            29           8736
          384            34          16192
          512            69          43776
         4096             1           5200
        16384             1          18832
     47695264             1       47695264*

*  - top most releasable chunk.
** - contiguous memory on top of heap.


----- allocated memory statistics -----

 fragment size       count          total
    (bytes)                        (bytes)
----------------  ----------  --------------
          160             2            320
          224             3            672
          240             6           1440
          256             2            512
          512            12           6144
         1024           167         171008
         2048             8          16384
         4096             3          12288
         8192             1           8192
        12288            14         172032
        16384             3          49152
        32768            38        1245184
        49152             5         245760
        65536             1          65536
        98304             4         393216
       131072            10        1310720
       196608             1         196608
       262144             4        1048576
       393216             5        1966080
       524288             2        1048576
       786432             2        1572864
      1048576            10       10485760
      1572864             6        9437184
      3145728             3        9437184
      6291456             2       12582912
      8388608             1        8388608
     12582912             2       25165824

MEMPOOL_GLOBAL_SHARED POOL STATS:

Non-mmapped bytes allocated =    129056768
Number of free chunks       =          701
Number of mmapped regions   =          101
Mmapped bytes allocated     =    227082240
Max memory footprint        =            0
Keepcost                    =        88112
Max contiguous free mem     =      3674016
Allocated memory in use     =    124352224
Free memory                 =      4704544

----- fragmented memory statistics -----

 fragment size       count          total
    (bytes)                        (bytes)
----------------  ----------  --------------
           32           314          10048
           48           281          13488
           64            41           2624
           96             7            672
          112            85           9520
          128            19           2432
          192             1            192
          240             1            240
          544             8           4352
          752             1            752
          800             1            800
          848             2           1696
         1088             5           5440
         1152             1           1152
         1984             3           6000
         2240             1           2272
         4096             1           4160
         6144             2          12800
         7680             1           8112
         8192             1           8496
        10752             2          21648
        12288             1          12640
        16384             1          16960
        20480             2          46240
        36864             1          38608
        65536             2         165872
       524288             1        3674016

----- allocated memory statistics -----

 fragment size       count          total
    (bytes)                        (bytes)
----------------  ----------  --------------
           64           724          46336
           80         67933        5434640
           96         80556        7733376
          112          2236         250432
          128         69075        8841600
          144          3058         440352
          160           522          83520
          176           404          71104
          192           368          70656
          208           268          55744
          224           141          31584
          240          3303         792720
          256          4152        1062912
          384         66221       25428864
          512          3499        1791488
          768           416         319488
         1024           930         952320
         1536           211         324096
         2048           238         487424
         3072            57         175104
         4096           374        1531904
         6144            67         411648
         8192           192        1572864
        12288           168        2064384
        16384           579        9486336
        24576            26         638976
        32768           124        4063232
        49152            52        2555904
        65536           196       12845056
        98304            23        2260992
       131072            58        7602176
       196608            23        4521984
       262144            11        2883584
       393216            16        6291456
       524288             7        3670016
       786432             8        6291456
      1048576            15       15728640
      1572864             5        7864320
      2097152            10       20971520
      3145728             6       18874368
      4194304             4       16777216
      8388608             1        8388608
     12582912             3       37748736

Summary for all pools:

Non-mmapped bytes allocated =    297881600
Number of free chunks       =          846
Number of mmapped regions   =            0
Mmapped bytes allocated     =    227082240
Max memory footprint        =    168824832
Keepcost                    =     47783376
Allocated memory in use     =    245388368
Free memory                 =     52493232

Can you tell me what MEMPOOL_GLOBAL_SHARED POOL actually is and why its monitored seperatley from the main memory?

Thanks in advance.

Bogdan Nita · ‎03-29-2018

As far as I know MEMPOOL_DMA is used mainly for vpn and the MEMPOOL_GLOBAL_SHARED is used for all other services.
I guess it made sense to monitor it when it wasn't allocated dynamically.

yangtao72 · ‎04-02-2020

Anybody found a solution regarding on this? We have similar problem that SNMP monitoring system alert very high Memory usage but it seems to be calculated by MEMPOOL_GLOBAL_SHARED, instead of real memory.

How can I get rid of MEMPOOL_GLOBAL_SHARED? Our version is Cisco Adaptive Security Appliance Version 9.12(3)7

thanks!

Tao Yang

yensheng_lin · ‎09-06-2021

We also have this kind of problem in our ASA 5515.

Here is the sh memo det output:

But I didn't see any log regarding to run out of memory.

The alert has triggered and shown on our monitor system.

=========================================

ahl-edgedpke-fw1# sh memo det
Free memory heap: 34192 bytes ( 0%)
Free memory system: 2852141257 bytes (65%)
Used memory:
Allocated memory in use: 775370088 bytes (18%)
Reserved memory (DMA): 499122176 bytes (11%)
Memory overhead: 255538440 bytes ( 6%)
----------------------------- ------------------
Total memory: 4382206153 bytes (100%)

Least free memory: 3107493969 bytes (71%)
Most used memory: 1274712184 bytes (29%)
MEMPOOL_HEAPCACHE_0 POOL STATS:

Non-mmapped bytes allocated = 314572800
Number of free chunks = 641
Number of mmapped regions = 0
Mmapped bytes allocated = 0
Max memory footprint = 314572800
Keepcost = 32
Max contiguous free mem = 384
Allocated memory in use = 314538608
Free memory = 34192

=============================

Here is our show version output:

Cisco Adaptive Security Appliance Software Version 9.8(1)
Firepower Extensible Operating System Version 2.2(1.47)
Device Manager Version 7.8(1)

Compiled on Wed 10-May-17 15:37 PDT by builders
System image file is "disk0:/asa981-lfbff-k8.SPA"
Config file at boot was "startup-config"

ahl-edgedpke-fw1 up 97 days 17 hours

Hardware: ASA5516, 8192 MB RAM, CPU Atom C2000 series 2416 MHz, 1 CPU (8 cores)
Internal ATA Compact Flash, 8000MB
BIOS Flash M25P64 @ 0xfed01000, 16384KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)

Milos_Jovanovic · ‎09-07-2021

Hi @yensheng_lin,

I always take a look at software version, and your's is quite old. Older software have bunch of bugs, and you should always keep up to date.

Next, there was a recent Field notice about high memory utilization, so please check if relevant to you.

BR

Milos