ASA 5525 0 free memory

[the-lebowski]

Have an ASA 5525 running 9.4(4)16 recently upgraded (two weeks ago).  Today it started consuming all its memory even upon reboot.  At first it started denying ssh and then users complained of VPN tunnels going up and down.  When I attempt to SSH I get this:

libc_fnctions.mmap(addr=0x0000000000000000, len=45056, prot=0, flag=2, fd=3, off=0) returned -1 errno=12!
process_create: out of stack memory
_listen_ssh: failed to create thread for interface 3 port 22

Show mem shows this:

fw1-5525# show memory detail
Free memory heap:                     798144 bytes ( 0%)
Free memory system:                  1832714 bytes ( 0%)
Used memory:
     Allocated memory in use:     3410471824 bytes (79%)
     Reserved memory (DMA):        381233398 bytes ( 9%)
     Memory overhead:              500631216 bytes (12%)
-----------------------------   ------------------
Total memory:                     4294967296 bytes (100%)

Least free memory:        503048570 bytes (12%)
Most used memory:        3791918726 bytes (88%)
MEMPOOL_DMA POOL STATS:

Non-mmapped bytes allocated =    230662144
Number of free chunks       =          154
Number of mmapped regions   =            0
Mmapped bytes allocated     =            0
Max memory footprint        =    230662144
Keepcost                    =     49364000
Max contiguous free mem     =     49364000
Allocated memory in use     =    181203648
Free memory                 =     49458496

I cant find for the life of me what is causing it. Can someone help me pinpoint what the problem is? 

[the-lebowski]

fw1-5525# sh flash
--#--  --length--  -----date/time------  path
%Error opening disk0:/ (Cannot allocate memory)

8238202880 bytes total (7905357824 bytes free)

fw1-5525# copy start tftp

Address or name of remote host []? 10.30.3.3

Destination filename [startup-config]? fw1-5525
No memory available

[Ajay Saini]

Hello,

This looks like a memory leak issue. I don't have access to Cisco tools, so can't help with debugging the crash. But as a quick fix, you can try to upgrade to versions 9.6 and beyond. 

Some process is currently consuming the memory and not releasing.

-

HTH
AJ

[the-lebowski]

Turned out it was just load, an offending host sending some 7k connections/sec to the ASA causing it to exhaust all of its memory trying to process it.