Virtual FTD Memory Keeps Filling Up

[zstamm]

I am working on a virtual FTD device that the memory keeps filling up. We have to reboot it every few weeks or else the device will become unstable because of high memory usage. Something that pulls from the Global Shared Pool is doing this. The ballooning pool will expand, but it will eventually take up the system memory as well. I ran the top command, and nothing really came up as taking all of the memory at that point.

What should I do to check what is taking up all of the memory? I could not find a known bug for this version that matches this issue. Is there a bug that I missed?

Cisco Firepower Threat Defense for VMware (75) Version 7.3.0 (Build 69)
Cisco Adaptive Security Appliance Software Version 9.19(1)
SSP Operating System Version 2.13(0.198)

------------------ show memory detail ------------------

Heap Memory:
Free Memory:
Global Shared Pool: 26977120 bytes ( 0% )
Message Layer Pool: 6494848 bytes ( 0% )
System: 782445580 bytes ( 6% )
Used Memory:
Global Shared Pool: 9256709280 bytes ( 76% )
Reserved (Size of DMA Pool): 335544320 bytes ( 3% )
Reserved for messaging: 1893760 bytes ( 0% )
MMAP usage: 393813280 bytes ( 3% )
System Overhead: 1450062560 bytes ( 12% )
------------------------------------- ----------------
Total Memory: 12253940748 bytes ( 100% )

[Marius Gunnerud]

First off, I would always recommend using the starred version as this tends to have the most stable release.  That being said, it sounds like you are hitting a memory leak of some sort.  I would suggest upgrading to 7.3.1 as a first step in trying to get this resolved.  The first releases in a minor train (7.3.0 in this case) is often the most unstable.

[MHM Cisco World]

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvn09583

[Marius Gunnerud]

Though I do recommend upgrading, Here are some commands you can use to identify what might be taking up memory:

from the FTD command prompt (>)

show access-list count or show access-list elements count (dont remember the exact syntax)
show memory detail
show memory system
show process memory

You can also go into expert mode and then go to cd /ngfw/var/log and then sudo less messages.  There is a lot of logs to go through here but if you have a date and time you had the issue last, you could filter on that and see if there is anything there that stands out.

[tvotna]

To clarify what @Marius Gunnerud said, memory management is performed by both underlying Linux system and Lina (ASA) as well. Below commands pertain to Linux memory management:

topsystem support utilization
show memory system
cat /ngfw/var/log/messages #search for oom-killer

And below commands pertain to ASA/Lina memory management:

show memory detail
show memory top

If you see in the "show memory detail" that GSP pool grows and eventually free System memory goes low, you probably have a memory leak inside ASA/Lina. Collect "show memory top" multiple times over time, to see which binsize grows (memory is allocated in fixed-size blocks). Then open a TAC case, as only TAC can decode memory addresses for you.

HTH