FTD-1010 - Critical High System Memory usage

[Micccc4]

Hi Everyone,

For not so long time ago we have deployed FTD 1010 (software v. 7.0.1) and pretty short after deployment it started to generate 'High Memory Usage' critical alerts. Configuration of that FP is minimalist and CPU is daround 4% -5% all the time. Nevertheless System Memory Usage reaches 90% constantly. 

Did any of you experience similar issue? Any advice on what to check, how to normalize it? 

Thanks in advance!

Below you will find some logs. Note I am missing direct CLI access and provided output is from commands being pushed via FMC:

#show memory all

Data Path
Free memory: 808371463 bytes (27%)
Used memory: 2180270784 bytes (73%)
--------------- ---------------
Total memory: 2988642247 bytes (100%)

Inspection Engine
Free memory: 1863548928 bytes (69%)
Used memory: 826404864 bytes (31%)
--------------- ---------------
Total memory: 2689953792 bytes (100%)

System
Free memory: 747515904 bytes ( 9%)
Used memory: 7556042752 bytes (91%)
--------------- ---------------
Total memory: 8303558656 bytes (100%)

#show cpu

CPU utilization for 5 seconds = 2%; 1 minute: 1%; 5 minutes: 1%

#show processes

Hardware: FPR-1010
Cisco Adaptive Security Appliance Software Version 9.16(2)5
ASLR enabled, text region 55e5f7983000-55e5fbf75665

PC SP STATE Runtime SBASE Stack Process TID
Mwe 0x000055e5fa456b45 0x00001506d0114e98 0x000055e61b987f40 0 0x00001506d010d030 32088/32768 zone_background_idb 146
Mwe 0x000055e5fb2dbf25 0x00001506cf3d1ae8 0x000055e61b987f40 0 0x00001506cf3ca030 30576/32768 webvpn_task 

...

...

[MHM Cisco World]

can I see 
show asp drop 
please can share the FPR version also

[balaji.bandi]

this is ok normal as i see - but is this effecting anything ?

[Rob Ingram]

@Micccc4 I had high memory usage on my FPR1010 and upgrading the FTD version resolved the issue.

[MHM Cisco World]

I think this is issue system allocated of memory 90%, that high. 

[Micccc4]

@Rob Ingram Do you happen to remember which software version you had when you experienced memory usage errors? We are currently running it on 7.0.1. thansk

[Rob Ingram]

@Micccc4 it was either 6.7 or 7.0, I've had no memory related issues with 7.2 or 7.3

[Micccc4]

Thanks for quick responses everyone.

@MHM Cisco World :

FTD 1010, SW 7.0.1

#show asp drop (wasn't cleared since September or October)

Frame drop:
IPSEC tunnel is down (ipsec-tun-down) 131
Invalid encapsulation (invalid-encap) 7
Invalid TCP Length (invalid-tcp-hdr-length) 9
No valid adjacency (no-adjacency) 752
No route to host (no-route) 25446
Flow is denied by configured rule (acl-drop) 10445777
Invalid SPI (np-sp-invalid-spi) 28
First TCP packet not SYN (tcp-not-syn) 2833631
Bad TCP flags (bad-tcp-flags) 1
TCP data send after FIN (tcp-data-past-fin) 1
TCP failed 3 way handshake (tcp-3whs-failed) 652592
TCP RST/FIN out of order (tcp-rstfin-ooo) 1448393
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 3359
TCP SYNACK on established conn (tcp-synack-ooo) 1917
TCP packet SEQ past window (tcp-seq-past-win) 145312
TCP invalid ACK (tcp-invalid-ack) 134
TCP RST/SYN in window (tcp-rst-syn-in-win) 1926
TCP packet failed PAWS test (tcp-paws-fail) 5426
Slowpath security checks failed (sp-security-failed) 48889985
ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 32
ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 581
DNS Inspect invalid domain label (inspect-dns-invalid-domain-label) 19
DNS Inspect id not matched (inspect-dns-id-not-matched) 1506
FP L2 rule drop (l2_acl) 146971
Interface is down (interface-down) 4
IKE new SA limit exceeded (ike-sa-rate-limit) 3996
Blocked or blacklisted by the firewall preprocessor (firewall) 14
Blocked or blacklisted by the session preprocessor (session-preproc) 69
Fragment reassembly failed (fragment-reassembly-failed) 334
Packet is blocked as requested by snort (snort-block) 3

Last clearing: Never

Flow drop:
Need to start IKE negotiation (need-ike) 8222
VPN decryption missing (vpn-missing-decrypt) 194
Inspection failure (inspect-fail) 3939142

Last clearing: Never

[Marius Gunnerud]

Unfortunately, the FTD and ASA have been plagued with memory leak issues for a while now.  The only version I have not seen this on yet is 7.2.x for FTD and 9.18.x for ASA (no sure about 9.19.x yet as I have not upgraded any to this).

Another possible reason could be configured rules.  Could you post the output of show access-list element-count

[Micccc4]

Thanks @Marius Gunnerud - there is just one rule set in the ACL with curren element count:

#show access-list element-count
Total number of access-list elements: 11

[MHM Cisco World]

I am new in FRP but I can help you to check some think that I see not right, 
Flow is denied by configured rule (acl-drop) 10445777
Slowpath security checks failed (sp-security-failed) 48889985

 the issue is there is many new conn and this not right..!!!
check this link for more info. about slow path. 
https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/217802-troubleshoot-firepower-threat-defense-ro.html

[Micccc4]

For your information.. Memory usage has increased another 2% last day so we went for reboot. 

If its still increasing we will plan for upgrade.

Will update ticket in about 1 week...