topic Re: FTD-1010 - Critical High System Memory usage in Network Security

FTD-1010 - Critical High System Memory usage

Micccc4 — Wed, 18 Jan 2023 14:42:54 GMT

Hi Everyone,

For not so long time ago we have deployed FTD 1010 (software v. 7.0.1) and pretty short after deployment it started to generate 'High Memory Usage' critical alerts. Configuration of that FP is minimalist and CPU is daround 4% -5% all the time. Nevertheless System Memory Usage reaches 90% constantly.

Did any of you experience similar issue? Any advice on what to check, how to normalize it?

Thanks in advance!

Below you will find some logs. Note I am missing direct CLI access and provided output is from commands being pushed via FMC:

#show memory all

Data Path
Free memory: 808371463 bytes (27%)
Used memory: 2180270784 bytes (73%)
--------------- ---------------
Total memory: 2988642247 bytes (100%)

Inspection Engine
Free memory: 1863548928 bytes (69%)
Used memory: 826404864 bytes (31%)
--------------- ---------------
Total memory: 2689953792 bytes (100%)

System
Free memory: 747515904 bytes ( 9%)
Used memory: 7556042752 bytes (91%)
--------------- ---------------
Total memory: 8303558656 bytes (100%)

#show cpu

CPU utilization for 5 seconds = 2%; 1 minute: 1%; 5 minutes: 1%

#show processes

Hardware: FPR-1010
Cisco Adaptive Security Appliance Software Version 9.16(2)5
ASLR enabled, text region 55e5f7983000-55e5fbf75665

PC SP STATE Runtime SBASE Stack Process TID
Mwe 0x000055e5fa456b45 0x00001506d0114e98 0x000055e61b987f40 0 0x00001506d010d030 32088/32768 zone_background_idb 146
Mwe 0x000055e5fb2dbf25 0x00001506cf3d1ae8 0x000055e61b987f40 0 0x00001506cf3ca030 30576/32768 webvpn_task

...

Re: FTD-1010 - Critical High System Memory usage

MHM Cisco World — Mon, 16 Jan 2023 21:09:33 GMT

can I see
show asp drop
please can share the FPR version also

Re: FTD-1010 - Critical High System Memory usage

balaji.bandi — Mon, 16 Jan 2023 21:13:33 GMT

this is ok normal as i see - but is this effecting anything ?

Re: FTD-1010 - Critical High System Memory usage

Rob Ingram — Mon, 16 Jan 2023 21:21:25 GMT

@Micccc4 I had high memory usage on my FPR1010 and upgrading the FTD version resolved the issue.

Re: FTD-1010 - Critical High System Memory usage

MHM Cisco World — Mon, 16 Jan 2023 21:22:54 GMT

I think this is issue system allocated of memory 90%, that high.

Re: FTD-1010 - Critical High System Memory usage

Micccc4 — Mon, 16 Jan 2023 22:10:46 GMT

Thanks for quick responses everyone.

@MHM Cisco World :

FTD 1010, SW 7.0.1

#show asp drop (wasn't cleared since September or October)

Frame drop:
IPSEC tunnel is down (ipsec-tun-down) 131
Invalid encapsulation (invalid-encap) 7
Invalid TCP Length (invalid-tcp-hdr-length) 9
No valid adjacency (no-adjacency) 752
No route to host (no-route) 25446
Flow is denied by configured rule (acl-drop) 10445777
Invalid SPI (np-sp-invalid-spi) 28
First TCP packet not SYN (tcp-not-syn) 2833631
Bad TCP flags (bad-tcp-flags) 1
TCP data send after FIN (tcp-data-past-fin) 1
TCP failed 3 way handshake (tcp-3whs-failed) 652592
TCP RST/FIN out of order (tcp-rstfin-ooo) 1448393
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 3359
TCP SYNACK on established conn (tcp-synack-ooo) 1917
TCP packet SEQ past window (tcp-seq-past-win) 145312
TCP invalid ACK (tcp-invalid-ack) 134
TCP RST/SYN in window (tcp-rst-syn-in-win) 1926
TCP packet failed PAWS test (tcp-paws-fail) 5426
Slowpath security checks failed (sp-security-failed) 48889985
ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 32
ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 581
DNS Inspect invalid domain label (inspect-dns-invalid-domain-label) 19
DNS Inspect id not matched (inspect-dns-id-not-matched) 1506
FP L2 rule drop (l2_acl) 146971
Interface is down (interface-down) 4
IKE new SA limit exceeded (ike-sa-rate-limit) 3996
Blocked or blacklisted by the firewall preprocessor (firewall) 14
Blocked or blacklisted by the session preprocessor (session-preproc) 69
Fragment reassembly failed (fragment-reassembly-failed) 334
Packet is blocked as requested by snort (snort-block) 3

Last clearing: Never

Flow drop:
Need to start IKE negotiation (need-ike) 8222
VPN decryption missing (vpn-missing-decrypt) 194
Inspection failure (inspect-fail) 3939142

Last clearing: Never

Re: FTD-1010 - Critical High System Memory usage

Micccc4 — Tue, 17 Jan 2023 08:41:07 GMT

@Rob Ingram Do you happen to remember which software version you had when you experienced memory usage errors? We are currently running it on 7.0.1. thansk

Re: FTD-1010 - Critical High System Memory usage

Marius Gunnerud — Tue, 17 Jan 2023 09:24:58 GMT

Unfortunately, the FTD and ASA have been plagued with memory leak issues for a while now. The only version I have not seen this on yet is 7.2.x for FTD and 9.18.x for ASA (no sure about 9.19.x yet as I have not upgraded any to this).

Another possible reason could be configured rules. Could you post the output of show access-list element-count

Re: FTD-1010 - Critical High System Memory usage

Micccc4 — Tue, 17 Jan 2023 09:35:56 GMT

Thanks @Marius Gunnerud - there is just one rule set in the ACL with curren element count:

#show access-list element-count
Total number of access-list elements: 11

Re: FTD-1010 - Critical High System Memory usage

Rob Ingram — Tue, 17 Jan 2023 09:43:14 GMT

@Micccc4 it was either 6.7 or 7.0, I've had no memory related issues with 7.2 or 7.3

Re: FTD-1010 - Critical High System Memory usage

Micccc4 — Wed, 18 Jan 2023 14:45:29 GMT

For your information.. Memory usage has increased another 2% last day so we went for reboot.

If its still increasing we will plan for upgrade.

Will update ticket in about 1 week...

Re: FTD-1010 - Critical High System Memory usage

MHM Cisco World — Wed, 18 Jan 2023 19:10:42 GMT

I am new in FRP but I can help you to check some think that I see not right,
Flow is denied by configured rule (acl-drop) 10445777
Slowpath security checks failed (sp-security-failed) 48889985

the issue is there is many new conn and this not right..!!!
check this link for more info. about slow path.
https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/217802-troubleshoot-firepower-threat-defense-ro.html

Re: FTD-1010 - Critical High System Memory usage

FGR — Tue, 27 May 2025 01:23:46 GMT

I have the same problem but it is caused by VPN Remote Access configuration.

With VPN Remote Access: 94% avg data plane memory

Without VPN Remote Access: 76% avg data plane memory

The default Health Policy warns at 90%.

Re: FTD-1010 - Critical High System Memory usage

FGR — Tue, 27 May 2025 01:58:08 GMT

Aha! I removed 2 of 3 "Secure Client Images", and the data plane memory usage dropped with 1GB.