Re: High Memory in FPR ASA 2120

johnlloyd_13 · ‎06-28-2023

hi,

i got a dual FPR 2120 in ASA appliance mode, active-standby HA and in multiple context.

there was a high memory incident wherein the primary-active unit hanged/no console prompt, need to login to secondary-standby, force a failover and reboot the primary unit.

the recent config change was only adding two /24 subnets with dynamic NAT/PAT on the outside interface. as interim, had to change from dynamic interface to dynamic <PUBLIC IP> for each of the subnet, i.e. 1 subnet > PAT to a separate public IP.

can someone advise if the high memory is due to high NAT? i thought FPR 2100 platforms are more "beefy" than previous 5500-x?

was it due to a lot of "inside" subnets PAT'd on a single public IP/"outside" interface?

will the change of the 2 subnets to PAT to a different IP will "help" to some degree offload the memory?

ciscoasa/pri/act/CONTEXT-A# sh xlate

77588 in use, 77589 most used

Flags: D - DNS, e - extended, I - identity, i - dynamic, r - portmap,

s - static, T - twice, N - net-to-net

ciscoasa/pri/act/CONTEXT-A# sh conn count

83665 in use, 89120 most used

<SNIP>

tvotna · ‎07-03-2023

How do you know that the issue was caused by high memory? Did you see console or syslog messages about memory allocation failures or collected "show memory detail" output?

77K xlates is not many for this platform, so it's unlikely that the issue was caused by PAT. It doesn't matter how many internal subnets you have or whether you PAT them to different IPs or same IP. Only the number of xlates and conns matters. Also "extended" PAT option can lead to high memory sometimes: CSCud84453 High Memory Utilization when PAT has "extended" option, but it's unlikely you use it.

If PAT pool is too small, you can run out of xlate slots, but this is another story. The "xlate per-session permit" per-session PAT feature can significantly improve scalability and reduce memory footprint in this case.

johnlloyd_13 · ‎07-03-2023

hi,

we saw the high memory graph shoot up to a max of around 10 GB on this platform. console wasn't responsive on the active FW that's why we forced standby to become active and reboot primary/old active.

i checked the bug link and it doesn't affect our current 9.16 code. the bug doesn't affect ASA version 9.6(3)1 and above.

i do see the 2x /24 subnets being offload from PAT to outside interface to its separate public IP (.13 and .14) got maxed out. the average memory went down to 4-5 GB for several days now.

TCP PAT pool outside, address 64.x.x.13, range 1024-65535, allocated 64512
UDP PAT pool outside, address 64.x.x.28, range 1-1023, allocated 0
UDP PAT pool outside, address 64.x.x.28, range 1024-65535, allocated 2035
TCP PAT pool outside, address 64.x.x.14, range 1-1023, allocated 0
TCP PAT pool outside, address 64.x.x.14, range 1024-65535, allocated 64512
ciscoasa/pri/act/INET#

tvotna · ‎07-04-2023

Right, you're running out of translation slots and need more public IP addresses in your PAT pool. You can either use single PAT statement like "nat (inside, outside) source dynamic ... pat-pool mypool" or few statements and PAT each/few subnets to a different public IP address. To remove xlates more quickly use per-session PAT feature "xlate per-session permit".

It's hard to say what caused high memory, but I agree that ~10GB is a lot. 2120 platform has 8GB of DRAM for dataplane if I remember correctly. Syslog, "show memory detail", "show memory top-usage", "show memory app-cache" can be used to troubleshoot, but you'll need to open TAC case to decode memory addresses. Also, "show chunkstat", but note that the output is huge.

johnlloyd_13 · ‎07-04-2023

hi,

thanks! i still see a LOT of xlate on these 2x subnet/public IP.

i suspect some P2P abuse going on downstream customer/device. i will just trace it and shutdown its switch port.

can you enlighten or how do you use the 'xlate per-session permit'?

ciscoasa/pri/act/INET# sh run | i xlate
timeout xlate 3:00:00
timeout pat-xlate 0:00:30

ciscoasa/pri/act/INET(config)# xlate per-session permit ?

configure mode commands/options:
tcp TCP protocol
udp UDP protocol
ciscoasa/pri/act/INET(config)# xlate per-session permit tcp ?

configure mode commands/options:
Hostname or A.B.C.D Match based on source network IPv4
address
Hostname/<0-128> or X:X:X:X::X/<0-128> Source IPv6 address/prefix
any4 Abbreviation for source address and
mask of 0.0.0.0 0.0.0.0
any6 Abbreviation for source prefix ::/0
host Use this keyword to configure source
host
ciscoasa/pri/act/INET(config)# xlate per-session permit tcp ?

configure mode commands/options:
Hostname or A.B.C.D Match based on source network IPv4
address
Hostname/<0-128> or X:X:X:X::X/<0-128> Source IPv6 address/prefix
any4 Abbreviation for source address and
mask of 0.0.0.0 0.0.0.0
any6 Abbreviation for source prefix ::/0
host Use this keyword to configure source
host

tvotna · ‎07-04-2023

Per-session PAT:

https://www.cisco.com/c/en/us/td/docs/security/asa/asa916/configuration/firewall/asa-916-firewall-config/nat-basics.html#ID-2090-000007b6

If ASA was upgraded from an older release, it is disabled by default. For new installations it is enabled by default (what you need). You can verify it with "show run xlate". Typically the following two commands is all you need to use this feature for IPv4 (to remove xlate as soon as connection is torn down), but you can try to add your own rules for UDP if some UDP-based protocol create lots of xlates.

xlate per-session permit tcp any4 any4

xlate per-session permit udp any4 any4 eq domain

Also, in this case "set connection timeout" should be used in policy-map to lower UDP idle timeout for offending protocol. Alternatively offending protocol or host can be blocked.

johnlloyd_13 · ‎07-04-2023

hi,

it's a new install and said commands are there by default.

most of the high NAT traffic are TCP. is there a similar "set connection timeout" for TCP? what is the command syntax/example?

ciscoasa/pri/act/INET# sh run all | i xlate
xlate per-session permit tcp any4 any4 <<<<
xlate per-session permit tcp any4 any6
xlate per-session permit tcp any6 any4
xlate per-session permit tcp any6 any6
xlate per-session permit udp any4 any4 eq domain <<<<
xlate per-session permit udp any4 any6 eq domain
xlate per-session permit udp any6 any4 eq domain
xlate per-session permit udp any6 any6 eq domain
xlate block-allocation size 512
xlate block-allocation maximum-per-host 4
no xlate block-allocation pba-interim-logging
timeout xlate 3:00:00

tvotna · ‎07-05-2023

Yes, you can set TCP idle timeout either globally ("timeout conn hh:mm:ss") or per class in policy-map ("set connection timeout idle"). This may not help much though, unless you have long-live TCP connection which don't pass any traffic and you want to get rid of them. There are few other timeouts, e.g. embryonic and half-closed. They can be used to teardown SYN-only connections and connections after one-way FIN. Reducing them may not help too, depending on your traffic profile. TCP Intercept ("set connection embryonic-conn-max" / "set connection per-client-embryonic-max") and/or per-client connection limiting ("set connection per-client-max") are typically more helpful to limit the number of SYN-only connections and established connections (e.g. per-client).

Anyway, it's possible that the only solution is to increase the PAT pool size.