Cisco ASA 5512-X ""Lua runtime: not enough memory"

[aaron-rousch]

Good day everyone.

Yesterday, I was working on a Cisco ASA 5512-X Firewall that was being used as a VPN Firewall when suddenly I was disconnected from the ASDM. My SSH session from Putty was closed, and I am unable to get back into any of them. 

I took a console cable directly connected to the Firewall. I was greeted with the usual log message but with the message "Lua runtime: not enough memory" which kept interrupting my inputs while I was troubleshooting. 

when I used the "show memory" command this is what caught my attention 

Free memory: 117768378 bytes (7%)
Used memory: 1628998880 bytes (93%)
------------- ------------------
Total memory: 1746767258 bytes (100%)

i assume the high used memory was why my remote sessions were cut off so abruptly and the reason why i can't get back in using ASDM or SSH.

Strangely enough, i was still able to connect and use the VPN without issues, others were still able to connect and use the VPN as well.

I have a feeling i am dealing with the Bug labeled CSCto76775 and i am currently in the process of backing up the running-config and submitting a downtime request form so i can Power-Cycle the device

I have uploaded two text files for reference, one is the "show version" of the device in question and another one is an abbreviated output of the "show memory detail"

My question is, is there another way i can free up the memory so i can re-gain access to the ASDM and re-establish SSH connections or is Power Cycling the only way?

Thank you for your time

Any help would be appreciated.

-Aaron 

PS: if this is in the wrong area, please let me know so i can re-ask the question in the correct location.

[JesseSmith8517]

I just saw it in code version 9.20.3.7 - Cisco ASAv10

[DG-120]

Also seen this issue on 9.20.3.7 this week the same as the previous poster (and also running ASAv10's) - We upgraded from 9.18.4.24 a couple of weeks ago in the hope it would resolve this issue, but I guess it hasn't.

Our primary device (lets call it ASA1) after the firmware update was at 25% free memory. This has gradually dropped pretty continuously since restart and finally dropped below 10% free today. It may reclaim 1% occasionally, but it's basically a continuous downward trend from the moment the device is made active, regardless of current workload.

Below 10% free memory on the active device seems to be the point at which the 'Lua runtime' error begins to occur and new AnyConnect/Putty sessions begin failing.  

The secondary device (ASA2) has been sat happily at around 23-24% free memory for the past couple of weeks. I've made ASA2 the active device today, it's currently still sat at 23% free. I restarted ASA1 and that is now back at 25% free and in standby.

I imagine over the next couple of weeks I will see ASA2 now gradually lose free memory, never regaining any regardless of it's workload, and eventually when it creeps towards 10% I will be forced to failover to ASA1 and start the process over again.

It's a simple 'fix' but it's also extremely frustrating.

[aaron-rousch]

So, is it a continuous problem regardless of updates? That's disheartening. I've started looking at ways to reduce the firewall's memory usage, remove unused network objects, and simplify ACLs and NAT rules, hoping that would help. 

[DG-120]

As far as I can tell it seems continuous regardless of load on the device or config. Our config hasn't changed much, just the firmware versions. The last 2 we have used both seem to have this issue. A few months ago the throughput on the devices was mostly the same, the firmware version is the only real change.

Just something I want to rule out, for other people seeing this same issue... do you happen to have any IKEv1/v2 VPN's to other devices, and if so what are they connecting to? Our ASA has a number of IKEv2 connections to Draytek Vigor devices at our other locations. The only other thing I can think of that would have changed is the firmware version running on those devices. It should have zero impact, but it's just a thought I would like to rule out.

Anyone else with this issue have any connections to Draytek Vigor's that have had firmware updates in the past couple of months?

[aaron-rousch]

The 5512-X is used for VPN access, it uses Cisco AnyConnect to connect remote workers to the internal network.  

[JesseSmith8517]

Good morning,

I want to add on some new information being tried out at my company. My coworker did some digging, using a tool on the 'show tech' output. He noticed evidence of a 'password spray attack' - I'm hoping that I entering the correct term used. After consulting with TAC, we're doing the following:
upgrading our ASAvs to the recommended code per train
implementing three additional threat-detection commands.

Here are the command being added:

threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
threat-detection service invalid-vpn-access
threat-detection service remote-access-authentication hold-down 10 threshold 10
threat-detection service remote-access-client-initiations hold-down 10 threshold 20

So far, we have about 81 ASAvs on the 9.20.3.x code that have the above command implemented.  I'm monitoring them daily to see if any of them experience any of the various memory issues.

I'll try to to keep posting if I learn anything new.