ASA with FTD suddenly stops passing traffic

[marh]

Hello!

We have Cisco ASA5508X with FTD on site. Twice in one month happened, tha ASA suddenly stops passing traffic. When the reload command is issued, it starts working for a few seconds then reloads itself. After reload works normally. I didn't find anything strange in syslog.

ASA version: 9.8(4)22

FTD module: 6.3.0-83

Did anyone experienced the same problem? What could be wrong?

We have no problems before.

[balaji.bandi]

there are many reasons, look at the logs most first thing I do.

i check the throughput of the interface, and NAT table - process consumption starts with.

is the same time for the 2 times? what is the latest any major network changes?

[marh]

There was no changes in network for quite a time. There was the same problem both times, not at the same time.
The problem is that this ASA is in production and I don't have time to explore. I must react quickly. I didn't find anything in syslog but it is very big so I could miss something.
One thing I noticed is that it starts working, when reload command is entered. But this last only few seconds, than ASA reload itselves. Could it be, tha ASA closes FTD module just before reload, so FTD could be the reason?
Regards,
AM

[balaji.bandi]

Still this is not good information for us to guide you better

you need to explain more, Model. a version of code running, what is your NMS interface monitoring ?

[Marius Gunnerud · Accepted solution]

I am assuming this is an ASA with the SFR module and not an ASA running FTD software?

We had a similar issue with our ASA5585 with SFR module.  After much troubleshooting it looked like it was the logging that was causing the issues though I never got the chance to verify this as we swapped these out for FTD4110s and the issue dissapeared.  The theory TAC and myself came up with was that the Firepower logging is very "chatty" and we had logging on almost all our rules, which are around 600 ACP rules.  The ASA module resources become overwhelmed by the amount of logs it needs to process.  I would suggest, if possible, turn off all "unecessary" logging and see if the situation is better.

[marh]

Yes Marius, you are right. ASA with SFR module of course. My bed.

I think that could be the reason. The Syslog is huge. I will lower the logging and observe the behaviour of ASA.

Thanks for the tip.

[marh]

Hello all!

It happened again. I set then SFR to monitor-only and it helped. I checked this twice so I'm know that it was the SFR that caused that. Now I'm searching trough Connection Events in Log and I can see, that suddenly there was only DNS requests (UDP 53) to varius public DNS server sent mostly from private DNS servers in LAN. No other traffic. For a few minutes.

Short after setting SFR to Monitor-only the traffic in Log normalizes again. Anyone sees that before?

This is very strange. I will try to upgrade SFR module and see if it helps.

[Marius Gunnerud]

Is this an ASA5585?  We had lots of issues with our ASA5585 SFR module, among them the same issue you mention here.  Other issues were that the module was for some reason causing a failover condition on the ASA itself.  As mentioned earlier, our suspicion was the amount of logging happening but never got to test it.  Our solution was instead to swap out the ASA5585 with FTD4110 which solved the issue.

I suggest opening a TAC case on this since there might need to be deeper troubleshooting of the SFR module.

[marh]

We have FMC with two SFR sensors on ASA5508X. I did an upgrade to FMC and both sensors to recommended version 6.4.0.9-62. I thought this will solve the problem but it didn't. 

After a week it happened again, this time the traffic stops only for VPN locations. Removing sending the traffic to FMC for about 10 minutes helps. And it happened only on one ASA. 

[Marius Gunnerud · Accepted solution]

I still believe this is due to the amount of ACP entries that have logging enabled on them. I suggest turning logging off on the ACP entries to see If the situation gets any better (not setting the lot limit to a lower value). Other than that you need to open a TAC case AS this might require a deepdive into the Firepower code.