topic Flash disk is almost full but there are no many files listed on ASA in Network Security

Flash disk is almost full but there are no many files listed on ASA

FernandoPerez75270 — Wed, 10 May 2023 20:12:50 GMT

Hi everyone,

I have two cisco ASA working on redundancy. One of them has an alarm of disk0 use superior to 80%. When I check de content of the flash, I don´t see enough files to justify such memory usage. How is the disk is being constantly filled (almost 20 MB per day) by itself? I am working with the version 9.16(4)18.

fw01# dir flash:

Directory of disk0:/

100 -rwx 151266800 09:51:18 Apr 18 2023 asa9-16-4-18-lfbff-k8.SPA
4 drwx 69632 15:41:54 May 10 2023 log
18 drwx 4096 03:08:00 May 11 2019 crypto_archive
19 drwx 4096 03:08:04 May 11 2019 coredumpinfo
6 -rwx 25394 09:53:01 Apr 18 2023 asa-cmd-server.log
102 -rwx 23346 13:02:58 Jul 24 2019 oldconfig_20xxjulxx-xx.cfg
104 -rwx 110401360 13:08:24 Nov 16 2022 asdm-7181-152.bin
7 -rwx 39 09:47:40 Apr 18 2023 snortpacketinfo.conf

5 file(s) total size: 261716939 bytes
7863623680 bytes total (1213321216 bytes free/15% free)

Thanks for your help.

Re: Flash disk is almost full but there are no many files listed on AS

Marvin Rhoads — Thu, 11 May 2023 10:06:11 GMT

Check the coredumpinfo folder.

Re: Flash disk is almost full but there are no many files listed on AS

FernandoPerez75270 — Thu, 11 May 2023 12:19:13 GMT

Hi Marvin,

thanks for the answer, the coredump info folder has one little file inside, but the log folder has a lot, with names like "asa_snmp.log.1-2023051023.backup" of 5 MB aprox:

fw01# dir flash:coredumpinfo/

Directory of disk0:/coredumpinfo/

20 -rwx 59 03:08:04 May 11 2019 coredump.cfg

1 file(s) total size: 59 bytes
7863623680 bytes total (2835738624 bytes free/36% free)

fw01# dir flash:log/

Directory of disk0:/log/

fw01# dir flash:log/

Directory of disk0:/log/

15109 -rwx 375 12:21:44 Jul 24 2019 asa-appagent.log
15110 -rwx 34131 12:52:26 Apr 28 2022 asa-cmd-server.log
26 -rwx 15929 09:49:12 Apr 18 2023 ma_ctx2000.log
5 -rwx 3048 09:49:12 Apr 18 2023 lina_monitor.log
11 -rwx 411897 08:16:00 May 11 2023 asa_snmp.log
15167 -rwx 6251109 08:13:37 May 11 2023 asa_snmp.log.1
8469 -rwx 6278775 20:27:28 Apr 30 2023 asa_snmp.log.1-2023050100.backup
8497 -rwx 5970152 21:27:35 Apr 30 2023 asa_snmp.log.1-2023050101.backup
8525 -rwx 6278771 22:27:41 Apr 30 2023 asa_snmp.log.1-2023050102.backup
8553 -rwx 5974218 23:27:48 Apr 30 2023 asa_snmp.log.1-2023050103.backup
8581 -rwx 6283925 00:27:49 May 01 2023 asa_snmp.log.1-2023050104.backup15111 -rwx 6291463 18:12:08 May 10 2023 asa_snmp.log.1-2023051022.backup
15115 -rwx 6034202 19:12:15 May 10 2023 asa_snmp.log.1-2023051023.backup
15119 -rwx 6285942 20:12:21 May 10 2023 asa_snmp.log.1-2023051100.backup

....and so on, Can I delete this backup files safely? Why are these files being generated?

thanks for the help.

Re: Flash disk is almost full but there are no many files listed on AS

MHM Cisco World — Thu, 11 May 2023 12:24:00 GMT

somebody run script that auto save the ASA config, but I think instead of using TFTP he use disk0

Re: Flash disk is almost full but there are no many files listed on AS

Marvin Rhoads — Thu, 11 May 2023 12:36:13 GMT

All the files in that folder can be safely deleted.

What @MHM Cisco World said or something similar is most likely happening. Some script-based process is telling the ASA to write a backup file to the log folder every day and there's no cleanup. After several years the disk is getting full of those files.

Re: Flash disk is almost full but there are no many files listed on AS

FernandoPerez75270 — Thu, 11 May 2023 12:45:37 GMT

Mmmm I suppose is an external script, there is no configuration related to backups in the ASA. I´m gonna delete those files and check some backup servers, seems to be the root of the problem. Thanks @MHM Cisco World and @Marvin Rhoads a lot for the help.

Re: Flash disk is almost full but there are no many files listed on AS

MHM Cisco World — Thu, 11 May 2023 13:39:09 GMT

You are so Welcome

Re: Flash disk is almost full but there are no many files listed on AS

jovalo — Mon, 26 Jun 2023 11:41:16 GMT

After upgrading from asa9-16-2-14 to asa9-16-4-18 I have the same issue. I delete the files ones in a while manually. Strange thing this is not mentioned in any release note for newer software versions as a fix. Will test asa9-16-4-27 soon to see whether that stops the logging.

Re: Flash disk is almost full but there are no many files listed on AS

FernandoPerez75270 — Mon, 26 Jun 2023 14:37:00 GMT

I opened a case with CISCO for this issue. They admin the bug and it is being discuss as an "internal case only". There are no releases to fix this so for the moment this snmp-log files have to be deleted manually.

Re: Flash disk is almost full but there are no many files listed on AS

tvotna — Mon, 26 Jun 2023 15:26:51 GMT

Please tell the engineer on the case that every bug for which a customer case has been opened must be made externally-visible. So, even if the bug was opened as internal, the engineer should add valid release notes enclosure to it and make it externally-visible. There are lots of other customers which would benefit from this action. Please also ask engineer to provide the bug id and post it here.

Re: Flash disk is almost full but there are no many files listed on AS

Isaac Smith — Wed, 26 Jul 2023 14:24:20 GMT

We also are having this issue. 9.16(4)18

Re: Flash disk is almost full but there are no many files listed on AS

FernandoPerez75270 — Wed, 26 Jul 2023 20:30:07 GMT

Yeap, it is a bug of the 9.16(4) release, the only solution know is a downgrade to 9.16(3), confirmed with Cisco through a support case. Please check all implications to downgrade to this version before doing the downgrade, but it is the only way to reclaim the lost free disk space for now.

Re: Flash disk is almost full but there are no many files listed on AS

jovalo — Thu, 27 Jul 2023 07:27:55 GMT

It looks like with 9.16(4)27 the problem is less / not there anymore. When I look at the disk0 graph read out with SNMP the line is now sort of stable since the last reboot time and that reboot was for the upgrade to 9.16(4)27.

Re: Flash disk is almost full but there are no many files listed on AS

tvotna — Thu, 27 Jul 2023 13:57:08 GMT

There is some fix in FXOS 2.10.1.271 and 2.12.0.498 which looks related:

CSCwb24306 Duplicate log entry for /mnt/disk0/log/asa_snmp.log

https://www.cisco.com/c/en/us/td/docs/security/firepower/fxos/fxos2101/release/notes/fxos2101_rn.html#reference_gmx_4g4_spb

I don't know how this bug was fixed and if this is exactly the issue you faced with. I guess, if it is FXOS, they could have implemented log rotation there... Can anybody with TAC access clarify? And what bug id was mentioned by TAC, even if it is internal?

Also, for some reason I don't see such SNMP log files in disk0:/log on my test Firepower 4145 running 9.16.4.14 / 2.10.1.234, although SNMP is configured and polling is in progress... Don't know why.

Re: Flash disk is almost full but there are no many files listed on AS

tvotna — Thu, 27 Jul 2023 14:25:50 GMT

Also, there is a CSCwa38996 https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa38996 with yet another awkward description. Fixed in 9.12.4.9 according to bug toolkit, but not mentioned at all in interim release notes. As usual.