FMC shows Disk Usage Sep 26, 2023 6:57 AM / using 89%: 3.0G

[gmore]

FMC is on 7.0.5. I am seeing disk space 89% and due to this my backup is failing.  Please guide how to fix this.

TEST-XXXX-FMC-01:/var/log$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.7G 3.0G 414M 89% /

[Marvin Rhoads]

You can start by identifying the largest files. Run the following command as root user (i.e., "sudo su -" first):

find /var -type f -exec du -Sh {} + | sort -rh | head -n 15

[Eric R. Jones]

You didn't mention it so I'm going to suggest checking System > Tools > Backup > Restore > Backup Management

Check "Firewall Management Backups" and find the last page. Select the check box and remove those backups until you get the 1st/2nd page.

Check the "Device Backups" and find the last page. Select the check box and remove those backups until you get to the 1st/2nd page.

We recently have run into an issue with our FPR 2110 where it hit over 80% and couldn't pass the check readiness portion of the upgrade process. I went in and removed old install files from 6.6.X as we are on 7.2.4. I found plenty of old files to remove and got it under 70%. The next day I ran the check readiness tool and it failed again because it went over 70%. This triggered the failure during check readiness. 

I can't find large enough files to remove it and the TAC engineer thins it's a bug. Waiting to schedule a webex to find out more.

[gmore]

thanks Eric !

There were not much files but I deleted 6-7 files but still same.

[Marvin Rhoads]

Did you check the command I suggested on 9-26?

You could also be hitting this bug below. the link has a work around.

https://bst.cisco.com/bugsearch/bug/CSCwb34240

[gmore]

I tried 9-26 and below are the large files.

4.4G /var/opt/CSCOpx/databases/vms/vms.db
4.3G /var/opt/CSCOpx/upgrade/fmc/dbdata/7.0.1/vms.db
3.3G /var/common/core_1696168400_US-VA-IAD-FMC-01.iri.int_mariadbd_6.11468
2.0G /var/sf/updates/Cisco_Firepower_Mgmt_Center_Upgrade-7.0.5-72.sh.REL.tar
1.7G /var/common/192.168.160.203-cd6d7f40-dd4f-11ea-be2d-98ca4c71559d-troubleshoot.tar.gz
1.1G /var/lib/mysql/sfsnort/packet_log_1568897640_0.MYD

[Eric R. Jones]

So we just cleaned up our issue via TAC. We went in and we ran through some commands to track down a bunch of files. Here is the output from my WebEx. I recommend opennng a webex unless your really confident/comfortable in your regex skills to parse the file for proper date formats and understand Linux/Unix commands. I don't have the mysql commands. It didn't require a reload of the device but we did restart a few of the processes so that disk space would be re-read.

"Webex summary: Disk space on /ngfw was showing 72% used. We removed older mysql directories from the device but this only regained 1% disk space. We used the command 'du -s ./* | sort -n -r | head -n 20' to drill down through the filesystem to identify what was taking up space. The directory '/ngfw/Volume/root2/ngfw/var/cisco/deploy/pkg/var/sf/lsp/active-lsp' was using the greatest amount of space, with several small 40 byte files and a large archive file which was the current lsp. We were unable to find anything else which could be deleted to free space."

[Marius Gunnerud]

In every instance of high disk usage issues I have seen it has been related to logrotate not being implemented correctly after upgrades.  During your TAC session did the engineer check the logrotate files?  @Marvin Rhoads has linked to the bug related to this issue, and I have also done a write-up on it in the link below.

https://community.cisco.com/t5/security-knowledge-base/ngfw-high-unmanaged-disk-usage/ta-p/4927059

[Marvin Rhoads]

There are several bugs that can cause this. As @Marius Gunnerud correctly noted, the logrotate one is the most common by far.

Another is related to Geodb files. That's why I asked to check for the largest files.

[Eric R. Jones]

That was one of the first things we looked at, GeoDB files.

[Eric R. Jones]

Yes, we went over that and it was working. We also have NFS syslog working.