Cisco FTD Upgrade fails due low disk space on ngfw

[vashan]

I am trying to upgrade from version 6.4 til 6.6.5 and the upgrade failes due low disk space on FTD. 

I have deleted old updates files , temp files manually. But i cannot free up additional disk space.

/ngfw

Size: 51G

Free: 13G

Used: 38G

Percent: 75%

Error message:

Checking free /root disk space...

[211015 22:49:16:353] Calculating free disk space (measured in Kbytes)...

[211015 22:49:16:358] Root disk free = 16984804K, Var disk free = 16984804K

[211015 22:49:16:359] Accounting for DE directory size (which will be copied)...

[211015 22:49:16:368] DEDir = 4371396

[211015 22:49:16:370] Root disk free = 16984804K, Var disk free = 12613408K

[211015 22:49:16:371] Root disk free = 16984804K, Var disk free = 12613408K

Checking free /var disk space...

Find largest table in /var/lib/mysql ...

Checking for 507904K of /var disk space to upgrade the database

Total disk space calculated based on mysql files

/ngfw/Volume/lib/mysql /ngfw/var/tmp/selfgz155806486

/ngfw/var/tmp/selfgz155806486

disk space required to backup database is 1906360

Total /var disk space needed for upgrade is 13551988

Fatal error: Not enough var disk space available.  You need at least 13551988K free to perform this upgrade.  You have 12613408K free.

Exit return value = 1

[Kasun Bandara]

Hi,

guess this will help you

https://community.cisco.com/t5/network-security/upgrade-ha-pair-of-ftdv-s-from-6-4-to-6-5/td-p/3934741

[Marius Gunnerud]

Which hardware are you upgrading on, is there ASA? if so which hardware model?  I haven't seen any FTD hardware being sold with less than 100Gig of disk space.  

If this is an ASA, you might be running into compatibility restrictions as well as disk space issues.  Check the compatibility guide to make sure your hardware supports 6.6.x

[vashan]

Hi Marius.

My hardware info:

2x Firepower 2110 in HA configuration.

Disk Usage:1 2021-10-16 16:19:36 /ngfw using 74%: 37G (14G Avail) of 51G

Upgrading from version 6.4.0.10 to 6.6.5

I have successfully upgraded FMC and Firepower-Device2 to version 6.6.5. I have problem upgrading Firepower-Device1.

Firepower-Device2: /ngfw using 34%: 17G (34G Avail) of 51G. 

Firepower-Device1: /ngfw using 74%: 37G (14G Avail) of 51G

[Marius Gunnerud]

then I would suggest entering expert mode and issue the command df -h

find the directory that is taking up the most disk space and then start drilling down into it to find what is taking up the majority of that space.

Once you find what is taking up disk space you will need to decide if you can delete those files or not.  I would recommend not deleting any files without consulting TAC.

[vashan]

Output of df-h

Filesystem Size Used Avail Use% Mounted on
rootfs 7.6G 512M 7.1G 7% /
devtmpfs 7.7G 595M 7.1G 8% /dev
tmpfs 7.7G 488K 7.7G 1% /run
tmpfs 7.7G 1.3M 7.7G 1% /var/volatile
/dev/sda1 923M 170M 707M 20% /opt/cisco/config
/dev/sda2 922M 64M 811M 8% /opt/cisco/platform/logs
/dev/sda3 11G 29M 11G 1% /var/data/cores
/dev/sda4 51G 37G 14G 74% /ngfw
/dev/sdb1 7.4G 3.3G 4.2G 44% /mnt/boot
cgroup_root 7.7G 0 7.7G 0% /dev/cgroups
tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup
tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup/pm
tmpfs 1.0M 0 1.0M 0% /var/data/cores/sysdebug/tftpd_logs

Do you know how to find which files or directories uses space on /ngfw?

[Marvin Rhoads]

It's Linux so any number of recursive file find syntax tricks will work.

For instance, from within the /ngfw directory, run the following as root:

du -ah ./ | sort -rh | head -20

That will give you the largest 20 subdirectories and files.

I have found core dumps and Snort rule updates as frequent contributors to overly full partitions that can cause upgrades to fail.

[Ilkin]

The output of df -h may not always be shown all the paths of interest. df -ah shows all the file systems and their mount points.

For example, sda4 is mounted at 2 points that both can contribute to high disk space utilization:

admin@firepower:~$ df -ah | egrep -i 'Filesys|sda4'
Filesystem Size Used Avail Use% Mounted on
/dev/sda4 81G 9.7G 71G 13% /opt/cisco/csp
/dev/sda4 51G 9.7G 41G 20% /ngfw
...

 Additionally, high disk space utilization can be caused by excessive logging and/or lack of log file rotation/compression. This can also be caused due to a software defect (e.g. CSCvt35774  'Missing rotate for SNMP log file causes high disk usage', etc.).

In case of doubts, a TAC case can be opened.

[j.cusick]

https://community.cisco.com/t5/security-documents/ngfw-current-disk-usage-77-higher-than-max-expected-75/ta-p/4524056

This will solve the /mnt/boot disk exceeding 75%