topic Re: FMC can't upgrade to 7.2.7 - Readiness fails. Bug? in Network Security

FMC can't upgrade to 7.2.7 - Readiness fails. Bug?

walwar — Fri, 03 May 2024 12:21:07 GMT

Getting below error whenever I try to upgrade to 7.2.7 from 7.2.5. We've removed a lot of stuff, but still it says not enough disk, are possibly hitting a bug?

****************** FAILURE SCRIPT: 1 ***********************************
[240502 12:10:36:808]
SCRIPT NAME: 000_start/410_check_disk_space.sh
RECOVERY MESSAGE: Not enough disk space available in /var(Filesystem:/dev/sda7) to perform the upgrade.(Current available disk space: 89676924KB. Minimum required disk space: 140617111KB).
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

find /Volume -type f -exec du -Sh {} + | sort -rh | head -n 15
35G     /Volume/lib/mysql/sfsnort/geolocation_ipv4_new.MYD
19G     /Volume/lib/monetdb/dbfarm/eventdb/bat/41/4121.theap
13G     /Volume/lib/monetdb/dbfarm/eventdb/bat/24/2406.theap
5.7G    /Volume/.swaptwo
5.6G    /Volume/lib/monetdb/dbfarm/eventdb/bat/02/30/23076.theap
2.7G    /Volume/7.2.5-208/opt/CSCOpx/upgrade/fmc/dbdata/7.0.5/vms.db
2.7G    /Volume/7.2.5-208/opt/CSCOpx/databases/vms/vms.db
1.6G    /Volume/7.2.5-208/sf/updates/Cisco_Secure_FW_Mgmt_Center_Upgrade-7.2.6-168.sh.REL.tar
994M    /Volume/7.2.5-208/sf/updates/Cisco_FTD_Upgrade-7.0.6-236.sh.REL.tar
923M    /Volume/7.2.5-208/sf/updates/Cisco_FTD_SSP_FP2K_Upgrade-7.0.6-236.sh.REL.tar
923M    /Volume/7.2.5-208/log/tid_process.log
865M    /Volume/7.2.5-208/sf/updates/Cisco_FTD_SSP_FP1K_Upgrade-7.0.6-236.sh.REL.tar
782M    /Volume/lib/monetdb/dbfarm/eventdb/sql_logs/sql/log.1158692
507M    /Volume/7.2.5-208/lib/rabbitmq/mnesia/rabbit@localhost/quorum/rabbit@localhost/00000003.wal
400M    /Volume/7.2.5-208/sf/sidns_download/tmp/Cisco_DNS_Intelligence_Feed

find /var -type f -exec du -Sh {} + | sort -rh | head -n 15
35G     /var/lib/mysql/sfsnort/geolocation_ipv4_new.MYD
2.7G    /var/opt/CSCOpx/upgrade/fmc/dbdata/7.0.5/vms.db
2.7G    /var/opt/CSCOpx/databases/vms/vms.db
1.6G    /var/sf/updates/Cisco_Secure_FW_Mgmt_Center_Upgrade-7.2.6-168.sh.REL.tar
994M    /var/sf/updates/Cisco_FTD_Upgrade-7.0.6-236.sh.REL.tar
923M    /var/sf/updates/Cisco_FTD_SSP_FP2K_Upgrade-7.0.6-236.sh.REL.tar
923M    /var/log/tid_process.log
865M    /var/sf/updates/Cisco_FTD_SSP_FP1K_Upgrade-7.0.6-236.sh.REL.tar
507M    /var/lib/rabbitmq/mnesia/rabbit@localhost/quorum/rabbit@localhost/00000003.wal
400M    /var/sf/sidns_download/Cisco_DNS_Intelligence_Feed
276M    /var/sf/usb-keyfob/usb-efi-installer-7.2.5-208-serial-1.img
275M    /var/lib/mysql/sfsnort/flow_chunk_1714328160_0.MYD
274M    /var/lib/mysql/sfsnort/flow_chunk_1714268160_0.MYD
273M    /var/lib/mysql/sfsnort/flow_chunk_1714140000_0.MYD
270M    /var/lib/mysql/sfsnort/flow_chunk_1714205760_0.MYD

Re: FMC can't upgrade to 7.2.7 - Readiness fails. Bug?

balaji.bandi — Sat, 04 May 2024 18:15:32 GMT

i see some old file 7.2.6 - if you upgrading from 7.2.5 to 7.2.7 - you may look remove some /var/log files if you dont need

if you have concern contact TAC can help to remove some of the stuff.

Re: FMC can't upgrade to 7.2.7 - Readiness fails. Bug?

Marvin Rhoads — Mon, 06 May 2024 13:41:43 GMT

Try trimming the geolocation files using the following commands (from expert mode, running as root user):

OmniQuery.pl -db mdb -e "truncate geolocation_ipv4;" OmniQuery.pl -db mdb -e "truncate geolocation_ipv6;" OmniQuery.pl -db mdb -e "truncate geolocation_ipv4_country;" OmniQuery.pl -db mdb -e "truncate geolocation_ipv6_country;"

Re: FMC can't upgrade to 7.2.7 - Readiness fails. Bug?

SFrahm — Wed, 08 May 2024 06:52:58 GMT

I have the exact same issue, but so far only if it is a FTDv - for some reason it wants between 14 and 16GB of free space (differs between firewalls oddly enough) and a FTDv has a drive with total 28GB of space. Never seen such a huge space requirement for upgrading before, normally it takes like 6-7GB. So I believe this is some kind of bug. Both 2100 and 1000 series went through like a charm.

Re: FMC can't upgrade to 7.2.7 - Readiness fails. Bug?

SFrahm — Wed, 08 May 2024 09:10:21 GMT

Just a little update, just ran the check on a 3100 pair with no issues.

@walwar are you having the issue on a FTDv or is it another platform?
Update: Just realized from your output it is a 2k platform 🙂
Update2: Found that clearing old files in /ngfw/var/cisco/deploy/pkg/var/sf/lsp/active-lsp got me enough space on all devices to do the upgrade.

Re: FMC can't upgrade to 7.2.7 - Readiness fails. Bug?

walwar — Wed, 08 May 2024 09:23:32 GMT

After a couple of hours troubleshooting I've found that this /var/lib/mysql/libdata1 is too large. Tried to update GEO but that failed as well, I think that this file is corrupted and can't be updated.

I am not brave enough to perform the following as I seen in this post that ACP was not working properly afterwards.

OmniQuery.pl -db mdb -e "truncate geolocation_ipv4;"
OmniQuery.pl -db mdb -e "truncate geolocation_ipv6;"
OmniQuery.pl -db mdb -e "truncate geolocation_ipv4_country;"
OmniQuery.pl -db mdb -e "truncate geolocation_ipv6_country;"

I've opened a TAC case and I'll update this as soon as it is resolved.