ASA5508: Unable to upgrade Firepower module from 6.7.0 to 7.0.6

[swscco001]

Hello everybody,

yesterday I tried to to upgrade Firepower module in a ASA5508 (rel. 9.16(4) from 6.7.0 to 7.0.6.

The FMC is running rel. 7.2.5.1.

There were no open deployments and the Readiness Check was successful.

After more than one hour I just got the erreor mesage: "Update Install failed" (see attached screen dump).

This was the same for a further same module at the upgrade from rel. 7.0.0.1 to 7.0.6.

I checked the CLI of the module but did not find the right log file. I just found the 
logged information of the Readiness Check:

root@fw-set-ips:/var/log/sf/Cisco_Network_Sensor_Upgrade-7.0.6# tail /var/log/sf/Cisco_Network_Sensor_Upgrade-7.0.6/upgrade_readiness/main_upgrade_script.log
[240125 10:15:24:486] SKIP 200_pre/998_fix_broken_rules.pl
[240125 10:15:24:693] SKIP 200_pre/999_enable_sync.sh
[240125 10:15:24:899] MAIN_UPGRADE_SCRIPT_END
[240125 10:15:25:756]  Readiness check completed....
[240125 10:15:25:779] Attempting to remove upgrade lock
[240125 10:15:25:782] Success, removed upgrade lock
[240125 10:15:25:794]
[240125 10:15:25:797] #######################################################
[240125 10:15:25:800] # UPGRADE READINESS CHECK COMPLETE  status : PASS #
[240125 10:15:25:804] #######################################################


root@fw-set-ips:/Volume/6.7.0/log/sf/Cisco_Network_Sensor_Upgrade-7.0.6# ls -l
total 12
-rw-r--r-- 1 root root  236 Jan 25 10:00 DBCheck.log
-rw-r--r-- 1 root root 1613 Jan 25 10:15 status.log
drwxr-xr-x 4 root root 4096 Jan 25 10:15 upgrade_readiness

root@fw-set-ips:/Volume/6.7.0/log/sf/Cisco_Network_Sensor_Upgrade-7.0.6# tail status.log
ui:[61%] Running script 000_start/113_EO_integrity_check.pl...
ui:[64%] Running script 000_start/250_check_system_files.sh...
ui:[68%] Running script 000_start/410_check_disk_space.sh...
ui:[71%] Running script 200_pre/001_check_reg.pl...
ui:[75%] Running script 200_pre/002_check_mounts.sh...
ui:[79%] Running script 200_pre/007_check_sru_install.sh...
ui:[82%] Running script 200_pre/015_verify_rpm.sh...
ui:[86%] Readiness Check completed successfully.
ui: Readiness Check has completed.
state:finished

There are no space problems on the firewall:

root@fw-set-ips:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.7G  765M  2.7G  22% /
devtmpfs        1.7G   80K  1.7G   1% /dev
/dev/vda7        65G   15G   48G  23% /var
none            1.7G  8.7M  1.7G   1% /dev/shm
tmpfs           1.7G     0  1.7G   0% /dev/cgroups
/dev/sda1        88M   40M   43M  49% /boot

Where can I find the log file that contains the reason for the failed upgrade?

Thanks a lot!




Bye
R.

[Marius Gunnerud]

There should be a status.log or upgrade_status.log file located under /ngfw/var/log/sf/Cisco_Network_Sensor_Upgrade-7.0.6.  In there you should be able to see the reason for the failure

[swscco001]

Hi Marius,

thanks for your reply!

The folder you gave me does not exist on the ASA5508-X running rel. 6.7.0.

> show version
-------------------[ fw-set-ips ]-------------------
Model                     : ASA5508 (72) Version 6.7.0 (Build 65)
UUID                      : 6dd2cc24-eadc-11e7-b52f-e416c72e0cd2
Rules update version      : 2024-01-24-001-vrt
VDB version               : 360
----------------------------------------------------

root@fw-set-ips:/# ls -l
total 81
drwxr-xr-x   7 root root  4096 Feb 18  2021 Volume
drwxr-xr-x   2 root root  4096 Nov  2  2020 bin
drwxr-xr-x   6 root root  1024 Feb 18  2021 boot
drwxr-xr-x   7 root bin   4096 Nov  2  2020 cisco
drwxr-xr-x  13 root root  3340 Jan 19 16:03 dev
drwxr-xr-x  46 root root  4096 Jan 29 12:03 etc
drwxr-xr-x   2 root root  4096 Oct  6  1997 home
drwxr-xr-x   5 root root  4096 Feb 18  2021 lib
drwxr-xr-x   3 root root  4096 Sep  9  2020 lib64
drwx------   2 root root 16384 Feb 18  2021 lost+found
drwxr-xr-x   5 root root  4096 Mar 16  2002 mnt
drwxr-xr-x   3 root root  4096 Feb 18  2021 new-root
drwxr-xr-x   3 root root  4096 Sep  9  2020 opt
dr-xr-xr-x 153 root root     0 Dec 21 22:30 proc
drwx------   4 root root  4096 Jan 25 09:57 root
drwxr-xr-x   2 root root  4096 Sep  9  2020 sbin
dr-xr-xr-x  11 root root     0 Dec 21 22:30 sys
drwxrwxrwt   3 root root  4096 Jan 29 12:17 tmp
drwxr-xr-x   2 root root  4096 Feb 18  2021 upgrade_workspace
-rw-r--r--   1 root root     0 Feb 18  2021 upgraded
drwxr-xr-x  20 root root  4096 Jul 24  2020 usr
drwxr-xr-x  19 root root  4096 Feb 18  2021 var

I also searched for the file name upgrade_status.log but it does not exist
in the correct folder apparently.

root@fw-set-ips:/# find / -name upgrade_status.log -print
/Volume/6.7.0/log/sf/Cisco_Network_Sensor_Upgrade-6.5.0/upgrade_status.log
/Volume/6.7.0/log/sf/Cisco_Network_Sensor_Upgrade-6.7.0/upgrade_status.log
/Volume/6.7.0/log/sf/Cisco_Network_Sensor_Patch-6.5.0.4/upgrade_status.log
/var/log/sf/Cisco_Network_Sensor_Upgrade-6.5.0/upgrade_status.log
/var/log/sf/Cisco_Network_Sensor_Upgrade-6.7.0/upgrade_status.log
/var/log/sf/Cisco_Network_Sensor_Patch-6.5.0.4/upgrade_status.log

It looks like there were no upgrade attempt to rel 7.0.6 on this firewall but the customer
has just one with the hostname fw-set-ips.

Could it be that the log file was deleted after a failed upgrade attempt
or do you have another explanation?

Thanks a lot!

Bye
R.

[Marius Gunnerud]

It is of course possible that the log file was deleted, but I doubt it.  It almost seems like the upgrade was aborted before the upgrade started.  

Since the upgrade readiness check was successful I am assuming that you successfully pushed the upgrade package to the firewall / SFR module from the FMC? My thought is there might not be enough storage on the device to unpack the upgrade file.

If there is no log file for the upgrade, then I think you would need to run a tail on the upgrade log during the next upgrade attempt to see the reason for the failure, if it fails again.