Firepower 1010 readiness check stuck at "Please Wait"

[Michael King]

I have a Firepower 1010, on 7.0.0-94 that I'm trying to upgrade to 7.4.2-172.

I ran the readiness check last night at 8pm.   12 hours later, at 8 am, it still says "Please wait" on the readiness check, and won't let me hit the upgrade button.

Following these instructions:

https://www.cisco.com/c/en/us/support/docs/security/secure-firewall-management-center/222169-monitor-and-resume-readiness-check-or-up.html

I went and found the logs.... which was less than helpful since they all indicate the readiness check is complete:

[361019 18:15:34:044] MAIN_UPGRADE_SCRIPT_END
[361019 18:15:34:056]  Readiness check completed....
[361019 18:15:35:431] Attempting to remove upgrade lock
[361019 18:15:35:434] Success, removed upgrade lock
[361019 18:15:35:450] Process is Disabled enabling Syncd
[361019 18:15:35:463] Syncd (normal) - Running 20758
Command: /ngfw/usr/local/sf/bin/Syncd.pl --persistent
PID File: /ngfw/var/sf/run/Syncd.pid
[361019 18:15:35:466]
[361019 18:15:35:468] #######################################################
[361019 18:15:35:476] # UPGRADE READINESS CHECK COMPLETE  status : PASS #
[361019 18:15:35:479] #######################################################

I reloaded the page, no change.

I rebooted the box.  No change.

I power cycled the box.  No change.

I used the above instructions, and reran the readiness check,  the UI updated to say something like "task 4 of 21, disk cleanup" and I reloaded the page after a few minutes, it went back to "Please Wait"

I have the firewall pulled out and running on my desk, so the only port connected right now is the management port, and it's direct to my laptop.

Suggestions?

[balaji.bandi]

Hope you have enough space ? are you using FMC or FDM to upgrade ?

get on to shell and check what is the logs

tail -f upgrade_status.log

[Michael King]

Using FDM (Do not have FMC)

I think i have enough space.   df -h tends to agree.

oot@Firepower-C:/ngfw/var/log/sf/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2/upgrade_readiness# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.3G  4.3M  3.3G   1% /
devtmpfs        3.3G  5.1M  3.3G   1% /dev
tmpfs           3.9G  504K  3.9G   1% /run
/dev/sda1       7.3G  2.2G  4.7G  32% /mnt/boot
/dev/sda2       923M  193M  683M  23% /opt/cisco/config
/dev/sda3       923M   33M  844M   4% /opt/cisco/platform/logs
/dev/sda5       149G   17G  133G  11% /opt/cisco/csp
/dev/sda4        28G   45M   26G   1% /var/data/cores
cgroup_root     3.9G     0  3.9G   0% /dev/cgroups
none            128M     0  128M   0% /dev/shm/snort
tmpfs           1.0M     0  1.0M   0% /var/data/cores/sysdebug/tftpd_logs

i'm not sure where upgrade_status.log is.

/ngfw/var/log/sf/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2/status.log is:

ui: Readiness in progress: (67% done). (200_pre/004_check_deploy_package.pl)
ui: Readiness in progress: (71% done). (200_pre/005_check_manager.pl)
ui: Readiness in progress: (76% done). (200_pre/006_check_snort.sh)
ui: Readiness in progress: (81% done). (200_pre/007_check_sru_install.sh)
ui: Readiness in progress: (86% done). (200_pre/009_check_snort_preproc.sh)
ui: Readiness in progress: (90% done). (200_pre/011_check_self.sh)
ui: Readiness in progress: (95% done). (200_pre/015_verify_rpm.sh)
ui: Readiness Check completed successfully.
ui: Readiness Check has completed.

and

/ngfw/var/log/sf/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2/upgrade_readiness/upgrade_readiness.log is blank (0 bytes)

and /ngfw/var/log/sf/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2/upgrade_readiness/upgrade_readiness_status.log has

TIMESTAMP:Sun Oct 19 18:15:24 UTC 2036 PERCENT:57%  MESSAGE:Running script 200_pre/001_check_reg.pl...
TIMESTAMP:Sun Oct 19 18:15:26 UTC 2036 PERCENT:62%  MESSAGE:Running script 200_pre/002_check_mounts.sh...
TIMESTAMP:Sun Oct 19 18:15:26 UTC 2036 PERCENT:67%  MESSAGE:Running script 200_pre/004_check_deploy_package.pl...
TIMESTAMP:Sun Oct 19 18:15:27 UTC 2036 PERCENT:71%  MESSAGE:Running script 200_pre/005_check_manager.pl...
TIMESTAMP:Sun Oct 19 18:15:28 UTC 2036 PERCENT:76%  MESSAGE:Running script 200_pre/006_check_snort.sh...
TIMESTAMP:Sun Oct 19 18:15:29 UTC 2036 PERCENT:81%  MESSAGE:Running script 200_pre/007_check_sru_install.sh...
TIMESTAMP:Sun Oct 19 18:15:29 UTC 2036 PERCENT:86%  MESSAGE:Running script 200_pre/009_check_snort_preproc.sh...
TIMESTAMP:Sun Oct 19 18:15:30 UTC 2036 PERCENT:90%  MESSAGE:Running script 200_pre/011_check_self.sh...
TIMESTAMP:Sun Oct 19 18:15:31 UTC 2036 PERCENT:95%  MESSAGE:Running script 200_pre/015_verify_rpm.sh...
TIMESTAMP:Sun Oct 19 18:15:34 UTC 2036 PERCENT:100%  MESSAGE:Readiness Check completed successfully.

[Marvin Rhoads]

If it doesn't have any production config that you need to save, it would be easiest to just re-image it straight away using the 7.4.2 image.

https://www.cisco.com/c/en/us/support/docs/security/firepower-1000-series/220642-reimage-a-secure-firewall-threat-defense.html

[Michael King]

I did think about that.... but the point of this was to test our upgrade processes.   Reimaging would be cheating 

[balaji.bandi]

May i was looking more of upgrade process :  the logs show clearly 2036 missed this bit, good catch.

TIMESTAMP:Sun Oct 19 18:15:24 UTC 2036 PERCENT:57%  MESSAGE:Running script 200_pre/001_check_reg.pl...
TIMESTAMP:Sun Oct 19 18:15:26 UTC 2036 PERCENT:62%  MESSAGE:Running script 200_pre/002_check_mounts.sh...
TIMESTAMP:Sun Oct 19 18:15:26 UTC 2036 PERCENT:67%  MESSAGE:Running script 200_pre/004_check_deploy_package.pl..

[Michael King]

So I got impatient, and I went to the 2nd step of the instructions i linked to:

https://www.cisco.com/c/en/us/support/docs/security/secure-firewall-management-center/222169-monitor-and-resume-readiness-check-or-up.html

and ran

root@Firepower-C:/ngfw/var/sf/updates# install_update.pl --detach --resume /var/sf/updates/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2-172.sh.REL.tar

it was unhappy

root@Firepower-C:/ngfw/var/sf/updates# install_update.pl --detach --resume /var/sf/updates/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2-172.sh.REL.tar
ARGV[0] = --detach
ARGV[1] = --resume
ARGV[2] = /var/sf/updates/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2-172.sh.REL.tar
bundle_filepath: /var/sf/updates/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2-172.sh.REL.tar
install_update.pl begins. bundle_filepath: /var/sf/updates/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2-172.sh.REL.tar
Use of uninitialized value $option in concatenation (.) or string at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 27.
 Makeself GetUpdate Info params FILEPATH : /var/tmp/upgrade-patch/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2-172.sh and OPTION:  at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 27.
FILEPATH directory name /var/tmp/upgrade-patch at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 40.
Inside GetInfo FILEPATH :/var/tmp/upgrade-patch/Cisco_FTD_SSP_FP1K_Upgrade-7.4.2-172.sh at /usr/local/sf/lib/perl/5.24.4/SF/Update/Makeself.pm line 265.
Use of uninitialized value in string at /usr/local/sf/lib/perl/5.24.4/SF/Update/StatusProc.pm line 197.
Use of uninitialized value in string at /usr/local/sf/lib/perl/5.24.4/SF/Update/StatusProc.pm line 197.
Use of uninitialized value in string at /usr/local/sf/lib/perl/5.24.4/SF/Update/StatusProc.pm line 197.

UI is still has the same place... so I don't think it actually did anything.

[Michael King]

Update, 10 minutes later the firewall rebooted and the upgrade screen showed up (Upgrade screen has new logo / login page).  I'll report back after the upgrade screen completes.

[Michael King]

So according to the console, the upgrade is still running.... but the UI threw this message

 Upgrade failed
The chosen certificate has already expired. Please apply an unexpired certificate..
See detailed info
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] ownership of '/ngfw/var/cisco/deploy/tmp' retained as www:www The chosen certificate has already expired. Please apply an unexpired certificate. ValidationException: The chosen certificate has already expired. Please apply an unexpired certificate. com.cisco.ngfw.onbox.importer.services.UpgradeSqliteImportService.importConfigFromSqlite(UpgradeSqliteImportService.java:190) com.cisco.ngfw.onbox.importer.services.UpgradeSqliteImportService.importConfig(UpgradeSqliteImportService.java:134) com.cisco.ngfw.onbox.importer.NGFWDBImporter.importConfigSqlite(NGFWDBImporter.java:315) com.cisco.ngfw.onbox.importer.NGFWDBImporter.main(NGFWDBImporter.java:173) Reporting error : The chosen certificate has already expired. Please apply an unexpired certificate. Fatal error: The chosen certificate has already expired. Please apply an unexpired certificate.

With a Cancel / Retry button.

The console is still showing a lot of activity, so I don't know what it's doing, but I'm going to let it sit for a few hours and see if it makes itself happy or not.

@Marvin Rhoads i fear that a reimage is going to happen no matter what.  We'll see what happens very soon.

[Marvin Rhoads]

It won't upgrade if the certificate used on the management interface is expired.

[Michael King]

So I'm unsure how..... but the clock got really messed up.

admin@Firepower-C:/$ sudo hwclock
2036-10-22 21:54:59.280437+00:00

I just reset the clock to current time, and I'm giving it another go.   I skipped running the readiness check.

[Michael King · Accepted solution]

So to wrap this up. I had many problems. 

1. My hardware clock was FAR out of date. (12 years?)
2. I ran into Bug CSCwd11825  https://bst.cisco.com/bugsearch/bug/CSCwd11825 FDM upgrade failure due to HTTPS cert expired.   I would also hazard a guess this is why my readiness check got "stuck"