Re: FPR2140 FDM upgrade 6.3 to 6.6 problem

KiloBravo · ‎11-30-2020

Hi,

I'm on my second attempt of trying to upgrade the FPR2140 to 6.6 from version 6.3. followed the 6.3 guide and it mentions that during the upgrade you get logged out of the GUI, and that it could take 30+ minutes to complete. The guide says not to log straight back in, but to wait (didn't say how long to wait so i figure 30 mins should be the minimum).

Just background info, this is fresh out the box, only has the basic config on it and i skipped device setup as it needed to be upgraded (running eval licenses, if that matters).

Uploaded the software via the GUI as the instructions mentioned, made sure any tasks were deployed and then hit install. As expected the GUI logged me out after a a minute or 2. I also had my console plugged into the FPR just to see when it reboots, but 30 mins later....nothing seemed to happen. I logged back into the GUI and it let me straight in and is still on version 6.3. Gave it another 10 minutes but nothing happened.

went for a a second attempt, this time had console access and typed in 'connect FTD' to see if anything happens there but everything else the same. Again, it logged me out, but 40 minutes later....nothing seems to be happening.

in version 6.6 apparently there is a show upgrade status sort of command but no such thing in 6.3...is there anyway to see what is (or isn't) happening?

Any other ideas?

Thanks in advance.

Marvin Rhoads · ‎11-30-2020

Connect to the unit via ssh (or console) and switch to /ngfw/var/log/sf. There should be a folder (directory) with the title of the upgrade.

FW1:/ngfw/var/log/sf$ ls -al | grep Cisco
drwxr-xr-x 11 root root   4096 Aug 14 03:58 Cisco_FTD_SSP_FP2K_Patch-6.6.0.1
drwxr-xr-x 12 root root   4096 Aug 14 03:01 Cisco_FTD_SSP_FP2K_Upgrade-6.6.0
drwxr-xr-x 12 root root   4096 Nov 10 11:55 Cisco_FTD_SSP_FP2K_Upgrade-6.7.0
<snip>

Switch into that folder and check the tail (last entries) of status.log. You can monitor the progress as it happens by using "tail -f status.log".

FW1:/ngfw/var/log/sf/Cisco_FTD_SSP_FP2K_Upgrade-6.7.0$ tail status.log
ui: Upgrade in progress: (97% done. 1 mins to reboot). Finishing the upgrade... (999_finish/999_y00_must_be_next_to_last_to_generate_integrity_data.sh)
ui: Upgrade in progress: (98% done.1 mins to reboot). Finishing the upgrade... (999_finish/999_y02_python2_pth_clean.sh)
ui: Upgrade in progress: (99% done.1 mins to reboot). Finishing the upgrade... (999_finish/999_z_complete_upgrade_message.sh)
ui: Upgrade complete
ui: Upgrade in progress: (99% done.1 mins to reboot). Finishing the upgrade... (999_finish/999_z_must_remain_last_finalize_boot.sh)
ui: Upgrade in progress: (99% done.1 mins to reboot). Finishing the upgrade... (999_finish/999_zz_install_bundle.sh)
ui: The system will reboot after FXOS platform upgrade completes followed by a firmware upgrade.
ui:System will reboot after FXOS platform upgrade completes followed by a firmware upgrade.
ui: Upgrade to 6.7.0 completed successfully.
state:finished

KiloBravo · ‎11-30-2020

Thanks for rsponding Marvin. Did as you mentioned and this is what I have:

admin@firepower:/ngfw/var/log/sf/Cisco_FTD_SSP_FP2K_Upgrade-6.6.1$ tail -f status.log
ui:[ 8%] [14 mins to go for reboot] Running script 000_start/125_verify_bundle.sh...
ui:[10%] [14 mins to go for reboot] Running script 000_start/410_check_disk_space.sh...
ui:[10%] [14 mins to go for reboot] Running script 200_pre/001_check_reg.pl...
ui:[11%] [14 mins to go for reboot] Running script 200_pre/002_check_mounts.sh...
ui:[18%] [12 mins to go for reboot] Running script 200_pre/202_disable_syncd.sh...
ui:[18%] [12 mins to go for reboot] Running script 200_pre/400_restrict_rpc.sh...
ui:[19%] [12 mins to go for reboot] Running script 200_pre/500_stop_system.sh...
ui:[20%] [12 mins to go for reboot] Running script 200_pre/505_revert_prep.sh...
ui:[20%] [12 mins to go for reboot] Running script 200_pre/600_ftd_onbox_data_export.sh...
ui:[20%] Fatal error: Error running script 200_pre/600_ftd_onbox_data_export.sh. For more details see /ngfw/var/log/sf/Cisco_FTD_SSP_FP2K_Upgrade-6.6.1/200_pre/600_ftd_onbox_data_export.sh.log on the device being upgraded.

Had a look at that log and found the following:

admin@firepower:/ngfw/var/log/sf/Cisco_FTD_SSP_FP2K_Upgrade-6.6.1/200_pre$ tail 600_ftd_onbox_data_export.sh.log
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 4 tries remaining ...
returnCode is 0
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 3 tries remaining ...
returnCode is 0
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 2 tries remaining ...
returnCode is 0
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 1 tries remaining ...
1
Timing out on waiting for tomcat to shut down, failing the upgrade
Exit return value = 1

So it seems to be having trouble shutting down the tomcat service. I've rebooted this box and tried again since, this is the latest log.

any thoughts/ideas?

KiloBravo · ‎11-30-2020

Hi Marvin,

Thanks for responding. Did as you mentioned which gave me the following:

admin@firepower:/ngfw/var/log/sf/Cisco_FTD_SSP_FP2K_Upgrade-6.6.1$ tail -f status.log
ui:[ 8%] [14 mins to go for reboot] Running script 000_start/125_verify_bundle.sh...
ui:[10%] [14 mins to go for reboot] Running script 000_start/410_check_disk_space.sh...
ui:[10%] [14 mins to go for reboot] Running script 200_pre/001_check_reg.pl...
ui:[11%] [14 mins to go for reboot] Running script 200_pre/002_check_mounts.sh...
ui:[18%] [12 mins to go for reboot] Running script 200_pre/202_disable_syncd.sh...
ui:[18%] [12 mins to go for reboot] Running script 200_pre/400_restrict_rpc.sh...
ui:[19%] [12 mins to go for reboot] Running script 200_pre/500_stop_system.sh...
ui:[20%] [12 mins to go for reboot] Running script 200_pre/505_revert_prep.sh...
ui:[20%] [12 mins to go for reboot] Running script 200_pre/600_ftd_onbox_data_export.sh...
ui:[20%] Fatal error: Error running script 200_pre/600_ftd_onbox_data_export.sh. For more details see /ngfw/var/log/sf/Cisco_FTD_SSP_FP2K_Upgrade-6.6.1/200_pre/600_ftd_onbox_data_export.sh.log on the device being upgraded.

admin@firepower:/ngfw/var/log/sf/Cisco_FTD_SSP_FP2K_Upgrade-6.6.1/200_pre$ tail 600_ftd_onbox_data_export.sh.log
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 4 tries remaining ...
returnCode is 0
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 3 tries remaining ...
returnCode is 0
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 2 tries remaining ...
returnCode is 0
Tomcat is running with PID 15551. Waiting for tomcat to shut down, 1 tries remaining ...
1
Timing out on waiting for tomcat to shut down, failing the upgrade
Exit return value = 1

Marvin Rhoads · ‎11-30-2020

As indicated, it timed out waiting for Tomcat service to shut down. I've only upgraded a handful of FTD devices to 6.6.1 and haven't encountered that particular error yet. (Nor have I on any of the many previous version upgrades I've done over the years.)

I would suggest taking this up in a TAC case as they may have better insight as to how to work around the issue.

KiloBravo · ‎11-30-2020

Thanks for your input Marvin.

Interestingly I've gone to the other firepower which is identical to this and is meant to be the secondary for this HA pair, using the same method, straight out the box...and its worked fine(!). same original version to upgraded version.

Is it still ill-advised to do the upgrade/re-image from the CLI?

Marvin Rhoads · ‎11-30-2020

cli is not the supported method so I don't recommend it unless it's under direction of the TAC.

If the device is brand new, reimage is certainly an option.

I did have one Firepower 2140 fail it's upgrade to 6.7. It appeared to be in the firmware upgrade following reload. A power cycle allowed it to continue and then successfully complete everything.

It seems like a meme, but sometimes it really does work to just turn the power off and then on again.

KiloBravo · ‎11-30-2020

I'm a firm believer in the old 'turn it off and on again' so I did give that a try as my first port of call after the first failure, but unfortunately it didn't come through for me this time round.

I'll try a re-image and see if that gets it sorted. Thanks again Martin.

johnlloyd_13 · ‎12-01-2020

hi,

sometimes files get corrupted during download or transferring to the device.

try to re-download the OS package and verify if checksum is the same in the download site.

KiloBravo · ‎12-02-2020

Hi John. thanks for your reply. I ended up just factory resetting it and trying the upgrade again which got it all up running.