Firepower 1120: FDM - unable to upgrade the cluster

[swscco001]

Hello everybody,

our customer has a cluster of two Firepower 1120 runnig rel. 7.3.0-69 with FDM.

I wanted to upgrade the cluster to rel. 7.4.1.

I downloaded the file from CCO and uploaded it to the standby device.

There were no open deployments.

The Readiness Check was successful.

Then I started the upgrade and it took a pretty long time while counting
up the percentage of progress (see screen dumps).

After the reboot the old 7.3.0-69 was booted again without any error message.

I repeated the whole procedure once again but with the same result.

The I performed a normal reboot but the standby device stayed at rel. 7.3.0-69.

What could be the reason for this issue and what can be done to upgrade the
cluster.

If you need any further information please let me know.

Thanks a lot for every hint!

Bye
R.

[Marvin Rhoads]

In the cli expert mode, check under /ngfw/log/sf - you should see a folder there with the 7.4 upgrade designation. Within that folder, check status.log file for the last handful of entries (tail status.log) and share that output please.

[swscco001]

H Marvin,

thanks for your fast reply!

There is no folder /ngfw/log/sf.

I found the /ngfw/var/log/sf with teh following content:

The date of the upgrade attempt was Jan. 12.

I searched the whole system and found a status.log file with the following content:

root@FW-Bermueller:/ngfw/var/log/sf# tail /opt/cisco/csp/applications/cisco-ftd.6.6.1.91__ftd_001_JMX2611X1S6O1OQB61/app_data/Volume/root2/ngfw/var/log/sf/Cisco_FTD_SSP_FP1K_Upgrade-7.4.1.1705079784.rollback/status.log
ui: Upgrade in progress: (35% done.26 mins to reboot). Updating configurations... (800_post/500_analysis_cleanup.pl)
ui: Upgrade in progress: (35% done.26 mins to reboot). Updating configurations... (800_post/720_update_devices.pl)
ui: Upgrade in progress: (35% done.26 mins to reboot). Updating configurations... (800_post/720_update_peers.pl)
ui: Upgrade in progress: (35% done.26 mins to reboot). Updating configurations... (800_post/780_remove_future_flagsconf.pl)
ui: Upgrade in progress: (35% done.26 mins to reboot). Updating configurations... (800_post/810_clean_upgrade_workflow.sh)
ui: Upgrade in progress: (35% done.26 mins to reboot). Updating configurations... (800_post/810_update_ld_conf.sh)
ui: Upgrade in progress: (35% done.26 mins to reboot). Updating configurations... (800_post/850_clear_eula.sh)
ui: Upgrade in progress: (38% done.25 mins to reboot). Updating configurations... (800_post/870_update_fireamp_cert.sh)
ui: Upgrade in progress: (38% done.25 mins to reboot). Updating configurations... (800_post/880_install_VDB.sh (in background: 800_post/100_ftd_onbox_data_import.sh))
ui:__[] Fatal error: Upgrade Failed: The chosen certificate has already expired. Please apply an unexpired certificate.. Returning to previous version (7.3.0)...

When I check the current wildcard certificate I see it is not expired yet (see attached screen dump).

Do you have any idea how to get fixed this?

Thanks a lot!

Bye
R.

[Marvin Rhoads]

Sorry about the path confusion. I was working from memory earlier.

The status.log file pinpoints the issue.

It could be that your older wildcard or possibly the device self-signed certificate is still in use.Check your URL bar in FDM and inspect to verify which certificate the device is using.

[swscco001]

Hi Marvin,

I found out that the expired wildcard certificate of 2022 is currently in use (see attached screen dump). 

There is already a valid wildcard certificate of 2023 on the nodes (see attached screen dump).

The question for me is now how to exchange the expired with the valid wildcard certificate to be used in the FDM? Is it enough to delete the expired one or is there any guide?

Thanks a lot!

Bye
R.

[Marvin Rhoads]

You should change the management web server certificate. Instructions to do so can be found here:

https://www.cisco.com/c/en/us/td/docs/security/firepower/730/fdm/fptd-fdm-config-guide-730/fptd-fdm-system.html#task_31B0F47D39444D6EB91A552A2B93B63E

Once that's done, you can delete the expired certificate and retry the upgrade.

[swscco001]

Hi Marvin,

I followed your link and chose the valid wildcard certificate for the Management Web Server on the active device (see screen dump). Then I logged out from FMC and closed Chrome.

I could login to the active device' FMC without problems but the setting was not replicated to the standby device unfortunately. I tried to set the valid wildcard certificate for the Management Web Server on the standby device but it rejected this attempt (see screen dump), even if the same valid wildcard certificate is stored on the standby device too.

Is there a way to force the replication of this configuration?

Thanks a lot!



Bye
R.

[Marvin Rhoads · Accepted solution]

I have not done this actual step since most of my systems are FMC-managed.

However, I would try forcing a failover and then log into the the newly Active unit (formerly Standby) and repeating the steps to change its management certificate from there.

[swscco001]

Hi Marvin,

I will do so at Friday and if I can change the certificate I will retry the upgrade.

I will let you know then.

Thanks a lot!



Bye
R.

[swscco001]

Hi Marvin,

after failover the wildcard certificate of 2023 was already there for the Management Web Server!

After that I retried the upgrade and it worked even if it took much time.

Thanks a lot for your useful hints!



Bye
R.