Fail Upgrading FMC 4600 from Version 6.6.1 (Build 91) to 6.6.5

[dodavis]

While attempting an FMC upgrade the upgrade failed with the following message in the CLI:

The Cisco 6.6.5 upgrade has halted, status:

  [48%] Fatal error: Error running script 800_post/021_reinstall_sru.sh. For more details see /var/log/sf/Cisco_Firepower_Mgmt_Center_Upgrade-6.6.5/800_post/021_reinstall_sru.sh.log on the device being upgraded.

Log files for the halted upgrade are located beneath:

  /var/log/sf/Cisco_Firepower_Mgmt_Center_Upgrade-6.6.5

If log files indicate upgrade failure please contact technical support.'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

When I check the 021_reinstall_sru.sh.log file it ends with the following:

[211122 15:17:31] BEGIN  installer/510_install_policy.pl

[211122 15:17:35]   FAILED  installer/510_install_policy.pl

[211122 15:17:35]   ====================================

[211122 15:17:35]     tail -n 10 //var/log/sf/sru-2021-11-18-001-vrt/installer/510_install_policy.pl.log

Package failure at /usr/local/sf/lib/perl/5.10.1/SF/SRUInstall.pm line 1384.

Printing stack trace:

        called from /usr/lib/perl5/site_perl/5.10.1/Error.pm (150)

        called from /usr/local/sf/lib/perl/5.10.1/SF/SRUInstall.pm (1384)

        called from /usr/local/sf/lib/perl/5.10.1/SF/SRUInstall.pm (332)

        called from /usr/lib/perl5/site_perl/5.10.1/Error.pm (393)

        called from /usr/local/sf/lib/perl/5.10.1/SF/SRUInstall.pm (335)

        called from lib/Install.pm (1075)

        called from installer/510_install_policy.pl (32)

[211122 15:17:35] Fatal error: Error running script installer/510_install_policy.pl, RC=1

[211122 15:17:35] Exiting, 1.

SRU install was unsuccessful

Exit return value = 1

Does anyone know what could cause this error in 510_install_policy.pl?    Would it be safe to simply run 'upgrade_resume.sh"?

Thanks!

[Marvin Rhoads]

Often when an upgrade fails on the reinstall_sru step it is because the device is not current on its SRU (Snort Rule Updates) prior to the upgrade having started.

I would check the SRU version/ date and, if you find it outdated, go ahead and update it first before resuming/retrying.

If, however, the SRU checks out as current already then I would open a TAC case to investigate for a more uncommon issue.

[dodavis]

Hi Marvin, 

Thanks for the information. I am working with the TAC right now. They suspected it may be related to bug CSCvr62480.

So far they tried modifying CSMagent.pm and adjusting an inactivity timer and restarting the update only to get the same error.

They backed out the update and rebooted. Now we are trying the upgrade again.

[PETER AGENGO]

When we first engaged TAC, we resumed synchronization and everything was green. Both instances in our HA deployment showed that the SRU and LSP was installed. However, during the initial triage, the TAC engineer navigated via CLI to the SRU directory to look at the installation log/history and to our surprise the log showed that the SRU install the previous day failed at 49 percent even though in the GUI it showed it was successfully install via the normal Remote Install after the Primary one got it. The version on the GUI and CLI also showed the current version running. So that was a very interesting discrepancy. 

[nspasov]

What Marvin said. I have only seen this if: 1) Your SRUs are out of date or 2) If something got corrupted during the RSU upgrades. In the 2nd case, you would definitely want to engage TAC. 

[dodavis · Accepted solution]

The TAC engineers discovered that we were running into bugs CSCvu23149 and CSCvw35657 titled "FMC upgrade fails at 800_port/021_reinstall due to 520_install_rules.pl failure".  After the engineer performed the workaround described in the bug report he was able to resume and complete the software upgrade from version 6.6.1 to 6.6.5. This resolved our issues.

Contents of the bug report:

Symptom:
Backup generation on Firepower management center fails with the database error: VMS backup failed.

Conditions:
This is due to corrupt index under rule_opts table. The following errors are seen:

Validation /var/tmp/backup5EGe/0/vms/database/vms.db failed.
Validation /var/tmp/backup5EGe/0/vms/database/vms.db failed.
ERROR(1735): Database vms backup failed. Check the log file for information.

2132 VALIDATE TASQL error (-196) -- Index 'SID_GID_ORD' for table 'rule_opts' would not be unique

Workaround:
After connecting to the database using OmniQuery.pl:
sdb>Alter INDEX SID_GID_ORD on rule_opts rebuild;

This should be done only on the primary FMC incase of HA configuration.

[PETER AGENGO]

I had the same issue upgrading from 7.0.1 to 7.0.4 recently with error messages below.

OUT: [221014 00:01:45:116] BEGIN 800_post/020_702_fix_users_and_roles.pl
OUT: [221014 00:01:53:368] END 800_post/020_702_fix_users_and_roles.pl
OUT: [221014 00:01:53:499] BEGIN 800_post/021_reinstall_sru.sh
OUT: [221014 00:04:11:008] FAILED 800_post/021_reinstall_sru.sh
OUT: [221014 00:04:11:009] ====================================
OUT: [221014 00:04:11:010] tail -n 10 /var/log/sf/Cisco_Firepower_Mgmt_Center_Upgrade-7.0.4/800_post/021_reinstall_sru.sh.log
OUT:
OUT: called from /usr/local/sf/lib/perl/5.24.4/SF/SRUInstall.pm (1406)
OUT: called from /usr/local/sf/lib/perl/5.24.4/SF/SRUInstall.pm (338)
OUT: called from /usr/lib64/perl/site_perl/5.24.4/Error.pm (394)
OUT: called from /usr/local/sf/lib/perl/5.24.4/SF/SRUInstall.pm (341)
OUT: called from lib/Install.pm (1075)
OUT: called from installer/520_install_rules.pl (24)
OUT:
OUT: [221014 00:04:10] Fatal error: Error running script installer/520_install_rules.pl, RC=1
OUT: [221014 00:04:10] Exiting, 1.
OUT: SRU install was unsuccessful
OUT:
OUT: [221014 00:04:11:653] MAIN_UPGRADE_SCRIPT_END
OUT: [221014 00:04:11:654] Fatal error: Error running script 800_post/021_reinstall_sru.sh
OUT: [221014 00:04:11:668] Exiting.
OUT: removed '/tmp/upgrade.lock/PID'
OUT: removed '/tmp/upgrade.lock/AQ_UUID'
OUT: removed '/tmp/upgrade.lock/status_log'
OUT: removed '/tmp/upgrade.lock/main_upgrade_script.log'
OUT: removed '/tmp/upgrade.lock/LSM'
OUT: removed '/tmp/upgrade.lock/UUID'
OUT: removed '/tmp/upgrade.lock/UPGRADE'
OUT: removed directory '/tmp/upgrade.lock'
OUT: [221014 00:04:11:685] Attempting to remove upgrade lock
OUT: [221014 00:04:11:686] Success, removed upgrade lock
RC: 256
The update failed!

I used the steps mentioned above that Dave got from TAC. I have a HA pair and synchronization was paused during the upgrades. So even though the TAC steps say to do it on the Primary only I actually did it on the Secondary instance as well because it was not clearing for some reason. After performing the steps I was able to upgrade with no issue. The issue was apparently also causing our backups to fail recently. That has also been resolve by doing the steps provided by TAC to Dave.