FMC upgrade to 6.4.0 - FAILED 200_pre/007_check_sru_install.sh

[CraigR]

I have 2 FMC devices in a HA pair. They are currently running 6.1.0.6 and I am looking to update them to 6.4.0 before upgrading them further to the latest release.

The 'upgrade readiness check' does not work from the GUI even when the HA is in a paused state. I thereby then broke the HA completely and run the readiness check from the CLI via the command 'install_update.pl --detach --readiness-check /var/sf/updates/upgrade_package_name

However when looking at the logs, the readiness check fails at the check
FAILED 200_pre/007_check_sru_install.sh

I look at the log for this check at /var/log/sf/Cisco...Upgrade-6.4.0/upgrade_readiness/200_pre/007_check_sru_install.sh.log and it states:

Starting script: 200_pre/007_check_sru_install.sh

Entering 200_pre/007_check_sru_install.sh....

previous SRU install completed succesfully

Lists a heap of files'

removed '/tmp/sru_sort'

SRU to be reinstalled later in upgrade is missing from the /var/sf/SRU/ directory!

2021-10-27-001-vrt

Please download Sourcefire_rule_update-2021-10-27-001-vrt.sh and put it under /var/sf/SRU/ then perform upgrade again


When I check under the /var/sf/SRU/ folder the vrt.sh file is there alongside its .lsm and .info counterparts.... 


As a side note that may or may not have some relationship. But when we had our 2 FMC in HA, the SRU rule updates applied to the active FMC would never be synced to the standby for whatever reason. Once an update was applied to the active, the HA sync would stay up fine, but it would state SRU content does not match. Thought this might be of use.

[balaji.bandi]

Looks you follow the steps correctly, something wrong here - Hope you have enough disk space, (if yes)

contact TAC is good option before you do anything else and totally mess up things go worst.

(Hope you have backup ?)

[CraigR]

Noticed that all the file names in the /var/sf/SRU are lower case, where as the error in the log was requesting a SRU file with capitals in it.

I copied the SRU file with this 'capitalised' naming convention, and the readiness check then passes.....

Go figure. Hopefully the upgrade works succesfully based on this same logic and doesnt screw up.

No support on these devices anymore so TAC aint an option sadly

[balaji.bandi]

Oh that good trick, ye we need to spend more time to get to know what was the errors. if no support, no option than investigate our time.

some of the areas we do not get access to as much as we like. so we need to rely on TAC.

[CraigR]

So finally getting round to running the actual upgrader, and its failing here.
[220224 01:35:35:478] END 800_post/400_update_sfipproxy.pl
[220224 01:35:35:613] BEGIN 800_post/500_update_correlation_rules.pl
[220224 01:35:37:434] END 800_post/500_update_correlation_rules.pl
[220224 01:35:37:437] FAILED 800_post/500_update_correlation_rules.pl
[220224 01:35:37:438] ====================================
[220224 01:35:37:439] tail -n 10 /var/log/sf/Cisco_Firepower_Mgmt_Center_Upgrade-6.4.0/800_post/500_update_correlation_rules.pl.log
[220224 01:35:38:125] MAIN_UPGRADE_SCRIPT_END
[220224 01:35:38:126] Fatal error: Error running script 800_post/500_update_correlation_rules.pl
[220224 01:35:38:129] Exiting.
[220224 01:35:38:131] Attempting to remove upgrade lock
[220224 01:35:38:132] Success, removed upgrade lock


The actual error from the 500_update_correlation_rules.pl is as follows.

**********************************************************
[220224 01:35:35:617] Starting script: 800_post/500_update_correlation_rules.pl
Entering script: 800_post/500_update_correlation_rules.pl
Loading rules from file...
Storing updated rules...
DBD::SQLAnywhere::db prepare failed: Table 'rna_policy_rules' not found (DBD: prepare failed) at /usr/local/sf/lib/perl/5.10.1/SF/EODataHandler/ComplianceRule.pm line 118.
Store Failed\n at /usr/local/sf/lib/perl/5.10.1/SF/EOHandler.pm line 3437.

Can't call method "execute" on an undefined value at /usr/lib/perl5/site_perl/5.10.1/Error.pm line 273
Error::subs::run_clauses('HASH(0xa26a200)', 'Can\'t call method "execute" on an undefined value at /usr/lo...', undef, 'ARRAY(0x9c92ab8)')
called at /usr/lib/perl5/site_perl/5.10.1/Error.pm line 390
Error::subs::try('CODE(0xa4ca658)', 'HASH(0xa26a200)')
called at /usr/local/sf/lib/perl/5.10.1/SF/EOHandler.pm line 3443
SF::EOHandler::_storeObject('HASH(0x9188b30)', 'HASH(0x8dbe240)')
called at /usr/local/sf/lib/perl/5.10.1/SF/EOHandler.pm line 1757
SF::EOHandler::storeObject('HASH(0x9188b30)', 'HASH(0x8dbe240)')
called at 800_post/500_update_correlation_rules.pl line 29
Error saving correlation rules: Can't call method "execute" on an undefined value
500_update_correlation_rules.pl.log (END)

Appears the error is around a SQL DB missing a table called 'rna_policy_rules'.....

Any ideas on a workaround or such?

[Marvin Rhoads]

When there are database issues, TAC will typically run DBCheck.pl as root user and take appropriate action based on the outcome there. It is not recommended to try your own work around in such cases - use the TAC and their much more extensive knowledge base.

[CraigR]

Thanks Marvin,

Running that command shows 182 fatal errors, with a bunch of missing tables and fields.

Hopefully TAC can help, or possibly we can get the 6.4.0 Restore ISO and just rebuild this one from scratch