FMC upgrade failed, VM restore failed, next steps?

stckaye — Thu, 18 Aug 2022 16:38:29 GMT

Good morning, I have two FTP 2110s in a HA pair managed by FMC. I am using the KVM version of the virtual FMC software. My upgrade from 6.6.5.2 to 7.0.4 failed (although the Readiness Check ran successfully), the VM became unresponsive. When I restored the VM file from just before the upgrade attempt it follows a consistent path (same thing three tries in a row so far):

The VM starts, no errors. I can ping and log in to the cli. I get the message "You have logged in while system startup is in progress. Please wait, some feature may be unavailable until startup is complete." At this time, the FMC web interface shows the "System processes are starting" animation. I can interact somewhat with the cli but I get a lot of errors of the following type:

System> show version
Failed get_sybase_password $VAR1 = bless( {
'-stacktrace' => 'System (/var/jre/bin/java -cp /var/opt/CSCOpx/MDC/tomcat/shared/lib/cs-lib-4.2.2.b.jar:/var/sf/lib64/:/lib64/:.: DBP) Failed at /usr/local/sf/lib/perl/5.10.1/SF/System/Privileged.pm line 5811
SF::System::Privileged::DecryptSybasePassword() called at /usr/local/sf/lib/perl/5.10.1/SF/SFDBI.pm line 138
SF::SFDBI::get_sybase_password() called at /usr/local/sf/lib/perl/5.10.1/SF/SFDBI.pm line 374
SF::SFDBI::connect() called at /usr/local/sf/lib/perl/5.10.1/SF/EODataHandler/Default.pm line 127
SF::EODataHandler::Default::bulkLoad(\'DE_Config\', undef, undef) called at /usr/local/sf/lib/perl/5.10.1/SF/EODataHandler/DetectionEngine.pm line 21
SF::EODataHandler::DetectionEngine::AUTOLOAD(\'DE_Config\', undef, undef) called at /usr/local/sf/lib/perl/5.10.1/SF/EOHandler.pm line 2466
SF::EOHandler::bulkLoad(\'DE_Config\') called at /usr/local/sf/lib/perl/5.10.1/SF/DetectionEngine.pm line 5936
SF::DetectionEngine::getPrimaryDetectionEngineUUID() called at /usr/local/sf/lib/perl/5.10.1/SF/CLI/version.pm line 120
SF::CLI::version::getSRUVersion() called at /usr/local/sf/lib/perl/5.10.1/SF/CLI/version.pm line 27
SF::CLI::version::show() called at /usr/local/sf/lib/perl/5.10.1/SF/CLI.pm line 88
SF::CLI::process_command(\'show\', \'version\') called at /usr/local/sf/bin/sfcli.pl line 7
',
'-stdout' => 'Error occurred during initialization of VM
java/lang/ClassNotFoundException: error in opening JAR file /var/jre/lib/rt.jar
',
'-line' => 5811,
'-file' => '/usr/local/sf/lib/perl/5.10.1/SF/System/Privileged.pm',
'-value' => 256,
'-text' => 'System (/var/jre/bin/java -cp /var/opt/CSCOpx/MDC/tomcat/shared/lib/cs-lib-4.2.2.b.jar:/var/sf/lib64/:/lib64/:.: DBP) Failed',
'-stderr' => '',
'-ui_message' => '',
'-package' => 'SF::System::Privileged'
}, 'Error::SFSystem' );
DBI connect('DBN=vms;Host=127.0.0.1:10033;CharSet=utf8','DBA',...) failed: Invalid user ID or password (DBD: login failed) at /usr/local/sf/lib/perl/5.10.1/SF/SFDBI.pm line 375
$VAR1 = [
'Connection failed',
bless( {
'-stacktrace' => 'Can\'t call method "prepare" on an undefined value at /usr/lib/perl5/site_perl/5.10.1/Error.pm line 273
Error::subs::run_clauses(\'HASH(0x43af680)\', \'Can\\\'t call method "prepare" on an undefined value at /usr/lo...\', undef, \'ARRAY(0x436dc48)\') called at /usr/lib/perl5/site_perl/5.10.1/Error.pm line 390
Error::subs::try(\'CODE(0x43b5b58)\', \'HASH(0x43af680)\') called at /usr/local/sf/lib/perl/5.10.1/SF/SFDBI.pm line 401
SF::SFDBI::connect() called at /usr/local/sf/lib/perl/5.10.1/SF/EODataHandler/Default.pm line 127
SF::EODataHandler::Default::bulkLoad(\'DE_Config\', undef, undef) called at /usr/local/sf/lib/perl/5.10.1/SF/EODataHandler/DetectionEngine.pm line 21
SF::EODataHandler::DetectionEngine::AUTOLOAD(\'DE_Config\', undef, undef) called at /usr/local/sf/lib/perl/5.10.1/SF/EOHandler.pm line 2466
SF::EOHandler::bulkLoad(\'DE_Config\') called at /usr/local/sf/lib/perl/5.10.1/SF/DetectionEngine.pm line 5936
SF::DetectionEngine::getPrimaryDetectionEngineUUID() called at /usr/local/sf/lib/perl/5.10.1/SF/CLI/version.pm line 120
SF::CLI::version::getSRUVersion() called at /usr/local/sf/lib/perl/5.10.1/SF/CLI/version.pm line 27
SF::CLI::version::show() called at /usr/local/sf/lib/perl/5.10.1/SF/CLI.pm line 88
SF::CLI::process_command(\'show\', \'version\') called at /usr/local/sf/bin/sfcli.pl line 7
',
'-file' => '/usr/local/sf/lib/perl/5.10.1/SF/SFDBI.pm',
'-text' => 'Can\'t call method "prepare" on an undefined value',
'-line' => '379',
'-package' => 'Error::subs'
}, 'Error::Simple' )
];

etc. After some time, both the cli and the website become unresponsive although the VM still answers pings.

Cisco TAC is looking to see if my backup VM is irretrievably corrupted. I am guessing given the above that the answer will be yes. If I set up a brand new FMC VM, will I be able to retrieve the config from the currently working firewalls, or is my only way forward a manual step by step rebuild? I can still access the cli on the firewalls. Thanks in advance for any advice or guidance.

Re: FMC upgrade failed, VM restore failed, next steps?

Marvin Rhoads — Fri, 19 Aug 2022 03:02:08 GMT

Besides the backup of the VM, did you have any local FMC application level backup? Normally they would be stored under /var/sf/backup.

Unfortunately, running device configurations cannot be restored/exported into an FMC

Re: FMC upgrade failed, VM restore failed, next steps?

stckaye — Fri, 19 Aug 2022 13:16:52 GMT

Thanks for responding! I do swear I ran the application level backup but /var/sf/backup is empty. I dug back further in our file system and pulled out a VM backup from April that works, we'll have a bit of rebuild to do but that's much better than rebuild from scratch. I now have about 5 different backups in different formats.

My outstanding question at this point is why the upgrade to 7.0.4 (which I have to face again soon) failed so badly, although the Readiness Check was positive. Do you have any insight? Thank you.

Re: FMC upgrade failed, VM restore failed, next steps?

Marvin Rhoads — Fri, 19 Aug 2022 15:55:13 GMT

The point where the upgrade failed can usually be seen via looking at the status.log file within /var/log/sf/<folder for 7.0.4 upgrade>. You can drill down into more detailed log files within subfolders based on what you see in status.log. However it usually requires TAC analysis and intervention to figure out what to do about whatever is shown there.

topic FMC upgrade failed, VM restore failed, next steps? in Network Security

FMC upgrade failed, VM restore failed, next steps?

Re: FMC upgrade failed, VM restore failed, next steps?

Re: FMC upgrade failed, VM restore failed, next steps?

Re: FMC upgrade failed, VM restore failed, next steps?