Re: FMC virtual upgrade failure

GaeMi · ‎05-06-2023

Today, I tried to upgrade my FMC 6.2.2.5 to 6.2.3 and I 800_post/880_install_VDB.sh error messages.

I didn't want to upgrade VDB, but I guess there was VDB upgrade process in FMC upgrade process.

It was might caused because I uploaded VDB 364 files before try to upgrade 6.2.3.

But I want to upgrade only FMC version to 6.2.3, not VDB.

And I captured VM snapshot and rebooted the FMC and retried it using command :

root@seatech:/var/sf/updates# upgrade_resume.sh

but I failed and got a error messsages 800_post/800_sensor_pull_vdb.pl.

At the moment, I can't access or control FMC policies and stuck in upgrade failure message.

I can't get a TAC support now. Can you give me some advices?

=== Error Messages ===

**********************************************************

[230506 02:23:41:937] Starting script: 800_post/880_install_VDB.sh

IS_MC = 1, IS_SENSOR = 0, IS_RNANS = 0, IS_RNA = 0

IS_RNASW = 0, IS_GIGABIT_SENSOR = 0

Entering 800_post/880_install_VDB.sh...

Checking Status of Mysql DB.

Number of Mysqld processes running: 1 Process ID = 20455

Checking Status of Sybase DB.

Number of Sybase DB processes running: 1 Process ID = 27415

'/var/sf/updates/Sourcefire_VDB_Fingerprint_Database-4.5.0-364.sh' -> '/var/sf/vdb/Sourcefire_VDB_Fingerprint_Database-4.5.0-364.sh'

Chose this fingerprint DB (probably delivered by this upgrade):

/var/sf/vdb/Sourcefire_VDB_Fingerprint_Database-4.5.0-364.sh

Verifying archive integrity... All good.

Uncompressing Sourcefire Vulnerability And Fingerprint Database Updates...................................................................

[230506 02:23:44] Lock //tmp/vdb.lock for install created successfully

[230506 02:23:44]

Authority UUID =

AQ UUID =

[230506 02:23:45] Using root directory

[230506 02:23:45] #####################

[230506 02:23:45] # UPGRADE STARTING

[230506 02:23:45] #####################

[230506 02:23:45]

[230506 02:23:45] BEGIN pre/000_start.sh

[230506 02:23:45] COMPLETED pre/000_start.sh

[230506 02:23:45]

[230506 02:23:45] BEGIN pre/005_check_low_end.pl

I have 1 from pre/005_check_low_end.pl

[230506 02:23:45] FAILED pre/005_check_low_end.pl

[230506 02:23:45] ====================================

[230506 02:23:45] tail -n 10 //var/log/sf/vdb-4.5.0-364/pre/005_check_low_end.pl.log

**********************************************************

[230506 02:23:45] Starting script: pre/005_check_low_end.pl

Current build and existing build:333,364 at pre/005_check_low_end.pl line 14.

Checking low-end devices compatibility: at pre/005_check_low_end.pl line 31.

/var/sf/updates/*_VDB_Fingerprint_Database-4.5.0-364*sh*

removing /var/sf/updates/Sourcefire_VDB_Fingerprint_Database-4.5.0-364.sh

deleted 1 VDB files because they are not compatible with this device in its current state. at pre/005_check_low_end.pl line 73.

VDB install cancelled: insufficient device memory. At least one of your managed devices or for device manager, this device cannot install the full VDB. Before you install VDB 363+, upgrade the management center or device manager. This allows you to install a smaller VDB package on lower memory devices. For more information, see the VDB release notes:'<a href=https://appid.cisco.com/relnotes></a>' at pre/005_check_low_end.pl line 75.

[230506 02:23:45] Fatal error: Error running script pre/005_check_low_end.pl

[230506 02:23:45] Exiting.

VDB Installation failed.

**********************************************************

[230506 03:03:40:343] Starting script: 800_post/880_install_VDB.sh

IS_MC = 1, IS_SENSOR = 0, IS_RNANS = 0, IS_RNA = 0

IS_RNASW = 0, IS_GIGABIT_SENSOR = 0

Entering 800_post/880_install_VDB.sh...

Checking Status of Mysql DB.

Number of Mysqld processes running: 1 Process ID = 25561

Checking Status of Sybase DB.

Number of Sybase DB processes running: 1 Process ID = 27415

cp: cannot stat '/var/sf/updates/Sourcefire_VDB_Fingerprint_Database-*.sh': No such file or directory

Chose this fingerprint DB (probably delivered by this upgrade):

/var/sf/vdb/Sourcefire_VDB_Fingerprint_Database-4.5.0-364.sh

Verifying archive integrity... All good.

Uncompressing Sourcefire Vulnerability And Fingerprint Database Updates...................................................................

[230506 03:03:44] Lock //tmp/vdb.lock for install created successfully

[230506 03:03:44]

Authority UUID =

AQ UUID =

[230506 03:03:44] Using root directory

[230506 03:03:44] #####################

[230506 03:03:44] # UPGRADE STARTING

[230506 03:03:44] #####################

[230506 03:03:44]

[230506 03:03:44] BEGIN pre/000_start.sh

[230506 03:03:44] COMPLETED pre/000_start.sh

[230506 03:03:44]

[230506 03:03:44] ** enabling SCRIPT_RECOVERY_MODE for pre/005_check_low_end.pl

[230506 03:03:44] BEGIN pre/005_check_low_end.pl

I have 1 from pre/005_check_low_end.pl

[230506 03:03:45] FAILED pre/005_check_low_end.pl

[230506 03:03:45] ====================================

[230506 03:03:45] tail -n 10 //var/log/sf/vdb-4.5.0-364/pre/005_check_low_end.pl.log

**********************************************************

[230506 03:03:44] Starting script: pre/005_check_low_end.pl

Current build and existing build:333,364 at pre/005_check_low_end.pl line 14.

Checking low-end devices compatibility: at pre/005_check_low_end.pl line 31.

/var/sf/updates/*_VDB_Fingerprint_Database-4.5.0-364*sh*

removing

deleted 0 VDB files because they are not compatible with this device in its current state. at pre/005_check_low_end.pl line 73.

VDB install cancelled: insufficient device memory. At least one of your managed devices or for device manager, this device cannot install the full VDB. Before you install VDB 363+, upgrade the management center or device manager. This allows you to install a smaller VDB package on lower memory devices. For more information, see the VDB release notes:'<a href=https://appid.cisco.com/relnotes></a>' at pre/005_check_low_end.pl line 75.

[230506 03:03:45] Fatal error: Error running script pre/005_check_low_end.pl

[230506 03:03:45] Exiting.

VDB Installation failed.

=== Error Messages 2 (after reboot) ===

**********************************************************

[230506 02:23:30:769] Starting script: 800_post/800_sensor_pull_vdb.pl

entering script

This script only runs on Sensors, done.

**********************************************************

[230506 03:03:36:797] Starting script: 800_post/800_sensor_pull_vdb.pl

entering script

This script only runs on Sensors, done.

**********************************************************

[230506 04:17:26:073] Starting script: 800_post/800_sensor_pull_vdb.pl

entering script

This script only runs on Sensors, done.

**********************************************************

[230506 05:00:19:583] Starting script: 800_post/800_sensor_pull_vdb.pl

entering script

This script only runs on Sensors, done.

**********************************************************

[230506 05:50:38:880] Starting script: 800_post/800_sensor_pull_vdb.pl

Can't locate strict.pm in @INC (@INC contains: /usr/lib/perl5/5.10.1/i386-linux /usr/lib/perl5/5.10.1 /usr/lib/perl5/site_perl/5.10.1/i386-linux /usr/lib/perl5/site_perl/5.10.1 /usr/local/sf/lib /usr/local/sf/lib/perl/5.10.1 /usr/local/sf/lib/perl/5.10.1/i386-linux .) at 800_post/800_sensor_pull_vdb.pl line 15.

BEGIN failed--compilation aborted at 800_post/800_sensor_pull_vdb.pl line 1

MHM Cisco World · ‎05-06-2023

https://bst.cisco.com/bugsearch/bug/CSCwd55058

GaeMi · ‎05-07-2023

Hello.

I checked your link and I tried to install VDB 364 / 362 / 333 in CLI and I failed in 364 / 362. Therefore, I reinstalled to 333. (333 is my original VDB before start 6.2.3 upgrade process)

And I tried to upgrade process, but I still got 800_sensor_pull_vdb.pl error.

[230507 01:34:04:704] Starting script: 800_post/800_sensor_pull_vdb.pl

Can't locate strict.pm in @INC (@INC contains: /usr/lib/perl5/5.10.1/i386-linux /usr/lib/perl5/5.10.1 /usr/lib/perl5/site_perl/5.10.1/i386-linux /usr/lib/perl5/site_perl/5.10.1 /usr/local/sf/lib /usr/local/sf/lib/perl/5.10.1 /usr/local/sf/lib/perl/5.10.1/i386-linux .) at 800_post/800_sensor_pull_vdb.pl line 15.

BEGIN failed--compilation aborted at 800_post/800_sensor_pull_vdb.pl line 15.

In addition, I stucked upgrade failure messages in FMC GUI, so I wasn't able to access FMC GUI.

I found out that when FMC is rebooted I can access FMC GUI about 2~3 minutes before stucking in upgrade failure GUI.

I found the some process let me redirect to upgrade failure GUI using in /var/sf/upgrade-ui/http_server/httpsd resources.

So I tried to move the httpsd folder temporaily.

root@FMC:/var/sf/upgrade-ui/http_server# mv httpsd /Volume/home/admin/

After mv command, I wasn't able to access FMC GUI, it was rejected.

Foutunately I was able to access FMC GUI after I reboot the FMC. (also I saw the FMC console log about missing /var/sf/upgrade-ui/http_server/httpsd after reboot FMC)

I can change FMC rule and deploy successfully, so I think it is useful for escaping upgrade failure GUI. (I don't recommend it of course, always TAC support is much better than my command)

I retried to readiness check and FMC upgrade to 6.2.3 in GUI twice. Readiness check was success.

However, I also still have an 800_sensor_pull_vdb.pl errors.

My FMC still in upgrade failure status, but I can operate FMC, so I think it isn't serious condition at the moment.

Can I get some useful advices?

Thank you.

brettius · ‎05-08-2023

I have the same issue, and the same VDB is causing the fuss. Seems I am unable to update from VDB 362 to 363 or 364 and because of this the upgrade is failing every time I try to do it. I am looking into it.

brettius · ‎05-08-2023

VDB install cancelled: insufficient device memory. At least one of your managed devices or for device manager, this device cannot install the full VDB. Before you install VDB 363+, upgrade the management center or device manager. This allows you to install a smaller VDB package on lower memory devices. For more information, see the VDB release notes:'<a href=https://appid.cisco.com/relnotes></a>' at pre/005_check_low_end.pl line 75.

Seems they have a lightweight version of the VDB but you have to contact TAC to get the alternate file.