FMC Upgrade Failed from 6.4.0 to 6.6.4 (.. while running 999_finish/989_flip_mbr.sh)

[AndreaQuerci]

in EVE-NG I simulated a simple network environment with an FMC (6.4.0-113) and two FTD in cluster (6.4.0-102). I also added two PCs in order to test the security policies and the traffic works as I expected:

I created this LAB in order to replicate an upgrade activity that I have to implement for a customer that uses the same firepower releases. mainly I need to check how to upgrade the FTD cluster without create any kind of disservice, .. so I started to upgrade the FMC using the file "Cisco_Firepower_Mgmt_Center_Upgrade-6.6.4-59.sh.REL" (the advised release) but the upgrade process gets an issue:

below is the log:

**********************************************************
[210608 18:36:59:109] Starting script: 999_finish/989_flip_mbr.sh
Entering 999_finish/989_flip_mbr.sh...

Fire Linux OS 6.6.4-3
Update Lilo, wth dir /
Arch is x86_64
Found device type UNKNOWN.
No device config variables file found for this device ( /usr/local/etc/devcfg.variables.UNKNOWN doesn't exist) 
1000:0030
1000:0054
1000:0056
1000:0058
The CPU_VENDOR is GenuineIntel, the family is 6
The machine id 8086:1237 and the SCSI_DEVICE is 1af4:1001
The arch is x86_64
The kernel version is 4.14.187-sf.core264
enter run_lilo_and_depmod()

  ### RUNNING LILO

LILO version 24.2 (released 22-November-2015)
  * Copyright (C) 1992-1998 Werner Almesberger  (until v20)
  * Copyright (C) 1999-2007 John Coffman  (until v22)
  * Copyright (C) 2009-2015 Joachim Wiedorn  (since v23)
This program comes with ABSOLUTELY NO WARRANTY. This is free software 
distributed under the BSD License (3-clause). Details can be found in 
the file COPYING, which is distributed with this software.

Warning: Ignoring entry 'default'
Fatal: raid_setup: stat("e;/dev/sda"e;)
Fatal error: Something went wrong running lilo in the chroot (/new-root)
Exit return value = 1

after this issue, I can't recover the FMC anymore! even if I restart it, after the login it is still stuck with the same error message that you saw in the screenshot.

how can I solve this issue? what is the script "989_flip_mbr.sh"? I'm scared that the same issue can happen in production during the activity. I hope no.

p.s.:

in my LAB, the FMC has 4 vCPU and 32GB of RAM.

[AigarsK]

Hi Andrea,

Wonder if you ever managed to locate what was the cause for upgrade failure?

I in similar manner am running a Lab and encountered same Upgrade Failure. I am upgrading 6.6.1 to 7.0.1

[cdukeslogzilla]

I tried the following, but after reboot, it still has errors saying the upgrade failed.

If anyone gets any further, please let us know.

1. Log into the console, go to expert mode and sudo to root

2. run `chroot /new_root`

3. edit /etc/lilo.conf and change `disk=/dev/sda` and also add a new line:

expert
sudo su -
# use the same password you used used to log in for ssh
vi /etc/lilo.conf 
boot=/dev/vda
disk=/dev/vda bios=0x80 max-partitions=7
# to save and exit vi, type :wq

Reboot, pray

[AigarsK]

Hi All who might encounter this.

I believe the issue here is with system variables. In all fairness, I doubt this will be ever fixed by Cisco as we are technically running FTD in unsupported environment.

Issue appears to be with 989_flip_mbr.sh script which is run at the end of upgrade process which is meant to set new boot disk and update boot loader description.

This issue is most likely caused by us running EVE-NG on VMware ESXi where the FTD is deployed as KVM image is EVE-NG nested virtualization. This causes checks on variables to fail to identify the correct script variables as for disk name, like vda vs sda as there are multiple checks scrips do to determine the platform it is deployed on, and I believe it gets some inputs to state that FTD is running on VMware.

I am sure some experienced Linux chap is able to fix this if they are familiar with LILO (boot loader), this would require replacing script or variables in temp folder where all upgrade scrips are extracted to before they are called by main upgrade process