Zero Downtime Upgrade to 8.4(5) Does NOT work

[thomas.rydzewski]

I'm working this in a lab for now, but I am attempting to upgrade from 8.2(5.26) to 8.4(5) on a failover pair of 5510s and have encountered an issue with the zero downtime upgrade process.

The whole thing appears to be very simple.  Upload the images to both firewalls, change the boot statement on the primary and issue a "failover reload-standby" from the primary/active ASA.

I've done this over a dozen times and each time, the secondary/standby ASA reboots to the original 8.2(5.26) version.

So, I thought about the major/minor version thing, and thought I'd upgrade from 8.2(5.26) to 8.3(2.37).  This process worked as advertised - so that must be it.  Right? Wrong!!!

Then I attempted to upgrade from 8.3(2.37) to 8.4(5) using the same procedures and the secondary ASA reboots each time to 8.3(2.37).

I thought I'd cheat a little, since it was a lab, and I could recover.  I issued the reload command directly on the secondary/standby firewall, and it booted up to 8.4(5).

I checked Cisco's site, but have not seen anything that indicates that this is an issue.  Any assistance with this would be greatly appreciated.

Thanks,

Tom

[Jouni Forss]

Hi,

Just thought I'd confirm this since you dont specifically mention doing it.

Have you saved the configuration either on the Primary/Active unit OR directly on the Secondary/Standby unit before you reload the Secondary unit from within the commandline of the Primary ASA?

That might be one reason the Secondary unit would still boot with the wrong software.

Though I'd imagine this is not the problem.

- Jouni

[thomas.rydzewski]

Yes.  I issued a write mem command after changing the boot statement.  After it didn't work, I also issued a write standby command, and then somewhere in the dozen or so attempts at upgrading, I logged directly into the secondary/standby ASA and issued a write mem command from there.

All with the same results.

[Jouni Forss]

Hmm..strange...

Is it possible that you define the new software file with "boot system flash:/image.bin" BUT dont remove the old "boot system flash:/image.bin" command which leads to the fact that the ASA uses the old image file?

Since the "boot system flash:/image.bin" doesnt overwrite the old one. Just adds a new line.

- Jouni

[thomas.rydzewski]

Nothing is impossible when human fingers are involved, but in this case, I was sure to delete the old boot statement - and I even verified this fact after the upgrade failed the first couple of times.

As mentioned before, the upgrade from 8.2(5.26) to 8.3(2.37) was completely successful.

boot system disk0:/asa845-k8.bin
no boot system disk0:/asa832-37-k8.bin

[Jouni Forss]

Im beginning to run out of ideas atleast

Have you been able to boot any device alone to that software?

Is there some possibility that the image file used is somehow corrupted/damaged. (Though I am not sure if ASA would in that case automatically ignore the command)

Maybe it tries to boot it but falls back to the existing image files when the original fails.

Have you tried loading the 8.4(5) image file again to the ASA?

I think you can verify the image file on the command line with the command "verify flash:/image.bin"

You can then compare the provided hash to the hash on the Cisco site where you download the software.

Again I am just guessing because it does seem so wierd.

- Jouni

[thomas.rydzewski]

I appreciate your imput on this - and since I posted this, I too have ran short of ideas (which is why I posted it in the first place)

I have performed the upgrade/downgrade process on the same ASAs about a dozen times in an effort to verify different configurations (we support in excess of 250 ASAs that we will be upgrading this year).  This was the first failover configuration that I have attempted the upgrade.  As I mentioned in the original post, the upgrade was successful when I issued a reboot command directly from the secondary/standby ASA. 

Definately not the image.

Since this is a lab, I was able to console directly into the secondary/standby ASA as it boots up, and the following  messages appear as it boots:

Launching BootLoader...
Boot configuration file contains 1 entry.

Loading disk0:/asa832-37-k8.bin... Booting...
Platform ASA5510

When it finishes booting, the output of a show run boot command shows:

MYPOSASA/pri/act(config)# sh run boot

boot system disk0:/asa845-k8.bin

I was concerned that the firewall was attempting to load a different config file, so I deleted everything from flash that wasn't an image and tried it again with the same results. 

So three images:

asa845-k8.bin

asa832-37-k8.bin

asa825-26-k8.bin

Both firewalls will boot to any of the above images in a standalone ASA configuration.

In an active/standby failover configuration - following Cisco's instructions - issuing the "failover reload-standby" command works to boot to the following images:

asa832-37-k8.bin

asa825-26-k8.bin

However, in an active/standby failover configuration - following Cisco's instructions - issuing the "failover reload-standby" command DOES NOT WORK with asa845-k8.bin.  Issuing a "reboot" command on either standby or the active firewall, with the asa845-k8.bin image as the ONLY boot statement successfully boots the ASA that the command is issued on to the 8.4(5) image, as expected.

I have not tested any other image(s), but my findings indicate an issue specific to version 8.4(5).

Sorry that this is so long-winded, but I just wanted to be sure everything is included.

Tom

[Jouni Forss]

Ah,

Sorry I completely managed to miss that you said it worked while booting directly from the unit to be rebooted

Sadly I dont have a pair of ASA at hand at the moment to test this out myself.

Personally when upgrading failover pairs I am always connecting to the host to be upgraded directly and not doing it through the Active device so if this is related to such an situation only I would probably never run into it myself.

If I do get the change to set this up at some point, I will let you know if I get the same issue.

I didnt find any mention of such a bug, atleast in the section stating the corrected bugs in the Release Notes for software 8.4(5)6 which to my understanding should be the most up to date 8 series software

- Jouni