Failover fails (turns off) after License Upgrade for 5520s.

[Robert Claxton]

Hi-

I recently did a license upgrade on an active/standy pair of 5520's (8.04) for a 2 to 50 SSL VPN upgrade. According to Cisco Document #70390 it should have been a no brainer...but it wasn't. According to the doc upgrading the activation key of the primary then doing a wr mem along with a physical shutdown causes the secondary to become active. But in my experience as soon as the new activation key was entered failover broke. This caused the secondary to remain inactive and I had to reboot both to get the network back up (of course only one could be active and so I took the secondary offline so I could upgrade it via serial). So I brought the secondary up (after activation key upgrade) and failover did not kickin. I did a sh failover on both machines and both said failover was off. I would do the failover command on the primary and secondary...nothing. It wasn't until I did a no failover active on the secondary that failover finally started to work properly.

Anyone else have this experience? I have a ticket open with Cisco for the Doc # 70390 for clarification via my experiences. Any input would be appreciated on this. I was working on the assumption that I would have 2 to 3 minute outage as the primary rebooted but turned out to be a longer outage than anticipated and higher ups were pissed...

Thxs. --Rob.

[Kureli Sankar]

Sorry to hear that.  I believe failover will be disabled when there is license mismatch between the two units. Which is what you have seen.

I read the document that you indicated and it does indicate it should have worked. It also indicates it only supports the following

• Cisco PIX 515, 515E, 525, and 535 Security Appliances with 7.x and later version

Also, according to this link below it clearly says that license requirement should be the same on both units for failover to work.

http://www.cisco.com/en/US/docs/security/asa/asa80/configuration/guide/failover.html#wp1053685

The licensed features (such as SSL VPN peers or security contexts, for example) on both security appliances participating in failover must be identical.

Unless we try it in the lab it is very hard to say what the exact behavior will be.

-KS

[Robert Claxton]

Yes, I realize that the licenses need to be identical, hence the question on the procedure. Curious for the documentation that it says "Pix/ASA " in the title for the procedure along with being in the Configuration Examples and Technotes for the ASA5500 Series. All I'm asking is for someone to review the procedure and edit the document as it is obviously incorrect for the ASA. I just don't want others to get burned like I did on it.

--Rob.

[jwilder]

Hi Rob, i am getting ready to upgrade my ASA5520's in Active/Stanby mode with new vpn licenses also. however i am upgrading them to anyconnect essentials. Will this license upgrade break my failover?

Thanks,

Jeff

[Robert Claxton]

Any upgrade to the licenses of one firewall (making them temporarily different will cause failover to turn off. If I did this again this is what I would do:

1. First turn off failover by issuing the no failover command on the active and wr mem.
2. This will disengage failover and the states of the boxes (active and standy will still be the same) until reboot.
3. On the active do the license upgrade and wr mem.
4. Serial into the standby and do the license upgrade, wr mem.
   
5. Power off the standby and reboot the active (This will be your network down time as the active reboots).
6. With the active working turn failover back on with the command failover active.
7. Turn the standby unit on. It should detect the hello message from the active and go into standy mode automatically (If the licenses match).

That should be it.  I would let the powers that be that there'll be an outage and its simply unavoidable. Give yourself plenty of time, go through the serial interfaces to observe syslog messages and...good luck. Let me know how it goes.

--Rob.

[jwilder]

Hi Rob, Thanks for the response. Unfortunately I was getting pressure to get the upgrade done so

I went through the old process and had the exact same problem you did. Fortunately you had already posted the solution in your first post so that worked for me also.

Having now gone through the process however, I would concur with your findings. The process you have described should work correctly and Cisco definitely needs to update their documentation for this process on the ASA.

Thanks again for your help!

Jeff

[Kureli Sankar]

I will take care of this issue.  I will get this document corrected.

Thanks for bringing it to our attention.

-KS

[jwilder]

Thank you!

[Kureli Sankar]

This cco document 70390 has been corrected.

Sorry it took so long.

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a00806b1c0f.shtml

-KS

[tonymitchell]

Hi Rob,

Firstly - thanks for posting all your findings with this issue, otherwise I would have followed the Cisco doc too!

I just have a couple of queries though... after failover is disabled and the ASA's remain in their current state, rather than use the console port to perform the license upgrade, would the "standby" ASA still be assigned the 'standby' IP address, even though failover is disabled? For example, the active had the inside ip address 1.1.1.1 and the standby was assigned 1.1.1.2 on its inside interface. Would this configuration state be the same when failvoer was disabled?

The reason for asking, is whether the upgrade could be performed remotely (whether this is a sensible idea is another thing!). So could I connect to the "standby" unit by IP to upgrade the license?

The second query is regarding the reboot of both units (as opposed to shutting down). Following the license upgrade on both units, could you just reboot both units ("standby" then "active"), and when they both come back up, enable failover? I would assume the previous "active" unit would again become active and failover restored??

Thanks,

Tony

[Panos Kampanakis]

For the first question, when they disabled failover the secondary should be in pseudo standby. state. If that is the case it should be using the stanby ip addresses and thus you will be able to what you want.

For the second question, again the secondary when coming up if it doesn't detect a mate it will stay pseudo standby so it won't be passing traffic. So if you establish failover then you can do the zero downtime upgrade, but if not then you won't avoid downtime.

I hope it helps.

PK

[tonymitchell]

Thanks PK for your quick post.

Would you know if Cisco have any plans release new firmware that would allow customers to perform license upgrades with zero downtime, and not break failover (like firmware updates)?

This would be very useful.

Thanks,

Tony

[Panos Kampanakis]

There are improvements coming soon.

Undisclosable for now.

PK

[Farooq Razzaque]

Dear Pkampana

I want to apply license to increase security context in FWSM which is running in Active-Active mode on VSS Core switches

As per below document, first we need to disable failover by entering 'no failover' command on active FWSM and then apply the license seperately on both FWSM.

I just want to know when i will disable the failover then standby move to pseudo-standby state. 

Will there be any services impact which are running behind the FWSM when disbaling the failover and then re-enabling the failover.

http://www.cisco.com/c/en/us/td/docs/security/fwsm/fwsm40/configuration/guide/fwsm_cfg/swcnfg_f.html#wp1073226

Appreciate your response

[Robert Claxton]

Hi Tony-

For your first question yes, the standby address should still be

active after turning off failover as both states are "frozen" to

avoid ip address conflict. As far as trying to do this remotely I

would simply say not advised w/ current software bugs for upgrading

licenses. If you have remote access via console port then it would be

ok.

For your second question rebooting would be ok. However do not do them

both at the sametime! This will cause both firewalls to be active and

ip conflicts abound. I will post a sequence I would recommend when I

get to an actual computer. Zero downtime in my opinion is not

achievable w/ current software. So it's important to communicate this

to higher ups in preparation. By pool on vaca now...

--Rob Claxton

Mobile:

On Feb 16, 2010, at 5:58, tonymitchell