ASA 5525 failover key failure

[netspazz]

I have a very strange issue. I have several firewall pairs, each running active/standby. I am in the process of upgrading firmware to 9.7(1)21. Once both firewalls have been rebooted they both in the active state because they loose failover connection. What I do is "no failover key 8 ####" (### is masking the real hash number) Once the key is removed then both firewalls communicate and the standby will go into its normal standby mode.  A sh failover will now show them up and communicating. If I issue the "failover key 8 ####" on the active then it is pushed over to the standby, but within a minute or so communication is stopped and the standby will go active again. The sh failover shows the other end failed. Only way to get them to communicate is without the failover key installed. This was working before the upgrade. Do I need to do something?

thanks

[Marius Gunnerud]

have you tried entering the command with an unencrypted key?

failover key 0 PRIVATEKEY

If this works it might be that that something happened to the encryption of the key during the upgrade.

[netspazz]

I have just entered the key with the (0) and is seems to be staying up for the moment. Would this mean I need to generate a new key or fix something else that reads the correct key? Is there a way to keep the current key?

[Marius Gunnerud]

Well it seems like during the upgrade the failover keys are migrated differently on each of the ASAs.  What I mean is that the encryption or the way they are encrypted seems to have been affected during the upgrade.  

As for keeping the current key I can only see two possibilities.  If you know the key enter it using the un encrypted format. Or you could try to copy the encrypted format from ASA1 to ASA2.

[netspazz]

Ok, this does make sense as the config files may have been copy and pasted. As far as the key values, how do I generate new shared keys for both sides? I have run the crypto keygen command for generating ssh keys, but this is different correct?

thanks

[Marius Gunnerud]

The failover key is not a generated key, it is a preshared key.  So you would just need to decide on a preshared key and then enter it into the failover key configuration on both ASAs.  It is worth mentioning that the failover key is legacy now and that failover ipsec pre-shared-key command is the prefered way to configure this now.

[netspazz]

This is where I am confused. I just tested 2 pairs of ASA's with the same upgraded firmware. I set both pairs to have "failover key 0 <random key>". I did this on the primary/active, this pushed the key over to the standby. I did a "write standby" just to make sure. All is well on both pairs.

Now stage 2 was to enable the encryption. I changed the failover key entry to "failover key 8 <random key>" on the primary. Again, I did a write standby to make sure. On the first pair it looked to stay up and I see the "key 8" on both primary and standby. On the second pair as soon as I entered the command on the primary I lost connection to the secondary and both units went active. I then logged into the secondary/active and re-entered the failover key with a "0". Logged into the primary/active and did the same and then communication was restored and the secondary went to standby.

I must be missing something, not sure why it would fail since the primary is sending over the failover key 8 with the right passcode, they match up.

thanks

[Marius Gunnerud]

First of all, it is important to understand that "Now stage 2 was to enable the encryption" is not what you seem to think it is.  This is just a preshared key but in a hashed format so it is not easily readable.  Now usually this is not encrypted and can be read using the command more system:running-config.  To encrypt this you would need to use a master passphrase.  so this could be why you might see it encrypted in the running configuration, if this is configured.  This would mean that the first set of ASAs where this worked might be configured with master passphrase, and the second pair where this fails do not understand the encrypted data.

Check to see if there is key config-key password-encryption <PASSPHRASE> in your configuration.

[netspazz]

I don't see a "key config-key" anywhere in my config. This does make sense, so it is already encrypted. The key forma is something like  "ZpoIY1dvBuWRhlv6lLffXDD2Ss63gIOy8NpkQ=" not world readable.  I also have "key 8 <letters> for my aaa-server, which is not authenticating, this was my next troubleshooting adventure. I have looked for the steps in the process of doing all this, but have not found a good guide or explanation on how it works. These keys must have been generated somehow if I am not mistaken.

[Florin Barhala]

It's a bit confusing, ok. Now why you really wish to use key 8 instead of key 0 for your production environment? After all stability should count first, right?

[netspazz]

Its a requirement so at this time it is still being implemented this way. I would hope that it would not be a problem, since it is a supported configuration and has worked in the past. At this point I just need to find the steps to re-do the key, just trying to find out how this is done.

[netspazz]

Ok, I have been doing some testing, and looks like I have figured out how to the keys to work, but only problem is I broke everything else.

1. I set my failover key to "failover key <password>"

2. ran "key config-key password-encryption <pass-phrase>"

3. Now my failover shows "failover key 8 <hashed password>"

I was happy, but this did not last. Luckily I kept my ssh window open as I tested by ssh to the primary and found my username/password was not working. Tried at the terminal and still no worky.

1. I now tried to set a new password "username admin <password> priv 15"

2. tried to login, still no work

3. tried to revert back to beginning by running "no key config-key password-encryption <pass-phrase>"

4. this looks to have removed the hashing

5. tried to login with admin, no work

6. tried again to change the password "username admin <password> priv 15"

7. no work, next created a new user and still could not log in with this newly created password.

This is very strange. I ended up reloading since I never wrote to memory.

Any ideas??

[Marius Gunnerud]

could you post the output of the following

show run aaa

show run ssh

show run http

[netspazz]

firewall1/pri/act# sh run aaa
aaa authentication http console RADIUS LOCAL
aaa authentication serial console RADIUS LOCAL
aaa authentication enable console RADIUS LOCAL
aaa authentication ssh console RADIUS LOCAL
aaa local authentication attempts max-fail 3

firewall1/pri/act# sh run ssh
ssh scopy enable
ssh stricthostkeycheck
ssh pubkey-chain
  server x.x.x.x
    key-hash sha256 <...H..A..S..H....>
ssh x.x.x.x 255.255.255.192 Management
ssh x.x.x.x 255.255.255.128 Management
ssh x.x.x.x 255.255.255.192 Management
ssh x.x.x.x 255.255.255.255 Management
ssh x.x.x.x 255.255.255.255 Management
ssh x.x.x.x 255.255.255.255 Management
ssh x.x.x.x 255.255.255.255 Management
ssh timeout 15
ssh version 2
ssh cipher encryption fips
ssh cipher integrity fips
ssh key-exchange group dh-group14-sha1

firewall1/pri/act# sh run http
http server enable
http x.x.x.x 255.255.255.192 Management
http x.x.x.x 255.255.255.192 Management
http x.x.x.x 255.255.255.192 Management
http x.x.x.x 255.255.255.255 Management

[Marius Gunnerud]

So what username is not working? the RADIUS I am assuming.  If RADIUS is not working does the locally defined user work?

Issue the following command to test your authentication, replace RADIUS-GROUP, USERNAME, and PASSWORD with the correct values.

test aaa-server authentication RADIUS-GROUP username USERNAME password PASSWORD