Failure of primary FW seconday now active-replacing primary

[nygenxny123]

We currently have an ASA 5520-per cisco docs you are supposed to configure failover groups, and designate one as the standby and one as primary

We recently had a power issue where one of our FW completely died

As far as failover groups

I see none of this in my ASA-yet when i do a show failover-the ASA says it is the secondary.

Here are are some config statements in my fw that i feel relate to failover..How would I configure the new replacement firewall..and I want to make the "secondary" primay when i do replace the dead-old primary.

PnetFW# sh failover

Failover On

Failover unit Secondary

Failover LAN Interface: fover GigabitEthernet0/3 (Failed - No Switchover)

Unit Poll frequency 1 seconds, holdtime 5 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 2 of 250 maximum

failover replication http

Version: Ours 7.2(1), Mate 7.2(1)

Last Failover at: 09:44:23 EST Dec 10 2008

This host: Secondary - Active --------------------

Active time: 6594955 (sec)

slot 0: ASA5520 hw/sw rev (1.1/7.2(1)) status (Up Sys)

Interface outside (xxx.xxx.46.245): Normal (Waiting)

Interface inside (172.20.1.1): Normal (Waiting)

Interface management (192.168.1.1): No Link (Not-Monitored)

slot 1: ASA-SSM-20 hw/sw rev (1.0/5.0(2)S152.0) status (Up/Up)

IPS, 5.0(2)S152.0, Up

Other host: Primary - Failed -----------------------

Active time: 60016171 (sec)

slot 0: ASA5520 hw/sw rev (1.1/7.2(1)) status (Unknown/Unknown)

Interface outside (6x.xxx.46.246): Unknown

Interface inside (172.20.1.2): Unknown

Interface management (0.0.0.0): Unknown (Not-Monitored)

slot 1: ASA-SSM-20 hw/sw rev (1.0/5.0(2)S152.0) status (Unknown/

Unknown)

IPS, 5.0(2)S152.0, Unknown

interface GigabitEthernet0/0

description External to Internet

nameif outside

security-level 0

ip address xx.xxx.xxx.245 255.255.255.240 standby xxx.xx.xxx.246

interface GigabitEthernet0/1

description Inside Handoff to dlb-Outside

nameif inside

security-level 100

ip address 172.20.1.1 255.255.255.248 standby 172.20.1.2

failover

failover lan unit secondary

failover lan interface fover GigabitEthernet0/3

failover polltime unit 1 holdtime 5

failover replication http

failover link fover GigabitEthernet0/3

failover interface ip fover 192.168.255.249 255.255.255.252 standby 192.168.255.

250

netFW# sh ip

System IP Addresses:

Interface Name IP address Subnet mask

Method

GigabitEthernet0/0 outside xxx.xxx.xxx.245 255.255.255.240

manual

GigabitEthernet0/1 inside 172.20.1.1 255.255.255.248

CONFIG

GigabitEthernet0/3 fover 192.168.255.249 255.255.255.252

unset

Management0/0 management 192.168.1.1 255.255.255.0

CONFIG

Current IP Addresses:

Interface Name IP address Subnet mask

Method

GigabitEthernet0/0 outside xxx.xxx.xxx.245 255.255.255.240

manual

GigabitEthernet0/1 inside 172.20.1.1 255.255.255.248

CONFIG

GigabitEthernet0/3 fover 192.168.255.250 255.255.255.252

unset

Management0/0 management 192.168.1.1 255.255.255.0

CONFIG

[jbalchunas · Accepted solution]

First thing to keep in mind is that your replacement firewall should be the same model, version of ASA OS and license-type as the original.

Configuration is rather simple (assuming the above is true and you are connecting to the same ports the old firewall was in).

(1) Configure the following lines on the new ASA:

failover

failover lan unit primary

failover lan interface fover GigabitEthernet0/3

failover polltime unit 1 holdtime 5

failover link fover GigabitEthernet0/3

failover interface ip fover 192.168.255.249 255.255.255.252 standby 192.168.255.

250

(2) Make sure that your failover interface is not shutdown. Save config.

(3) Mount your new firewall and connect to the same ports your old one was using.

(4) Connect to your console port and power the ASA on.

As long as the new firewall is configured to let it know that it is part of a failover group (the commands we just used) and where the failover ASA is, it will have the config pushed to it when it turns on. The secondary ASA will see a response on its failover interface when the new firewall is turned on, check it's state and then push the config over.

It will not necessarily make the new primary firewall active. The active firewall should remain the standby, and will so until either the standby firewall encounters an event that makes it in worse condition than its partner (like an interface going down) or the other is manually set to be active with the 'failover active' command.

Hope that helps.

[nygenxny123]

great..a few questions.

So I dont need the command

hostname(config)#failover group 1

When is this really necessary.

And there is no need for me to change

the active firewally command statement

from secondary to primary

and configure the new asa as secondary?

My fear is preempt and pushing a blank configuration to the firewall that is working

[jbalchunas]

The 'failover group' command is only for active/active pairs in multiple context mode.

REF:http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/ef.html#wp1927152

And the new ASA will not push over a blank config, it will receive the config from the active partner. The pair is controlled by the currently active firewall.

You can read more about failover here:

http://www.cisco.com/en/US/docs/security/asa/asa80/configuration/guide/failover.html

[nygenxny123]

thx

[jbalchunas]

You're welcome.

[marcelnjkoks]

I have a question about this.

I had to do this twice in de last few months, both times the primary firewall died and the secondary took over and became active.

The new firewall was configured like above (failover unit primary). Because it doens't see another unit it becomes active while configuring the basics as mentioned earlier.

When the new primary firewall is booted and connected to the seconday which is active, you have the situation that both firewalls think they are active. In my case, both times, i saw the primary firewall replicate to the standby and sending the almost empty config.

Both times i ended up having to restore the config by hand, which i saved up front.

Anyone else experienced this behavior?

Edit; need to mention, i only connected the failover interface, not the other interfaces.

i did that after replication.

[jbalchunas]

I can't say that I have seen this. I just tested the procedures I listed above on a pair of old PIX-525s and got the desired results twice - once with Serial-based failover and again with LAN-based failover. I do have to note that one command is missing during the LAN-based failover. Aside from un-shutting the LAN interface, the command 'failover lan enable' must also be issued.

I configured both PIXs and verified that failover was active between the ethernet1 interfaces (test 1), and also used the failover cable for the PIX (test 2). While consoled into the Secondary (Standby) PIX, I shut off the Primary (Active) and watched it failover. Current state is now Standby (Active).

I disconnected all cables, simulating replacing the hardware and turned back on the Primary (still did not connect any cables). I erased the startup config and reloaded again (simulating new blank config). I configured ONLY the 6 lines above (plus 'failover lan enable' for LAN-based), but did change the hostname for verification of replication. Then saved the config to startup.

I then shutdown the Primary (yes, he believed he was Active also and could not detect Mate). Reconnected the LAN-based cable (ethernet1) and Failover cable (separate tests) and turned the firewall back on.

As expected, it booted up, detected that there was a Failover mate and then received a copy of the config from the Secondary (Active) to the Primary (Standby). Active firewall was still the Secondary as there were no qualifying events for it to relinquish its Active state:

DUMMY#

CISCO SYSTEMS PIX FIREWALL

Embedded BIOS Version 4.3.207 01/02/02 16:12:22.73

Compiled by morlee

256 MB RAM

...

Cryptochecksum (unchanged):

Type help or '?' for a list of available commands.

DUMMY>

Detected an Active mate

Beginning configuration replication from mate.

End configuration replication from mate.

FIREWALL> en

Password:

FIREWALL# sh fail

Failover On

Cable status: Normal

Failover unit Primary

Failover LAN Interface: N/A - Serial-based failover enabled

Unit Poll frequency 1 seconds, holdtime 5 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 0 of 250 maximum

Version: Ours 7.2(2)18, Mate 7.2(2)18

Last Failover at: 19:47:55 UTC Mar 10 2009

This host: Primary - Standby Ready

Active time: 0 (sec)

Other host: Secondary - Active

Active time: 305 (sec)

and

DUMMY>

Detected an Active mate

Beginning configuration replication from mate.

End configuration replication from mate.

FIREWALL> en

Password:

FIREWALL# sh fail

Failover On

Cable status: N/A - LAN-based failover enabled

Failover unit Primary

Failover LAN Interface: fover Ethernet1 (up)

Unit Poll frequency 1 seconds, holdtime 5 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 0 of 250 maximum

Version: Ours 7.2(2)18, Mate 7.2(2)18

Last Failover at: 20:12:31 UTC Mar 10 2009

This host: Primary - Standby Ready

Active time: 0 (sec)

Other host: Secondary - Active

Active time: 1691 (sec)

So, that leaves me with a few questions:

(1) What hardware are you using, PIX or ASA?

(2) What version are you running?

(3) Was it serial- or LAN-based failover?

(4) Do you have any console-captures of your config and bootup?

[marcelnjkoks]

To first answer your questions;

Both times the hardware was ASA 5510.

Software 8.04

Lan based failover

I did not save the console messages of the primary unit i was connected to, but it said somethin like: replication started to mate.

And after a short while it reported finished.

Then i discovered it had send it's empty config to the fully configured secondary.

I was wondering, what mechanism does it use when you have two perfectly good units, both active on connect (the failover LAN that is)?

I could image that the primary/active unit wins this battle from the secondary/active unit. Which would explain what i have seen both instances.

[jbalchunas]

Second question first - When a new failover mate powers up, regardless of Primary or Secondary, the Active partner does not automatically change over to the configured Primary. An event has to occur to cause the current Active mate (Primary or Secondary) to be in a less-healthy state than its partner, i.e. an interface goes down, power-failure, etc. When a less-healthy state occurs, only then will it change over.

For example - if the current Standby has two interfaces in a down state and the current Active has only one, the current Active stays Active. If it loses another interface, it still won't failover because it is still not 'less-healthier' than its mate. If it loses a third interface, then failover to the Standby occurs.

Now the first question - I do not know what packets are sent across the failover links as the replacement ASA is powered up. I have to research that, but imagine that it goes something like this -

(1) Power up thinking it is Primary Active

(2) 'Mate detection' packets are constantly being sent by both ASAs if they are configure for failover.

(3) The two ASAs find each other. The newly booted ASA should determine that its mate is the current Active for a longer period of time, and if it is as healthy or less healthier (some interfaces not connected), that it defers to its mate and receives configuration from it.

The only instance I can think of where the newly powered ASA wins is if it is healthier than current Active mate; i.e. it has more UP interfaces, etc. But that contradicts your statement about not having any interfaces connected, aside from the LAN failover. This occurrence wouldn't happen if that were true.

I can test that theory when I return to work tomorrow.