Replacement of primary unit failed! (ASA5510 active/standby)

[amaerklin]

Hi all,

I have an issue bringing up my RMA'd primary ASA unit.

So what happened so far:

1. primary unit failed

2. secondary took over and is now secondary - active (as per sh fail)

2. requested RMA at Cisco

3. got ASA and checked that Lic (SSL), OS (8.2.2) and ASDM are at the same level as the secondary

4. issued wr erase and reloaded

5. copied the following commands to the new (RMA) primary unit:

failover lan unit primary

failover lan interface Failover Ethernet3

failover interface ip Failover 172.x.x.9 255.255.255.248 standby 172.x.x.10

int eth3

no shut

failover

wr mem

6. installed primary unit into rack

7. plugged-in all cables (network, failover, console and power)

8. fired up the primary unit

9. expected that the unit shows:

Detected an Active mate

Beginning configuration replication from mate.

End configuration replication from mate.

10. but nothing happened on primary unit

So can anyone give me assistance on what is a valid and viable approach in replacing a failed primary unit? Is there a missing step that hinders me to successfully replicate the secondary - active config to the primary - standby unit.

I was looking for help on the net but unfortunately I was not able to find anything related to ASA55xx primary unit replacement with a clear guideline or step by step instructions.

Any comments or suggestions are appreciated, and might help others who are in the same situation.

Thanks,

Nico

[varrao]

Hi Nico,

I would request you to kindly explain, what does "but nothing happened on primary unit" means?? Did it not copy the config at all?? Can you please provide the output from both the firewalls:

show run failover

show failover history

show failover

show version

This would help.

Thanks,

Varun

[amaerklin]

Hi Varun,

Thanks for catching-up this thread.

Here you go:

sh run fail on secondary - active:

failover

failover lan unit secondary

failover lan interface Failover Ethernet0/3

failover key *****

failover link Failover Ethernet0/3

failover interface ip Failover 172.x.x.9 255.255.255.248 standby 172.x.x.10

sh fail hist on secondary - active:

asa1# sh fail hist

==========================================================================

From State                 To State                   Reason

==========================================================================

23:47:15 CEST Feb 19 2011

Not Detected               Negotiation                No Error

23:47:19 CEST Feb 19 2011

Negotiation                Cold Standby               Detected an Active mate

23:47:21 CEST Feb 19 2011

Cold Standby               Sync Config                Detected an Active mate

23:47:36 CEST Feb 19 2011

Sync Config                Sync File System           Detected an Active mate

23:47:36 CEST Feb 19 2011

Sync File System           Bulk Sync                  Detected an Active mate

23:47:50 CEST Feb 19 2011

Bulk Sync                  Standby Ready              Detected an Active mate

10:34:09 CEDT Sep 3 2011

Standby Ready              Just Active                HELLO not heard from mate

10:34:09 CEDT Sep 3 2011

Just Active                Active Drain               HELLO not heard from mate

10:34:09 CEDT Sep 3 2011

Active Drain               Active Applying Config     HELLO not heard from mate

10:34:09 CEDT Sep 3 2011

Active Applying Config     Active Config Applied      HELLO not heard from mate

10:34:09 CEDT Sep 3 2011

Active Config Applied      Active                     HELLO not heard from mate

==========================================================================

sh fail on secondary - active

asa1# show fail

Failover On

Failover unit Secondary

Failover LAN Interface: Failover Ethernet0/3 (up)

Unit Poll frequency 1 seconds, holdtime 15 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 2 of 110 maximum

Version: Ours 8.2(2), Mate 8.2(2)

Last Failover at: 10:34:09 CEDT Sep 3 2011

        This host: Secondary - Active

                Active time: 441832 (sec)

                slot 0: ASA5510 hw/sw rev (2.0/8.2(2)) status (Up Sys)

                  Interface Outside (x.x.x.14): Normal (Waiting)

                  Interface Inside (x.x.x.11): Normal (Waiting)

                slot 1: empty

        Other host: Primary - Failed

                Active time: 40497504 (sec)

                slot 0: ASA5510 hw/sw rev (2.0/8.2(2)) status (Unknown/Unknown)

                  Interface Outside (x.x.x.15): Unknown

                  Interface Inside (x.x.x.12): Unknown

                slot 1: empty

Stateful Failover Logical Update Statistics

        Link : Failover Ethernet0/3 (up)

        Stateful Obj    xmit       xerr       rcv        rerr

        General         2250212    0          64800624   309

        sys cmd         2250212    0          2249932    0

        up time         0          0          0          0

        RPC services    0          0          0          0

        TCP conn        0          0          46402635   309

        UDP conn        0          0          21248      0

        ARP tbl         0          0          15921639   0

        Xlate_Timeout   0          0          0          0

        IPv6 ND tbl     0          0          0          0

        VPN IKE upd     0          0          96977      0

        VPN IPSEC upd   0          0          108174     0

        VPN CTCP upd    0          0          19         0

        VPN SDI upd     0          0          0          0

        VPN DHCP upd    0          0          0          0

        SIP Session     0          0          0          0

        Logical Update Queue Information

                        Cur     Max     Total

        Recv Q:         0       17      203259096

        Xmit Q:         0       1       2250212

show ver on secondary - active

asa1# sh ver

Cisco Adaptive Security Appliance Software Version 8.2(2)

Device Manager Version 6.2(5)53

Compiled on Mon 11-Jan-10 14:19 by builders

System image file is "disk0:/asa822-k8.bin"

Config file at boot was "startup-config"

asa1 up 200 days 12 hours

failover cluster up 1 year 108 days

Hardware:   ASA5510, 256 MB RAM, CPU Pentium 4 Celeron 1600 MHz

Internal ATA Compact Flash, 256MB

Slot 1: ATA Compact Flash, 64MB

BIOS Flash M50FW080 @ 0xffe00000, 1024KB

Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)

                             Boot microcode   : CN1000-MC-BOOT-2.00

                             SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03

                             IPSec microcode  : CNlite-MC-IPSECm-MAIN-2.04

0: Ext: Ethernet0/0         : address is 0022.55cf.7420, irq 9

1: Ext: Ethernet0/1         : address is 0022.55cf.7421, irq 9

2: Ext: Ethernet0/2         : address is 0022.55cf.7422, irq 9

3: Ext: Ethernet0/3         : address is 0022.55cf.7423, irq 9

4: Ext: Management0/0       : address is 0022.55cf.741f, irq 11

5: Int: Not used            : irq 11

6: Int: Not used            : irq 5

Licensed features for this platform:

Maximum Physical Interfaces    : Unlimited

Maximum VLANs                  : 100

Inside Hosts                   : Unlimited

Failover                       : Active/Active

VPN-DES                        : Enabled

VPN-3DES-AES                   : Enabled

Security Contexts              : 2

GTP/GPRS                       : Disabled

SSL VPN Peers                  : 10

Total VPN Peers                : 250

Shared License                 : Disabled

AnyConnect for Mobile          : Disabled

AnyConnect for Cisco VPN Phone : Disabled

AnyConnect Essentials          : Disabled

Advanced Endpoint Assessment   : Disabled

UC Phone Proxy Sessions        : 2

Total UC Proxy Sessions        : 2

Botnet Traffic Filter          : Disabled

This platform has an ASA 5510 Security Plus license.

Serial Number: xxx

Running Activation Key:xxxx

Configuration register is 0x1

Configuration last modified by enable_1 at 10:05:32.149 CEDT Fri Jul 15 2011

[varrao]

Hi Nico,

On the secondary you have the failover key entered as well, you need to make sure that you need to have the same key on Primary as well. If you are not sure about the key, kindly use the following command on the secondary ASA to find out the key:

more system:running-config | in failover

This would tell you the key, and then enter the key on primary as well.

Hope this helps.

Thanks,

Varun

[varrao]

On the Primary you should have the following commands:

failover lan unit primary

failover lan interface Failover Ethernet3

failover interface ip Failover 172.x.x.9 255.255.255.248 standby 172.x.x.10

failover link Failover Ethernet0/3

failover key *****

failover

failover lan unit secondary

failover lan interface Failover Ethernet0/3

failover key *****

failover link Failover Ethernet0/3

failover interface ip Failover 172.x.x.9 255.255.255.248 standby 172.x.x.10

failover

Thanks,

Varun

[amaerklin]

Hi Varun,

Thanks for your help!

I will modify or extend my config on the primary ASA without all the cables plugged-in.

Afterwards  it should be straight forward to fire up  the primary unit and the  secondary will replicate its config to the  primary unit, which will  then be placed in standby mode?

After a failover active on the primary unit this unit will become the master again, right?

Many thanks and regards,

Nico

[varrao]

Hi Nico,

The Secvondary firewall would remian active when you fire up the Primary, the primary would go into the standby state. Do let me know how it goes.

Thanks,

Varun

[amaerklin]

Hi Varun,

I was able to put in the commands you mentioned and afterwards i fired up the ASA with all cables plugged in, and synchronisation started without any issues.

After some time i did a failover active on the standby unit to make it the active one and all went back to normal.

Many thanks for your valuable support!

best regards,

Nico

[varrao · Accepted solution]

Hi Nico,

Glad it worked as expected without any issues, let me know if you have any other issues

You can mark this thread as answered and do rate helpful posts.

Thanks,

Varun

[DeanBlanscett]

My experance was that the replacement (RMA)  primary ASA copied it's blank configuration to the Secondary insted of seeing that there was an active mate!!!!!!!!!!!

Good thing I had a backup of the configuration!!!!!!

[JEFF SPRADLING]

I was just preparing to replace the primary ASA in an HA pair and could not find a solid answer to this question.  I found that, indeed, the primary ASA started replicating it's blank config to the secondary as soon as I connected the LAN Failover cable.

Here's the steps to keep this from happening:

configure the primary for failover -

failover lan unit primary

failover lan interface LANFail GigabitEthernet0/2

failover replication http

failover link stateful GigabitEthernet0/3

failover interface ip LANFail 172.16.100.1 255.255.255.0 standby 172.16.100.2

failover interface ip stateful 172.16.101.1 255.255.255.0 standby 172.16.101.2    

Configure all interfaces with the primary IP (no standby needed at this point)

'no shut' on all active interfaces

no failover active         <------- (critical! Forces the primary to standby)

connect lan failover cable (the only one needed at this point)

Secondary will start replicating to primary.

Once the replication is complete (show failover, ensure primary is "standby ready", you can connect the remaining cables and do a 'failover active' on the primary.

Hope this helps others...