topic Re: ASA HA Failure in Network Security

ASA HA Failure

abtt-39 — Mon, 11 Sep 2023 15:05:10 GMT

Hello

I have a pair of ASAs in HA mode. Primary active, secondary standby ready.

But since September 1, the secondary has become active and the primary has become standby ready.

When I connect in ssh on the interface:

ASA/act# sh failover
Failover On
Failover unit Secondary
Failover LAN Interface: bckfail GigabitEthernet1/8 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 410 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.16(4)14, Mate 9.16(4)14
Serial Number: Ours JAD24330TGU, Mate JAD24330TA2
Last Failover at: 13:30:44 CEDT Sep 1 2023
This host: Secondary - Active
Active time: 873756 (sec)
slot 1: ASA5516 hw/sw rev (3.4/9.16(4)14) status (Up Sys)
Interface Outside (89.90.218.155): Normal (Monitored)
Interface Inside (10.39.6.5): Normal (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Primary - Standby Ready
Active time: 14668767 (sec)
slot 1: ASA5516 hw/sw rev (3.4/9.16(4)14) status (Up Sys)
Interface Outside (89.90.218.154): Normal (Monitored)
Interface Inside (10.39.6.50): Normal (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)

ASA/act#sh failover history

==========================================================================
From State To State Reason
==========================================================================
17:55:15 CEST Mar 15 2023
Not Detected Negotiation No Error

17:55:19 CEST Mar 15 2023
Negotiation Cold Standby Detected an Active mate

17:55:20 CEST Mar 15 2023
Cold Standby Sync Config Detected an Active mate

17:55:40 CEST Mar 15 2023
Sync Config Sync File System Detected an Active mate

17:55:40 CEST Mar 15 2023
Sync File System Bulk Sync Detected an Active mate

17:55:56 CEST Mar 15 2023
Bulk Sync Standby Ready Detected an Active mate

13:30:43 CEDT Sep 1 2023
Standby Ready Just Active Interface check
This host:0
Other host:0

13:30:44 CEDT Sep 1 2023
Just Active Active Drain Interface check
This host:0
Other host:0

13:30:44 CEDT Sep 1 2023
Active Drain Active Applying Config Interface check
This host:0
Other host:0

13:30:44 CEDT Sep 1 2023
Active Applying Config Active Config Applied Interface check
This host:0
Other host:0

13:30:44 CEDT Sep 1 2023
Active Config Applied Active Interface check
This host:0
Other host:0

If i do the same on the second :

ASA/act#failover exec standby sh failover

Last Failover at: 13:30:43 CEDT Sep 1 2023
This host: Primary - Standby Ready
Active time: 14668767 (sec)
slot 1: ASA5516 hw/sw rev (3.4/9.16(4)14) status (Up Sys)
Interface Outside (89.90.218.154): Normal (Monitored)
Interface Inside (10.39.6.50): Normal (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Secondary - Active
Active time: 874152 (sec)
slot 1: ASA5516 hw/sw rev (3.4/9.16(4)14) status (Up Sys)
Interface Outside (89.90.218.155): Normal (Monitored)
Interface Inside (10.39.6.5): Normal (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 2: SFR5516 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)

ASA/act#failover exec standby sh failover history

==========================================================================
From State To State Reason
==========================================================================
17:48:47 CEST Mar 15 2023
Not Detected Negotiation No Error

17:48:51 CEST Mar 15 2023
Negotiation Cold Standby Detected an Active mate

17:48:52 CEST Mar 15 2023
Cold Standby Sync Config Detected an Active mate

17:49:12 CEST Mar 15 2023
Sync Config Sync File System Detected an Active mate

17:49:12 CEST Mar 15 2023
Sync File System Bulk Sync Detected an Active mate

17:49:27 CEST Mar 15 2023
Bulk Sync Standby Ready Detected an Active mate

17:51:42 CEST Mar 15 2023
Standby Ready Just Active Other unit wants me Active

17:51:42 CEST Mar 15 2023
Just Active Active Drain Other unit wants me Active

17:51:42 CEST Mar 15 2023
Active Drain Active Applying Config Other unit wants me Active

17:51:42 CEST Mar 15 2023
Active Applying Config Active Config Applied Other unit wants me Active

17:51:42 CEST Mar 15 2023
Active Config Applied Active Other unit wants me Active

13:30:43 CEDT Sep 1 2023
Active Failed Interface check
This host:0
Other host:0

13:37:27 CEDT Sep 1 2023
Failed Standby Ready Interface check
This host:0
Other host:0

13:38:15 CEDT Sep 1 2023
Standby Ready Cold Standby Configuration mismatch

13:38:16 CEDT Sep 1 2023
Cold Standby Sync Config Configuration mismatch

13:39:00 CEDT Sep 1 2023
Sync Config Sync File System Configuration mismatch

13:39:00 CEDT Sep 1 2023
Sync File System Bulk Sync Configuration mismatch

13:39:15 CEDT Sep 1 2023
Bulk Sync Standby Ready Configuration mismatch

==========================================================================

Could someone explain the mismatch configuration to me at the end?

and what do the different lines dated September 1st mean?

The "failover lan unit primary" line is on the primary (standby)

And I would like to put it back in the correct order. Let the primary become active again, and the secondary, stand by.

I believe i must use the command “no failover active” on the active firewall , Correct?

Or "failover active" on the standby?

Re: ASA HA Failure

Marvin Rhoads — Mon, 11 Sep 2023 15:16:22 GMT

Either of the commands you mentioned at the end of your posting will work.

It's not clear from the information shown exactly what happened on 1 September to cause the issue.

Re: ASA HA Failure

balaji.bandi — Mon, 11 Sep 2023 19:24:04 GMT

Bulk Sync Standby Ready Configuration mismatch - looks for me config mismatch (or sometime i see the License issue also)

when was the last time it was tested ?

both command works, easy way is reboot current primary so original primary become active.

Re: ASA HA Failure

abtt-39 — Mon, 16 Oct 2023 09:59:41 GMT

Hello,

I had this problem again.
I'm glad it hasn't been tested for a long time. But it was functional.

==========================================================================

From State To State Reason

==========================================================================

17:51:42 CEST Mar 15 2023

Active Config Applied Active Other unit wants me Active

13:30:43 CEDT Sep 1 2023

Active Failed Interface check

This host:0

Other host:0

13:37:27 CEDT Sep 1 2023

Failed Standby Ready Interface check

This host:0

Other host:0

13:38:15 CEDT Sep 1 2023

Standby Ready Cold Standby Configuration mismatch

13:38:16 CEDT Sep 1 2023

Cold Standby Sync Config Configuration mismatch

13:39:00 CEDT Sep 1 2023

Sync Config Sync File System Configuration mismatch

13:39:00 CEDT Sep 1 2023

Sync File System Bulk Sync Configuration mismatch

13:39:15 CEDT Sep 1 2023

Bulk Sync Standby Ready Configuration mismatch

17:29:09 CEDT Oct 3 2023

Standby Ready Just Active Other unit wants me Active

17:29:09 CEDT Oct 3 2023

Just Active Active Drain Other unit wants me Active

17:29:09 CEDT Oct 3 2023

Active Drain Active Applying Config Other unit wants me Active

17:29:09 CEDT Oct 3 2023

Active Applying Config Active Config Applied Other unit wants me Active

17:29:09 CEDT Oct 3 2023

Active Config Applied Active Other unit wants me Active

01:47:56 CEDT Oct 12 2023

Active Failed Interface check

This host:0

Other host:0

01:54:40 CEDT Oct 12 2023

Failed Standby Ready Interface check

This host:0

Other host:0

01:55:24 CEDT Oct 12 2023

Standby Ready Cold Standby Configuration mismatch

01:55:25 CEDT Oct 12 2023

Cold Standby Sync Config Configuration mismatch

01:56:09 CEDT Oct 12 2023

Sync Config Sync File System Configuration mismatch

01:56:09 CEDT Oct 12 2023

Sync File System Bulk Sync Configuration mismatch

01:56:23 CEDT Oct 12 2023

Bulk Sync Standby Ready Configuration mismatch

==========================================================================

#sh failover state

State Last Failure Reason Date/Time

This host - Primary

Standby Ready Ifc Failure 01:47:56 CEDT Oct 12 2023

Other host - Secondary

Active Comm Failure 17:29:51 CEDT Oct 3 2023

====Configuration State===

Sync Done

Sync Done - STANDBY

====Communication State===

Mac set

Re: ASA HA Failure

balaji.bandi — Mon, 16 Oct 2023 10:35:51 GMT

i would go back and investigate is there any Physical issue ?cables connected ? (how they connected back to back or using any switch ?)

Re: ASA HA Failure

abtt-39 — Mon, 16 Oct 2023 11:08:43 GMT

In fact between the 2 ASAs, there is a switch (9200). the 2 ASAs are in 2 separate buildings, these 2 buildings are connected by fiber and there is a switch in one building, a switch in the other.
This is also useful because we have a second pair of ASAs, in fact the 2 pairs are crossed. An active ASA Primary in one building, the second of the Failover pair on stand-by ready in the other. And vice versa for the second paire.
Note that on the other pair of ASAs, which goes through the same switches (with 2 vlans created to partition the 2 internet connections), I have no problems.

I will try to look at these switches if I see any errors on the ports. These switches cannot be reached directly, I have to plug into the console port to watch.