Firewall ASA showing failover state as Failed

[suneetmalhotra]

The reachability of each interface can be done from either leg, still my asa failover is in failed state.

Below is the show failover result from secondary leg (acting as active box)

Failover On
Failover unit Secondary
Failover LAN Interface: failoverlan GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 18 of 566 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.13(1)16, Mate 9.13(1)16
Serial Number: Ours <CENSORED>, Mate <CENSORED>
Group 1 last failover at: 06:58:56 GMT Apr 7 2022
Group 2 last failover at: 05:35:17 GMT Apr 7 2022

This host: Secondary
Group 1 State: Active
Active time: 1744602 (sec)
Group 2 State: Active
Active time: 1749621 (sec)

slot 0: ASA5545 hw/sw rev (1.0/9.13(1)16) status (Up Sys)
admin Interface management (12.12.48.9): Normal (Not-Monitored)
C1-PIPSEC1 Interface Outside (12.12.48.99): Normal (Waiting)
C1-PIPSEC1 Interface CA_Internal (12.12.53.126): Normal (Not-Monitored)
C1-PIPSEC1 Interface CA_External (12.12.53.142): Normal (Not-Monitored)
C1-PIPSEC1 Interface CA_Management (12.12.53.158): Normal (Not-Monitored)
C1-PIPSEC1 Interface Inside (12.12.48.113): Unknown (Waiting)
C1-PIPSEC1 Interface D_G_C_I (12.12.49.125): Normal (Waiting)
C3-PIGW Interface Outside (13.13.100.13): Normal (Waiting)
C3-PIGW Interface Mickey (0.0.0.0): Normal (Not-Monitored)
C3-PIGW Interface O-T-I (12.12.53.13): Normal (Not-Monitored)
C3-PIGW Interface S-PROD (12.12.53.30): Unknown (Waiting)
C3-PIGW Interface S-ADM (12.12.53.46): Unknown (Waiting)
C3-PIGW Interface S-ESX (12.12.53.54): Unknown (Waiting)
C3-PIGW Interface S-ADM_QA (12.12.53.62): Unknown (Waiting)
C3-PIGW Interface S-QA (12.12.53.70): Unknown (Waiting)
C3-PIGW Interface GIP-2 (12.12.53.77): Normal (Not-Monitored)
C3-PIGW Interface GIP-3 (12.12.53.85): Normal (Not-Monitored)
C3-PIGW Interface GIP-4 (12.12.53.93): Normal (Not-Monitored)
C3-PIGW Interface GIP-1 (12.12.53.101): Normal (Not-Monitored)
C3-PIGW Interface Inside (12.12.52.113): Unknown (Waiting)
C3-PIGW Interface DMZ-3 (13.13.100.69): Failed (Waiting)
C3-PIGW Interface DMZ-2 (13.13.100.61): Normal (Waiting)
C3-PIGW Interface PEN-IPMI (12.12.53.166): Normal (Not-Monitored)
C3-PIGW Interface PEN-MGMT (12.12.53.110): Normal (Not-Monitored)
C3-PIGW Interface PEN-SERVICE (12.12.53.189): Normal (Not-Monitored)
C2-PSSL1 Interface Outside (12.12.50.99): Normal (Waiting)
C2-PSSL1 Interface Inside (12.12.50.113): Unknown (Waiting)
C2-PSSL1 Interface D_G_C_S (12.12.51.254): Failed (Waiting)
C4-PRA Interface Outside (12.12.54.99): Normal (Waiting)
C4-PRA Interface Inside (12.12.54.113): Normal (Waiting)
C4-PRA Interface D_G_C_R (12.12.55.254): Failed (Waiting)

Other host: Primary
Group 1 State: Failed
Active time: 42299808 (sec)
Group 2 State: Failed
Active time: 728 (sec)

slot 0: ASA5545 hw/sw rev (1.0/9.13(1)16) status (Up Sys)
admin Interface management (12.12.48.10): Normal (Not-Monitored)
C1-PIPSEC1 Interface Outside (12.12.48.100): Normal (Waiting)
C1-PIPSEC1 Interface CA_Internal (12.12.53.125): Normal (Not-Monitored)
C1-PIPSEC1 Interface CA_External (12.12.53.141): Normal (Not-Monitored)
C1-PIPSEC1 Interface CA_Management (12.12.53.157): Normal (Not-Monitored)
C1-PIPSEC1 Interface Inside (12.12.48.114): Normal (Waiting)
C1-PIPSEC1 Interface D_G_C_I (12.12.49.126): Normal (Waiting)
C3-PIGW Interface Outside (13.13.100.14): Normal (Waiting)
C3-PIGW Interface Mickey (0.0.0.0): Normal (Not-Monitored)
C3-PIGW Interface O-T-I (12.12.53.14): Normal (Not-Monitored)
C3-PIGW Interface S-PROD (12.12.53.29): Normal (Waiting)
C3-PIGW Interface S-ADM (12.12.53.45): Normal (Waiting)
C3-PIGW Interface S-ESX (12.12.53.53): Normal (Waiting)
C3-PIGW Interface S-ADM_QA (12.12.53.61): Normal (Waiting)
C3-PIGW Interface S-QA (12.12.53.69): Normal (Waiting)
C3-PIGW Interface GIP-2 (12.12.53.78): Normal (Not-Monitored)
C3-PIGW Interface GIP-3 (12.12.53.86): Normal (Not-Monitored)
C3-PIGW Interface GIP-4 (12.12.53.94): Normal (Not-Monitored)
C3-PIGW Interface GIP-1 (12.12.53.102): Normal (Not-Monitored)
C3-PIGW Interface Inside (12.12.52.114): Normal (Waiting)
C3-PIGW Interface DMZ-3 (13.13.100.70): Normal (Waiting)
C3-PIGW Interface DMZ-2 (13.13.100.62): Normal (Waiting)
C3-PIGW Interface PEN-IPMI (12.12.53.165): Normal (Not-Monitored)
C3-PIGW Interface PEN-MGMT (12.12.53.109): Normal (Not-Monitored)
C3-PIGW Interface PEN-SERVICE (12.12.53.190): Normal (Not-Monitored)
C2-PSSL1 Interface Outside (12.12.50.100): Normal (Waiting)
C2-PSSL1 Interface Inside (12.12.50.114): Normal (Waiting)
C2-PSSL1 Interface D_G_C_S (12.12.51.253): Normal (Waiting)
C4-PRA Interface Outside (12.12.54.100): Normal (Waiting)
C4-PRA Interface Inside (12.12.54.114): Normal (Waiting)
C4-PRA Interface D_G_C_R (12.12.55.253): Normal (Waiting)

Stateful Failover Logical Update Statistics
Link : failovertstate GigabitEthernet0/3 (up)
Stateful Obj xmit xerr rcv rerr
General 18388453021 0 20106534738 815
sys cmd 5884606 0 5884603 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 7030569350 0 9597360502 0
UDP conn 11007061799 0 10480932177 0
ARP tbl 18842 0 19445150 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 197017059 0 1818790 0
SIP Tx 147290675 0 965535 0
SIP Pinhole 610647 0 127893 815
Route Session 0 0 0 0
Router ID 0 0 0 0
User-Identity 43 0 88 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0
Umbrella Device-ID 0 0 0 0

Show failover history

==========================================================================
Group From State To State Reason
==========================================================================
05:33:10 GMT Apr 7 2022
1 Standby Ready Bulk Sync No Error

05:33:10 GMT Apr 7 2022
2 Standby Ready Bulk Sync No Error

05:33:10 GMT Apr 7 2022
0 Bulk Sync Standby Ready Configuration mismatch

05:33:15 GMT Apr 7 2022
2 Bulk Sync Standby Ready No Error

05:33:20 GMT Apr 7 2022
1 Bulk Sync Standby Ready No Error

05:35:17 GMT Apr 7 2022
2 Standby Ready Just Active Failover state check

05:35:17 GMT Apr 7 2022
2 Just Active Active Drain Failover state check

05:35:17 GMT Apr 7 2022
2 Active Drain Active Applying Config Failover state check

05:35:17 GMT Apr 7 2022
2 Active Applying Config Active Config Applied Failover state check

05:35:17 GMT Apr 7 2022
2 Active Config Applied Active Failover state check

06:58:56 GMT Apr 7 2022
1 Standby Ready Just Active Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
1 Just Active Active Drain Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
1 Active Drain Active Applying Config Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
1 Active Applying Config Active Config Applied Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
1 Active Config Applied Active Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
0 Standby Ready Just Active Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
0 Just Active Active Drain Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
0 Active Drain Active Applying Config Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
0 Active Applying Config Active Config Applied Interface check
This host:0
Other host:0

06:58:56 GMT Apr 7 2022
0 Active Config Applied Active Interface check
This host:0
Other host:0

show failover state

State Last Failure Reason Date/Time
This host - Secondary
Group 1 Active Ifc Failure 07:07:08 GMT Apr 7 2022
Group 2 Active Ifc Failure 07:07:08 GMT Apr 7 2022
Other host - Primary
Group 1 Failed Ifc Failure 17:50:55 GMT Dec 3 2020
Group 2 Failed Ifc Failure 17:50:55 GMT Dec 3 2020

====Configuration State===
Sync Done
Sync Done - STANDBY
====Communication State===
Mac set

[johnd2310]

Hi,

What is the state of the other unit? When you run those commands on the other unit what do you get. Are all the interfaces on your firewalls connected properly?

Thanks

John

[MHM Cisco World]

C1-PIPSEC1 Interface Outside (12.12.48.99): Normal (Waiting)

!

C1-PIPSEC1 Interface Outside (12.12.48.100): Normal (Waiting)

Waiting <- meaning that Outside is not reachable, there is switch between the two ASA check the VLAN map or the subnet wrong IP assign for both.

[suneetmalhotra]

I can ping each of the interfaces from one leg to another

For e.g. on C1 context, Outside interface

<Hostname>/C1-PIPSec1/stby# ping outside 12.12..48.99
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 12.12.48.99, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Similarly from other unit.,

HOSTNAME/C1-PIPSec1/act# ping Outside 12.12.48.100
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.48.100, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

[MHM Cisco World]

allocate-interface Ethernet0.1 inside_context1 <- are this config done ??

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/91336-pix-activeactive-config.html

[suneetmalhotra]

Yes. Interfaces are part of the right contexts. I did not touched/configured the ASA in anyway. The failed state showed up after I upgraded the switches in between the two ASAs. I am guessing after the reboot of both the switches.

[johnd2310]

Hi,

What is the model of switches and what  software version did you upgrade to?

Thanks

John

[MHM Cisco World]

As I mention before the issue with VLAN, 
when you add SW between two ASA do you allow all VLAN in trunk? does sub interface or VLAN tag in SW and ASA is same ??

[suneetmalhotra]

@johnd2310 

Cisco 9300 switches (9348 Model) - Ver: 17.6.2

FW1>>FailoverLAN>>>>>SW1>>>>>Po1<<<<<<<<<<SW2<<<<<<FailoverLAN<<FW2

FW1>>FailoverState>>>>>SW1>>>>>Po1<<<<<<<<<<SW2<<<<<<FailoverState<<FW2

Refer: 

ASA Failover State showing Failed after switch stacking/reboot - Cisco Community

[MHM Cisco World]

As you not relay about vlan mismatch and allow vlan in trunk, i assume that it ok.

So 

We know that this issue arise from add l2 device between two asa, and l2 dealing with mac address, I see before many issue about asa failover use same mac add,

And solution for this  is use real mac address in one peer of failover.

[paul driver]

Hello

Did you by any chance try to save the config of the standby fw, try the following.

active fw
copy run start
reload 

[MHM Cisco World]

any Update friend after config MAC address ??? are issue solved ??

[suneetmalhotra · Accepted solution]

Thank you everyone for your help.

The issue got resolved after I run "failover reset" command on the failed unit.