Re: ASA5545 active/standby failover problem

wayne wan · ‎01-04-2023

I have a pair of ASA5545 firewalls configured as Active/standby mode.

I added an addon network adapter card on them and configured the management port on this card.
It's because the original management port needed to be the IPS(Firepower) port.

There was an incident occured as the programs on servers complained cannot connect to the default gateways.
I checked on the front panel , both firewalls ' active LED was on which indicated both are Active.

After further checking, there was hardware failure on the addon network adapter card occured on one of the firewall.
(I confirmed this as I can see the LED alarm is on the back panel and also I exported the configruation and compared with the saved version.
I saw the following GigbitEthernet1 lines are all missing on the defective firewall.

interface GigabitEthernet1/0
management-only
nameif management
security-level 100
ip address 192.168.10.102 255.255.255.0 standby 192.168.10.101
!
interface GigabitEthernet1/1
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/2
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/3
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/4
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/5
shutdown
no nameif
no security-level
no ip address
!

I connected the console port and find that both of them prompted the following messages.

WARING: Failover message decryption failure. Please make sure both units have the same failover shared key and crypto license
or system is not out of memory.

I rebooted both of the firwalls one by one and both active status still presisted.
The network is better but still, some of the programs will down due to the connection problem to gateways.
Finally, we need to power disconnected the good firewall , and leaving the defective firewall to be active. This made the network stable again.

Does anyone know why the failover failed?

Thanks a lot!

Wayne Wan

=====================

This is my configuration

=====================
!
hostname fk01ssc-1
xlate per-session deny tcp any4 any4
xlate per-session deny tcp any4 any6
xlate per-session deny tcp any6 any4
xlate per-session deny tcp any6 any6
xlate per-session deny udp any4 any4 eq domain
xlate per-session deny udp any4 any6 eq domain
xlate per-session deny udp any6 any4 eq domain
xlate per-session deny udp any6 any6 eq domain
names

!
interface GigabitEthernet0/0
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/0.2
description Gigabit Connection to ek01ssc
vlan 2
nameif ek01ssc
security-level 50
ip address 10.73.2.11 255.255.255.240 standby 10.73.2.12
!
interface GigabitEthernet0/1
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/1.10
description Gigabit Connection to ek11ssc Core Zone
vlan 10
nameif ek11ssc_vlan_10
security-level 50
ip address 170.11.10.254 255.255.255.0 standby 170.11.10.253
!
interface GigabitEthernet0/1.11
description Gigabit Connection to ek11ssc Core Zone
vlan 11
nameif ek11ssc_vlan_11
security-level 50
ip address 170.11.11.254 255.255.255.0 standby 170.11.11.253
!
interface GigabitEthernet0/1.12
description Gigabit Connection to ek11ssc Core Zone
vlan 12
nameif ek11ssc_vlan_12
security-level 50
ip address 170.11.12.254 255.255.255.0 standby 170.11.12.253
!
interface GigabitEthernet0/1.13
description Gigabit Connection to ek11ssc Core Zone
vlan 13
nameif ek11ssc_vlan_13
security-level 50
ip address 170.11.13.254 255.255.255.0 standby 170.11.13.253
!
interface GigabitEthernet0/1.14
description Gigabit Connection to ek11ssc Core Zone
vlan 14
nameif ek11ssc_vlan_14
security-level 50
ip address 170.11.14.254 255.255.255.0 standby 170.11.14.253
!
:

:

interface GigabitEthernet0/4
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/5
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/6
description LAN Failover Interface
!
interface GigabitEthernet0/7
description STATE Failover Interface
!
interface Management0/0
management-only
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/0
management-only
nameif management
security-level 100
ip address 192.168.10.102 255.255.255.0 standby 192.168.10.101
!
interface GigabitEthernet1/1
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/2
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/3
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/4
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/5
shutdown
no nameif
no security-level
no ip address
!
:
:
pager lines 24
logging enable
logging standby
logging trap debugging
logging asdm informational
logging host management 192.168.10.181
logging permit-hostdown
mtu ek01ssc 1500
mtu ek11ssc_vlan_10 1500
mtu ek11ssc_vlan_11 1500
mtu ek11ssc_vlan_12 1500
mtu ek11ssc_vlan_13 1500
mtu ek11ssc_vlan_14 1500
mtu ek21ssc_vlan_20 1500
mtu ek21ssc_vlan_21 1500
mtu ek31ssc_vlan_30 1500
mtu ek31ssc_vlan_31 1500
mtu ek31ssc_vlan_32 1500
mtu ek31ssc_vlan_34 1500
mtu management 1500
failover
failover lan unit secondary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6
monitor-interface ek01ssc
monitor-interface ek11ssc_vlan_10
monitor-interface ek11ssc_vlan_11
monitor-interface ek11ssc_vlan_12
monitor-interface ek11ssc_vlan_13
monitor-interface ek11ssc_vlan_14
monitor-interface ek21ssc_vlan_20
monitor-interface ek21ssc_vlan_21
monitor-interface ek31ssc_vlan_30
monitor-interface ek31ssc_vlan_31
monitor-interface ek31ssc_vlan_32
monitor-interface ek31ssc_vlan_34
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-792-152.bin
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
arp rate-limit 32768
:
:

marce1000 · ‎01-04-2023

- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx10845

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

wayne wan · ‎01-05-2023

Hi,

yes, I read this docuemnt before.

But I experienced this probelm twice in last year.

Before the first incident happened, I configured the failover in 2018 and I have already reloaded/failover a lot of times during testing and applying patch.
So I don't feel it's related.

Then, after first incident happened, I used a spare ASA5545 (which is one of the running pair in the test system) and I loaded the configure into it to replace the defective one.

After the replacemnet, I am sure I had done the failover testing.
Then the second incident happened. I checked all firewalls now, including the defective firewall, were enabled the Encryption license (3DES/AES) (using show ver to check).

Did I misunderstand something?

Regards,

Wayne Wan

wayne wan · ‎01-05-2023

I also want to mention that after I loaded the configuration to the spare firewall.

I run the line "failover key <real key>" on both firewalls and let them to do the synchronization again.

Marius Gunnerud · ‎01-05-2023

What type of "plugin" / module have you installed in the ASAs? Are any other configuration being synchronised with the standby unit or are all configurations not replicated?

--
Please remember to select a correct answer and rate helpful posts

paul driver · ‎01-05-2023

Hello

@wayne wan wrote:

This is my configuration

=====================
!
hostname fk01ssc-1
failover
failover lan unit secondary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6
:

I assume what you have posted is the Secondary ASA correct?

Can you post the following please from both FWs:
sh mode
sh cluster
sh interface ip brief
sh run failover
sh failover

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

wayne wan · ‎01-05-2023

This the healthy firewall (default secondary)

show mode
Security context mode: single
fk01ssc-1# show cluster
ERROR: % Incomplete command
fk01ssc-1# show cluster info
Clustering is not configured
fk01ssc-1# show interface ip brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset up up
GigabitEthernet0/0.2 10.23.2.12 YES CONFIG up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/1.10 170.11.10.253 YES CONFIG up up
GigabitEthernet0/1.11 170.11.11.253 YES CONFIG up up
GigabitEthernet0/1.12 170.11.12.253 YES CONFIG up up
GigabitEthernet0/1.13 170.11.13.253 YES CONFIG up up
GigabitEthernet0/1.14 170.11.14.253 YES CONFIG up up
GigabitEthernet0/2 unassigned YES unset up up
GigabitEthernet0/2.20 170.11.20.253 YES CONFIG up up
GigabitEthernet0/2.21 170.11.21.253 YES CONFIG up up
GigabitEthernet0/3 unassigned YES unset up up
GigabitEthernet0/3.30 170.11.30.253 YES CONFIG up up
GigabitEthernet0/3.31 170.11.31.253 YES CONFIG up up
GigabitEthernet0/3.32 170.11.32.253 YES CONFIG up up
GigabitEthernet0/3.34 170.11.34.253 YES CONFIG up up
GigabitEthernet0/4 unassigned YES unset administratively down down
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 192.168.20.2 YES unset up up
GigabitEthernet0/7 192.168.20.6 YES unset up up
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up up
Internal-Data0/1 unassigned YES unset up up
Internal-Data0/2 unassigned YES unset up up
Internal-Data0/3 169.254.1.1 YES unset up up
Management0/0 unassigned YES unset up up
GigabitEthernet1/0 192.168.10.101 YES CONFIG up up
GigabitEthernet1/1 unassigned YES unset administratively down down
GigabitEthernet1/2 unassigned YES unset administratively down down
GigabitEthernet1/3 unassigned YES unset administratively down down
GigabitEthernet1/4 unassigned YES unset administratively down down
GigabitEthernet1/5 unassigned YES unset administratively down down
fk01ssc-1# show run failover
failover
failover lan unit secondary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6
fk01ssc-1# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: LAN_Link GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 500 milliseconds, holdtime 2 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 13 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.6(4)42, Mate 9.6(4)42
Serial Number: Ours FCH1942J8FM, Mate FCH1942J8NT
Last Failover at: 00:42:29 HKST Dec 21 2022
This host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.12): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.253): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.253): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.253): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.253): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.253): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.253): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.253): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.253): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.253): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.253): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.253): Normal (Monitored)
Interface management (192.168.10.101): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0-764) status (Up/Up)
ASA FirePOWER, 5.4.0-764, Up, (Monitored)
Other host: Primary - Active
Active time: 1411346 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.11): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.254): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.254): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.254): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.254): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.254): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.254): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.254): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.254): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.254): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.254): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.254): Normal (Monitored)
Interface management (192.168.10.102): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0.12-17) status (Up/Up)
ASA FirePOWER, 5.4.0.12-17, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : State_Link GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 188157 0 59077632 0
sys cmd 188157 0 188157 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 55862238 0
UDP conn 0 0 2230639 0
ARP tbl 0 0 796597 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 0 0 0 0
Router ID 0 0 0 0
User-Identity 0 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 23 62664268
Xmit Q: 0 1 188158

This is the replaced one (default primary)

show mode
Security context mode: single
fk01ssc-1# show cluster
ERROR: % Incomplete command
fk01ssc-1# show cluster info
Clustering is not configured
fk01ssc-1# show interface ip brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset up up
GigabitEthernet0/0.2 10.23.2.11 YES CONFIG up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/1.10 170.11.10.254 YES CONFIG up up
GigabitEthernet0/1.11 170.11.11.254 YES CONFIG up up
GigabitEthernet0/1.12 170.11.12.254 YES CONFIG up up
GigabitEthernet0/1.13 170.11.13.254 YES CONFIG up up
GigabitEthernet0/1.14 170.11.14.254 YES CONFIG up up
GigabitEthernet0/2 unassigned YES unset up up
GigabitEthernet0/2.20 170.11.20.254 YES CONFIG up up
GigabitEthernet0/2.21 170.11.21.254 YES CONFIG up up
GigabitEthernet0/3 unassigned YES unset up up
GigabitEthernet0/3.30 170.11.30.254 YES CONFIG up up
GigabitEthernet0/3.31 170.11.31.254 YES CONFIG up up
GigabitEthernet0/3.32 170.11.32.254 YES CONFIG up up
GigabitEthernet0/3.34 170.11.34.254 YES CONFIG up up
GigabitEthernet0/4 unassigned YES unset administratively down down
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 192.168.20.1 YES unset up up
GigabitEthernet0/7 192.168.20.5 YES unset up up
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up up
Internal-Data0/1 unassigned YES unset up up
Internal-Data0/2 unassigned YES unset up up
Internal-Data0/3 169.254.1.1 YES unset up up
Management0/0 unassigned YES unset up up
GigabitEthernet1/0 192.168.10.102 YES CONFIG up up
GigabitEthernet1/1 unassigned YES unset administratively down down
GigabitEthernet1/2 unassigned YES unset administratively down down
GigabitEthernet1/3 unassigned YES unset administratively down down
GigabitEthernet1/4 unassigned YES unset administratively down down
GigabitEthernet1/5 unassigned YES unset administratively down down
fk01ssc-1# show run failover
failover
failover lan unit primary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6
fk01ssc-1# show failover
Failover On
Failover unit Primary
Failover LAN Interface: LAN_Link GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 500 milliseconds, holdtime 2 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 13 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.6(4)42, Mate 9.6(4)42
Serial Number: Ours FCH1942J8NT, Mate FCH1942J8FM
Last Failover at: 00:39:39 HKST Dec 21 2022
This host: Primary - Active
Active time: 1411431 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.11): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.254): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.254): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.254): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.254): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.254): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.254): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.254): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.254): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.254): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.254): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.254): Normal (Monitored)
Interface management (192.168.10.102): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0.12-17) status (Up/Up)
ASA FirePOWER, 5.4.0.12-17, Up, (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.12): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.253): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.253): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.253): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.253): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.253): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.253): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.253): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.253): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.253): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.253): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.253): Normal (Monitored)
Interface management (192.168.10.101): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0-764) status (Up/Up)
ASA FirePOWER, 5.4.0-764, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : State_Link GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 59082938 0 211943 0
sys cmd 188242 0 188242 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 55867219 0 22304 0
UDP conn 2230814 0 1081 0
ARP tbl 796653 0 315 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 9 0 0 0
Router ID 0 0 0 0
User-Identity 1 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 25 3036007
Xmit Q: 0 30 59847409

This is the defective one (original primary, now down)

term p 0
fk01ssc-1# show mode
Security context mode: single
fk01ssc-1# show cluster
ERROR: % Incomplete command
fk01ssc-1# show cluster info
Clustering is not configured
fk01ssc-1# show interface ip brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset down down
GigabitEthernet0/0.2 10.23.2.11 YES CONFIG down down
GigabitEthernet0/1 unassigned YES unset down down
GigabitEthernet0/1.10 170.11.10.254 YES CONFIG down down
GigabitEthernet0/1.11 170.11.11.254 YES CONFIG down down
GigabitEthernet0/1.12 170.11.12.254 YES CONFIG down down
GigabitEthernet0/1.13 170.11.13.254 YES CONFIG down down
GigabitEthernet0/1.14 170.11.14.254 YES CONFIG down down
GigabitEthernet0/2 unassigned YES unset down down
GigabitEthernet0/2.20 170.11.20.254 YES CONFIG down down
GigabitEthernet0/2.21 170.11.21.254 YES CONFIG down down
GigabitEthernet0/3 unassigned YES unset down down
GigabitEthernet0/3.30 170.11.30.254 YES CONFIG down down
GigabitEthernet0/3.31 170.11.31.254 YES CONFIG down down
GigabitEthernet0/3.32 170.11.32.254 YES CONFIG down down
GigabitEthernet0/3.34 170.11.34.254 YES CONFIG down down
GigabitEthernet0/4 unassigned YES unset administratively down down
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 192.168.20.1 YES unset down down
GigabitEthernet0/7 192.168.20.5 YES unset down down
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up down
Internal-Data0/1 unassigned YES unset up up
Internal-Data0/2 unassigned YES unset up up
Internal-Data0/3 169.254.1.1 YES unset up up
Management0/0 unassigned YES unset down down
fk01ssc-1# show run failover
failover
failover lan unit primary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6
fk01ssc-1# show failover
Failover On
Failover unit Primary
Failover LAN Interface: LAN_Link GigabitEthernet0/6 (Failed - No Switchover)
Reconnect timeout 0:00:00
Unit Poll frequency 500 milliseconds, holdtime 2 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 12 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.6(4)42, Mate 9.6(4)42
Serial Number: Ours FCH1945768K, Mate FCH1942J8FM
Last Failover at: 23:43:52 HKST Dec 20 2022
This host: Primary - Active
Active time: 1414944 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.11): No Link (Waiting)
Interface ek11ssc_vlan_10 (170.11.10.254): No Link (Waiting)
Interface ek11ssc_vlan_11 (170.11.11.254): No Link (Waiting)
Interface ek11ssc_vlan_12 (170.11.12.254): No Link (Waiting)
Interface ek11ssc_vlan_13 (170.11.13.254): No Link (Waiting)
Interface ek11ssc_vlan_14 (170.11.14.254): No Link (Waiting)
Interface ek21ssc_vlan_20 (170.11.20.254): No Link (Waiting)
Interface ek21ssc_vlan_21 (170.11.21.254): No Link (Waiting)
Interface ek31ssc_vlan_30 (170.11.30.254): No Link (Waiting)
Interface ek31ssc_vlan_31 (170.11.31.254): No Link (Waiting)
Interface ek31ssc_vlan_32 (170.11.32.254): No Link (Waiting)
Interface ek31ssc_vlan_34 (170.11.34.254): No Link (Waiting)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0.12-17) status (Up/Up)
ASA FirePOWER, 5.4.0.12-17, Up, (Monitored)
Other host: Secondary - Failed
Active time: 12084 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Unknown/Unknown)
Interface ek01ssc (10.23.2.12): Unknown (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.253): Unknown (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.253): Unknown (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.253): Unknown (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.253): Unknown (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.253): Unknown (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.253): Unknown (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.253): Unknown (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.253): Unknown (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.253): Unknown (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.253): Unknown (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.253): Unknown (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0-764) status (Unknown/Unknown)
ASA FirePOWER, 5.4.0-764, Unknown, (Monitored)

Stateful Failover Logical Update Statistics
Link : State_Link GigabitEthernet0/7 (down)
Stateful Obj xmit xerr rcv rerr
General 11137 0 4 0
sys cmd 5 0 4 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 10721 0 0 0
UDP conn 353 0 0 0
ARP tbl 57 0 0 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 0 0 0 0
Router ID 0 0 0 0
User-Identity 1 0 0 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 12 71
Xmit Q: 0 30 11203

wayne wan · ‎01-05-2023

Hello,

Please see my FWs' information as below.

Thank you!

Wayne Wan

This is for the Secondary (healthy one)

show mode
Security context mode: single

fk01ssc-1# show cluster
ERROR: % Incomplete command
fk01ssc-1# show cluster info
Clustering is not configured

fk01ssc-1# show interface ip brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset up up
GigabitEthernet0/0.2 10.23.2.12 YES CONFIG up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/1.10 170.11.10.253 YES CONFIG up up
GigabitEthernet0/1.11 170.11.11.253 YES CONFIG up up
GigabitEthernet0/1.12 170.11.12.253 YES CONFIG up up
GigabitEthernet0/1.13 170.11.13.253 YES CONFIG up up
GigabitEthernet0/1.14 170.11.14.253 YES CONFIG up up
GigabitEthernet0/2 unassigned YES unset up up
GigabitEthernet0/2.20 170.11.20.253 YES CONFIG up up
GigabitEthernet0/2.21 170.11.21.253 YES CONFIG up up
GigabitEthernet0/3 unassigned YES unset up up
GigabitEthernet0/3.30 170.11.30.253 YES CONFIG up up
GigabitEthernet0/3.31 170.11.31.253 YES CONFIG up up
GigabitEthernet0/3.32 170.11.32.253 YES CONFIG up up
GigabitEthernet0/3.34 170.11.34.253 YES CONFIG up up
GigabitEthernet0/4 unassigned YES unset administratively down down
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 192.168.20.2 YES unset up up
GigabitEthernet0/7 192.168.20.6 YES unset up up
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up up
Internal-Data0/1 unassigned YES unset up up
Internal-Data0/2 unassigned YES unset up up

Internal-Data0/3 169.254.1.1 YES unset up up
Management0/0 unassigned YES unset up up
GigabitEthernet1/0 192.168.10.101 YES CONFIG up up
GigabitEthernet1/1 unassigned YES unset administratively down down
GigabitEthernet1/2 unassigned YES unset administratively down down
GigabitEthernet1/3 unassigned YES unset administratively down down
GigabitEthernet1/4 unassigned YES unset administratively down down
GigabitEthernet1/5 unassigned YES unset administratively down down

fk01ssc-1# show run failover
failover
failover lan unit secondary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6

fk01ssc-1# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: LAN_Link GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 500 milliseconds, holdtime 2 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 13 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.6(4)42, Mate 9.6(4)42
Serial Number:
Last Failover at: 00:42:29 HKST Dec 21 2022
This host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.12): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.253): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.253): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.253): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.253): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.253): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.253): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.253): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.253): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.253): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.253): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.253): Normal (Monitored)
Interface management (192.168.10.101): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0-764) status (Up/Up)
ASA FirePOWER, 5.4.0-764, Up, (Monitored)
Other host: Primary - Active
Active time: 1411346 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.11): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.254): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.254): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.254): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.254): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.254): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.254): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.254): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.254): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.254): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.254): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.254): Normal (Monitored)
Interface management (192.168.10.102): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0.12-17) status (Up/Up)
ASA FirePOWER, 5.4.0.12-17, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : State_Link GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 188157 0 59077632 0
sys cmd 188157 0 188157 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 55862238 0
UDP conn 0 0 2230639 0
ARP tbl 0 0 796597 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 0 0 0 0
Router ID 0 0 0 0
User-Identity 0 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 23 62664268
Xmit Q: 0 1 188158

This is for the Primary (replaced one)

show mode
Security context mode: single

fk01ssc-1# show cluster
ERROR: % Incomplete command
fk01ssc-1# show cluster info
Clustering is not configured

fk01ssc-1# show interface ip brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset up up
GigabitEthernet0/0.2 10.23.2.11 YES CONFIG up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/1.10 170.11.10.254 YES CONFIG up up
GigabitEthernet0/1.11 170.11.11.254 YES CONFIG up up
GigabitEthernet0/1.12 170.11.12.254 YES CONFIG up up
GigabitEthernet0/1.13 170.11.13.254 YES CONFIG up up
GigabitEthernet0/1.14 170.11.14.254 YES CONFIG up up
GigabitEthernet0/2 unassigned YES unset up up
GigabitEthernet0/2.20 170.11.20.254 YES CONFIG up up
GigabitEthernet0/2.21 170.11.21.254 YES CONFIG up up
GigabitEthernet0/3 unassigned YES unset up up
GigabitEthernet0/3.30 170.11.30.254 YES CONFIG up up
GigabitEthernet0/3.31 170.11.31.254 YES CONFIG up up
GigabitEthernet0/3.32 170.11.32.254 YES CONFIG up up
GigabitEthernet0/3.34 170.11.34.254 YES CONFIG up up
GigabitEthernet0/4 unassigned YES unset administratively down down
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 192.168.20.1 YES unset up up
GigabitEthernet0/7 192.168.20.5 YES unset up up
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up up
Internal-Data0/1 unassigned YES unset up up
Internal-Data0/2 unassigned YES unset up up

Internal-Data0/3 169.254.1.1 YES unset up up
Management0/0 unassigned YES unset up up
GigabitEthernet1/0 192.168.10.102 YES CONFIG up up
GigabitEthernet1/1 unassigned YES unset administratively down down
GigabitEthernet1/2 unassigned YES unset administratively down down
GigabitEthernet1/3 unassigned YES unset administratively down down
GigabitEthernet1/4 unassigned YES unset administratively down down
GigabitEthernet1/5 unassigned YES unset administratively down down

fk01ssc-1# show run failover
failover
failover lan unit primary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6

fk01ssc-1# show failover
Failover On
Failover unit Primary
Failover LAN Interface: LAN_Link GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 500 milliseconds, holdtime 2 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 13 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.6(4)42, Mate 9.6(4)42
Serial Number: Ours
Last Failover at: 00:39:39 HKST Dec 21 2022
This host: Primary - Active
Active time: 1411431 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.11): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.254): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.254): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.254): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.254): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.254): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.254): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.254): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.254): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.254): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.254): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.254): Normal (Monitored)
Interface management (192.168.10.102): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0.12-17) status (Up/Up)
ASA FirePOWER, 5.4.0.12-17, Up, (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.12): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.253): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.253): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.253): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.253): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.253): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.253): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.253): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.253): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.253): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.253): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.253): Normal (Monitored)
Interface management (192.168.10.101): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0-764) status (Up/Up)
ASA FirePOWER, 5.4.0-764, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : State_Link GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 59082938 0 211943 0
sys cmd 188242 0 188242 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 55867219 0 22304 0
UDP conn 2230814 0 1081 0
ARP tbl 796653 0 315 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 9 0 0 0
Router ID 0 0 0 0
User-Identity 1 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 25 3036007
Xmit Q: 0 30 59847409

This is for the Primary (defective one, now down and disconnected)

show mode
Security context mode: single
fk01ssc-1# show cluster
ERROR: % Incomplete command
fk01ssc-1# show cluster info
Clustering is not configured
fk01ssc-1# show interface ip brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES unset up up
GigabitEthernet0/0.2 10.23.2.11 YES CONFIG up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/1.10 170.11.10.254 YES CONFIG up up
GigabitEthernet0/1.11 170.11.11.254 YES CONFIG up up
GigabitEthernet0/1.12 170.11.12.254 YES CONFIG up up
GigabitEthernet0/1.13 170.11.13.254 YES CONFIG up up
GigabitEthernet0/1.14 170.11.14.254 YES CONFIG up up
GigabitEthernet0/2 unassigned YES unset up up
GigabitEthernet0/2.20 170.11.20.254 YES CONFIG up up
GigabitEthernet0/2.21 170.11.21.254 YES CONFIG up up
GigabitEthernet0/3 unassigned YES unset up up
GigabitEthernet0/3.30 170.11.30.254 YES CONFIG up up
GigabitEthernet0/3.31 170.11.31.254 YES CONFIG up up
GigabitEthernet0/3.32 170.11.32.254 YES CONFIG up up
GigabitEthernet0/3.34 170.11.34.254 YES CONFIG up up
GigabitEthernet0/4 unassigned YES unset administratively down down
GigabitEthernet0/5 unassigned YES unset administratively down down
GigabitEthernet0/6 192.168.20.1 YES unset up up
GigabitEthernet0/7 192.168.20.5 YES unset up up
Internal-Control0/0 127.0.1.1 YES unset up up
Internal-Data0/0 unassigned YES unset up up
Internal-Data0/1 unassigned YES unset up up
Internal-Data0/2 unassigned YES unset up up
Internal-Data0/3 169.254.1.1 YES unset up up
Management0/0 unassigned YES unset up up
GigabitEthernet1/0 192.168.10.102 YES CONFIG up up
GigabitEthernet1/1 unassigned YES unset administratively down down
GigabitEthernet1/2 unassigned YES unset administratively down down
GigabitEthernet1/3 unassigned YES unset administratively down down
GigabitEthernet1/4 unassigned YES unset administratively down down
GigabitEthernet1/5 unassigned YES unset administratively down down
fk01ssc-1# show run failover
failover
failover lan unit primary
failover lan interface LAN_Link GigabitEthernet0/6
failover polltime unit msec 500 holdtime 2
failover polltime interface msec 500 holdtime 5
failover key *****
failover link State_Link GigabitEthernet0/7
failover interface ip LAN_Link 192.168.20.1 255.255.255.252 standby 192.168.20.2
failover interface ip State_Link 192.168.20.5 255.255.255.252 standby 192.168.20.6
fk01ssc-1# show failover
Failover On
Failover unit Primary
Failover LAN Interface: LAN_Link GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 500 milliseconds, holdtime 2 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 13 of 316 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.6(4)42, Mate 9.6(4)42
Serial Number: Ours
Last Failover at: 00:39:39 HKST Dec 21 2022
This host: Primary - Active
Active time: 1411431 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.11): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.254): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.254): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.254): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.254): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.254): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.254): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.254): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.254): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.254): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.254): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.254): Normal (Monitored)
Interface management (192.168.10.102): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0.12-17) status (Up/Up)
ASA FirePOWER, 5.4.0.12-17, Up, (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5545 hw/sw rev (1.0/9.6(4)42) status (Up Sys)
Interface ek01ssc (10.23.2.12): Normal (Monitored)
Interface ek11ssc_vlan_10 (170.11.10.253): Normal (Monitored)
Interface ek11ssc_vlan_11 (170.11.11.253): Normal (Monitored)
Interface ek11ssc_vlan_12 (170.11.12.253): Normal (Monitored)
Interface ek11ssc_vlan_13 (170.11.13.253): Normal (Monitored)
Interface ek11ssc_vlan_14 (170.11.14.253): Normal (Monitored)
Interface ek21ssc_vlan_20 (170.11.20.253): Normal (Monitored)
Interface ek21ssc_vlan_21 (170.11.21.253): Normal (Monitored)
Interface ek31ssc_vlan_30 (170.11.30.253): Normal (Monitored)
Interface ek31ssc_vlan_31 (170.11.31.253): Normal (Monitored)
Interface ek31ssc_vlan_32 (170.11.32.253): Normal (Monitored)
Interface ek31ssc_vlan_34 (170.11.34.253): Normal (Monitored)
Interface management (192.168.10.101): Normal (Monitored)
slot 1: SFR5545 hw/sw rev (N/A/5.4.0-764) status (Up/Up)
ASA FirePOWER, 5.4.0-764, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : State_Link GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 59082938 0 211943 0
sys cmd 188242 0 188242 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 55867219 0 22304 0
UDP conn 2230814 0 1081 0
ARP tbl 796653 0 315 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 9 0 0 0
Router ID 0 0 0 0
User-Identity 1 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 25 3036007
Xmit Q: 0 30 59847409

wayne wan · ‎01-05-2023

Sorry, I had replied but I can't see my reply. Don't know if the configuration is too long?

I tried to attach here.

wayne wan · ‎01-05-2023

I made a more detail of my problem as below.

Hope it can be more understandable.

Case 1) Occurred on APR-2022
fk01ssc-1 (Primary) is active and fk02ssc-1 (Secondary) has the defective add-on network adapter card.
The management port was configured on the add-on network adapter card.
Thus, syslog can still be transferred to the CSM for log consolation.

log in CSM
==========
4/11/22 10:47:03AM Alert (Primary) Lost Failover communications with mate on interface management
4/11/22 10:47:03AM Alert (Primary) Testing Interface management
4/11/22 10:47:03AM Alert (Primary) Testing on Interface management Passed
4/11/22 10:47:03AM Alert (Primary) Testing Interface ek31ssc_vlan_34
:
4/11/22 10:47:03AM Alert (Primary) Testing Interface ek21ssc_vlan_20
4/11/22 10:47:03AM Alert (Primary) Lost Failover communications with mate on interface ek31ssc_vlan_32
:
4/11/22 10:47:03AM Alert (Primary) Lost Failover communications with mate on interface ek31ssc_vlan_34
4/11/22 10:47:03AM Alert (Primary) Testing Interface ek01ssc
4/11/22 10:47:03AM Alert (Primary) Testing on interface ek31ssc_vlan_31 Passed
:
4/11/22 10:47:03AM Alert (Primary) Testing on interface ek01ssc Passed
:
4/11/22 10:47:03AM Alert (Primary) Testing on interface ek11ssc_vlan_13 Status Undetermined
4/11/22 10:47:03AM Alert (Primary) No respone from other firewall (reason code = 4) No response from failover mate
4/11/22 10:47:03AM Alert (Primary) No respone from other firewall (reason code = 3) No response from failover mate

We connect the console to the firewall (not sure primary or secondary) , a lot of these message prompted.

Number of interfaces on Active and Standby are not consistent.
If the problem persists, you should disable and re-enable failover on the Standby.

After rebooted Secondary, since the add-on network adapter card is defective, we can see the rule cannot applied to the management interface. And also, the messages
“Number of interfaces on active and Standby are not consistent. If the problem persists, you should disable and re-enable failover on the standby.” got prompted after the Secondary was rebooted.

====================================================================================
Console on Secondary
====================================================================================
mtu management 1500
^
ERROR: % Invalid input detected at '^' marker.

http 192.168.10.0 255.255.255.0 management
^
ERROR: % Invalid input detected at '^' marker.

ssh 192.168.10.0 255.255.255.0 management
^
ERROR: % Invalid input detected at '^' marker.

Number of interfaces on Active and Standby are not consistent.
If the problem persists, you should disable and re-enable failover on the Standby.
====================================================================================

We connected the console to the Primary(active) and observed the same error message was prompted. We checked on the front panel both active LEDs were on.

====================================================================================
Console on Primary
====================================================================================
Number of interfaces on Active and Standby are not consistent.
If the problem persists, you should disable and re-enable failover on the Standby.

Switching to Failed state
====================================================================================

My question for this case is that why the whole network seems so busy such that some of the program were down due to they cannot communicatie to the gateway when both firewalls were powered on.
The acitve firewall was up and the defective firewall is standby, there was no failover occurred.
I also experienced when use the ASDM to connect the gateway port, it take very long time.
The problem was quiet down when we power disconnected the defective firewall.

Case 2) Occurred on DEC-2022

We replaced the defected firewall (Secondary) in APR. Now, on this case, the Primary firewall got the defective add-on network adapter card.

The symtopms were nearlly the same like the first case. Some of the programs were up and down between 04:52 PM to 06:09 PM .

After the incident was resloved, we checked on the CSM and find the follwoing log messages

12/20/22 04:52:49 PM Alert (Secondary) Switching to ACTIVE - HELLO not heard from mate.
12/20/22 06:16:16 PM Alert (Secondary) Failover interface Failed
12/20/22 06:16:16 PM Alert (Secondary) No response from other firewall (reason code = 4). No response from failover mate

The main difference is that at 18:09PM, we want to power disconnect the defective firewall to quiet down the problem base on last time's experience.
However, we did it on the wrong firewall, we power discconected the Secondary. We reveiwed the log after the incident and understand that The Secondary firewall is active and switched to active
at 04:52 since the primary got the defective add-on adapter card.

7 minutes later the Secondary is up. At that time period, we observed the network is still "busy" and programs were up and down.
Then we reload the Primary (the defective) one. Things became worse, many programs are totally down.
Then, we realised the Primary (the defective) is the real active and it needed to be up.

At this point, we observed the warning message prompted on both console .
"WARNING: Failover message decryption failure. Please make sure both units have the same failover shared key and crypto license or system is not out of memory"

So, we need to power disconnected the Secondary (healthy one) to make the network quiet down.

Finaly, we replaced the Primary firewall with a spare one to solve the problem.

For this case, the first part is the same as the case we faced in Apr.
For the second part, is that the "Failover message decryption failure" was triggered due to we stop the Secondary (healthy) firewall?

Marius Gunnerud · ‎01-06-2023

Is the HA pair stable if you remove the failover key?

--
Please remember to select a correct answer and rate helpful posts