topic Re: ASA Random Failovers in Network Security

ASA Random Failovers

NETAD — Fri, 21 Feb 2020 15:12:37 GMT

Hello I'm working on a ASA 5512 ver 9.4(1) that keeps failing over randomly like once every other week.

When the failed firewall is reboot, it comes back like normal. Can you please assist? Last I did was to disable the service-card monitoring and this is still happening.

here's the last messages from the show failover history:

03:59:44 EST Jan 25 2018

Standby Ready Just Active HELLO not heard from mate

03:59:44 EST Jan 25 2018

Just Active Active Drain HELLO not heard from mate

03:59:44 EST Jan 25 2018

Active Drain Active Applying Config HELLO not heard from mate

03:59:44 EST Jan 25 2018

Active Applying Config Active Config Applied HELLO not heard from mate

03:59:44 EST Jan 25 2018

Active Config Applied Active HELLO not heard from mate

sh failo state

State Last Failure Reason Date/Time

This host - Secondary

Active Ifc Failure 09:08:58 EST May 29 2017

inside: No Link

Other host - Primary

Failed Comm Failure 03:59:44 EST Jan 25 2018

Re: ASA Random Failovers

Francesco Molino — Thu, 25 Jan 2018 20:05:47 GMT

Hi

Can you share your config and the output of show failover?
Here it says that you got ifc failure on inside.

Re: ASA Random Failovers

NETAD — Thu, 25 Jan 2018 20:33:09 GMT

Here's the config. I checked the inside interface on the switch side and the firewall and they both look good. When we reboot the failed firewall this ifc failure goes away.

ciscoasa/sec/act# sh run fail

failover

failover lan unit secondary

failover lan interface FAILOVER+STATE GigabitEthernet0/5

failover link FAILOVER+STATE GigabitEthernet0/5

failover interface ip FAILOVER+STATE 169.254.255.1 255.255.255.252 standby 169.254.255.2

Re: ASA Random Failovers

NETAD — Thu, 25 Jan 2018 21:21:54 GMT

Here's the show failover.

ciscoasa/sec/act# sh fail

Failover On

Failover unit Secondary

Failover LAN Interface: FAILOVER+STATE GigabitEthernet0/5 (up)

Reconnect timeout 0:00:00

Unit Poll frequency 1 seconds, holdtime 15 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 4 of 114 maximum

MAC Address Move Notification Interval not set

Version: Ours 9.4(1), Mate 9.4(1)

Last Failover at: 03:59:44 EST Jan 25 2018

This host: Secondary - Active

Active time: 44302 (sec)

slot 0: ASA5512 hw/sw rev (1.0/9.4(1)) status (Up Sys)

Interface outside (208.75.175.97): Normal (Waiting)

Interface inside (192.168.250.2): Normal (Waiting)

Interface DMZ (192.168.1.1): Normal (Waiting)

Interface PUBLIC_WIFI (172.31.255.1): Normal (Waiting)

Interface BCMSSpec (173.43.121.1): Normal (Not-Monitored)

Interface BCMSStaff (154.43.175.1): Normal (Not-Monitored)

slot 1: SFR5512 hw/sw rev (N/A/5.4.0-764) status (Up/Up)

ASA FirePOWER, 5.4.0-764, Up

Other host: Primary - Failed

Active time: 841905 (sec)

slot 0: ASA5512 hw/sw rev (1.0/9.4(1)) status (Unknown/Unknown)

Interface outside (208.75.175.98): Unknown (Monitored)

Interface inside (192.168.250.3): Unknown (Monitored)

Interface DMZ (192.168.1.253): Unknown (Monitored)

Interface PUBLIC_WIFI (172.31.255.2): Unknown (Monitored)

Interface BCMSSpec (173.43.121.2): Unknown (Not-Monitored)

Interface BCMSStaff (154.43.175.2): Unknown (Not-Monitored)

slot 1: SFR5512 hw/sw rev (N/A/5.4.0-764) status (Unknown/Unknown)

ASA FirePOWER, 5.4.0-764, Unknown

Stateful Failover Logical Update Statistics

Link : FAILOVER+STATE GigabitEthernet0/5 (up)

Stateful Obj xmit xerr rcv rerr

General 1485994 0 338087620 4311

sys cmd 1136394 0 1136391 0

up time 0 0 0 0

RPC services 0 0 0 0

TCP conn 226204 0 207005024 3

UDP conn 113503 0 119573402 4

ARP tbl 9840 0 10303060 0

Xlate_Timeout 0 0 0 0

IPv6 ND tbl 0 0 0 0

VPN IKEv1 SA 10 0 13373 0

VPN IKEv1 P2 16 0 17903 0

VPN IKEv2 SA 0 0 0 0

VPN IKEv2 P2 0 0 0 0

VPN CTCP upd 0 0 0 0

VPN SDI upd 0 0 0 0

VPN DHCP upd 0 0 0 0

SIP Session 0 0 7750 0

SIP Tx 0 0 4401 0

SIP Pinhole 0 0 3123 666

Route Session 1 0 0 3638

Router ID 0 0 0 0

User-Identity 26 0 23193 0

CTS SGTNAME 0 0 0 0

CTS PAC 0 0 0 0

TrustSec-SXP 0 0 0 0

IPv6 Route 0 0 0 0

STS Table 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 25 365785225

Xmit Q: 0 90 1495942

Re: ASA Random Failovers

Francesco Molino — Fri, 26 Jan 2018 05:03:52 GMT

It's not able to see the primary unit interfaces.
How the standby interface is connected? Through a vlan or direct cable?

If you use a vlan can you validate full connectivity between them?

Can you share config of primary asa?

Re: ASA Random Failovers

NETAD — Fri, 26 Jan 2018 06:43:33 GMT

DO you mean how the failover link is connected? And do you need the full config?

Re: ASA Random Failovers

johnlloyd_13 — Fri, 26 Jan 2018 10:04:28 GMT

hi,

are the active and standby ASA units directly connected on their G0/5?

kindly post a show int g0/5 output to check for any errors.

Re: ASA Random Failovers

NETAD — Fri, 26 Jan 2018 13:46:32 GMT

They seem to be directly connected.

ciscoasa/sec/act# sh fail
Failover On
Failover unit Secondary
Failover LAN Interface: FAILOVER+STATE GigabitEthernet0/5 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 114 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.4(1), Mate 9.4(1)
Last Failover at: 03:59:44 EST Jan 25 2018
This host: Secondary - Active
Active time: 44302 (sec)
slot 0: ASA5512 hw/sw rev (1.0/9.4(1)) status (Up Sys)
Interface outside (208.75.175.97): Normal (Waiting)
Interface inside (192.168.250.2): Normal (Waiting)
Interface DMZ (192.168.1.1): Normal (Waiting)
Interface PUBLIC_WIFI (172.31.255.1): Normal (Waiting)
Interface BCMSSpec (173.43.121.1): Normal (Not-Monitored)
Interface BCMSStaff (154.43.175.1): Normal (Not-Monitored)
slot 1: SFR5512 hw/sw rev (N/A/5.4.0-764) status (Up/Up)
ASA FirePOWER, 5.4.0-764, Up
Other host: Primary - Failed
Active time: 841905 (sec)
slot 0: ASA5512 hw/sw rev (1.0/9.4(1)) status (Unknown/Unknown)
Interface outside (208.75.175.98): Unknown (Monitored)
Interface inside (192.168.250.3): Unknown (Monitored)
Interface DMZ (192.168.1.253): Unknown (Monitored)
Interface PUBLIC_WIFI (172.31.255.2): Unknown (Monitored)
Interface BCMSSpec (173.43.121.2): Unknown (Not-Monitored)
Interface BCMSStaff (154.43.175.2): Unknown (Not-Monitored)
slot 1: SFR5512 hw/sw rev (N/A/5.4.0-764) status (Unknown/Unknown)
ASA FirePOWER, 5.4.0-764, Unknown

Stateful Failover Logical Update Statistics
Link : FAILOVER+STATE GigabitEthernet0/5 (up)
Stateful Obj xmit xerr rcv rerr
General 1485994 0 338087620 4311
sys cmd 1136394 0 1136391 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 226204 0 207005024 3
UDP conn 113503 0 119573402 4
ARP tbl 9840 0 10303060 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 10 0 13373 0
VPN IKEv1 P2 16 0 17903 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 7750 0
SIP Tx 0 0 4401 0
SIP Pinhole 0 0 3123 666
Route Session 1 0 0 3638
Router ID 0 0 0 0
User-Identity 26 0 23193 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 25 365785225
Xmit Q: 0 90 1495942

Re: ASA Random Failovers

Francesco Molino — Fri, 26 Jan 2018 15:19:29 GMT

This output says that failover is passing through g0/5 but behind the hood, is it a direct cable or is there a switch or multiple switch transporting a vlan?

The 2nd asa is in failed status and interfaces attached to it are in unknown state.

Can you validate the physical or logical path between them?

Re: ASA Random Failovers

NETAD — Fri, 26 Jan 2018 15:28:40 GMT

They just confirmed they’re directly connected.

Re: ASA Random Failovers

Francesco Molino — Fri, 26 Jan 2018 15:30:54 GMT

Can you post the output of sh int g0/5?

Re: ASA Random Failovers

NETAD — Fri, 26 Jan 2018 16:06:30 GMT

ciscoasa/sec/act# sh int g0/5
Interface GigabitEthernet0/5 "FAILOVER+STATE", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN/STATE Failover Interface
MAC address f44e.0522.29f2, MTU 1500
IP address 169.254.255.2, subnet mask 255.255.255.252
215693704 packets input, 222179420058 bytes, 0 no buffer
Received 827 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
16934623 packets output, 1701954458 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 10 interface resets
0 late collisions, 0 deferred
3 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (485/444)
output queue (blocks free curr/low): hardware (459/366)
Traffic Statistics for "FAILOVER+STATE":
215693607 packets input, 214542825912 bytes
16934498 packets output, 1340471394 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 0 bytes/sec
1 minute output rate 0 pkts/sec, 6 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 0 bytes/sec
5 minute output rate 0 pkts/sec, 6 bytes/sec
5 minute drop rate, 0 pkts/sec
ciscoasa/sec/act#

Re: ASA Random Failovers

NETAD — Sun, 28 Jan 2018 23:56:28 GMT

Hi Francesco, anything from that show inter gi0/5?

Re: ASA Random Failovers

Francesco Molino — Mon, 29 Jan 2018 00:31:13 GMT

Nothing wrong on this output.

You said when you reboot everything is back normal?

Can you reboot the standby and paste the show failover again?

Do you see something on logs?

Re: ASA Random Failovers

NETAD — Tue, 30 Jan 2018 15:11:38 GMT

This is after the reload.

ciscoasa/pri/act# sh fail state

State Last Failure Reason Date/Time

This host - Primary

Active None

Other host - Secondary

Standby Ready None

====Configuration State===

Sync Done - STANDBY

====Communication State===

ciscoasa/pri/act# sh fail

Failover On

Failover unit Primary

Failover LAN Interface: FAILOVER+STATE GigabitEthernet0/5 (up)

Reconnect timeout 0:00:00

Unit Poll frequency 1 seconds, holdtime 15 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 4 of 114 maximum

MAC Address Move Notification Interval not set

Version: Ours 9.4(1), Mate 9.4(1)

Last Failover at: 13:17:10 EST Jan 27 2018

This host: Primary - Active

Active time: 246742 (sec)

slot 0: ASA5512 hw/sw rev (1.0/9.4(1)) status (Up Sys)

Interface outside (208.75.175.97): Normal (Monitored)

Interface inside (192.168.250.2): Normal (Monitored)

Interface DMZ (192.168.1.1): Normal (Monitored)

Interface PUBLIC_WIFI (172.31.255.1): Normal (Monitored)

Interface BCMSSpec (173.43.121.1): Normal (Not-Monitored)

Interface BCMSStaff (154.43.175.1): Normal (Not-Monitored)

slot 1: SFR5512 hw/sw rev (N/A/5.4.0-764) status (Up/Up)

ASA FirePOWER, 5.4.0-764, Up

Other host: Secondary - Standby Ready

Active time: 206229 (sec)

slot 0: ASA5512 hw/sw rev (1.0/9.4(1)) status (Up Sys)

Interface outside (208.75.175.98): Normal (Monitored)

Interface inside (192.168.250.3): Normal (Monitored)

Interface DMZ (192.168.1.253): Normal (Monitored)

Interface PUBLIC_WIFI (172.31.255.2): Normal (Monitored)

Interface BCMSSpec (173.43.121.2): Normal (Not-Monitored)

Interface BCMSStaff (154.43.175.2): Normal (Not-Monitored)

slot 1: SFR5512 hw/sw rev (N/A/5.4.0-764) status (Up/Up)

ASA FirePOWER, 5.4.0-764, Up

Stateful Failover Logical Update Statistics

Link : FAILOVER+STATE GigabitEthernet0/5 (up)

Stateful Obj xmit xerr rcv rerr

General 7634432 0 40390 10

sys cmd 32931 0 32931 0

up time 0 0 0 0

RPC services 0 0 0 0

TCP conn 4861505 0 5132 4

UDP conn 2478582 0 1947 6

ARP tbl 260210 0 359 0

Xlate_Timeout 0 0 0 0

IPv6 ND tbl 0 0 0 0

VPN IKEv1 SA 405 0 5 0

VPN IKEv1 P2 770 0 6 0

VPN IKEv2 SA 0 0 0 0

VPN IKEv2 P2 0 0 0 0

VPN CTCP upd 0 0 0 0

VPN SDI upd 0 0 0 0

VPN DHCP upd 0 0 0 0

SIP Session 0 0 0 0

SIP Tx 0 0 0 0

SIP Pinhole 0 0 0 0

Route Session 16 0 0 0

Router ID 0 0 0 0

User-Identity 14 0 10 0

CTS SGTNAME 0 0 0 0

CTS PAC 0 0 0 0

TrustSec-SXP 0 0 0 0

IPv6 Route 0 0 0 0

STS Table 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 25 41188

Xmit Q: 0 7 8349280

Re: ASA Random Failovers

Francesco Molino — Wed, 31 Jan 2018 03:25:15 GMT

It happens several times right?
Make sure to save all logs to be able to see what's going on when it fails.
You can send logs to a syslog server to save them