Secondary/Stdby ASA suddenly stopped responding

[Ve Con]

Suddenly the primary/active ASA (5515-x with FirePOWER) reported "lost failover communications with mate on interface ..." (all interfaces) 

primary ASA then tested all the interfaces and result with "passed".  Then it threw another error message "no response from other firewall (reason code = 4)"  

Switching to ACTIVE - HELLO no heard from mate

I had to go to power cycled the secondary ASA and it came up fine

before powering off the secondary ASA, it has all the green lights on just like with the primary/active ASA.  The alarm and VPN lights were off as expected, same with other ASA too.

Can anyone guide me where to look for the reason why the secondary ASA suddenly stop responding? 

I looked at the versions on both ASA, they look all the same.  The Source FirePOWER sensors status look good on ASDM status too

Thanks!

[AlexPi]

Sorry if I am asking something that you might already done, but have you checked the fail-over interfaces (on each firewall) for errors or anything unusual? Check for input errors, collisions, interface resets (that you have not initiated) etc. Also check for speed and/or duplex mismatch between the two firewalls and even try setting them manually instead of auto.

Also check if the cable which connects the two fail-over interfaces between them is in good condition.

[Ve Con]

Hi Alex,

While the 2ndary was unresponsive, I did the "show tech" on the primary/active ASA, and saved it out to a document the result.

Looking back into the 'show tech' result document:
1) --show interface --- has 0 collisions, 0 output errors, 0 deferred, 0 late collisions for all the reported interfaces, 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored or abort, 0 input/output reset drops

2) The FAILOVER interface has this line:

0 output errors, 0 collisions, 1 interface resets

While other Interfaces has this line:

0 output errors, 0 collisions, 2 interface resets


I don't understand what it means by 2 interfaces resets vs 1 interface resets
Can someone help?

auo-Duplex(full-duplex), auto-speed
No mismatch reported

Failover interface shows as "is up, line protocol is up"

How should I change it to manual speed? What will it affect?

Thanks!

[Florin Barhala]

Very smart from you to use "show tech" before reboot.
I have no idea so far except review show tech output when it reaches show failover or any failover related message. Can you post it?

[Ve Con]

Hi Florin,

Here the output of the "show failover" from the "show tech" result before power cycling the secondary ASA (i changed the IP addresses for security purposes):

Failover On 
Failover unit Primary
Failover LAN Interface: LAN-FAILOVER GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 5 of 114 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.6(4)3, Mate 9.6(4)3
Serial Number: Ours FCH1902K9CS, Mate FCH1902K9DK
Last Failover at: 16:08:25 EDT May 10 2018
	This host: Primary - Active 
		Active time: 6643 (sec)
		slot 0: ASA5515 hw/sw rev (1.0/9.6(4)3) status (Up Sys)
		  Interface INTERNAL-1 (x.x.x.3): Normal (Waiting)
		  Interface DMZ-2 (x.x.x.3): Normal (Waiting)
		  Interface EXTERNAL-P (x.x.x.250): Normal (Waiting)
		  Interface INTERNAL (x.x.x.9): Normal (Waiting)
		  Interface management (0.0.0.0): Normal (Waiting)
		slot 1: SFR5515 hw/sw rev (N/A/6.1.0.6-85) status (Up/Up)
		  ASA FirePOWER, 6.1.0.6-85, Up, (Monitored)
	Other host: Secondary - Failed 
		Active time: 6572299 (sec)
		slot 0: ASA5515 hw/sw rev (1.0/9.6(4)3) status (Unknown/Unknown)
		  Interface INTERNAL-1 (x.x.x.2): Unknown (Monitored)
		  Interface DMZ-2 (x.x.x.2): Unknown (Monitored)
		  Interface EXTERNAL-P (x.x.x.249): Unknown (Monitored)
		  Interface INTERNAL (x.x.x.8): Unknown (Monitored)
		  Interface management (0.0.0.0): Unknown (Waiting)
		slot 1: SFR5515 hw/sw rev (N/A/6.1.0.6-85) status (Unknown/Unknown)
		  ASA FirePOWER, 6.1.0.6-85, Unknown, (Monitored)

Stateful Failover Logical Update Statistics
	Link : LAN-FAILOVER GigabitEthernet0/2 (up)
	Stateful Obj 	xmit       xerr       rcv        rerr      
	General		882524     0          22726070   139       
	sys cmd  	876611     0          876609     0         
	up time  	0          0          0          0         
	RPC services  	0          0          0          0         
	TCP conn 	3523       0          11386569   0         
	UDP conn 	244        0          278819     0         
	ARP tbl  	2141       0          10182899   0         
	Xlate_Timeout  	0          0          0          0         
	IPv6 ND tbl  	0          0          0          0         
	VPN IKEv1 SA 	0          0          0          0         
	VPN IKEv1 P2 	0          0          0          0         
	VPN IKEv2 SA 	0          0          0          0         
	VPN IKEv2 P2 	0          0          0          0         
	VPN CTCP upd 	0          0          0          0         
	VPN SDI upd 	0          0          0          0         
	VPN DHCP upd 	0          0          0          0         
	SIP Session 	0          0          132        0         
	SIP Tx 	0          0          99         0         
	SIP Pinhole 	0          0          0          0         
	Route Session 	4          0          0          139       
	Router ID 	0          0          0          0         
	User-Identity 	1          0          943        0         
	CTS SGTNAME 	0          0          0          0         
	CTS PAC 	0          0          0          0         
	TrustSec-SXP 	0          0          0          0         
	IPv6 Route 	0          0          0          0         
	STS Table 	0          0          0          0         

	Logical Update Queue Information
	 	 	Cur 	Max 	Total
	Recv Q: 	0 	30 	26532088
	Xmit Q: 	0 	30 	883675

[AlexPi]

Having a quick look to the little info you posted from the show tech I find it weird that you have interface resets but without any errors on the interfaces. As far as I can tell this may also indicate a mismatch at the duplex and speed or issues with your cable, or issues with the fail-over protocol itself as it chats between the two interfaces.

As currently your fail-over is not functioning basically I would try to set the duplex and speed at full and 1000. To do that I would go to the port (on each firewall ideally) since I am not sure if having fail-over issues also messes up with the replication from primary to secondary.

To set those parameters go to the port which the fail-over uses and do duplex full and speed 1000. If you see no difference remove those commands and interfaces will go back to auto.

Also I would still try changing the physical medium that connects the two fail-over ports.

Hope that helps.

[Ve Con]

Hi Alex,

Not sure what you mean by "As currently your fail-over is not functioning". The result of "show failover state" doesn't report any error:

Result of the command: "show failover state"

State Last Failure Reason Date/Time
This host - Primary
Active None
Other host - Secondary
Standby Ready None

====Configuration State===
Sync Done
Sync Done - STANDBY
====Communication State===
Mac set

From my understanding (pls correct me if i am wrong),  the failover over doesn't happen if it still able to communicate with other interfaces on the other ASA.  In this case, it lost communication to all the interfaces all at once, that's why I didn't think about the cable issue. 

Unless if the failover port communication has issue, then it fails communicate to the rest of interfaces on other ASA, (if it is how it supposes to work) then i think the cable issue is possible.  

Will changing the speed possibly cause any interruption to the ASA? It is production, i don't want to change anything outside of maintenance time window.

Thanks for your helps!

[Ve Con]

Update information:

When i turned off the secondary with "no failover"

Use FireSIGHT to generate troubleshooting files, it caused secondary to be locked up again that I had to power cycle, couldn't do anything with the console 

Same result if I leave failover = on on 2ndary ASA

One thing i am not sure if this works as designed:

1) when "no failover" on 2ndary, the result of "show failover" shows the failover off on 2ndary, but on primary it shows as on for "show failover" command

2) "show failover state" displays "secondary disabled" on both firewalls

 ===========

Today I did the Rule update on FireSIGHT, when it tried to deploy to the 2ndary ASA, that failed and caused the 2ndary ASA to lock up as well.  I had to hard reset it.