topic Re: ASA Active/Standby failover issues. in Network Security

ASA Active/Standby failover issues.

K-Grev — Thu, 15 Oct 2020 02:30:59 GMT

Hi all,

Occasionally (twice a month or so) our ASA 5585's will fail over to the standby unit. I haven't been able to understand why this is happening so I'm reaching out for help.

Looking through logs around the time of the failover I'm unable to see anything that would trigger this event.

Adding some "show" information from both ASA's in hopes it has something useful.

***************** Firewall 1 *********************

B1560-38A-R1R4-FW1-EXT/sec/act# show run failover
failover
failover lan unit secondary
failover lan interface LAN-FAIL Port-channel1
failover polltime unit 1 holdtime 5
failover polltime interface 1 holdtime 5
failover key *****
failover link LAN-FAIL Port-channel1
failover interface ip LAN-FAIL 192.168.100.1 255.255.255.252 standby 192.168.100.2

----------------

B1560-38A-R1R4-FW1-EXT/sec/act# show failover history
==========================================================================
From State To State Reason
==========================================================================
00:02:05 UTC Apr 26 2013
Not Detected Negotiation No Error

00:02:29 UTC Apr 26 2013
Negotiation Cold Standby Detected an Active mate

00:02:31 UTC Apr 26 2013
Cold Standby Sync Config Detected an Active mate

19:52:39 UTC Sep 10 2020
Sync Config Sync File System Detected an Active mate

19:52:39 UTC Sep 10 2020
Sync File System Bulk Sync Detected an Active mate

19:52:54 UTC Sep 10 2020
Bulk Sync Standby Ready Detected an Active mate

19:53:00 UTC Sep 10 2020
Standby Ready Failed Interface check
This host:1
single_vf: Outside
Other host:0

19:53:06 UTC Sep 10 2020
Failed Standby Ready Interface check
This host:0
Other host:0

21:49:05 UTC Oct 8 2020
Standby Ready Just Active Other unit wants me Active

21:49:05 UTC Oct 8 2020
Just Active Active Drain Other unit wants me Active

21:49:05 UTC Oct 8 2020
Active Drain Active Applying Config Other unit wants me Active

21:49:05 UTC Oct 8 2020
Active Applying Config Active Config Applied Other unit wants me Active

21:49:05 UTC Oct 8 2020
Active Config Applied Active Other unit wants me Active

==========================================================================

----------

B1560-38A-R1R4-FW1-EXT/sec/act# show fail state

State Last Failure Reason Date/Time
This host - Secondary
Active Ifc Failure 19:53:33 UTC Sep 10 2020
Other host - Primary
Standby Ready Ifc Failure 21:49:05 UTC Oct 8 2020
inside: No Link

====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set

----------------

B1560-38A-R1R4-FW1-EXT/sec/act# show failover statistics
tx:11651090
rx:17094117

--------------------

B1560-38A-R1R4-FW1-EXT/sec/act# show fai
B1560-38A-R1R4-FW1-EXT/sec/act# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: LAN-FAIL Port-channel1 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 5 seconds
Interface Poll frequency 1 seconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 2 of 1049 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.9(2), Mate 9.9(2)
Serial Number: Ours JAD19080262, Mate JAD16260523
Last Failover at: 21:49:05 UTC Oct 8 2020
This host: Secondary - Active
Active time: 533981 (sec)
slot 0: ASA5585-SSP-60 hw/sw rev (3.0/9.9(2)) status (Up Sys)
Interface management (10.2.21.138): Normal (Not-Monitored)
Interface inside (10.2.5.34): Normal (Monitored)
Interface Outside (10.2.0.114): Normal (Monitored)
slot 1: empty
slot 1: empty
Other host: Primary - Standby Ready
Active time: 2426323 (sec)
slot 0: ASA5585-SSP-60 hw/sw rev (2.2/9.9(2)) status (Up Sys)
Interface management (10.2.21.140): Normal (Not-Monitored)
Interface inside (10.2.5.36): Normal (Monitored)
Interface Outside (10.2.0.116): Normal (Monitored)
slot 1: empty
slot 1: empty

Stateful Failover Logical Update Statistics
Link : LAN-FAIL Port-channel1 (up)
Stateful Obj xmit xerr rcv rerr
General 9970155 0 47746303 8434
sys cmd 394694 0 394694 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 1341545 0 11876437 0
UDP conn 8026043 0 34326770 315
ARP tbl 308 0 1546 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 553 0 10879 0
VPN IKEv1 P2 1809 0 38147 0
VPN IKEv2 SA 202070 0 1064828 0
VPN IKEv2 P2 600 0 5322 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 56 0
SIP Tx 0 0 42 0
SIP Pinhole 0 0 0 0
Route Session 603 0 272 8119
Router ID 0 0 0 0
User-Identity 1930 0 27310 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 21 113346385
Xmit Q: 0 20 15872958

----------------------------

******************* Firewall 2 ***********************

Firewall 2

B1560-38A-R1R4-FW1-EXT/pri/stby# show run failover
failover
failover lan unit primary
failover lan interface LAN-FAIL Port-channel1
failover polltime unit 1 holdtime 5
failover polltime interface 1 holdtime 5
failover key *****
failover link LAN-FAIL Port-channel1
failover interface ip LAN-FAIL 192.168.100.1 255.255.255.252 standby 192.168.100.2

----------

B1560-38A-R1R4-FW1-EXT/pri/stby# show failover history
==========================================================================
From State To State Reason
==========================================================================
19:51:49 UTC Sep 10 2020
Not Detected Negotiation No Error

19:52:17 UTC Sep 10 2020
Negotiation Just Active No Active unit found

19:52:17 UTC Sep 10 2020
Just Active Active Drain No Active unit found

19:52:17 UTC Sep 10 2020
Active Drain Active Applying Config No Active unit found

19:52:17 UTC Sep 10 2020
Active Applying Config Active Config Applied No Active unit found

19:52:17 UTC Sep 10 2020
Active Config Applied Active No Active unit found

21:49:05 UTC Oct 8 2020
Active Failed Interface check
This host:1
single_vf: inside
Other host:0

21:49:16 UTC Oct 8 2020
Failed Standby Ready Interface check
This host:0
Other host:0

==========================================================================

-------------

B1560-38A-R1R4-FW1-EXT/pri/stby# show fail state

State Last Failure Reason Date/Time
This host - Primary
Standby Ready Ifc Failure 21:49:44 UTC Oct 8 2020
Other host - Secondary
Active Ifc Failure 19:53:00 UTC Sep 10 2020
Outside: No Link

====Configuration State===
Sync Done
====Communication State===
Mac set
------------------

B1560-38A-R1R4-FW1-EXT/pri/stby# show failover statistics
tx:17094790
rx:11652228

------------------------

B1560-38A-R1R4-FW1-EXT/pri/stby# show failover
Failover On
Failover unit Primary
Failover LAN Interface: LAN-FAIL Port-channel1 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 5 seconds
Interface Poll frequency 1 seconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 2 of 1049 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.9(2), Mate 9.9(2)
Serial Number: Ours JAD16260523, Mate JAD19080262
Last Failover at: 21:49:05 UTC Oct 8 2020
This host: Primary - Standby Ready
Active time: 2426323 (sec)
slot 0: ASA5585-SSP-60 hw/sw rev (2.2/9.9(2)) status (Up Sys)
Interface management (10.2.21.140): Normal (Not-Monitored)
Interface inside (10.2.5.36): Normal (Monitored)
Interface Outside (10.2.0.116): Normal (Monitored)
slot 1: empty
slot 1: empty
Other host: Secondary - Active
Active time: 533992 (sec)
slot 0: ASA5585-SSP-60 hw/sw rev (3.0/9.9(2)) status (Up Sys)
Interface management (10.2.21.138): Normal (Not-Monitored)
Interface inside (10.2.5.34): Normal (Monitored)
Interface Outside (10.2.0.114): Normal (Monitored)
slot 1: empty
slot 1: empty

Stateful Failover Logical Update Statistics
Link : LAN-FAIL Port-channel1 (up)
Stateful Obj xmit xerr rcv rerr
General 47785879 0 9952414 623
sys cmd 394696 0 394696 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 11876841 0 1341518 0
UDP conn 34343637 0 8009859 48
ARP tbl 1546 0 308 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 16647 0 376 0
VPN IKEv1 P2 38147 0 1809 0
VPN IKEv2 SA 1073244 0 201290 0
VPN IKEv2 P2 5322 0 600 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 56 0 0 0
SIP Tx 42 0 0 0
SIP Pinhole 0 0 0 0
Route Session 8391 0 28 575
Router ID 0 0 0 0
User-Identity 27310 0 1930 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 18 15856868
Xmit Q: 0 27 113363658
---------------------

Thanks for any advice or help for what I should look for.

Re: ASA Active/Standby failover issues.

MHM Cisco World — Thu, 15 Oct 2020 02:45:52 GMT

the Reason the primary is failed

because of inside interface is down,

other time outside is down,

do you connect the both asa to same sw for inside and outside ?

Re: ASA Active/Standby failover issues.

K-Grev — Thu, 15 Oct 2020 02:50:55 GMT

Thanks for your reply.

Outside goes to a Cisco 6500, inside is a Cisco 4500X. Well there is an IPS inline before the 4500X also.

Re: ASA Active/Standby failover issues.

balaji.bandi — Thu, 15 Oct 2020 09:32:53 GMT

inside: No Link

this shows something gone wrong inside interface, check Physical Path or any congestion in the link - which failed to establish connection so failover took place.

or do you have any recent upgrade or network changes ?

Re: ASA Active/Standby failover issues.

Marius Gunnerud — Thu, 15 Oct 2020 12:19:19 GMT

Have you checked the logs on the switch the ASA connects to if the interface there has been flapping?

Re: ASA Active/Standby failover issues.

K-Grev — Thu, 15 Oct 2020 22:10:41 GMT

I checked the logs on the 4500X and the 6500. Neither of them seem to report anything happening.

Re: ASA Active/Standby failover issues.

K-Grev — Thu, 15 Oct 2020 22:11:28 GMT

No recent network changes, I'll try to check history of port usage.

Re: ASA Active/Standby failover issues.

K-Grev — Fri, 16 Oct 2020 01:09:01 GMT

Sorry, checked again and found these logs.

2020-10-08 16:49:05 Local2.Notice B1560-38A-R1R4-RTR1-INT 250580: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: Interface Te2/1/31 left the port-channel Po25
2020-10-08 16:49:06 Local2.Notice B1560-38A-R1R4-RTR1-INT 250584: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: STANDBY:Interface Te2/1/31 left the port-channel Po25
2020-10-08 16:49:17 Local2.Notice B1560-38A-R1R4-RTR1-INT 250589: Oct 8 2020 21:49:16 UTC: %EC-5-BUNDLE: Interface Te2/1/31 joined port-channel Po25
2020-10-08 16:49:18 Local2.Notice B1560-38A-R1R4-RTR1-INT 250592: Oct 8 2020 21:49:16 UTC: %EC-5-BUNDLE: STANDBY:Interface Te2/1/31 joined port-channel Po25

2020-10-08 16:49:05 Local2.Notice B1560-38A-R1R4-RTR1-INT 250581: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: Interface Te1/1/31 left the port-channel Po25
2020-10-08 16:49:06 Local2.Notice B1560-38A-R1R4-RTR1-INT 250585: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: STANDBY:Interface Te1/1/31 left the port-channel Po25
2020-10-08 16:49:17 Local2.Notice B1560-38A-R1R4-RTR1-INT 250590: Oct 8 2020 21:49:16 UTC: %EC-5-BUNDLE: Interface Te1/1/31 joined port-channel Po25
2020-10-08 16:49:18 Local2.Notice B1560-38A-R1R4-RTR1-INT 250593: Oct 8 2020 21:49:16 UTC: %EC-5-BUNDLE: STANDBY:Interface Te1/1/31 joined port-channel Po25

2020-10-08 16:49:05 Local2.Notice B1560-38A-R1R4-RTR1-INT 250582: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: Interface Te1/1/30 left the port-channel Po26
2020-10-08 16:49:06 Local2.Notice B1560-38A-R1R4-RTR1-INT 250586: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: STANDBY:Interface Te1/1/30 left the port-channel Po26
2020-10-08 16:49:18 Local2.Notice B1560-38A-R1R4-RTR1-INT 250595: Oct 8 2020 21:49:18 UTC: %EC-5-BUNDLE: Interface Te1/1/30 joined port-channel Po26
2020-10-08 16:49:20 Local2.Notice B1560-38A-R1R4-RTR1-INT 250597: Oct 8 2020 21:49:18 UTC: %EC-5-BUNDLE: STANDBY:Interface Te1/1/30 joined port-channel Po26

2020-10-08 16:49:05 Local2.Notice B1560-38A-R1R4-RTR1-INT 250583: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: Interface Te2/1/30 left the port-channel Po26
2020-10-08 16:49:06 Local2.Notice B1560-38A-R1R4-RTR1-INT 250587: Oct 8 2020 21:49:04 UTC: %EC-5-UNBUNDLE: STANDBY:Interface Te2/1/30 left the port-channel Po26
2020-10-08 16:49:17 Local2.Notice B1560-38A-R1R4-RTR1-INT 250591: Oct 8 2020 21:49:16 UTC: %EC-5-BUNDLE: Interface Te2/1/30 joined port-channel Po26
2020-10-08 16:49:18 Local2.Notice B1560-38A-R1R4-RTR1-INT 250594: Oct 8 2020 21:49:16 UTC: %EC-5-BUNDLE: STANDBY:Interface Te2/1/30 joined port-channel Po26

Po25 is for FW1

Po26 is for FW2

Re: ASA Active/Standby failover issues.

Marius Gunnerud — Fri, 16 Oct 2020 08:28:56 GMT

Have you made sure that there have not been any changes to the Port-channel configuration at either end? i.e. all interfaces are using the same bundle protocol?

Have you checked if there are any interface errors under show interface command on both switch and ASA?

Check the port-channel status on the switch, show etherchannel summary

Re: ASA Active/Standby failover issues.

K-Grev — Fri, 16 Oct 2020 23:48:04 GMT

So it turns out that these times coincide with someone messing with the portchannel config.

Thanks for your time!