Secondary ASA repeatedly Testing

[CiscoMedMed]

After a brief network flap on my secondary ASA 5525 the secondary unit is perpetually testing then passing. 
It's weird that link status would change for those three interfaces at the same time because those are three
separate devices they're connected to and nothing else on those devices went down nor did those switches reboot. Any thought on this? Thanks. 

May 13 2021 09:11:48 BOS-ASA01 : %ASA-1-105007: (Secondary) Link status 'Down' on interface OUTSIDE
May 13 2021 09:11:48 BOS-ASA01 : %ASA-1-105007: (Secondary) Link status 'Down' on interface DMVPNDMZ
May 13 2021 09:11:48 BOS-ASA01 : %ASA-1-105007: (Secondary) Link status 'Down' on interface management
May 13 2021 09:18:28 BOS-ASA01 : %ASA-1-105006: (Secondary) Link status 'Up' on interface OUTSIDE
May 13 2021 09:18:28 BOS-ASA01 : %ASA-1-105006: (Secondary) Link status 'Up' on interface DMVPNDMZ
May 13 2021 09:18:30 BOS-ASA01 : %ASA-1-105006: (Secondary) Link status 'Up' on interface management
May 13 2021 09:18:30 BOS-ASA01 : %ASA-1-104004: (Secondary) Switching to OK.
May 13 2021 09:18:38 BOS-ASA01 : %ASA-1-105003: (Secondary) Monitoring on interface OUTSIDE waiting
May 13 2021 09:18:38 BOS-ASA01 : %ASA-1-105003: (Secondary) Monitoring on interface DMVPNDMZ waiting
May 13 2021 09:18:38 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:18:38 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:18:41 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Failed
May 13 2021 09:18:41 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Failed
May 13 2021 09:18:58 BOS-ASA01 : %ASA-1-104004: (Secondary) Switching to OK.
May 13 2021 09:19:13 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:13 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:19:15 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:19:15 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Passed
May 13 2021 09:19:28 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:28 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:19:29 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:19:29 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Passed
May 13 2021 09:19:43 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:43 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:19:44 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:19:44 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Passed
May 13 2021 09:19:58 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:58 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:19:58 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Passed
May 13 2021 09:19:59 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:13 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:20:13 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:20:14 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:14 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Passed
May 13 2021 09:20:28 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:20:28 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:20:29 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:29 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Passed
May 13 2021 09:20:43 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:20:43 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ
May 13 2021 09:20:43 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:43 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface DMVPNDMZ Passed
May 13 2021 09:20:58 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:20:58 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface DMVPNDMZ

[Sheraz.Salim]

I would check the Physical links/spanning tree state of these interface/s (vlan) and also I shall look at the logs on the switch these ASA interface are connected. For example

May 13 2021 09:11:48 BOS-ASA01 : %ASA-1-105007: (Secondary) Link status 'Down' on interface OUTSIDE
May 13 2021 09:18:28 BOS-ASA01 : %ASA-1-105006: (Secondary) Link status 'Up' on interface OUTSIDE
May 13 2021 09:18:30 BOS-ASA01 : %ASA-1-104004: (Secondary) Switching to OK.
May 13 2021 09:18:38 BOS-ASA01 : %ASA-1-105003: (Secondary) Monitoring on interface OUTSIDE waiting
May 13 2021 09:18:38 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:18:41 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Failed
May 13 2021 09:18:58 BOS-ASA01 : %ASA-1-104004: (Secondary) Switching to OK.
May 13 2021 09:19:13 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:15 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:19:28 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:29 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:19:43 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:44 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:19:58 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:19:59 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:13 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:20:14 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:28 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:20:29 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:43 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE
May 13 2021 09:20:43 BOS-ASA01 : %ASA-1-105009: (Secondary) Testing on interface OUTSIDE Passed
May 13 2021 09:20:58 BOS-ASA01 : %ASA-1-105008: (Secondary) Testing Interface OUTSIDE

there is a 7minutes difference between OUTSIDE interface went down at 09:11 and cambe up at 09:18.

check the logs this firewall is connected to the L2/L3 devices.

105008

Error Message %ASA-1-105008: (Primary) Testing interface interface_name.

Explanation Testing of a specified network interface has occurred. This testing is performed only if the ASA fails to receive a message from the standby unit on that interface after the expected interval. Primary can also be listed as Secondary for the secondary unit.

Recommended Action None required.

105009

Error Message %ASA-1-105009: (Primary) Testing on interface interface_name {Passed|Failed}.

Explanation The result (either Passed or Failed) of a previous interface test has been reported. Primary can also be listed as Secondary for the secondary unit.

Recommended Action None required if the result is Passed. If the result is Failed, you should check the network cable connection to both failover units, that the network itself is functioning correctly, and verify the status of the standby unit.

could you share the output of your ASA.

show run failover

show failover history

show failover status

!

You can run debug on the active/Standby firewall to check if heartbeat is working or if there any issues in it "debug fover txip". can you also ping the ASA failover links.

[CiscoMedMed · Accepted solution]

So the answer turned out to be in the upstream Cat3K. The Smart Install feature had a vulnerability 

that has since been corrected. That explains why the OUTSIDE and DMVPNDMZ were both being

affected at the same time while being separate physical connections and INSIDE was not ever 
impacted.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg76186

https://software.cisco.com/download/home/286280258/type/282046477/release/Gibraltar-16.12.5b