Cisco ASA 5510 keeps 'disabling failover'

[drstokes76]

We have 2 ASA 5510s configured in a Active/Passive pair.

We've had a couple of incidents lately where the interconnect between our 2 sites goes down and the secondary firewall instead of switching to Active and staying there until it see the primary Active again and then failing back to Standby it does a 'disabling failover'.

This obviously causes us problems because we then have both firewalls thinking they're active with the same internal and external IP addresses.

Has anyone else experienced this?  Could it be a config issue?

We're running 7.20 on the ASAs AFAI can remember.

Any help would be appreciated as it seems to do this everytime there's a failure now.

Thanks

[Maykol Rojas]

Hello,

On normal circunstances, how is the filover looking on the show failover? Are all the interfaces on normal mode? Is the standby Unit as standby ready? Have you tested failover before? Normally when failover is disabled is because they dont have the same features on the license.

Let mek now.

Mike

[drstokes76]

Hi Mike

All the failover statuses are as normal.  Both are running the same version of software and license.  They've been in a pair for over 3 years, we failed over to the secondary because we're getting a lot of packet loss on the internal interface of the primary and can't work out why just yet.  The secondary was active for 3 days and then all of a sudden after the interconnect went down the primary went active and then the secondary disabled failover. Output of show failover below:

#show failover

Failover On
Failover unit Primary
Failover LAN Interface: Failover  Ethernet0/3 (up)
Unit Poll frequency 1 seconds, holdtime 15  seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface  Policy 1
Monitored Interfaces 2 of 250 maximum
Version: Ours 7.2(5), Mate  7.2(5)
Last Failover at: 14:43:12 UTC Feb 6 2012
        This  host: Primary - Active
                Active time: 11077  (sec)
                slot 0: ASA5510 hw/sw rev (1.1/7.2(5)) status (Up  Sys)
                  Interface outside (xx.xxx.144.4): Normal  (Waiting)
                  Interface inside (10.0.10.1): Normal  (Waiting)
                slot 1: empty
        Other host: Secondary -  Failed
                Active time: 15 (sec)
                slot 0:  ASA5510 hw/sw rev (1.1/7.2(5)) status (Unknown/Unknown)
                   Interface outside (xx.xxx.146.5): Normal
                  Interface inside  (10.0.10.2): Normal
                slot 1: empty

[Maykol Rojas]

Can you take the show failover history of both units?

Mike

[drstokes76]

This is from the primary

==========================================================================

From State                 To State                   Reason

==========================================================================

Active Applying Config     Active Config Applied      Other unit wants me Active

Active Config Applied      Active                     Other unit wants me Active

Active                     Standby Ready              Set by the config command

Standby Ready              Bulk Sync                  No Error

Bulk Sync                  Standby Ready              No Error

Standby Ready              Bulk Sync                  No Error

Bulk Sync                  Standby Ready              No Error

Standby Ready              Bulk Sync                  No Error

Bulk Sync                  Standby Ready              No Error

Standby Ready              Just Active                ACK not received for failover message

Just Active                Active Drain               ACK not received for failover message

Active Drain               Active Applying Config     ACK not received for failover message

Active Applying Config     Active Config Applied      ACK not received for failover message

Active Config Applied      Active                     ACK not received for failover message

Active                     Standby Ready              Set by the config command

Standby Ready              Just Active                Failover state check

Just Active                Active Drain               Failover state check

Active Drain               Active Applying Config     Failover state check

Active Applying Config     Active Config Applied      Failover state check

Active Config Applied      Active                     Failover state check

==========================================================================

This is from the secondary

==========================================================================

From State                 To State                   Reason

==========================================================================

Not Detected               Negotiation                No Error

Negotiation                Cold Standby               Detected an Active mate

Cold Standby               Sync Config                Detected an Active mate

Sync Config                Sync File System           Detected an Active mate

Sync File System           Bulk Sync                  Detected an Active mate

Bulk Sync                  Standby Ready              Detected an Active mate

Standby Ready              Just Active                ACK not received for failover message

Just Active                Active Drain               ACK not received for failover message

Active Drain               Active Applying Config     ACK not received for failover message

Active Applying Config     Active Config Applied      ACK not received for failover message

Active Config Applied      Active                     ACK not received for failover message

Active                     Cold Standby               Failover state check

Cold Standby               Disabled                   HA state progression failed

Disabled                   Negotiation                Set by the config command

Negotiation                Cold Standby               Detected an Active mate

Cold Standby               Sync Config                Detected an Active mate

Sync Config                Sync File System           Detected an Active mate

Sync File System           Bulk Sync                  Detected an Active mate

Bulk Sync                  Standby Ready              Detected an Active mate

Standby Ready              Failed                     Interface check

==========================================================================

I've just run failover on the secondary again so guess this accounts for the last 7 lines.

[Maykol Rojas]

Bingo,

From State                 To State                   Reason

Cold Standby               Disabled                   HA state progression failed

What is the exact version you are running?

Mike

[drstokes76]

They're both

Cisco Adaptive Security Appliance Software Version 7.2(5)

[Maykol Rojas]

Checking for common cases, they point to a bug, but seems like the version you are running is not affected. Some other cases shown that after re-enable the failover, the issue was not present.

I would recommend you to schedule maintenance window, and fail the units back and forth and see if you can reproduce the issue. If you do, please gather:

debug fover msg

debug fover sync

Mike

[ganesh.cs]

Hi Maykol/Darren,

Did you manage to resolve this issue ?.

I am having the same issue here, running at 7.2(4)30. Cisco TAC says the bug is only for 8.0.

They will check with their development team and revert.

Please advice,

Gan

[Maykol Rojas]

What is your TAC case number? I can double check.

Mean time, check the following:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsz13255

It sounds like a bug to me, but I can go ahead and double check the outputs taken by the engineer that owns your ticket. Layer 1 has nothing to do with HA progression failed nor ASA stuck in Bulk sync.

Mike

[ganesh.cs]

Hello Maykol,

TAC case is 622094833. When we initially had the firewall disabling the failover, TAC suggest it could be related to high connections and recommended to remove on of our log servers. We have removed one of the log servers now. Will check on the progress.

I manage to bump into this bug when I was looking into the error and Cisco TAC is suppose to revert on the status.

Our firewall is running at 7.2(x). The bug is for 8.0. My question to them, will the older version be included and vulnerable to the bug.

Cheers

Gan

[Ramraj Sivagnanam Sivajanam]

Hi Bro

Personally, I don' think this is a bug related issue. I suspect this is L1 issue instead. Are the heartbeat cables between both the Cisco FWs directly connected or via a L2 switch?

I'm assuming your configuration is correct, but if you'd like, you could paste your latest "show running-config interface" and "show running-config failover" here and I will assist you.

[ganesh.cs]

Hi Ramraj,

The firewall is configured normally.

interface Ethernet0/0

description xxxxxx

speed 100

duplex full

nameif outside

security-level 0

ip address x.x.x.x 255.255.255.248 standby x.x.x.x

!

interface Ethernet0/1

description xxxx

speed 100

duplex full

nameif dmz3

security-level 90

ip address x.x.x.x  standby x.x.x.x

!

interface Ethernet0/2

speed 100

duplex full

shutdown

no nameif

security-level 10

no ip address

!

interface Ethernet0/3

shutdown

no nameif

no security-level

no ip address

!

interface Management0/0

description xxxxx

speed 100

duplex full

nameif fwmgmt

security-level 99

ip address x.x.x.x x.x.x.x standby x.x.x.x

management-only

!

interface GigabitEthernet1/0

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/1

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/2

shutdown

no nameif

no security-level

no ip address

!

interface GigabitEthernet1/3

description LAN/STATE Failover Interface

speed 1000

duplex full

failover

failover lan unit primary

failover lan interface state GigabitEthernet1/3

failover link state GigabitEthernet1/3

failover interface ip state x.x.x.1 255.255.255.252 standby x.x.x.2

==============================================

[Ramraj Sivagnanam Sivajanam]

Hi Bro

Are the heartbeat cables between both the Cisco FWs directly connected or via a L2 switch?