Failover didn't worked

[size57]

hello All,

I have faced a strange problem recently.

I am using ASA5520 with OS verison 8.0.4 in ative standby failover.

We have observed that failover not happned my network was down for 15 minutes. when i checked i was not able login to primary ASA.

Same time secondary ASA was not allowing login, after some time logged into secondary firewall bu it was showing standby,

Then forcefully secondary to active and everything started working fine.

Also observed "no failover" command in my primary ASA and i was not able to ping any of production interfaces,

But failover link i was able to ping.

My queries are :

1) Why failover didn't worked

2) Why after forcing to secondary manually to active it started working fine

3) how is that possible "no failover" command automatically present there in configuration.

is there any bug relaetd to this,

[Shrikant Sundaresh]

Hey,

Could you please provide us with the configuration on the Primary ASA, and also the "show version" output of both the primary and the secondary ASAs?

Also please provide the output of "sh failover history"

When the Primary was not working correctly, and secondary was still in standby, then there was no ASA with the correct(active) IP addresses. As a result things weren't working. When you made the Secondary Active, the correct ip address was now present on a working device. Hence things started working.

I don't think this would be a bug, though I shall look into it.

However, the most important thing is: Are you sure that the primary ASA had caused the 15 minute network outage?

Since you were unable to reach both, I am inclined to think that the switch to which both are connected to, could have been down, or if you were trying to access it from the internet, then your internet connection could have been down during that time.

One more question: You said you couldn't ping any of the Primary's interfaces. From where were you trying to ping??

Kindly let me know the above, and I shall try to help you out further.

-Shrikant

[size57]

I tried to ping from seconadry firewall to primary firewall all interfaces ip addresses. found only failover link reachable.

automatically no failover command appeadred on primary firewall.

below is show failover history :

Active Drain               Active Applying Config     HELLO not heard from mate

10:56:39 IST Apr 2 2011
Active Applying Config     Active Config Applied      HELLO not heard from mate

10:56:39 IST Apr 2 2011
Active Config Applied      Active                     HELLO not heard from mate

10:56:42 IST Apr 2 2011
Active                     Failed                     Other unit reports that I am failed

10:56:48 IST Apr 2 2011
Failed                     Cold Standby               Failover state check

10:56:49 IST Apr 2 2011
Cold Standby               Just Active                Failover state check

10:56:49 IST Apr 2 2011
Just Active                Active Drain               Failover state check

10:56:49 IST Apr 2 2011
Active Drain               Active Applying Config     Failover state check

             
10:56:49 IST Apr 2 2011
Active Applying Config     Active Config Applied      Failover state check

10:56:49 IST Apr 2 2011
Active Config Applied      Active                     Failover state check

17:20:12 IST Apr 15 2011
Active                     Failed                     Other unit reports that I am failed

17:20:22 IST Apr 15 2011
Failed                     Cold Standby               Failover state check

17:20:29 IST Apr 15 2011
Cold Standby               Just Active                Failover state check

17:20:29 IST Apr 15 2011
Just Active                Active Drain               Failover state check

17:20:29 IST Apr 15 2011
Active Drain               Active Applying Config     Failover state check

17:20:29 IST Apr 15 2011
Active Applying Config     Active Config Applied      Failover state check

17:20:29 IST Apr 15 2011
Active Config Applied      Active                     Failover state check

18:10:18 IST Apr 15 2011
Active                     Failed                     Other unit reports that I am failed

18:10:19 IST Apr 15 2011
Failed                     Cold Standby               Recovered from communication failure

18:10:34 IST Apr 15 2011
Cold Standby               Disabled                   HA state progression failed

sh version :

Cisco Adaptive Security Appliance Software Version 8.0(4)
Device Manager Version 6.1(5)

Compiled on Thu 07-Aug-08 20:53 by builders
System image file is "disk0:/asa804-k8.bin"
Config file at boot was "startup-config"

JUSTPRIMARY up 41 days 5 hours

Hardware:   ASA5520, 512 MB RAM, CPU Pentium 4 Celeron 2000 MHz
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xffe00000, 1024KB

Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)
                             Boot microcode   :  CN1000-MC-BOOT-2.00
                             SSL/IKE microcode:  CNLite-MC-SSLm-PLUS-2.03
                             IPSec microcode  :  CNlite-MC-IPSECm-MAIN-2.05
0: Ext: GigabitEthernet0/0  : address is 0024.14d1.f140, irq 9
1: Ext: GigabitEthernet0/1  : address is 0024.14d1.f141, irq 9
2: Ext: GigabitEthernet0/2  : address is 0024.14d1.f142, irq 9
3: Ext: GigabitEthernet0/3  : address is 0024.14d1.f143, irq 9
4: Ext: Management0/0       : address is 0024.14d1.f13f, irq 11
5: Int: Not used            : irq 11
6: Int: Not used            : irq 5

              Licensed features for this platform:
Maximum Physical Interfaces  : Unlimited
Maximum VLANs                : 150      
Inside Hosts                 : Unlimited
Failover                     : Active/Active
VPN-DES                      : Enabled  
VPN-3DES-AES                 : Enabled  
Security Contexts            : 2        
GTP/GPRS                     : Disabled 
VPN Peers                    : 750      
WebVPN Peers                 : 2        
AnyConnect for Mobile        : Disabled 
AnyConnect for Linksys phone : Disabled 
Advanced Endpoint Assessment : Disabled 
UC Proxy Sessions            : 2       

This platform has an ASA 5520 VPN Plus license.

Serial Number:

Running Activation Key:

Configuration register is 0x1
Configuration last modified by enable_15 at 18:06:49.060 IST Fri Apr 15 2011

[Shrikant Sundaresh]

Hi,

I had requested the sh version of both devices to make sure that they have the same licenses (which is a prerequisite for failover).

However, since you provided only 1, I cannot determine that. Maybe you can just look it up and let me know if they are the same currently. I think they should be.

If you see the last line of the output of "show failover history", you would see that the failover was disabled on the ASA.

18:10:34 IST Apr 15 2011
Cold Standby               Disabled                   HA state progression failed

This is when the "no failover" command comes automatically on the ASA.

Can you please tell me which ASA you took the "show failover history" from? Also, do you have any logs collected around 6:10 PM IST??

Judging by the reasons provided in the output of "sh fail history", i think there is some problem in the communication between the two ASAs.

You might wanna look into the duplex/speed settings of the failover link if it is through a switch. Maybe try changing the wires even.

Then enable failover, and test if it fails over correctly, during a maintenance window.

Hope this helps.

-Shrikant

P.S.: Please mark the question as answered if it has been resolved. Do rate helpful posts. Thanks.

[size57]

Thank you Shrikanth for the your inouts.

Both ASA are having same license. Even this setup is running from last 2 years.

Also the last log from "show failover history" is showing "HA state progression failed" is confusing.

I tried simulating this through GNS and oberved that if somebody manually give no failover then output of "show failover history" wuld be something like.

00:23:34 UTC Nov 30 1999
Active                     Disabled                   Set by the CI config cmd

please comment some thing on this.

Secondly you said " This is when the "no failover" command comes automatically on the ASA.", but my question is how it is possible "no failover" command appeared automaticaly.

And yes definately we have checked the speed/duplex was fine. But yes problem symptoms indicate problem with switch connected to primary ASA.

We are already working on it, but till no luck also there are logs available during the incident period.

[Shrikant Sundaresh]

Hi,

Basically, if the Failover state goes to Disabled for any reason other than "Set by the CI config cmd" then the "no failover" command comes automatically into the configuration. I shall try to find details on what "HA state progression failed" means, but that is the reason why ASA decided to disable Failover; and the only way to disable failover, is to add the "no failover" command to the configuration.

When the ASA adds the command you won't see "Set by the CI config cmd" in the reason. This means that the command was manually entered in the CLI of the ASA.

If you have logs, I would suggest you analyze them for the Primary ASA, to see what exactly happened, during the 15 minute network outage.

Hope this helps.

-Shrikant

P.S.: Please mark the question as answered, if it has been resolved. Do rate all helpful posts. Thanks.

[size57]

Dear Shrikanth,

Thank you for all your help.

Below is the error observed.

%PIX|ASA-1-105001: (Primary) Disabling failover.

explaination is available but not matching to my scenario.

[Shrikant Sundaresh]

Hi,

I went through the explanation of the syslog as well. I don't think all possible reasons for seeing that are explained there.

I think you should look into the syslogs that led up to the 105001 syslog. Those would indicate as to why Failover needed to be disabled.

If you are not comfortable sharing the syslogs on the forums or the syslog file is too large to be attached, you could open a TAC case, if you have a contract entitled to TAC support. An analysis of the syslogs, with a knowledge of the ip scheme in your topology, would help in finding out why the failover got disabled.

Hope this helps.

-Shrikant

P.S: Please mark the question as answered, if it has been resolved. Do rate helpful posts. Thanks.