[toc:faq]
Introduction
Sometimes when you have two ASAs in failover, and you enter a command like "svc image ...", if you aren't consoled in you may lose connectivity to the ASA itself and when you reconnect the the command isn't there anymore. This document explains why this happens as well as the solution for it.
Prerequisites
Requirements
For information regarding configuration failover on an ASA please go through the following documents:
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_configuration_example09186a00807dac5f.shtml
Explanation
Consider the following syslogs:
Nov 30 2011 12:58:33 test-asa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=411,op=30,my=Active,peer=Standby Ready.
Nov 30 2011 12:58:33 test-asa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_CLIENT_NEGOTIATED_VERSION, my state Active, peer state Standby Ready.
>> At this point the secondary is active, and the primary is standby.
Nov 30 2011 12:58:33 test-asa : %ASA-5-111008: User 'enable_15' executed the 'svc image disk0:/anyconnect-win-3.0.4235-k9.pkg' command.
>> User enters command 'svc image disk0:/anyconnect-win-3.0.4235-k9.pkg'
Nov 30 2011 12:58:33 test-asa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=406,op=130,my=Active,peer=Active.
Nov 30 2011 12:58:33 test-asa : %ASA-6-720028: (VPN-Secondary) HA status callback: Peer state Active.
Nov 30 2011 12:58:33 test-asa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Active.
Nov 30 2011 12:58:34 test-asa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=406,op=80,my=Active,peer=Standby Ready.
Nov 30 2011 12:58:34 test-asa : %ASA-6-720028: (VPN-Secondary) HA status callback: Peer state Standby Ready.
Nov 30 2011 12:58:34 test-asa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Standby Ready.
Nov 30 2011 12:58:34 test-asa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=406,op=130,my=Active,peer=Active.
Nov 30 2011 12:58:34 test-asa : %ASA-6-720028: (VPN-Secondary) HA status callback: Peer state Active.
Nov 30 2011 12:58:34 test-asa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Active.
>> As soon as the user enters the command, both devices become active
Nov 30 2011 12:58:34 test-asa : %ASA-1-103004: (Secondary) Other firewall reports this firewall failed.
Nov 30 2011 12:58:34 test-asa : %ASA-1-104002: (Secondary) Switching to STNDBY - Other unit reports that I am failed
Nov 30 2011 12:58:34 test-asa : %ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=52,op=12,my=Failed,peer=Active.
Nov 30 2011 12:58:34 test-asa : %ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_FAILED, my state Failed, peer state Active.
Nov 30 2011 12:58:34 test-asa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=405,op=20,my=Failed,peer=Active.
Nov 30 2011 12:58:34 test-asa : %ASA-6-720027: (VPN-Secondary) HA status callback: My state Failed.
Nov 30 2011 12:58:34 test-asa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_MY_STATE, my state Failed, peer state Active.
>> Secondary firewall which was active until now is informed by the Primary firewall that it has apparently failed, this explains why both devices became active. However, on receiving this communication, the secondary firewal, to diffuse the dual active fw situation switches to failed.
Nov 30 2011 12:58:48 test-asa : %ASA-6-611101: User authentication succeeded: Uname: admin
Nov 30 2011 12:58:48 test-asa : %ASA-6-605005: Login permitted from 10.11.9.60/1491 to inside:10.11.254.1/ssh for user "admin"
Nov 30 2011 12:58:50 test-asa : %ASA-5-502103: User priv level changed: Uname: enable_15 From: 1 To: 15
>> Because the secondary "failed" the user was kicked out and is forced to log back in.
Nov 30 2011 12:58:54 test-asa : %ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=101,op=18,my=Sync Config,peer=Active.
Nov 30 2011 12:58:54 test-asa : %ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_STANDBY_CONFIG, my state Sync Config, peer state Active.
>>At this point the secondary has the configuration "svc image disk0:/anyconnect-win-3.0.4235-k9.pk", but the primary doesn't since it hasn't synced with the secondary yet. However, because the secondary "failed" it now tries to sync with the active device, which is the primary.
Nov 30 2011 12:58:54 test-asa : %ASA-1-709006: (Secondary) End Configuration Replication (STB)
>> Once configuration is replicated, the secondary will no longer have the svc image command because it synced from a device that didn't have the command.
This explains why the command got removed entirely, but it doesn't explain why the primary ASA suddenly thought that the secondary had failed. In order to understand that consider the following portion of the configuration:
failover polltime unit msec 400 holdtime 4
failover polltime interface msec 500 holdtime 5
This means that the devices are polling each other every 400msec to see if the other device is still active. Now the command we tried to execute is something that's very likely to cause a CPU hog for at least a second, which means that during that second that secondary firewall is in no state to reply to the primary firewall, thus the primary firewall assumes that the secondary is "dead" and send out the communication:
Nov 30 2011 12:58:34 test-asa : %ASA-1-104002: (Secondary) Switching to STNDBY - Other unit reports that I am failed
Solution
Modify the unit poll time to a more reasonable value allowing the ASAs to complete such exhaustive commands.
