Can't access secondary ASA with Active/Active Failover configura

shaunoc46 · ‎04-12-2011

Hello everyone. Hopefully someone will have the knowledge to help me out here. I'm working with two ASA 5510's that are loaded with the same hardware/software and set up with Active/Active failover. Before we started to have issues both of the ASA's would be reachable through Telnet and SSH. Now the secondary ASA isn't even pingable and the only way to access it is through the console port. The ASA's have the exact same configuration for failover and sync just fine when I do a copy run start or write mem command on the Primary ASA, the secondary ASA receives the config just fine from the primary. The other issue I have is when I initiate a failover to the secondary ASA I drop connection altogether and am forced to do a no failover active command to set the Primary ASA as being active once more. I'm not sure if I'm having a hardware issue with my inside interface or maybe it's as something as simple as a bad cable, but I'd like to get some thoughts around here while I'm replacing the cable to eliminate that cause. Below are some command line responses I get from the two ASA's.

ASA's IOS running:

Cisco Adaptive Security Appliance Software Version 8.2(1)
Device Manager Version 6.2(1)

asa821-k8.bin

Show run failover from Primary ASA:

DC-Edge-FW1# show run failover
failover
failover lan unit primary
failover lan interface Failover Ethernet0/2
failover key *****
failover link state Ethernet0/3
failover interface ip Failover 10.0.0.2 255.255.255.0 standby 10.0.0.1
failover interface ip state 10.0.1.2 255.255.255.0 standby 10.0.1.1

Show run failover from Standby ASA:

DC-Edge-FW1# show run failover
failover
failover lan unit secondary
failover lan interface Failover Ethernet0/2
failover key *****
failover link state Ethernet0/3
failover interface ip Failover 10.0.0.2 255.255.255.0 standby 10.0.0.1
failover interface ip state 10.0.1.2 255.255.255.0 standby 10.0.1.1

Show failover state from Primary:

DC-Edge-FW1# show failover state

               State          Last Failure Reason      Date/Time
This host -   Primary
               Active         None
Other host -   Secondary
               Failed         Ifc Failure              13:05:52 CDT Apr 12 2011
                              inside: Failed

====Configuration State===
        Sync Done
        Sync Done - STANDBY
====Communication State===
        Mac set

Show failover state from Secondary:

DC-Edge-FW1# show failover state

               State          Last Failure Reason      Date/Time
This host -   Secondary
               Failed         Ifc Failure              13:05:52 CDT Apr 12 2011
                              inside: Failed
Other host -   Primary
               Active         None

====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set

** I did a debug fover ifc command on the secondary ASA where the failed inside interface resides and here are the results:***

fover_health_monitoring_thread: ifc_check() group: 0, - time = 6298430
fover_health_monitoring_thread: vPifNum = 0x4, Shut Down
fover_health_monitoring_thread: ifc_check() - group 0 HW failed 1 (mate 0)

Show interface command from Primary ASA:

Interface Ethernet0/1 "inside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
        Description: Interface to Inside Secure Network
        MAC address 0018.73d6.ba25, MTU 1500
        IP address 10.1.0.4, subnet mask 255.255.255.0
        4597452 packets input, 2171926666 bytes, 0 no buffer
        Received 127506 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        1 L2 decode drops
        5413224 packets output, 4612290138 bytes, 0 underruns

show interface command from secondary ASA:

Interface Ethernet0/1 "inside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
        Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps)
        Description: Interface to Inside Secure Network
        MAC address 0025.45ce.ab33, MTU 1500
        IP address 10.1.0.5, subnet mask 255.255.255.0
        13230 packets input, 1163949 bytes, 0 no buffer
        Received 10029 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 L2 decode drops
        7743 packets output, 1015689 bytes, 0 underruns

I've also ensured that the appropriate VLAN's are enabled on the switching ports these two ASA's are connected through by the inside interface. I've tried to reboot the secondary ASA and have also did a failover reset command on it but no luck so far. Anyone have any ideas on where to start or maybe have seen these issues before? I'd really like to make sure our failover configuration works like it's supposed too. If any additional information is needed please just let me know.

Thanks!!!

shaunoc46 · ‎04-12-2011

Cable has been replaced with the same issues existing.

brquinn · ‎04-12-2011

Shaun,

If you are connected via telnet/ssh to the active ASA and then run 'no failover active', it is expected that your session will drop. When the standby unit becomes active, it assumes the IP and MAC address from the previously active unit.

After failing over, you should be able to connect back to the same IP and check the 'show failover' output to verify that the other unit is now active. You can always access the standby unit via the standby IP address.

Note: Please do not confuse Active/Standby with Primary/Secondary. Active/Standby define the current state of the ASA. Primary/Secondary define the role.

Regarding the dropped ICMP... does the issue follow the standby ASA when you failover or does it always affect the same unit?

Thanks,

Brendan

shaunoc46 · ‎04-12-2011

Thanks for your replay Brendan. Earlier today I exectured the command "failover active" on the standby ASA to force it to failover and become the primary. The configurations got pushed across the ASA's and they synced up but I lost all connection until I was able to do a no failover active on the standby so that the Primary ASA once again became the active. Both of the ASA's current state are Active/Active. The role of one ASA is set to Primary and the other ASA's role is set to Standby. I'm unable to reach the ASA with the assigned role of Standby. I'm unable to ping, telnet, or SSH to the standby IP from anywhere, including the primary ASA and the switch connected to the standby ASA. I'm even unable to ping the switch that the standby ASA is connected too from that standby ASA. All traffic over the inside interface just hits 10.1.0.5 and then nothing after that, almost like it's having a routing issue but the same config's are on the two ASA's so they have the exact same routing table. I'm able to ping all of the failover links between the two ASA's but the main inside interface on the standby ASA is unreachable. What would cause the inside interface of the standby ASA to result in failure? Thanks again!

DC-Edge-FW1# show failover state

               State          Last Failure Reason      Date/Time
This host -   Secondary
               Failed         Ifc Failure              13:48:24 CDT Apr 12 2011
                              inside: Failed
Other host -   Primary
               Active         None

====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set

brquinn · ‎04-12-2011

Shaun,

Do you have any data when the Secondary ASA was active and the traffic was failing? A quick packet capture on the interface will tell you if the ASA is receiving your traffic. Do you log to a syslog server? Is there a gap in the logs during the time the Secondary was active?

After a failover, the newly active unit sends out Gratuitious ARP messages to update the mac address tables on your switches. (otherwise the switch would continue to forward your traffic to the wrong ASA) Did you check the mac-address-table on the adjacent switch to make sure the entry changed properly? Also, check to make sure you don't have a static entry.

Thanks,

Brendan

shaunoc46 · ‎04-14-2011

I was pretty annoyed with this so took a break, came back and started from scratch and it ended up being a VLAN configuration error between a couple of stacked switches. Once the VLAN issues got resolved the ASA's worked perfectly. It's always an easier fix than you think. Thanks for trying to help me out on this!

brquinn · ‎04-14-2011

Good catch! It didn't smell like an ASA issue to me. I'm glad you found it. :-)

Thanks,

Brendan

Erick.Valle · ‎04-29-2012

Shaun, would you be able to indicate what the Vlan issue that you had was and what it took to resolved?

Can't access secondary ASA with Active/Active Failover configuration