Getting failover working again after upgrade from 8.2.2 to 8.4.2

Ken Stieers · ‎09-06-2011

Ok, some history. When we had 8.2.2, we bought a Mobile license to make the iPads running AnyConnect happy.

I applied it, but since we'd only purchased one license, it broke failover. 8.4 lets you share tracking licenses, and since we were planning on the upgrade to 8.4.x anyway, I figured no big deal, I'll get that straightened out when I do the upgrade.

Did the upgrade this weekend, and I still can't get things happy, the boxes don't see one-another:

Here's a show failover on the primary:

Failover On
Failover unit Primary
Failover LAN Interface: failover GigabitEthernet0/3 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 6 of 160 maximum
failover replication http
Version: Ours 8.4(2), Mate Unknown
Last Failover at: 12:19:08 CDT Sep 3 2011
        This host: Primary - Active
                Active time: 279890 (sec)
                slot 0: ASA5520 hw/sw rev (2.0/8.4(2)) status (Up Sys)
                  Interface outside (0.0.0.0): Normal (Waiting)
                  Interface Opus-Inet (72.172.144.100): Normal (Waiting)
                  Interface Qwest-Inet (63.224.24.11): Normal (Waiting)
                  Interface Qwest-VPN (208.45.168.115): Normal (Waiting)
                  Interface inside (172.16.15.100): Normal (Waiting)
                  Interface DMZ (172.25.0.1): Normal (Waiting)
                slot 1: empty
        Other host: Secondary - Failed
                Active time: 0 (sec)
                slot 0: empty
                  Interface outside (0.0.0.0): Unknown (Waiting)
                  Interface Opus-Inet (72.172.144.101): Unknown (Waiting)
                  Interface Qwest-Inet (63.224.24.12): Unknown (Waiting)
                  Interface Qwest-VPN (208.45.168.116): Unknown (Waiting)
                  Interface inside (172.16.15.101): Unknown (Waiting)
                  Interface DMZ (172.25.0.2): Unknown (Waiting)
                slot 1: empty

Stateful Failover Logical Update Statistics
Link : failover GigabitEthernet0/3 (up)

The secondary shows basically the same thing when I go to turn on failover..., but then since it says it can't find the active box it tries to go active and then things fall over...

Here are the failover configs as they stand at the moment.

***Primary***

failover

failover lan unit primary

failover lan interface failover GigabitEthernet0/3

failover replication http

failover link failover GigabitEthernet0/3

failover interface ip failover 172.16.13.1 255.255.255.252 standby 172.16.13.2

monitor-interface Opus-Inet

monitor-interface Qwest-Inet

monitor-interface Qwest-VPN

***Secondary***

no failover

failover lan unit secondary

failover lan interface failover GigabitEthernet0/3

failover replication http

failover link failover GigabitEthernet0/3

failover interface ip failover 172.16.13.1 255.255.255.252 standby 172.16.13.2

monitor-interface Opus-Inet

monitor-interface Qwest-Inet

monitor-interface Qwest-VPN

As soon as I issue the "failover" command on the secondary, it all goes sideways.

A "debug fover cable" on the primary shows arps coming back from all of the interfaces...

fover_health_monitoring_thread: fover_lan_check() Failover Interface TEST started

fover_health_monitoring_thread: send_mate_arp(0x9) - 172.16.13.2

fover_health_monitoring_thread: send_mate_arps(3) - 72.172.144.101

fover_health_monitoring_thread: send_mate_arps(5) - 63.224.24.12

fover_health_monitoring_thread: send_mate_arps(6) - 208.45.168.116

fover_health_monitoring_thread: send_mate_arps(7) - 172.16.15.101

fover_health_monitoring_thread: send_mate_arps(8) - 172.25.0.2

ARP Thread: ARP response received (9) - 172.16.13.2

ARP Thread: ARP response received (3) - 72.172.144.101

ARP Thread: ARP response received (5) - 63.224.24.12

ARP Thread: ARP response received (6) - 208.45.168.116

ARP Thread: ARP response received (7) - 172.16.15.101

ARP Thread: ARP response received (8) - 172.25.0.2

fover_health_monitoring_thread: fover_lan_check() Failover LAN Check

fover_health_monitoring_thread: fover_lan_check() Failover Interface OK

fover_health_monitoring_thread: fover_lan_check() Failover LAN Check

fover_health_monitoring_thread: fover_chk_my_down_ifcs() Local unit has 0 down ifcs

Any suggestions on getting this happy without taking down my firewall would be most welcome!

Thanks

Ken

Maykol Rojas · ‎09-06-2011

Ken,

Can you try just connecting the Failover Link on the secondary Unit? Then check if you can ping the secondary IP for the failover link. If you can, configuration will replicate from the primary to the standby. Once the configuration is replicated... connect the cables... wait for a period of time, if the interfaces dont come to a normal state, go ahead and check if you can ping each of the interfaces that show fail.

If you cannot ping them from the primary, check if you have a valid arp for them... If you do, track on the switches were those packets are going.

Mike

Ken Stieers · ‎09-07-2011

Hey Mike,

Neither unit can ping the other with just the failover link in place. The cable between the 2 is a crossover cable, and I have link lights.. "show interface" on both units shows the interface and line protocol as up.

Arp caches look correct...

A "show failover" shows them as connected, and that switchover can occur...

Is there an access list that needs to be set??

If there's a clue stick around, please hit me with it!

Ken

Ken Stieers · ‎09-08-2011

Got it fixed.

Did a write erase, bounced the standby unit and then reconfiged the failover stuff.

One clue was that there were syslog messages about traffic to/from the failover address being blocked. Instead of tracking that mess down, I figured that blowing it clean and resyncing from scratch would be easier...

It was.

Thanks for the help.