ASA Failover: physical interface failure?

[WStoffel1]

My active/standby failed over during a maintenance window last night and when I failed it back to my primary I noticed my standby unit went to a failed status:

Other host: Secondary - Failed

Namely because

Interface dmz (192.168.254.253): Failed (Waiting)

I'm not quite sure what this means or how to troubleshoot it.

My primary ip for the dmz is 192.168.254.254.

From the primary i can ping 192.168.254.254, but i cannot ping 192.168.253.253.  DMZ is up on the primary.

On the standby I can ping 192.168.254.253, but I cannot ping 192.168.254.254, and that interface is up.  show ip int brief on the standby:

Interface                  IP-Address      OK? Method Status                Protocol

GigabitEthernet0/2         192.168.254.254 YES CONFIG up                    up

I can go on standby and shut that interface and no shut it and it comes back online for a minute then fails again.

I've started to run debug commands but i'm kinda flying blind

debug fover ifc gives me

fover_health_monitoring_thread: ifc_check() - group 0 HW failed 1 (mate 0)

Making me think i have a failed physical interface on the standby, yet the interface is up up so I'm not quite sure what i'm reading.

Any help is appreciated as always.

Thanks

EDIT:  I shut that interface down completely on the standby, the failover status becomes

This host: Secondary - Standby Ready

Also, if I no shut the interface but remove MONITORING, the failover status remains

This host: Secondary - Standby Ready

Since this is my DMZ i prefer to have that interface monitored.  Any thoughts?

[Jouni Forss]

Hi,

I'd assume that since you need to use the "monitor-interface" command that the DMZ is on a Trunk interface?

What happens to the other interfaces on that Trunk when DMZ is showing Failed?

Is it possible for you to give the whole output of the "show failover" when the problem is on on the Secondary/Standby/Failed unit. At that moment the Primary firewall should be operation normally right?

I would still double/triple check the all the configurations between the ASAs (All the switches etc that provide the L2 connectivity between the ASAs for the Failover interface polling to work). Check that each Vlan is configured on the correct Trunks between the 2 ASAs.

- Jouni

[WStoffel1]

Question on your question first,

"I'd assume that since you need to use the "monitor-interface" command that the DMZ is on a Trunk interface?"

I didn't need the monitor-int command, it's not trunked, but the no monitor-int command worked to remove it.  DMZ is my physical gig0/2 interface. 

These two units are at separate physical locations, so the failover link is trunked across a fiber ring.  I'll check that first.  But I would have thought if there was an issue there it would affect all the interfaces.  I was almost thinking the interface itself failed, hardware wise, but it's up up.

Thanks as always.  I'll post back later today when I've verified everything across the failover link.

[Jouni Forss]

To be honest I have very rarely had to troubleshoot Failover and in those cases it has been either totally mystical problem or a really easy one to solve.

Maintanance and Maintanance window (or upgrade, updates etc) often means = expect problems So I just imagine there would always be some possiblity of missconfiguration causing problems with the failover connections.

Then again I find it annoying that there is a good selection of command references for the Cisco ASA but when you consider the debugs you are mostly left to find their meaning online. So you dont always know the full meaning or implication of some messages.

I would consider checking the following things (If possible and if nothing else has been found)

• Use the "show arp" command on both ASAs and see if there is even a marking in the ARP table for the Standby or Active unit depending on which you are logged into. If you dont see anything with the ARP then theres clearly something wrong between the connection of the ASAs
  • Try to "ping" the other ASA just incanse before you issue the "show arp"
• As an addition to the above, try to check the same for interfaces other than the DMZ. Is the same kind of output shows for others too. (= no ARP table markings for the remote ASA IP)
• Consider mirroring traffic from the switch physical ports to some other ports and take capture with wireshark. You should be able to see the Failover Hello (or whatever it would be called) between the interfaces
• Consider doing more debugs and copy/paste outputs here
• Consider changing the switch port used to connect  to the ASA with the interface problem
• Check the interface counters on the site expiriencing the problems

- Jouni

[nefkensp]

If the interface is in waitjng mode, it means that the asa's cant see eachother on that interface.

I would check the physical path, vlan's duplicate ip's to verify that on layer 2 there's no hickup between secondary and primary

Hth

Pieter-Jan

Sent from Cisco Technical Support iPhone App

[WStoffel1]

The quick update is the vlans all exist through the entire path.  The interface in question isn't trunked.  I have another interface that isn't trunked and it's setup the exact same way, and I can enable monitoring on that interface and it works as expected. 

But I haven't had the time to really look at it the past couple of days so it's taken a back seat.  For now I just turned off monitoring which keeps the secondary in a Ready state.  Double edged sword of course since if that interface fails, there's no failing over, sort of defeating the purpose.  Sometimes the political issues win out in IT

[WStoffel1]

We found a loop in our network.  So until we get the core upgrade project completed and swapped I'm just leaving this interface unmonitored.

Ironically enough the loop was introduced when we moved this ASA and another internet router to this dr location.  But STP was doing it's job...which had been manually configured to allow for the original setup.  Long story but undocumented.

Thanks.