ASA outside interface stops responding/passing traffic

[mbroberson1]

Wanted to see if anyone has come across this issue.

Have an ASA 5545-X running 9.12(3)9 used solely to terminate AnyConnect client sessions, there have been several incidents where the ASA outside interface would stop passing traffic and would stop replying to pings and also drop AnyConnect client sessions. To restore connectivity, we reboot the ASA. At first thought it was related to AnyConnect configuration, but after several TAC cases, TAC says the AnyConnect configuration is good and they can find no issues with the show tech-support and outputs (supplied outputs when the issue is occurring) provided to point to an ASA issue. There have been no changes recalled in the past few months when these incidents started occurring to correlate. One output have not yet recorded, but will when the issue occurs again is a show arp on the ASA and the same on our internet routers, that's my next troubleshooting step. Was also wondering if I may need to set static arp entries on both the ASA and internet routers, perhaps this might be a best practice for edge devices such as these? Any suggestions are appreciated!

Regards,

Brandon

[Marius Gunnerud]

Before any more troubleshooting, create a baseline for when everything is working fine.

ping from ASA to internet

ping from a host to internet

show conn address <test PC IP>

show arp | in <IP of default route>

show arp | in <IP of test PC>

If you have access to the ISP routers check the ARP table for the MAC of the ASA as well as ping test to the ASA outside IP.

When the issue is happening go through the troubleshooting steps again and see what is changing.  It is quite possible that the ISP routers have, for whatever reason, wrong MAC address information for the ASA.

[mbroberson1]

Hi Marius,

I will check these, I have recorded a working baseline as your suggestion with the current arp table on both the ASA and my internet (ISP facing) routers and will also collect the other info you suggested. If by access to the ISP routers you are referring to my internet routers that interface with my ISP routers I assume and not the actual ISP routers at my internet service provider.

Regards,

Brandon

[Marius Gunnerud]

To start with it is fine with just the routers you have control over.  If everything looks fine on them, then we will need to get the ISP involved.

[mbroberson1]

Ok, my internet routers are where the ARP entry is held for this ASA (my public IP/Subnet ARP table), and other ASA's that seem to not be experiencing this issue. It appears isolated to just this particular ASA although TAC can find nothing wrong with it based off the outputs supplied.

Thanks

[Marius Gunnerud]

Is it a standalone ASA or an HA pair?

My assumption is that it is not an issue with the ASA but more likely either the Internet facing router on your network or the ISP router.

When the issue is happening try the following:

ping 8.8.8.8 from ASA

ping 8.8.8.8 from your internet facing router

show arp on ASA

show ip arp on internet facing router

Basically do the same troubleshooting steps on the routers as you have done on the ASA.

[mbroberson1]

It is actually an HA pair, when during one of the incidents that occurred, I actually started troubleshooting down that path and removed the fail-over unit from the pairing but the issue continued to occur. Somewhat a process of elimination. In either event, I have some additional good troubleshooting measures to try during the next event.

Regards,

Brandon

[Richard Burts]

Brandon

A couple of things are not clear in your post so have a couple of questions and a couple suggestions:

- I assume that you have checked syslog messages at the time the problem is active and there are not any significant log messages that relate to this?

- have you checked the failover history? Is there any correlation between failover events and this problem?

- I wonder if you did a shut and no shut on the outside interface if it would resolve the issue without need a reboot?

- I wonder if at the time the problem was occurring if you did a manual failover to the standby unit if it would resolve the issue?

[jchromcik]

Has anyone found and resolution to this, I have several ASA5505s with a very similar issue.   Any inbound connections to the firewall doesn't work however outbound from the firewall works just fine.   All L2L tunnels as well as any type of VPN client stops working..until a reboot is done.  Then everything works fine again.   ISP router is never touched.

[Richard Burts]

I am not clear about some parts of your description of the problem. You say that outbound from the ASA works fine. So that means that anything initiated from the ASA does get a response correctly? It is just things initiated from outside that are not accepted?

When the problem occurs L2L vpn that have been working then stop working? If you bring the vpn down and then attempt to initiate the vpn from your network what happens?

In my previous response I posted some questions and suggestions. Can you respond to them in terms of what is happening in your situation?

[jchromcik]

I am not clear about some parts of your description of the problem. You say that outbound from the ASA works fine. So that means that anything initiated from the ASA does get a response correctly? It is just things initiated from outside that are not accepted? 

Is this correct.

When the problem occurs L2L vpn that have been working then stop working? If you bring the vpn down and then attempt to initiate the vpn from your network what happens?

Have several L2L tunnels as well as vpn clients....when they go down they stay down...even if you try to reconnect from the other ends or the vpn users clients.  This include Windows VPN and Anyconnect clients.

I don't just have one ASA is that is having this issue I have several and they are all running asa924-k8.bin

[Marius Gunnerud]

When you say you have several ASAs with the same issue, do they all connect with L2L VPN to the same headend ASA?

Could you post the configuration for a local and remote ASA setup that are experiencing the issue?  It almost sounds like there might be a rekey timer mismatch between the headend and remote ASAs.

You say that you are also experiencing the issue with AnyConnect?  The next time the issue happens with AnyConnect could you create a DART file and post it here?

[sansumclinic]

I have this problem too, our ASA outside interface randomly stops passing traffic and stops replying to pings and drops all AnyConnect client sessions. This happens 2 - 3 times a day. Like you, we see the Arp entries in the ASA. Did you ever find a solution?