Azure Traffic Manager Probes being dropped by ASA after upgrade

[is.infrastructure1]

After upgrading ASA to version 9.18.4 we are seeing Azure Traffic Manager probes being dropped (or discarded). 

In the logs we see 'TCP access denied by ACL from <traffic manager IP> to Outside Interface /443' 

It works fine when reverting back to 9.16.4 

Not configuration changed were made

Has anyone seen this happen? What is it in the config that allows the probes? 

[tvotna]

It seems there are two discussions for the same issue:

https://community.cisco.com/t5/network-security/asa-dropping-azure-traffic-manager-http-health-probes/td-p/5079384

ASA only needs "webvpn" / "enable Outside" (or "http server enable" / "http <subnet> <mask> Outside") in order to respond to SYN packets over TCP/443. Do you have it enabled? Do you have any control-plane ACLs assigned (check with "show run access-group")?

What happens after SYN, when GET request arrives, is a big question. I'm not aware of any filtering by request URL, although who knows...

You can collect "capture cap-asp type asp-drop match ..." for TCP/443 to the firewall IP address. Then display it with "show capture cap-asp [packet-number <N>]". It will show memory address where drop happened, such as "Drop-location: frame 0x0000564694f7624e". TAC will be able to decode it and tell you exactly where drop happened.

[is.infrastructure1]

We have webvpn enabled and http server enabled (I assume this is for ASDM). We have ensure that the control plane ACL is replicated from the working ASA. 

In the logs we are seeing TCP Reset-O from the Azure Traffic Manager IP  addresses. 

I've attached a picture of the logs. The ASA seems to allow the probes for a minute or so, then it times out

Can you help with what is happening here please

[tvotna]

Most of the connections timed out after 30 seconds of inactivity, which is a typical TCP SYN timeout, so it looks like TCP 3WHS didn't complete.

Do you reload the ASA and after that probes start working for a minute or so? What do you mean by "control plane ACL is replicated"? Is it a failover pair and you send probes to a standby IP address, or you just mean that control plane ACL is identical between working and non-working ASAs running different versions of code?

Please provide

show run webvpn
show run http

Then start captures:

capture cap-out int OUTSIDE_INET match tcp any host <ASA-IP>
capture cap-asp type asp-drop match any host <ASA-IP>

and periodically collect:

show resource usage all
show asdm sessions
show quota management-session
show asp table socket
show asp table socket stats detail
show asp drop
show processes | grep Unicorn

Then copy captures off the device:

copy /pcap capture:cap-out ftp://.......
copy /pcap capture:cap-asp ftp://.......
no capture cap-out
no capture cap-asp

Then provide collected information.

[is.infrastructure1]

Hi tvotna,

Yes that is correct, the issue is happening when I upgrade the Firepower ASA - which has ASA Software, and reload it. It shows as 'Active' on Azure Traffic Manager for 90 seconds or so, and then goes to 'Downgraded'

I just meant we have the same control plane ACLs on the non working ASA as we do the working ASA

The most recent version that seems to work (doesn't have this issue) is cisco-asa-fp2k.9.19.1.SPA

This is the capture we are seeing - I have replaced our external IP address with xxx.xxx.xxx.xxx

We have tried allowing http all, and can connect reach the ASA from the outside network. 

This is cap-asp

Many thanks 

[is.infrastructure1]

As you can see from the capture, the ASA is only responding to a small number of packets, not all. I believe this is why it is showing as 'degraded' 

Is there any reason why upgrading the version of code would stop the ASA from responding to all 

This is a capture from the working ASA - as you can see here the ASA is responding to the probes from Azure Traffic Manager

Many thanks 

[tvotna]

We have very little information to judge and what we have is confusing. From syslog shared earlier it appears that inbound connections to TCP/443 are created. From the non-working capture we see that the ASA doesn't respond to TCP/443 SYNs at all. SYN/ACK is never sent. It sends only TCP RST in response to TCP RST coming from the client. Finally you mentioned "We have tried allowing http all, and can connect reach the ASA from the outside network". This is confusing. I recommend you opening a TAC case and let TAC engineer troubleshoot the issue live on Webex. You'll probably need to schedule maintenance window, reload the ASA with the non-working version so that the engineer can start capture right after device comes online and follow the exact troubleshooting procedure I shared earlier.

The traffic manager may report that device is reachable for 90 seconds or so, but this might not mean the device has ever responded to TCP/443 SYN packets. This possibility needs to be ruled out or confirmed. Then configuration of control plane ACL and http x.x.x.x y.y.y.y needs to be verified too. It is possible to verify this in the low-level firewall ASP tables as well. The commands I shared earlier can help understand what happens with TCP/443 socket and various limiters. Etc.

[MHM Cisco World]

the HTTP to ASA is pass through rule before Allow or deny 
I draw the rule it  must pass 
the capture you share meaning that the Azure is send "S" i.e. it try SYNC the TCP connection and ASA not reply 
check this points and I waiting your reply 

MHM

[MHM Cisco World]

So you have control acl' 

And you config http 0.0.0.0 0.0.0.0 also ?

If yes the subnet config with http command override the control acl' 

I.e. if you permit traffic in control acl and dont specify subnet in http command that not work.

MHM

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113685-asa-threat-detection.html

it can Thread detection drop the packet try excluded the IP 

MHM