I have an ASA 5516x w/FirePOWER running ASA version 9.5.(2).10 and with 6.0.1 SFR module. When we begin running backups over a VPN tunnel that is connected to an outside interface on this ASA we begin having high latency from our internal network on the inside interface to both the DMZ (devices behind the DMZ interface on the ASA) and out to the internet. Traceroutes that are performed during these back ups show the latency from our internal subnet to the DMZ subnet jump from a norm of around 1 - 2ms to hundreds of milliseconds. This issue is also seen with traceroutes going from our DMZ to both the internal subnet, and internet. However, the issue is NOT seen with pings and traceroutes going from our ASA to the internal subnet, DMZ subnet, and internet. This latency causes issues with connectivity for users trying to connect to a web server in the DMZ.
High Latency During Backups:
Internal subnet --> DMZ: YES - hundreds of milliseconds
Internal subnet --> Internet (4.2.2.2): Yes - Latency in the hundreds of milliseconds
DMZ subnet --> Internal subnet: Yes - Latency in the hundreds of milliseconds
DMZ subnet --> Internet (4.2.2.2): Yes - Latency in the hundreds of milliseconds
Internal subnet --> Internal subnet: No latency at all
ASA --> Internal subnet: No latency at all
ASA --> DMZ: No latency at all
ASA --> Internet(4.2.2.2): No latency at all
For our internal subnet we have a core switch that sits in front of the ASA and forwards all traffic to it. While performing traceroutes during the backups from devices in the internal subnet to both the DMZ and Internet I can see that there is NO latency to our core switch, but latency in the hundreds of milliseconds to our ASA. That would lead me to believe that the latency is being caused by the ASA. To further bolster that claim, traceroutes done during backups from our DMZ subnet to the internet also show latency in the hundreds of milliseconds. The kicker is, as you can see above, that there is no latency when I ping or run traceroutes from the ASA to our internal subnet, DMZ, or internet. There is also NO speed or duplex mismatches on any of the relevant interfaces on any device, nor are there any dropped packets or errors.There is also no high CPU for any devices or processes on any of the devices in the path when these backups are occurring, and no saturated links.
All this leads me to believe that the ASA is causing the latency during the inspection of the packets. I'm not sure if FirePOWER or some other process on the ASA is the culprit. When these backups occur it would involve many servers beginning to transfer files, and the traffic would all hit the ASA as it attempts to go over the VPN tunnel. I'm thinking that maybe something along the lines of TCP Normalization, or "TCP Stream Coalescing" is causing extra packet processing if, for example, packets were hitting the ASA out of order and it was attempting reorder them...but I honestly don't know very much about that at all. Like I said, it definitely looks like the ASA is causing the latency because I can ping the internet and the DMZ with no latency whatsoever which leads me to believe that has something to do with the processing of packets.
Ultimately, I'm sure that I will have to run captures and debugs the next time we perform the backups in hopes of seeing something that might help determine what is happening, but I really don't know how to troubleshoot this beyond that. Can anyone think of what steps I can take to try and resolve this issue? Any suggestions about commands, captures, or debugs to run would be greatly appreciated.
