Hi all,
We have been using anyconnect for years mostly without any problems. However, last week, sometime between Thursday and Saturday, nearly every VPN client running Windows started experiencing a severe throughput issue on TCP sockets that start by uploading multiple megabytes of data into the VPN. My IT department has been trying to schedule a support call for weeks, originally for a different matter, and somehow still has not been able to get one scheduled, so I thought I would check here out of desperation.
If the software on the VPN client (a windows laptop) opens a socket to a machine in our data center and attempts to upload for example 16mb of data, the throughput will start out high (maybe 5MiB/s) and around 1MB into the transfer it will suddenly drop to 21 bytes per second... for the rest of the lifetime of that socket. That's comparable to the speed of the very first modem created in 1958.
If I look carefully at what the receiver is receiving, one packet of size 1310 bytes shows up exactly every 60 seconds. 60 seconds seems very conspicuous--like a timeout in some configuration setting. This is far too slow to be usable. The sender thinks it has sent hundreds more kilobytes than the receiver has received (at least from userland's perspective) -- so that data is somewhere in the middle, being released to the receiver very slowly.
Observations:
- If the socket downloads even a single byte before starting to upload large quantities of data, the issue never occurs. It only occurs if the traffic on the socket is upload-only until it hits the throughput problem.
- If we artificially limit the throughput of the upload to 640KiB/s (e.g., by sleeping for 100ms after sending each 64kb chunk), the problem never occurs.
- If someone with a truly slow internet connection that can only pull off 270KiB/s does the upload, the problem never occurs for them.
- If the VPN client is running Linux or MacOS, then the upload goes fine.
- The issue can be reproduced regardless of what OS the receiving end of the socket is running.
- The payload being uploaded does not seem to matter. E.g., just zeros will do it.
- The issue does not occur if performing the same upload without going through the VPN, by e.g., uploading to an AWS server.
- Lowering the MTU on the VPN interface or the physical interface to 576 on the windows client has no effect.
The easiest tool to use to repro the problem is netcat, but any socket that starts out with ~1MB of traffic going into the VPN encounters the issue. e.g.:
On client:
nc somewhere 1234 < a_big_file
On server:
nc -l 1234 | dd of=/dev/null status=progress
I have been using a python script with vanilla windows python to run diagnostics where I do things like limit the upload rate and print out detailed progress messages. You can also use procmon on windows and strace on linux to monitor the syscalls.
I believe the hardware is a pair of cisco firepower ftd 2110s. I do not have access to this hardware, but my IT department does. However, they seem to be at a loss for how to proceed, and getting them to work on this is seemingly an uphill battle, so anything I can do without access to the firepowers would be good.
The Windows clients are running AnyConnect 4.10.08029.
I am trying to get IT to open a port on the firewall so that we might try the same upload to the same server going through the same network hardware without using the VPN. If it encounters the same problem, then we could at least rule out AnyConnect.
I would also like to start doing packet captures at the two ends where I do have access to see if anything odd is observable.
How would you proceed?
