We recently deployed a server farm where we noticed an increase of 200ms with processing versus when we had a single server in place. Can someone advise what the expected increase latency should be when moving a group of servers behind the ACE Load Balancers? Is the 200ms normal?
No, it's not normal that the ACE introduces a 200ms delay, even though it can happen in some situations. Let me explain.
The ACE has two different ways of treating the L7 connections internally, that we call "proxied" and "unproxied". In essence, the proxied mode means that the traffic will be processed by one of the CPU (normally to inspect/modify the L7 data), while, on the unproxied mode, the ACE sets up a hardware shortcut that allows forwarding traffic without the need to do any processing on it.
For a L7 connection, the ACE will proxy it at the beginning, and, once all the L7 processing has been done it will unproxy the connection to save resources. Before it goes ahead with the unproxying, it needs to see the ACK for the last L7 data sent, but some clients don't send it and instead expect the ACE to continue sending data. The default timeout for this wait is precisely 200 ms, so it's likely to be the cause for your issue.
This wait, on a Internet environment can introduce around 100-200ms of delay for each HTTP request, which can end up adding into a very big delay, so it's possible to configure a RTT threshold, above which, the ACE will no longer try to unproxy connections. With that in mind, I would suggest setting the threshold to 0 to ensure to keep connections always proxied (which should avoid the 200ms delay you are seeing). To do this, you would nee to configure a parameter map like the one below and add it to your VIP
parameter-map type connection
set tcp wan-optimization rtt 0
Even though this setting may avoid your issue, it also has some drawbacks. The main one is that the ACE appliance only supports up to 128K simultaneous L7 connections in proxied state (which includes also the connections towards the servers, so, it would be 64K for client connections), so, if the amount of simultaneous connections reaches that limit, new connections would be dropped. The second issue, although not so impacting, would be that the maximum number of connections per second supported would also go down slightly due to the increased processing needed.