Greetings,
ASA 5505 Security+
7.2(3)
Situation: server behind an ASA firewall configured with embryonic connection limiting via it's primary static translation. The number for the embryonic connection limiting was set too low, at 50. I say too low because the server, during 'idle' periods of the day, averaged a sustained embryonic connection count of between 40-60 as monitored by the ASA. We ran into problems with this configuration where during these relatively idle server traffic periods, clients would suffer from dropped HTTP connections, resulting in a half-loaded webpage. After further investigation and several tcpdump reads later, I found that while reproducing the issue that the firewall (while performing TCP intercept) would sometimes try and forward an 'ACK' response directly to the server without ever building the proper SYN, as it should do while 'patching' the connections together after a host is validated via TCP intercept/SYN cookies/Emb connection limiting. Here's a real-life example, only the IP's have been changed to protect their true identities :
A rather simple traffic flow:
Client -> ASA(w/ static xlate) -> Server
Client = 1.1.1.1
Server Global (ASA) = 10.10.10.220
Server Real = 192.168.100.220
Example capture from ASA (captured on BOTH inside and outside interfaces) of the server being sent an ACK w/ out initial SYN:
1: 17:31:26.121499 802.1Q vlan#2 P0 1.1.1.1.46711 > 10.10.10.220.80: S 1245822626:1245822626(0) win 65535 <mss 1380,nop,wscale 3,nop,nop,timestamp 648513996 0,sackOK,eol>
2: 17:31:26.121529 802.1Q vlan#2 P0 10.10.10.220.80 > 1.1.1.1.46711: S 706741255:706741255(0) ack 1245822627 win 0 <mss 1380>
3: 17:31:26.140098 802.1Q vlan#2 P0 1.1.1.1.46711 > 10.10.10.220.80: . ack 706741256 win 65535
4: 17:31:26.140221 802.1Q vlan#1 P0 1.1.1.1.46711 > 192.168.100.220.80: . ack 3094998809 win 65535
5: 17:31:26.141777 802.1Q vlan#1 P0 192.168.100.220.80 > 1.1.1.1.46711: R 3094998809:3094998809(0) win 0
6: 17:31:26.141807 802.1Q vlan#2 P0 10.10.10.220.80 > 1.1.1.1.46711: R 706741256:706741256(0) win 0
As you can see, the ASA uses SYN cookies to attempt to validate our host @ 1.1.1.1. Once the client returns with the "Hey, I'm legit" ACK response, the ASA tries sending it directly to our true server @ 192.168.100.220 but without building the connection correctly by first performing a back-end SYN, SYN+ACK with the server. Therefor the server RST's the request as invalid, lacking the first 2 steps of the TCP handshake.
Here's a theory/question that I came up with, and hopefully somebody can help validate, answer, or disprove it.
If SYN cookies are actived on the ASA, it then validates a new connection's initial SYN, but then SYN cookies DE-activates due to falling under threshold BEFORE the validated connection sends back it's legitimate ACK response, is the connection still fully proxied by the ASA? (ie. SYN, SYN+ACK response to server first) Or does only the ACK from the real client get forwarded if the response comes back by the time TCP intecept is disabled?
I ask this because I believe what is happening with the ASA is that TCP intercept feature is 'flapping' on and off because the number of sustained half-open connections can be monitored via show local-host as going anywhere between 40 - 60, with our activation limit set at 50.
ASA: Emb conn limit = 50 and count >50, TCP intecept feature enabled.
CLIENT: 'SYN' for new www connection to SERVER
ASA: 'SYN, ACK' to CLIENT for SERVER.
ASA: Emb conn limit =50 and count <50, TCP intecept feature disabled.
CLIENT: 'ACK' to 3-way handshake to ASA.
ASA: Half-open conn state still exists from TCP intercept challenge but TCP intercept now disabled, forward ACK directly to SERVER.
SERVER: 'ACK' received unexpectedly without initial SYN or SYN+ACK, reseting connection.
Interesting side note: The issue, as experienced by the end clients, was not reported during busy or peak usage periods on the webserver, where the embryonic connection count was held to well over 50 at most times. I believe this left the TCP intercept feature in an always on state, rather than on, then off, then on again as we see during times that we can reproduce the problem.
Anybody want to take a stab? Any insight or discussion would be greatly appreciated.
-Buck
