06-20-2014 06:54 AM
We have host in our network which tests reachability of ACE's VIP address at regular intervals. The test sequence consists of 4 TCP packets (SYN, SYN-ACK, FIN-ACK, RST-ACK; see picture attached) and causes incrementation of "dropped conns" counter in show service-policy output.
ACE30# sh service-policy XYZ detail | inc drop
dropped conns : 266812
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
dropped conns: 238177
dropped conns : 7
ACE30# sh service-policy XYZ detail | inc drop
dropped conns : 266813
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
dropped conns: 238178
dropped conns : 7
Is this normal behavior of ACE? Is there a way how to get rid of the dropped cons counter incrementation.
Petr
06-20-2014 04:39 PM
Hi Petr,
You should see why the connections are getting dropped. Is it due to some class-map condition not met or something else. The best way would be to take pcaps and see why ACE is dropping the connection. There is no other way to get rid of these dropped connections. The dropped connections are those which you didn't want ACE to allow at first place.
Regards,
Kanwal
06-23-2014 12:53 AM
Hi Kanwal,
In L3,L4 policy following class is configured:
class-map match-any VIP-ST
2 match virtual-address 10.105.137.171 tcp eq https
In L7 policy class class-default is configured only.
I have taken pcaps using NAM module. The results are attached. As you can see no data are sent from client to VIP of ACE. The communication consists of control packets (SYN,FIN,RST) only.
It seems to me that ACE cannot manage those kind of test connections and increments dropped connections counter.
Kind regards
Petr
06-23-2014 09:34 AM
Hi Petr,
Something is not right here. If you see TCP 3-way handshake is not completing and client is sending a FIN ACK instead of ACK. There are retransmissions from both sides untill ACE sends the RST. ACE is expecting ACK for 3-way handshake to complete but instead gets FIN ACK.
In L4 load balancing, ACE just forwards the packet from front end to backend. So we shall see this FIN ACK at the backend server and if we will look at the backend packet capture the RST would be from server too. Now, this behavior might change if we have SSL termination since ACE acts as a proxy while doing SSL termination (behavior like L7 LB) and backend connection is not opened unless we get HTTP GET at the front end. While in L7 we need to do LB on the basis of info in http header, in ssl termination we may not so i am not very sure if we need to wait for HTTP GET or just open a backend connection as soon as we get SYN at the front end.
But if in your case it is just L4 LB(classification on TCP PORT 443), to TS further i would request to get a simultaneous pcaps so that we can see both front end as well as back end communication i.e from client to ace and ace to server and see what is going on.
But in any case client has no business to send FIN ACK without completing 3 way handshake.
Regards,
Kanwal
06-24-2014 01:25 AM
Hi Kanwal,
it seems to me that ACE is unable to handle those SYN,SYN-ACK,FIN sequences and expects (as you've mentioned) usual 3-way handshacking
I had thought that the sequence SYN,SYN-ACK,FIN is incorrect but then I found TCP state-machine graph which describes this sequence as valid.
I agree it is strange when client finishes connection before it is completly established. But in this case the connection is created by host which only tests reachability of VIP. In this case such behavior makes sense in my opinion.
You can see graph of ACE to real server communication in vip-droped-con.gif. I also attach pcap screenshot from Wireshark (All packets are duplicated in this graph. This is caused by NAM. In real communication packets are not duplicated).
Regards
Petr
06-24-2014 08:09 AM
Hi Petr,
I would be interested to see the behavior when you have "Normalization" disabled. Can you do "no normalization" and test again?
http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Troubleshooting_Connectivity
Look at this link and look at section:
TCP L4 connection set up with Normalization enabled.
If you see the connection only turns to ESTAB when ACK is seen from the client. Otherwise IN is SYN SEEN and OUT is SYNACK. And in our case ACK is never received. This is like embryonic connection. And ACE will send RST to both front end as well as back end to clear the connection. This is because of security feature and is due to NORMALIZATION.
Configuring the Timeout for an Embryonic Connection
Occasionally, the TCP three-way handshake for a connection may not complete for some reason. This type of connection is called an embryonic connection. To configure a timeout for embryonic connections, use the set tcp timeout embryonic command in parameter map connection configuration mode. The syntax of this command is as follows:
set tcp timeout embryonic seconds
The seconds argument is an integer from 0 to 4294967295 seconds. The default is 5 seconds. A value of 0 specifies that the ACE does not time out an embryonic connection.
Hope this helps.
Regards,
Kanwal
Note: Please mark the answers if they helped.
06-27-2014 02:04 AM
Hi Kanwal,
When I set "no normalization" problem is solved. Disadvantage of this appoach is that by this command all trafic on interface is affected.
I've also tried to tune timeout for embrionic connection.
When I had set it to 0, dropped conns counter stopped to increase. Client which sends those "SYN,FIN" packets ends communication after 30 seconds using RST. This cause that connection ends and dropped conns counter does not increase.
Unfortunately for some reason sometimes happens that client doesn't send this final RST packet. This cause that number of active connection increases ...
ACE30-hto2/TEST-WEBAPP# sh service-policy XYZ | inc conn
curr conns : 9 , hit count : 2279841
dropped conns : 385467
conns per second : 0
conn-rate-limit : - , drop-count : -
ACE30-hto2/TEST-WEBAPP# sh service-policy XYZ | inc conn
curr conns : 22 , hit count : 2283653
dropped conns : 385467
conns per second : 0
conn-rate-limit : - , drop-count : -
When I set timeout to 120, those "non RST" connections are cleared but of course dropped conns counter increases ...
I guess I will try to reconfigure the probe.
Kanwal, thanks for your suggestions!
Kind regards
Petr
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide