ACE module (VIP) sends RST,ACK for SYN from client

[Suresh Nadansundaram]

I am facing this behavior where the ACE module sends [RST, ACK] as a response to a SYN from the client. The simultaneous packet captures at the ACE and the backend server does not show that the SYN from the client reaches the real server. The backend server appears to be up and runnign all the time. The time difference between teh SYN and the [RST, ACK] is negligible.

SYN - Arrival Time: Apr  1, 2013 14:22:05.350478000 Central Daylight Time

[RST, ACK] - Arrival Time: Apr  1, 2013 14:22:05.352178000 Central Daylight Time

So it doesn't look like the response is coming from the real server.

The application works fine for 80% of the time.

What could be the reason. The real servers are webservices proxies. These conenctions crosses firewall as the ACE and the realserevrs are in the DMZ and the client in the core network. The client is a Mainframe system.

The dropped connection at the VIP level and the conenction failures at the server farm and the real serevrs for this server farm keep on increasing.

Any help is appreciated.

I am atatching the simultaneous captures infront of the ACE and at the REAL servers. To start with look in to the frames 97,98,99 and 100 in the ACE_capture file.         

[Cesar Roque]

Hi Suresh,

Is this happening in a specific time frame? Is the same behavior for all the clients?

---------------------
Cesar R
ANS Team

[chrhiggi]

Suresh-

  Do you also have a showtech you can attach to this thread (as well as the VIP IP you are hitting) ?

Regards,

Chris Higgins

[csco10387876]

I don't know if this isue was resolved but I have a similar problem with an ACE sending reset directly after receiving a syn from clients.

[Suresh Nadansundaram]

Yes, this issue was resolved. I have opened a Cisco TAC case and they guided us to the root cause. I would suggest to open a TAC case.

Here is what we found in our environment - This is a combination of issues at multiple levels. The client that makes the call to the VIP ends the SSL conversation with a RST. Looks like the SSL layer on the client is sending a "close notify" event prematuarely. We are still tracing it down. Same time the real server, for some reason, responding to the RST with an ACK. Since this ACK comes to ACE after the communication between the client and the ACE effectively completed, it keeps the connection open in the connection table for that client. While this connection stays open in the connection table, if a SYN comes from the client that matches the same IP port combination then ACE closes that conenction with a RST as a connection for that IP and port is already open in the conenction table. The timeout value in our environment was setup to very high (8 hours), whih made the issue worse as more and more orphaned connections accumilated in the conenction table.

The issue at ACE level was resolved by reducing the timeout to 20 minutes. Though this move doesn't fix the issue completely it removes the conenctions from the table much quicker. Hence chance for the conflict is very less. We are still troubleshooting the SSL layer at the client side and the ACK after the RST issue at the real server side.