CSS11503 sends TCP RST to both client and server during active HTTP request/response.

[kevin.smith@reuters.com]

Hello. Any ideas on why our CSS11503 (8.10) would send TCP RST to both client and server involved in an active HTTP connection? CSS was actively managing the client-side and server-side TCP connections involving sequential XML soap request/response pairs prior to this time. There is no idle time in the connection, another client XML request simply comes in after server responds to previous one, and CSS starts forwarding the individual TCP segments of that request to the server. The server ACKs to the CSS on the first two segments (typically this application request involves 5-6 TCP segments) and issues "HTTP/1.1 100 Continue" presumably waiting for the remaining 4 TCP segments the client had sent. However with no delay the CSS then sends its own TCP RST to both client and server. Up to this point communication was fine and similar requests between the same client/server pair handled by CSS without issue. Neither client nor server initiated a TCP RST. All this is verified with sniffer capture of the CSS traffic via its one-arm port. This all takes place over single TCP connection pair (client-CSS and CSS-server).

It is not an idle timeout issue as there is no gap in data transfer. Even if CSS had timed out the flow in the middle of transfer I believe it would only send RST to the sender-side if it sends another next packet for the invalid flow. I don't believe this traffic would invoke any security attack response by the CSS, as its an already-established connection with normal HTTP data transfer. So I am puzzled as to why the CSS suddenly decides this connection is over. CSS shows no logged events during this time and the same backend server is handling multiple connections from other clients at the time. Any ideas?  Thanks in advance.

[chrhiggi]

By default.. the CSS would just pass through a 100 Continue, there must be other factors at work. If the CSS is looking for a L5 match in the response (i.e. you have cookie sticky configured to check the headers) - It is possible the CSS sends a reset because it cannot parse the header.

There are bugs that relate to why a reset would be seen like CSCek45031.

Check "show service" for "Pipelined Data" - if that counter incriments when you see the 100 continue, you are hitting the bug.

[kevin.smith@reuters.com]

TAC identified a likely bug below. The only issue I see is that our content rule is straight L4, not L5, so there should be no L5 inspection here. I'm not sure if the bug still applies but else it does match the symptoms.

CSCsu35876

POST with lots of data quickly fails (spanning packets)
Symptom:

The CSS11500 is configured for a Layer 5 (L5) content rule. A HTTP POST is sent to the CSS quickly followed my many packets of the data (this is the POST data). The CSS11500 can fail to send the last data packet to the backend server which results in the connection failing.

[chrhiggi · Accepted solution]

Kevin

  The CSS should never reset a flow mid-stream on a L4 flow because it is not doing anything at a L5 level that would cause that to occur. In L4 mode, the CSS is simply modifying the IP and ports in the L4 header.  I would be suspicious that maybe the rule you are discussing is L4, but another rule with the same IP as that rule is L5 - which makes all rules with that IP L5.  Can you share entire configuration (minus any secure information like passwords)?

Chris

[kevin.smith@reuters.com]

Chris,

Yes there are 4 rules in total associated with this IP address. Two are L5 rules which point to different services for various functions like redirect if the particular URL is hit. One is an SSL rule which ssl-proxy handles, and the last is the straight L4 port 80 rule which we see being hit in this case.

Now that you mention that concept, it makes sense. Since the the CSS has to first match the destination IP to a content rule, if there are multiple configured including L5, it has to do L5 inspection for any packet targetting that IP, to ensure the right one is used.

Is that about right? So in this case, the bug ID provided does explain it.  Thanks for the replies.

Kevin.

[chrhiggi]

Correct.  When a connection (TCP SYN) arrives, the CSS checks all of the rules it has that match the IP and port on the traffic. If any rule with that ip/port combination requires we need more data (i.e. a url, a cookie, an http header.) - The CSS must terminate the connection before choosing a rule.  The termination looks like this:

Client->CSS *SYN* 

CSS->client *SYN,ACK*

Client->CSS *ACK*

Client->CSS *GET*

CSS->Client  *ACK*

At this point, the CSS checks for whatever the L5 rule required and matches based on the most specific rule that the GET pertains to.

Hope that helps!

[kevin.smith@reuters.com]

Thanks, it does help. But one last question, once the connection is established and the flow is created, does the L5 inspection still occur for HTTP operations, in this case POST, even if the chosen rule is not L5 ?

In our case the connection/flow were established well in advance and multiple similar POST requests handled without error, when suddenly this particular POST resulted in the error and the TCP connection reset. Again, this was on a pre-existing flow.

Thanks,

Kevin.

[chrhiggi]

Kevin,

The CSS does recheck each client requests to make sure it does not need to be rebalanced to a new server.  This document has a bit more on that part of the equation:

http://www-search.cisco.com/en/US/products/hw/contnetw/ps792/products_tech_note09186a0080093e06.shtml

Chris

[kevin.smith@reuters.com]

Thank you Chris.