topic ACE Module max-parse-length buffer behavior in Application Networking

ACE Module max-parse-length buffer behavior

A.Swinnen — Mon, 01 Mar 2010 09:32:58 GMT

Hi All,

I configured X-Forwarded-For insertion via an Action-List. I also configured persistence rebalance to be sure that the header is inserted in every HTTP 1.1 request (and not only in the first request of the TCP connection).

However, in some cases, the ACE doesn't insert the X-Forwarded-For header into the request:

- A TCP connection is setup, an HTTP1.1 request is done. The X-Forwarded-For header is correctly inserted

- A few HTTP1.1 requests are done over the existing TCP connection, the X-Forwarded-For header is inserted as expected

- Some more HTTP1.1 requests are done over the existing TCP connection, the X-Forwarded-For header isn't inserted any more.

I configured the maximal header length to 2048bytes. According to my network analyzer (traces token after the ACE), the HTTP header never exceeded 2048 bytes, the maximum I saw was about 1700bytes.

I also want to mention that this configuration is used in combination with SSL termination.

I was thinking about the way the max-parse-length buffer is used. Is this a circular buffer, or are all headers within a TCP connection added up and put into the buffer?

I am using an ACE10 blade with release A2 (1.5a)

parameter-map type http PERSISTENCE.REBALANCE-PRM
persistence-rebalance
header modify per-request
set header-maxparse-length 2048
length-exceed continue

Re: ACE Module max-parse-length buffer behavior

Gilles Dufour — Mon, 01 Mar 2010 10:11:20 GMT

You can check if you had parse-length error with

show np [1|2] me-stats "-shttp"

AceC6k2/MinK# show np 1 me-stats "-shttp -v" | i arse
Parse result LB msgs sent:                        0             0
Parse result Inspect msgs sent:                   0             0
Static parse errors:                              0             0
Max parselen errors:                              0             0
AceC6k2/MinK#

Also, be aware that a static parse errors will stop the "persistent rebalance" function.

The max-parselen will be used to set the size of the buffer that will be used to store the HTTP headers.

We can store only up to the max-parse length. If we do not find what we are looking for and have not enough space to store the complete header, we report a max parselen error.

Gilles.

Re: ACE Module max-parse-length buffer behavior

A.Swinnen — Mon, 01 Mar 2010 10:21:23 GMT

Hi Gilles,

I do see parse-len errors:

LBWIN2SF1-ACE1/NC# sh np 1 me-stats "-shttp -v" | inc arse
Parse result LB msgs sent:                  5449539             5
Parse result Inspect msgs sent:                   0             0
Static parse errors:                              0             0
Max parselen errors:                           1280             0
LBWIN2SF1-ACE1/NC# sh stats http

+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 1089534    , TCP data msgs sent       : 1646146
Inspect parse result msgs : 0          , SSL data msgs sent       : 4819007
                      sent
TCP fin msgs sent         : 316053     , TCP rst msgs sent:       : 3796
Bounced fin msgs sent     : 27         , Bounced rst msgs sent:   : 0
SSL fin msgs sent         : 319854     , SSL rst msgs sent:       : 13
Drain msgs sent           : 823880     , Particles read           : 13375300
Reuse msgs sent           : 0          , HTTP requests            : 1078758
Reproxied requests        : 0          , Headers removed          : 0
Headers inserted          : 818857     , HTTP redirects           : 0
HTTP chunks               : 17334      , Pipelined requests       : 0
HTTP unproxy conns        : 168299     , Pipeline flushes         : 0
Whitespace appends        : 0          , Second pass parsing      : 0
Response entries recycled : 512397     , Analysis errors          : 0
Header insert errors      : 0          , Max parselen errors      : 545
Static parse errors       : 0          , Resource errors          : 0
Invalid path errors       : 0          , Bad HTTP version errors : 0
Headers rewritten         : 5074       , Header rewrite errors    : 0
Unproxy msgs sent         : 168299
LBWIN2SF1-ACE1/NC#

However, I am pretty sure that the HTTP headers do not exceed 2048 bytes (when leaving the ACE). So what are possible explanations for those parselen errors:

- The ACE counts the HTTPS headers, which are larger in encrypted form

- The headers are not limited to HTTP, but also the IP and TCP headers are counted (and maybe even the Ethernet Header)? If that's the case, I should re-check my sniffing trace.

- The parselen errors come from another Loadbalancing configuration in the same context. Is is possible to see the parse-len errors per service-policy?

Thanks for your help!

Re: ACE Module max-parse-length buffer behavior

Gilles Dufour — Mon, 01 Mar 2010 10:35:21 GMT

Parse length is only for the HTTP header.

If you are absolutely sure the header is less than 2000 bytes, than it comes from another application.

Send me the trace where you have the problem. I can replay it in my lab and see what we get.

Gilles.

Re: ACE Module max-parse-length buffer behavior

A.Swinnen — Mon, 01 Mar 2010 11:35:14 GMT

Hi Gilles,

I uploaded the capture (at the output) and I also uploaded my configuration. Just to warn you: might get a headache from reading this config

In thetrace you will see double X-Forwarded-For headers, that's because I tried to configure the header-insertion directly in the policy-map and with an action-list.

For exemple the request at 13h53 18.104 749 contains the X-Forwarded-For correctly, while the request at 14h40 16.075 698 doesn't...

Kind Regards,

Arnout

Re: ACE Module max-parse-length buffer behavior

Gilles Dufour — Mon, 01 Mar 2010 14:26:00 GMT

Frame 4907 + 4908 is a GET with a size of 1460 + 719 = 2159 bytes.

This is higher than your max parselen.

All requests after this one in the same connection do not have the x-forwarded-for header.

I would have to verify the code, but it seems like once we hit the max parselen error for a connection, we stop inspecting that connection. Even for new requests.

I would suggest to increase your max parselen to 3000 bytes.

Regards,

Gilles.

Re: ACE Module max-parse-length buffer behavior

A.Swinnen — Mon, 01 Mar 2010 14:45:04 GMT

Not parsing any longer once the max-parse-length has been hit seems me to be reasonable decision, as the alternative is to drop the connection...

Thanks for the great help!