Load Balancing Using ArrowPoint or Server Cookies

chris_luton · ‎02-13-2005

We were Load Balancing a 3 webserver application on our WAN using Sticky-Srcip. However, we recently went live with the application on the Internet, so we turned ArrowPoint cookies on to ensure multiple clients from the same site were load balanced. (Many of our large offices come in through the Internet for this app.) In any case, we have had a large number of complaints that users are getting timed out of the application at login. I managed to find a user that could reproduce the issue regularly. When reviewing the HTTP headers sent and received on the client, some initial requests were POSTs. When this is the case, the CSS is injecting the arrow-point cookie in an "HTTP 1.1 100 Continue" message back to the client. The client browser then ignores the cookie, does not send it in the next request, and gets load balanced by the CSS again which is where clients get timed out by the app.

----------------------------------------------

Arrow Point Cookies?

Is there a way to ensure the CSS does not send the arrowpoint cookie in the HTTP 100 continue messages? The RFC that touches on HTTP 1xx messages states client agents do not have to process 100 messages, so this appears to be occurring by design in the browser.

----------------------------------------------

Server Based Cookies?

I've tried using server based cookies. Our servers send JSESSIONID cookies, however I have been unable to determine how to set static data in the JRUN generated cookie for each individual server. Here is an example of what the cookie looks like.

jsessionid=1394801108141313375

This cookie is automatically generated by JRUN and is based on the system date and time among other magical things. The length of the data varies as well. I have tried to use cookie balancing using hash-crc32 and hash-xor which makes matters worse so far.

----------------------------------------------

Any thoughts would be appreciated!

Chris

Gilles Dufour · ‎02-13-2005

Chris,

let me first say that the HTTP 100 message is produced by the server and not the CSS.

The CSS can't generate a response.

All it can do is intereept the response and insert its cookie.

So, if you want to provent the HTTP 100 message, look how to do it on your server.

Then, the http 100 indicates to the client to continue sending the post-data.

This is not a new http request, so this info should be forwarded to the same server, which should eventually generate an HTTP 200 response which should also contain the cookie and the client should store it and use it.

I don't think changing who insert the cookie will fix the problem as the http 100 is generated by the server.

Do you see the post data and the next http response from the server ?

Does it include a cookie ?

Can you post the filtered sniffer trace so we can have a look.

Regards,

Gilles.

chris_luton · ‎02-14-2005

Here are the HTTP Headers as monitored on the client side. Notice the good.txt file includes a GET as it's initial request. All works fine in this case. However, the initial request in the bad.txt is a POST. This is odd since the URL was opened using the same shortcut in both incidents and the browser was closed between each trace that was taken. I've also reviewed the shortcut with notepad to verify it does not include unwanted data such as the JSESSIONID info....etc.

Once you have reviewed the HTTP headers, I have these questions.

1. IIS is sending the 100 Continue messages as you mention, but why is the CSS injecting the cookie in a 100 response that is not typically processed by the client? The bad.txt file shows the client receiving two ARPT cookies because the first cookie in the 100 continue response was ignored.

2. I know Cisco is not really in the business of troubleshooting browser behaviour. But do you know why the browser would behave differently....GET in one request and a POST in the next? We do not wish to get into modifying the browser, so I'm hoping we can provide a solution on the server side that will allow the browser to function this way if it chooses to do so. Do you think it would make sence to push the state management up a level to the cookie handed out by JRUN? This way, the cookie would not be handed back in a 100 response from IIS, and we could tell the CSS to monitor the JRUN cookie. Of course this would require we determine how to manage this cookie either by modifying to cookie to have static data for each server, or by using the right method of hashing...etc.

Chris

chris_luton · ‎02-14-2005

Here are the HTTP Headers as monitored on the client side. Notice the good.txt file includes a GET as it's initial request. All works fine in this case. However, the initial request in the bad.txt is a POST. This is odd since the URL was opened using the same shortcut in both incidents and the browser was closed between each trace that was taken. I've also reviewed the shortcut with notepad to verify it does not include unwanted data such as the JSESSIONID info....etc.

Once you have reviewed the HTTP headers, I have these questions.

1. IIS is sending the 100 Continue messages as you mention, but why is the CSS injecting the cookie in a 100 response that is not typically processed by the client? The bad.txt file shows the client receiving two ARPT cookies because the first cookie in the 100 continue response was ignored.

2. I know Cisco is not really in the business of troubleshooting browser behaviour. But do you know why the browser would behave differently....GET in one request and a POST in the next? We do not wish to get into modifying the browser, so I'm hoping we can provide a solution on the server side that will allow the browser to function this way if it chooses to do so. Do you think it would make sense to push the state management up a level to the cookie handed out by JRUN? This way, the cookie would not be handed back in a 100 response from IIS, and we could tell the CSS to monitor the JRUN cookie. Of course this would require we determine how to manage this cookie either by modifying to cookie to have static data for each server, or by using the right method of hashing...etc.

Chris

Gilles Dufour · ‎02-16-2005

Chris,

the CSS inject the cookie in the first server response.

I said the next response should contain a cookie and I was wrong.

Only the first response will contain a cookie unless the connection is remapped to another server.

I'm going to check with the developpers if we can do something on the CSS to inject the cookie in the next response following an HTTP 100.

In the meantime, you will have to find a way to inject a static cookie on the server.

Regarding the GET and POST, apparently the working solution sends a GET to another webpage and from there it sends a POST to login.

The not-working browser is sending the POST directly.

Looks like they cached the GET or bookmarked the post.

Gilles.

chris_luton · ‎02-16-2005

Thanks Gilles! Please let me know what you find from the development team. It would be nice if we could do something on the CSS so that cookies were not injected in a 100 response.

Regarding static cookies, I am looking into this option as well. Our JSESSIONID session cookie as generated by JRUN has not static content. So I have post on Macromedia's forums and opened a call to see if we can prefix static data for each of our servers. Still waiting on this.

Another note regarding the browsers behaviour...for whatever it's worth. I think this may be a bug with either IE6, or a bug with what our JRUN server is telling the browser to cache. This problem only occurs in random situations and is not a global problem for us. Also note that I did use the same shortcut for both HTTP header traces and the browser was completely closed between traces. When viewing the shortcut with notepad, it had the correct link with no POST or JSESSIONID info.

Thanks Gilles for the info!!

chris_luton · ‎02-15-2005

Did my double post scare you off? Any other thoughts or ideas?

Thanks!

Chris

Gilles Dufour · ‎02-16-2005

Chris,

I found a way to suppress the 100 continue on the server.

http://www.kbalertz.com/kb_832985.aspx

Gilles.

chris_luton · ‎02-16-2005

Thanks Gilles. I did see this KB article on Microsoft's website, but we would prefer to avoid customizing an ISAPI filter in house as we do not have a C programmer that is familiar with ISAPI filters and IIS specifically. We could probably compile the code and get it installed, but if it caused a problem, we'd probably be back to where we started. :) Thanks for the info thought.

Gilles, are you still going to see if development can come up with something that will ensure the CSS does not insert cookies in a 100 response?

Thanks!

Chris

Gilles Dufour · ‎02-16-2005

Chris,

I did check with development and they will not change the way it works.

Too much effort for what is considered a server issue.

The recommendation is to modify the server.

Gilles.

chris_luton · ‎02-16-2005

Ok, thanks for trying.

Chris

Gilles Dufour · ‎02-19-2005

I went into the code and found an easy way to implement a solution to integrated the cookie in both the 100 resonse and in the next one.

I provided the code to the developpers and they will implement it.

I think that deserves a little high 5 :-)

Gilles.

chris_luton · ‎02-21-2005

Sweet Gilles! High 5 for sure! Ok, so give me the skinny bro. What is the ETA on the fixed code and how do I go about obtaining it? :)

Chris

chris_luton · ‎09-25-2005

Gilles. We're looking at using ArrowPoint cookies again. Do you know if this 100 Continue issue was ever resolved?

Thanks!

Chris