Arrow Point Cookies Not Injecting Consistently

[chris_luton]

Cisco,

Please review the attached files. You will see the arrowpoint cookie is injected into the 200 response in the good.txt but not in the bad.txt. Our users continue to be timed out due to issues with arrow point cookeis. Please help. This is similar to the example I posted in "Load Balancing Using ArrowPoint or Server Cookies".

Gilles?

Chris

[seilsz]

Chris,

Can you post the full captures, including L2 and L3 headers?

Thanks,

Zach

[chris_luton]

Zach,

Sorry but I don't and I'm not sure that I could if I tried. The problem occurs on random systems throughout the world and is nearly impossible to catch a workstation while it experiences this issue. Someone will call about the problem, then I'll go to their desk and I cannot duplicate it on their system. However, I can tell you there are no proxy servers or content filters in the mix, and that we use Pix firewalls. The two systems I have managed to catch, one in London and one in Belgium, have pointed back to ArrowPoint Cookies. In one case, it was due to the CSS inserting cookies in a 100 response from IIS as discussed in my previous post from last week. And now the second is this 200 response in which the CSS is not inserting an ARPT cookie .9 percent of the time. In any case, this has been a very difficult issue for us to troubleshoot and we are fairly convinced that ARPT cookies just DO NOT WORK for us although this is what Cisco recommended. Perhaps it is an issue with the specific OS we are running. In any case, have there been no other customers that have complained about this same problem? If so, what was done to resolve the issue? Newer version of the OS? Alternative method of load balancing...which is what were looking at today.

Thanks for the reply Zach.

[Gilles Dufour]

Chris,

Zach asked for the complete trace.

This would help us see if the requests are part of the same connection or not.

Do you have 'no persistent' configured ?

Could you post your config as well.

Thanks,

Gilles.

[chris_luton]

Hey Gilles! I understand that Zach was asking for a trace. I'm assuming you or Zach want to see a sniffer trace of:

1. The application working correctly

2. The application not working correctly in the case of the missing cookie in the 200 response.

I could probably get you 1, but 2 would be very difficult as I explained in my last post. A very small percentage of 16000 named users experience the issue in many different countries. I have control of the network where the CSS resides, but the data that comes in from remote sites is PATed by the remote site's Pix. Even if it wasn't PATed, this issue occurs so randomly and I absolutely cannot duplicate it. I've been lucky in obtaining HTTP header info on two remote systems.

I really do wish I could get a trace. But hopefully the configs will shed some light.

I apologize for the stressed tone guys. Thanks for your help!

"content L3_Rule1b" is the ruleset specific to this issue.

Thanks!

Chris

[Gilles Dufour]

Chris,

the CSS only insert the cookie in the first response.

If you have persistent connection, the first response will get the cookie but not the others.

If you go through a proxy, the first client will get the cookie but not the others.

This is why you need the command 'no persistent' on all your content rule.

Your current issue is probably only seen by users going through a proxy.

This 'no persistent' should definitely solve ALL your problems including the one we discussed before as there will be no POST without an initial GET.

Regards,

Gilles.

PS: if you read "how netpro ratings work" you will see it is important to rate the answers that you are given - this gives visibility to users of the forum and help us get recognition

[chris_luton]

Thanks Gilles. I wish I were as confident as you. :) I will review the documentation regarding the 'no persistent' and give it a test. If it works for us, I will come back and give you the highest rating known to mankind.

[chris_luton]

Hey Gilles. I promise, I want to give you a good rating. :) Before I do though, I want to better understand how "no persistent" will resolve our issue.

I've read through the following document (as well as others) and it states that you should use "no persistent" in your content rule for a number of things. However, I don't see how it pertains to us.

http://www.cisco.com/en/US/customer/products/hw/contnetw/ps789/products_tech_note09186a0080093e06.shtml

Can you elaborate as to how this setting will resolve these issues? We are not using a proxy server at any of the sites that are experiencing the issue. All users get a "Set-Cookie". It's just that 1% of them get it too late due to a 100 response or an intial 200 response the CSS chose not to append the ARPT to. Keep in mind though, we have 16,000 named users so 1% is actually a good chunk of users.

Please just give me a brain dump of how you see this resolving the issue. Then maybe I will see the light.

Thanks Gilles!

Chris

[Gilles Dufour]

Without a complete trace showing the problem, it is only possible for us to guess what could cause this.

There is no rule in the CSS that says apply this 99%-of -the-time-only.

If you have the problem with 1% of the users, it's because they have something in common. - like a proxy server.

Don't you have one user that can recreate this all the time ?

I still believe that the *trace* you provided is the result of a persistent connection.

The GET was probably part of another session for which a cookie has been returned already [maybe in an HTTP100 message].

I checked the list of existing bugs for your version and there is only one that could lead to a cookie not being inserted but it also refers to persistent connection and 'no persistent'.

BTW, you should upgrade due to the following 2 bugs

CSCeb66864

CSCec26257

Finally, since you can't provide us the sniffer trace of a failure, I would still suggest to go for 'no persistent' and 'persistence reset remap' [forgot to mention it in my previous post].

These commands will allow the CSS to parse every request separately and introduce the cookie each time.

Regards,

Gilles.

[chris_luton]

I understand a trace would greatly improve the ability to resolve the issue. And I don't expect you to pull the exact resolution out of the air without this data. I'm simply requesting assistance for the limited information that I am able to obtain.

My point in stating it is only 1% of our users across all sites was to show you that it is probably NOT a site specific issue. It is also NOT the same 1% of users each day which would suggest it is NOT a client specific issue. The users that are affected by the problem change on a daily basis. I apologize if I wasn't clear on this point.

Regarding your belief on the persistent connection, I have not stated that I disagree with you. I have simply asked you to elaborate as to why you believe this to be the case. How would making the change alter the flow of traffic or decisions on the CSS specifically so that this issue "may" no longer occur.

The reason I'm challenging your suggestion, is due to the fact that we do NOT see this problem from a specific site, nor from specific users. It is RANDOM. Which would to me "suggest" it to be a bug...such as in the case of the 100 response...or some other abnormality.

Again, I'm asking for the best feedback I can get based on the "not so best" data I'm providing you. And so far, I think you have done an EXCELLENT job Gilles.

I will continue to read documentation about the settings you suggest here to try to understand your logic as it applies to our config. Please let me know if you have any other comments.

Thanks Gilles!

Chris

[seilsz]

Chris,

I was able to reproduce this problem fairly easily - I just reloaded http://tc.urscorp.com/tc4/dhtml// ~20 times, clearing the cookie each time.

Full sniffer traces are attached. Use filter:

ip.addr == 65.171.24.118

to just see what we're interested in.

Looking at the traces, it does appear that you could be running into CSCeb66864 (as Gilles mentioned). You can verify this by using 'no persistent' on the content rule (as Gilles also mentioned). Based on this, looks like you may need to upgrade code.

~Zach

[seilsz]

I wasn't able to attach the traces, so I emailed them to you both.

~Zach

[chris_luton]

Thanks guys!! You've both been very helpful. I'll try your suggestions when I return to the states next week. Once I have, I'll return and update you.

Cheers!

Chris

[chris_luton]

Ok, one more question. I see that CSCec26257 and CSCeb66864 are both resolved in 7.2. However, which OS do you two recommend....off the record. I want to ensure I don't cause alternative problems by upgrading our OS.

Chris

[Gilles Dufour]

both 7.10 and 7.20 are end of life.

Which means no more bug fixing.

I would recommend the latest 7.30 or 7.40(1.07)s

Regards,

Gilles.