Sticky session reset by ¿ACE or real server?

[rogelioalvez]

Hello team.

I am looking for hints to debug cookie-based sessions that are failing to work across my ACE. Basically, the user types http://10.150.3.130/iwsupport, and that shoud be distributed across a farm of servers hidden behind the ACE.The servers set a cookie PHPSESSID=<value> when this URL is requested.

The customer tells me that he thinks that the problem arises when he requests access to the VIP with the POST command (please see the attached wireshark capture, line 52). His browser receives the following message:

Based on the original requirements, I configured the ACE, whose related section of the configuration is the following:

sticky http-cookie PHPSESSID STICKY_SERVERS
  timeout 720
  serverfarm TEST_SERVERFARM
  replicate sticky

class-map type http loadbalance match-all iwsupport
  match http url /iwsupport.*

policy-map type loadbalance http first-match TEST_POLICY
    class iwsupport
    sticky-serverfarm STICKY_SERVERS
  class class-default
    serverfarm TEST_SERVERFARM

class-map match-all VIP-130
  match virtual-address 10.150.3.130 tcp eq www

policy-map multi-match CLIENT_VIPS
  class VIP-130
    loadbalance vip inservice
    loadbalance policy TEST_POLICY
    loadbalance vip icmp-reply active

I would appreciate your hints to get session information, debugs, or whatever it could be useful in order to see why this is not working properly.

Thank you very much in advance

Rogelio Alvez

Argentina

[Kanwaljeet Singh]

Hi Rogelio,

Taking pcap on ACE itself should help in this case. It should clearly show if RST is coming from server or ACE.

Does this work if you bypass ACE?

access-list acl1 line 40 extended permit ip any any (maybe filter just on your client IP)

!

capture mycap interface vlan (that you want to capture - or specify global intead of interface) access-list acl1 bufsize 2048 circ

!

capture mycap start

run your tests.... (ping, telnet, ssh, etc.)

capture mycap stop

!

copy capture mycap disk0:mycapturefile

!

show capture mycap status

show capture mycap detail

!

then when all done...

capture mycap remove

!

NOTES:

bufsize:1 - 2147483647 kilobytes (default is 64K)

circ: wraps buffer when buffer is full

!

May want to be more specific on ACL, do the following:

access-list acl1 line 10 extended permit ip host 1.2.3.4 any

access-list acl1 line 11 extended permit ip any host 1.2.3.4 !!!! do both directions of traffic, need 2 ACL statements !!!!

Once you have saved the pcap, ftp to ACE and get the pcap file.

Also, please note that this is a cpu intensive process and hence should be stopped immediately after testing.

Regards,

Kanwal

[mmusitani]

Hello Kanwaljeet:

I will ask the customer to connect a PC to the environment in order to get FTP access to the balancer. For the time being, I just have console access via a quite complicated remote connection.

I will let you know as soon as I have an ACE capture with me.

Thanks a lot, rogelio

[Kanwaljeet Singh]

Hi Rogello,

Do you see on server itself if POST request sent by client reached server or not? And if yes what did server reply? If you don't see POST request on the server then most probably it is the ACE which is sending the RST.

the outputs suggested by Jorge should help us and of course the suggested changes.

The changes will ensure that ACE parses upto 65535 bytes which is to ensure that ACE doesn't drop connection because it couldn't read which it was told to because it was way too far in the packet. By default ACE parses up to 4096 bytes.

Regarding persistence rebalance, When the first HTTP request comes in, the ACE will match the request to a layer-7 class-map  and load balance it to one of the servers within the serverfarm associated with that class-map.   The ACE will then also match all subsequent requests on the same TCP connection to a layer 7  class-map.  If the subsequent request matches the same layer 7 class-map as the previous  request, then it will be sent to the same server as the previous request.  If it matches a  different layer 7 class-map, then it will be load balanced to one of the servers within the  serverfarm of the newly matched layer-7 class-map according to the serverfarm’s predictor.

I doubt this will make any difference since without rebalance the traffic would be sent to the same server which i guess is not a problem here.

switch/Admin(config-parammap-http)# parsing non-strict--->This is a valid command and should work fine.

For allocating resources you can go to resource class and use limit resource command to allocate resources.

You can send the data at kanwalsi@cisco.com. Also, it would be good to have 2-3 instances of outputs while you do testing so that we can see the difference if any fail counter is increasing.

Regards,

Kanwal

[rogelioalvez]

Hello Kanwal:

I will ask the end user to log into the real servers and see whether the POST comes the selected server.

As of today, the customer told me that he logged in into all the real servers at the same time (for testing purposes), and verified that an end user connection coming from a single PC was alway redirected to the same real server as expected by the load balancing algorithm.

I replied to Jorge (one of the contributors on this discussion) that the ACE did not let me ingress the "parsing non-strict" command. It does not exists as an option on that section associated to the "parameter-map ..." command. Perhaps this is because i am running SW A4(2.0) release.

regards, rogelio

[Kanwaljeet Singh]

Need to check on that command parsing non-strict. I don't have LB running A4 with me. I have A5 and it works fine.

[rogelioalvez]

Hello Kanwal.
I forgot to answer to one of your questions. Yes, the session flows without problems if we bypass the ACE.

regards, Rogelio

[Jorge Bejarano]

Hi Rogelio,

Some questions, here.

Is this an ACE module or an ACE 4710? and what version?

Can you upload these outputs?

#show service-policy CLIENT_VIPS class-map iwsupport

#show service-policy CLIENT_VIPS class-map iwsupport detail

#show serverfarm TEST_SERVERFARM

#show serverfarm TEST_SERVERFARM detail

#show stats http

#show stats loadbalance

Can you include the configuration of the serverfarm?

I do not see any probe, can you configure one like this?

probe http testing

port 80

expect status 200 200

open 3

serverfarm TEST_SERVERFARM

probe testing --------> apply it like this

Cheers,

Jorge

[Jorge Bejarano · Accepted solution]

Hi Rogelio,

Also I would suggest you to apply a http parameter like this:

parameter-map type http PARAMETER-HTTP

  case-insensitive 

  persistence-rebalance

  set header-maxparse-length 65535

  set content-maxparse-length 65535

  length-exceed continue

  parsing non-strict

policy-map multi-match CLIENT_VIPS
  class VIP-130
    loadbalance vip inservice
    loadbalance policy TEST_POLICY
    loadbalance vip icmp-reply active

  appl-parameter http advanced-options PARAMETER-HTTP ---> apply it like this

Cheers,

Jorge

[rogelioalvez]

Hello Jorge:

I have a powerpoint file with the printscreens of the show commands you asked me to get from the ACE, but i do not know how to upload it here in the discussion.

If you could make me know some emails of yours, i could send it offline. Please let me know how to proceed.

In the meantime, I will apply the suggested commands to the equipment. ¿Could you please tell me what is the purpose of them once applied?

Thanks a lot in advance. rogelio

[rogelioalvez]

Hello Jorge:

The "parsing non-strict" command is not a valid sentence within the parameter-map structure. ¿Would this be a problem?

By other hand, I believe that in the past I had seen a recommendation to make a system reservation for sticky resources. ¿ Is there a command to make that possible?

regards,rogelio

[mmusitani]

Hello Jorge:

This is an appliance. ACE 4710, actually a cluster, running A4(2.0)

There already is a probe inside the serverfarm. The piece of configuration related to servers, probe and serverfarm is the following:

Regarding the "show commands", I can not include them here, but I will try to upload a file with the printscreens of the remote PC from which I am getting access to the ACE.

Thanks a lot, Rogelio

probe tcp TEST-farm

port 80

rserver hots SERVER_17

ip address 10.150.3.17

inservice

rserver host SERVER_18

ip address 10.150.3.18

inservice

rserver host SERVER_28

ip address 10.150.3.28

inservice

rserver host SERVER_29

ip address 10.150.3.29

inservice

　

serverfarm host TEST_SERVERFARM

probe TEST-farm

rserver SERVER_17

no inservice

rserver SERVER_18

inservice

rserver SERVER_28

inservice

rserver SERVER_29

inservice

[Jorge Bejarano]

Hello Rogelio,

I just looked at it and the parsing non-strict is available in the version A4(2.3) or higher.

And here you have my mail: jobejara@cisco.com

Something there are malformed packets or invalids characters included on the headers and/or cookies then that´s why we wanted to check with those outputs.

Also, with the other outputs we should see the client packets and servers packets incrementing, if everything is working properly, otherwise if we do not see them incrementing it might help us to determine if we are seeing an asymmetric flow or something like that.

Can you show us your #show resource usage all?

Can you show us #show probe TEST-farm detail and #show probe TEST-farm ?

Another test which you can do is to check with only one server at the time, meaning turn off all the servers and just leave one active, then if it works we may consider a problem with the sticky configuration.

By the way, can you describe a little bit your topology? Is there any firewall in between the traffic? Any proxy in front or behind the ACE? is the ACE the default gateway of the servers?

Jorge

[rogelioalvez]

Hi Jorge:

The customer told me that now it is working!!!!

¿ What could have been the effect -the benefit- of the persistance-rebalance command (and perhaps of the other commands suggested by you) on the good behavior that they observed?

I will email you the entire diagram and configuration. It is quite simple:

     - No proxies. No firewalls in the middle.

     - The ACE is doing bridging instead of routing between the farm and the "client side" interfaces. So the ACE is not  a default gateway for servers, because it is just doing bridging. The servers point to a layer3 switch (located on the client side of the ACE) as default gateway.

  - clients may be on the "client side" VLAN itself (same subnet of the servers, so the ACE bridges between clients and servers) or beyond the Layer 3 switch (the ACE bridges between the servers and the Layer 3 switch that routes for the clients)

- I had to use bridging on the ACE because unfortunatelly there were clients on the same subnet of the servers. So I split that subnet in two VLANs and made the ACE bridge between them.

Lucky me, there are no clients on the same VLAN of the server farm. Otherwise, I would have had to NAT for these clients in order to hide the clients from the server farm!

I will send a package with the very simple configuration and a VISIO diagram. Also, I will send the printscreen captures of the SHOW commands you asked me to gather.

It is very difficult to get the information because I am connected through the Internet to a laptop that it is in turn connected via its console cable to the ACE´s console. So it is not possible to get in touch to the LAN interfaces of the balancer. Every time someone asks me to test, I have to ask in turn to the end users to try and see what happens...

thank you very much again, rogelio

[Jorge Bejarano]

Hello Rogelio,

I just replied back with the details which you asked us for.

It is great it is working now.

It was a pleasure to help you.

Jorge