07-17-2012 02:41 PM
Hello team.
I am looking for hints to debug cookie-based sessions that are failing to work across my ACE. Basically, the user types http://10.150.3.130/iwsupport, and that shoud be distributed across a farm of servers hidden behind the ACE.The servers set a cookie PHPSESSID=<value> when this URL is requested.
The customer tells me that he thinks that the problem arises when he requests access to the VIP with the POST command (please see the attached wireshark capture, line 52). His browser receives the following message:
Based on the original requirements, I configured the ACE, whose related section of the configuration is the following:
sticky http-cookie PHPSESSID STICKY_SERVERS
timeout 720
serverfarm TEST_SERVERFARM
replicate sticky
class-map type http loadbalance match-all iwsupport
match http url /iwsupport.*
policy-map type loadbalance http first-match TEST_POLICY
class iwsupport
sticky-serverfarm STICKY_SERVERS
class class-default
serverfarm TEST_SERVERFARM
class-map match-all VIP-130
match virtual-address 10.150.3.130 tcp eq www
policy-map multi-match CLIENT_VIPS
class VIP-130
loadbalance vip inservice
loadbalance policy TEST_POLICY
loadbalance vip icmp-reply active
I would appreciate your hints to get session information, debugs, or whatever it could be useful in order to see why this is not working properly.
Thank you very much in advance
Rogelio Alvez
Argentina
Solved! Go to Solution.
07-17-2012 06:31 PM
Hi Rogelio,
Also I would suggest you to apply a http parameter like this:
parameter-map type http PARAMETER-HTTP
case-insensitive
persistence-rebalance
set header-maxparse-length 65535
set content-maxparse-length 65535
length-exceed continue
parsing non-strict
policy-map multi-match CLIENT_VIPS
class VIP-130
loadbalance vip inservice
loadbalance policy TEST_POLICY
loadbalance vip icmp-reply active
appl-parameter http advanced-options PARAMETER-HTTP ---> apply it like this
Cheers,
Jorge
07-17-2012 05:59 PM
Hi Rogelio,
Taking pcap on ACE itself should help in this case. It should clearly show if RST is coming from server or ACE.
Does this work if you bypass ACE?
access-list acl1 line 40 extended permit ip any any (maybe filter just on your client IP)
!
capture mycap interface vlan
!
capture mycap start
run your tests.... (ping, telnet, ssh, etc.)
capture mycap stop
!
copy capture mycap disk0:mycapturefile
!
show capture mycap status
show capture mycap detail
!
then when all done...
capture mycap remove
!
NOTES:
bufsize:1 - 2147483647 kilobytes (default is 64K)
circ: wraps buffer when buffer is full
!
May want to be more specific on ACL, do the following:
access-list acl1 line 10 extended permit ip host 1.2.3.4 any
access-list acl1 line 11 extended permit ip any host 1.2.3.4 !!!! do both directions of traffic, need 2 ACL statements !!!!
Once you have saved the pcap, ftp to ACE and get the pcap file.
Also, please note that this is a cpu intensive process and hence should be stopped immediately after testing.
Regards,
Kanwal
07-17-2012 07:03 PM
Hello Kanwaljeet:
I will ask the customer to connect a PC to the environment in order to get FTP access to the balancer. For the time being, I just have console access via a quite complicated remote connection.
I will let you know as soon as I have an ACE capture with me.
Thanks a lot, rogelio
07-17-2012 07:45 PM
Hi Rogello,
Do you see on server itself if POST request sent by client reached server or not? And if yes what did server reply? If you don't see POST request on the server then most probably it is the ACE which is sending the RST.
the outputs suggested by Jorge should help us and of course the suggested changes.
The changes will ensure that ACE parses upto 65535 bytes which is to ensure that ACE doesn't drop connection because it couldn't read which it was told to because it was way too far in the packet. By default ACE parses up to 4096 bytes.
Regarding persistence rebalance, When the first HTTP request comes in, the ACE will match the request to a layer-7 class-map and load balance it to one of the servers within the serverfarm associated with that class-map. The ACE will then also match all subsequent requests on the same TCP connection to a layer 7 class-map. If the subsequent request matches the same layer 7 class-map as the previous request, then it will be sent to the same server as the previous request. If it matches a different layer 7 class-map, then it will be load balanced to one of the servers within the serverfarm of the newly matched layer-7 class-map according to the serverfarm’s predictor.
I doubt this will make any difference since without rebalance the traffic would be sent to the same server which i guess is not a problem here.
switch/Admin(config-parammap-http)# parsing non-strict--->This is a valid command and should work fine.
For allocating resources you can go to resource class and use limit resource command to allocate resources.
You can send the data at kanwalsi@cisco.com. Also, it would be good to have 2-3 instances of outputs while you do testing so that we can see the difference if any fail counter is increasing.
Regards,
Kanwal
07-17-2012 07:57 PM
Hello Kanwal:
I will ask the end user to log into the real servers and see whether the POST comes the selected server.
As of today, the customer told me that he logged in into all the real servers at the same time (for testing purposes), and verified that an end user connection coming from a single PC was alway redirected to the same real server as expected by the load balancing algorithm.
I replied to Jorge (one of the contributors on this discussion) that the ACE did not let me ingress the "parsing non-strict" command. It does not exists as an option on that section associated to the "parameter-map ..." command. Perhaps this is because i am running SW A4(2.0) release.
regards, rogelio
07-17-2012 08:20 PM
Need to check on that command parsing non-strict. I don't have LB running A4 with me. I have A5 and it works fine.
07-17-2012 07:42 PM
Hello Kanwal.
I forgot to answer to one of your questions. Yes, the session flows without problems if we bypass the ACE.
regards, Rogelio
07-17-2012 06:14 PM
Hi Rogelio,
Some questions, here.
Is this an ACE module or an ACE 4710? and what version?
Can you upload these outputs?
#show service-policy CLIENT_VIPS class-map iwsupport
#show service-policy CLIENT_VIPS class-map iwsupport detail
#show serverfarm TEST_SERVERFARM
#show serverfarm TEST_SERVERFARM detail
#show stats http
#show stats loadbalance
Can you include the configuration of the serverfarm?
I do not see any probe, can you configure one like this?
probe http testing
port 80
expect status 200 200
open 3
serverfarm TEST_SERVERFARM
probe testing --------> apply it like this
Cheers,
Jorge
07-17-2012 06:31 PM
Hi Rogelio,
Also I would suggest you to apply a http parameter like this:
parameter-map type http PARAMETER-HTTP
case-insensitive
persistence-rebalance
set header-maxparse-length 65535
set content-maxparse-length 65535
length-exceed continue
parsing non-strict
policy-map multi-match CLIENT_VIPS
class VIP-130
loadbalance vip inservice
loadbalance policy TEST_POLICY
loadbalance vip icmp-reply active
appl-parameter http advanced-options PARAMETER-HTTP ---> apply it like this
Cheers,
Jorge
07-17-2012 07:29 PM
Hello Jorge:
I have a powerpoint file with the printscreens of the show commands you asked me to get from the ACE, but i do not know how to upload it here in the discussion.
If you could make me know some emails of yours, i could send it offline. Please let me know how to proceed.
In the meantime, I will apply the suggested commands to the equipment. ¿Could you please tell me what is the purpose of them once applied?
Thanks a lot in advance. rogelio
07-17-2012 07:37 PM
Hello Jorge:
The "parsing non-strict" command is not a valid sentence within the parameter-map structure. ¿Would this be a problem?
By other hand, I believe that in the past I had seen a recommendation to make a system reservation for sticky resources. ¿ Is there a command to make that possible?
regards,rogelio
07-17-2012 07:21 PM
Hello Jorge:
This is an appliance. ACE 4710, actually a cluster, running A4(2.0)
There already is a probe inside the serverfarm. The piece of configuration related to servers, probe and serverfarm is the following:
Regarding the "show commands", I can not include them here, but I will try to upload a file with the printscreens of the remote PC from which I am getting access to the ACE.
Thanks a lot, Rogelio
probe tcp TEST-farm
port 80
rserver hots SERVER_17
ip address 10.150.3.17
inservice
rserver host SERVER_18
ip address 10.150.3.18
inservice
rserver host SERVER_28
ip address 10.150.3.28
inservice
rserver host SERVER_29
ip address 10.150.3.29
inservice
serverfarm host TEST_SERVERFARM
probe TEST-farm
rserver SERVER_17
no inservice
rserver SERVER_18
inservice
rserver SERVER_28
inservice
rserver SERVER_29
inservice
07-17-2012 08:38 PM
Hello Rogelio,
I just looked at it and the parsing non-strict is available in the version A4(2.3) or higher.
And here you have my mail: jobejara@cisco.com
Something there are malformed packets or invalids characters included on the headers and/or cookies then that´s why we wanted to check with those outputs.
Also, with the other outputs we should see the client packets and servers packets incrementing, if everything is working properly, otherwise if we do not see them incrementing it might help us to determine if we are seeing an asymmetric flow or something like that.
Can you show us your #show resource usage all?
Can you show us #show probe TEST-farm detail and #show probe TEST-farm ?
Another test which you can do is to check with only one server at the time, meaning turn off all the servers and just leave one active, then if it works we may consider a problem with the sticky configuration.
By the way, can you describe a little bit your topology? Is there any firewall in between the traffic? Any proxy in front or behind the ACE? is the ACE the default gateway of the servers?
Jorge
07-18-2012 04:07 AM
Hi Jorge:
The customer told me that now it is working!!!!
¿ What could have been the effect -the benefit- of the persistance-rebalance command (and perhaps of the other commands suggested by you) on the good behavior that they observed?
I will email you the entire diagram and configuration. It is quite simple:
- No proxies. No firewalls in the middle.
- The ACE is doing bridging instead of routing between the farm and the "client side" interfaces. So the ACE is not a default gateway for servers, because it is just doing bridging. The servers point to a layer3 switch (located on the client side of the ACE) as default gateway.
- clients may be on the "client side" VLAN itself (same subnet of the servers, so the ACE bridges between clients and servers) or beyond the Layer 3 switch (the ACE bridges between the servers and the Layer 3 switch that routes for the clients)
- I had to use bridging on the ACE because unfortunatelly there were clients on the same subnet of the servers. So I split that subnet in two VLANs and made the ACE bridge between them.
Lucky me, there are no clients on the same VLAN of the server farm. Otherwise, I would have had to NAT for these clients in order to hide the clients from the server farm!
I will send a package with the very simple configuration and a VISIO diagram. Also, I will send the printscreen captures of the SHOW commands you asked me to gather.
It is very difficult to get the information because I am connected through the Internet to a laptop that it is in turn connected via its console cable to the ACE´s console. So it is not possible to get in touch to the LAN interfaces of the balancer. Every time someone asks me to test, I have to ask in turn to the end users to try and see what happens...
thank you very much again, rogelio
07-18-2012 08:58 AM
Hello Rogelio,
I just replied back with the details which you asked us for.
It is great it is working now.
It was a pleasure to help you.
Jorge
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide