09-11-2012 02:49 PM
We've got ACE30s (active/standby) running A5(1.2), and a context that's front-ending one of our major applications, doing SSL termination on the client side and SSL initiation on the back side:
parameter-map type ssl FrontEndSSL-Param
rehandshake enabled
parameter-map type ssl BackendSSL-param
authentication-failure ignore
parameter-map type http UP-ContentParseLength
persistence-rebalance
set header-maxparse-length 32767
set content-maxparse-length 32767
parameter-map type connection WEBAPPS_TCP_P
set timeout inactivity 180
set tcp timeout half-closed 180
ssl-proxy service SSL_PSRVICE_CTOOLS
key ctools.key
cert ctools.crt
chaingroup COMODO-INSTANTSSL
ssl advanced-options FrontEndSSL-Param
ssl-proxy service backend-ssl
ssl advanced-options BackendSSL-param
class-map match-any ctools-https
3 match virtual-address 192.168.234.10 tcp eq https
policy-map type loadbalance first-match ctools_l7_policy
class class-default
:
ssl-proxy client backend-ssl
policy-map multi-match ctools_ssl_l4_policy
class ctools-https
loadbalance vip inservice
:
nat dynamic 72 vlan 72
appl-parameter http advanced-options UP-ContentParseLength
ssl-proxy server SSL_PSRVICE_CTOOLS
connection advanced-options WEBAPPS_TCP_P
(It's running in one-arm mode with source NAT, and using RHI to inject a host route for the VIP so we can do IP anycast across multiple data centers for high availability; but we do this for other services that are working OK so I don't think this is the problem...)
We need to stick clients to the back-end server that they initially connect to, but don't want to do it based on client IP address (we sometimes have many clients sitting behind a single NAT address); and we know that stickiness based on SSL session ID isn't a good idea. So we've set this up to use cookie insertion:
serverfarm host CTOOLS
predictor response app-req-to-resp
probe HTTPS_PROBE
rserver tahoe 443
conn-limit max 500 min 490
inservice
rserver tavera 443
conn-limit max 500 min 490
inservice
:
:
sticky http-cookie CTcookie CTOOLS-sticky
cookie insert browser-expire
replicate sticky
serverfarm CTOOLS backup Outage
policy-map type loadbalance first-match ctools_l7_policy
class class-default
sticky-serverfarm CTOOLS-sticky
insert-http DEST_Port header-value "%pd"
insert-http DEST_IP header-value "%id"
insert-http SRC_Port header-value "%ps"
insert-http I_AM header-value "SSL_INIT"
insert-http X-Client-IP header-value "%is"
ssl-proxy client backend-ssl
policy-map multi-match ctools_ssl_l4_policy
class ctools-https
loadbalance vip inservice
loadbalance policy ctools_l7_policy
loadbalance vip icmp-reply active
loadbalance vip advertise active
nat dynamic 72 vlan 72
appl-parameter http advanced-options UP-ContentParseLength
ssl-proxy server SSL_PSRVICE_CTOOLS
connection advanced-options WEBAPPS_TCP_P
The ACE shows cookies for this group:
ACE30-ASBDC-1/TL-PROD-ASB# show sticky cookie-insert group CTOOLS-sticky
Cookie | HashKey | rserver-instance
------------+----------------------+----------------------------------------+
R2536245497 | 5183849923197467535 | CTOOLS/tahoe:443
R2274919243 | 1819252748455616984 | CTOOLS/tavera:443
R3735812906 | 4189234711381892064 | CTOOLS/tempest:443
R2985606557 | 13173125335060717162 | CTOOLS/terrain:443
R3002567455 | 9439662962898967148 | CTOOLS/tigra:443
R1961897160 | 14710369893090908424 | CTOOLS/titan:443
R499081919 | 6217826711974382744 | CTOOLS/tornado:443
R442350102 | 16043823351255228140 | CTOOLS/torrent:443
R3040870402 | 15912262023226993721 | CTOOLS/tosca:443
R508933855 | 4513105838401809319 | CTOOLS/townsman:443
R1259946260 | 3311717765801371676 | CTOOLS/tracker:443
R3378012421 | 4759463036538773497 | Outage/RedirectOutage:0
And we've got lots of connections:
ACE30-ASBDC-1/TL-PROD-ASB# show serverfarm CTOOLS
serverfarm : CTOOLS, type: HOST
total rservers : 11
state : ACTIVE
DWS state : DISABLED
---------------------------------
----------connections-----------
real weight state current total failures
---+---------------------+------+------------+----------+----------+---------
rserver: tahoe
192.168.0.201:443 8 OPERATIONAL 93 1999973 126183
rserver: tavera
192.168.0.135:443 8 OPERATIONAL 115 1954862 115132
rserver: tempest
192.168.0.207:443 8 OPERATIONAL 94 342550 17205
rserver: terrain
192.168.0.209:443 8 OPERATIONAL 91 337468 18796
rserver: tigra
192.168.0.202:443 8 OPERATIONAL 91 2074019 125814
rserver: titan
192.168.0.155:443 8 OPERATIONAL 97 629418 36974
rserver: tornado
192.168.0.203:443 8 OPERATIONAL 111 2084020 121127
rserver: torrent
192.168.0.208:443 8 OPERATIONAL 102 357320 19392
rserver: tosca
192.168.0.136:443 8 OPERATIONAL 125 2145770 126751
rserver: townsman
192.168.0.204:443 8 OPERATIONAL 101 2111449 128722
rserver: tracker
192.168.0.205:443 8 OPERATIONAL 95 1980421 120746
But the sticky database for group CTOOLS-sticky is empty:
ACE30-ASBDC-1/TL-PROD-ASB# show sticky database group CTOOLS-sticky
ACE30-ASBDC-1/TL-PROD-ASB#
And the Apache logs on the back-end servers are showing that clients are *not* getting stuck.
What am I missing here?
09-11-2012 03:05 PM
...I forgot to include this:
ACE30-ASBDC-1/TL-PROD-ASB# show stats sticky
+------------------------------------------+
+----------- Sticky statistics ------------+
+------------------------------------------+
Total sticky entries reused : 0
prior to expiry
Total active sticky entries : 21
Total active reverse sticky entries : 0
Total active sticky conns : 0
Total static sticky entries : 21
Total sticky entries from Global Pool : 0
Total insertion failures due to lack of resources : 0
09-11-2012 05:41 PM
Hi Kurt,
Your configuration looks good. You will not see anything in sticky database since ACE is not learning the cookies dynamically here. It is inserting it so you should see the entries in show sticky cookie-insert group.
We can see total active sticky entries as well in stats. You need to test using one client and take a pcap to see what exactly is going on. May be client is closing the browser which is deleting the cookie and hence client is going to a different server. Not sure just assuming. Pcaps should clear things here.
Regards,
Kanwal
09-11-2012 07:12 PM
Interesting; sure would be nice if some of this explanation made it into the config guide... ;-)
I don't have packet captures, but the apache logs (as explained to me - I'm a router monkey, not a webmaster) show traffic that looks to me like it's misdirected. For example, for one user over a 10-second period we saw this (anonymized - sorry about the lengthy excerpt) on the server "tigra":
141.211.14.186 - USERNAME [09/Sep/2012:21:50:39 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 66466718-3e38-4997-8394-ad9875fa280e.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 6933ba51-14a9-4483-8255-b7687453a1f5.tahoe Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 6933ba51-14a9-4483-8255-b7687453a1f5.tahoe Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:45 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 a76046e0-0796-47b3-94f9-96c79726ecc3.tosca Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - a76046e0-0796-47b3-94f9-96c79726ecc3.tosca Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 53dd9763-0ef2-47b0-9de2-fb28b26dbba5.townsman Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 53dd9763-0ef2-47b0-9de2-fb28b26dbba5.townsman Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166
If you look at the text right before the "Mozilla" on each line, you'll see the names of several other servers - townsman, tahoe, tosca, maybe a few others. Also, in none of these do I see anything that looks like one of the cookies that the ACE should be inserting ("Rnnnnnnnnn"); but as I said this isn't my area of expertise.
The folks running this application have had a number of complaints about this ever since classes started and load picked way up; it may be that it was just not noticed before, or wasn't reproducible enough to be diagnosed. But they say that they haven't identified any obvious commonalities among the users having this problem. Which doesn't mean that it isn't a client/browser problem, just that I don't know either way...
And how do we extract cookies from a pcap when the traffic is encrypted?
09-11-2012 08:22 PM
Hi Kurt,
To check the pcaps you will have to decrypt them using private keys and that is normally done to troubleshoot issues where SSL is involved.
You need to check if client came with cookie or not since after the cookie has been inserted by ACE, the client should come up with that cookie in next request and ACE will use that cookie value to stick it to the same server. I just verified in lab that ACE will forward the cookie as it is to the server which makes me believe that in above requests client didn't come with cookie.
Regards,
Kanwal
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide