topic No sticky database entries seen with end-to-end SSL and cookie i in Application Networking

No sticky database entries seen with end-to-end SSL and cookie insertion

KURT HILLIG — Tue, 11 Sep 2012 21:49:05 GMT

We've got ACE30s (active/standby) running A5(1.2), and a context that's front-ending one of our major applications, doing SSL termination on the client side and SSL initiation on the back side:

parameter-map type ssl FrontEndSSL-Param

rehandshake enabled

parameter-map type ssl BackendSSL-param

authentication-failure ignore

parameter-map type http UP-ContentParseLength

persistence-rebalance

set header-maxparse-length 32767

set content-maxparse-length 32767

parameter-map type connection WEBAPPS_TCP_P

set timeout inactivity 180

set tcp timeout half-closed 180

ssl-proxy service SSL_PSRVICE_CTOOLS

key ctools.key

cert ctools.crt

chaingroup COMODO-INSTANTSSL

ssl advanced-options FrontEndSSL-Param

ssl-proxy service backend-ssl

ssl advanced-options BackendSSL-param

class-map match-any ctools-https

3 match virtual-address 192.168.234.10 tcp eq https

policy-map type loadbalance first-match ctools_l7_policy

class class-default

ssl-proxy client backend-ssl

policy-map multi-match ctools_ssl_l4_policy

class ctools-https

loadbalance vip inservice

nat dynamic 72 vlan 72

appl-parameter http advanced-options UP-ContentParseLength

ssl-proxy server SSL_PSRVICE_CTOOLS

connection advanced-options WEBAPPS_TCP_P

(It's running in one-arm mode with source NAT, and using RHI to inject a host route for the VIP so we can do IP anycast across multiple data centers for high availability; but we do this for other services that are working OK so I don't think this is the problem...)

We need to stick clients to the back-end server that they initially connect to, but don't want to do it based on client IP address (we sometimes have many clients sitting behind a single NAT address); and we know that stickiness based on SSL session ID isn't a good idea. So we've set this up to use cookie insertion:

serverfarm host CTOOLS

predictor response app-req-to-resp

probe HTTPS_PROBE

rserver tahoe 443

conn-limit max 500 min 490

inservice

rserver tavera 443

conn-limit max 500 min 490

inservice

sticky http-cookie CTcookie CTOOLS-sticky

cookie insert browser-expire

replicate sticky

serverfarm CTOOLS backup Outage

policy-map type loadbalance first-match ctools_l7_policy

class class-default

sticky-serverfarm CTOOLS-sticky

insert-http DEST_Port header-value "%pd"

insert-http DEST_IP header-value "%id"

insert-http SRC_Port header-value "%ps"

insert-http I_AM header-value "SSL_INIT"

insert-http X-Client-IP header-value "%is"

ssl-proxy client backend-ssl

policy-map multi-match ctools_ssl_l4_policy

class ctools-https

loadbalance vip inservice

loadbalance policy ctools_l7_policy

loadbalance vip icmp-reply active

loadbalance vip advertise active

nat dynamic 72 vlan 72

appl-parameter http advanced-options UP-ContentParseLength

ssl-proxy server SSL_PSRVICE_CTOOLS

connection advanced-options WEBAPPS_TCP_P

The ACE shows cookies for this group:

ACE30-ASBDC-1/TL-PROD-ASB# show sticky cookie-insert group CTOOLS-sticky

Cookie | HashKey | rserver-instance

------------+----------------------+----------------------------------------+

R2536245497 | 5183849923197467535 | CTOOLS/tahoe:443

R2274919243 | 1819252748455616984 | CTOOLS/tavera:443

R3735812906 | 4189234711381892064 | CTOOLS/tempest:443

R2985606557 | 13173125335060717162 | CTOOLS/terrain:443

R3002567455 | 9439662962898967148 | CTOOLS/tigra:443

R1961897160 | 14710369893090908424 | CTOOLS/titan:443

R499081919 | 6217826711974382744 | CTOOLS/tornado:443

R442350102 | 16043823351255228140 | CTOOLS/torrent:443

R3040870402 | 15912262023226993721 | CTOOLS/tosca:443

R508933855 | 4513105838401809319 | CTOOLS/townsman:443

R1259946260 | 3311717765801371676 | CTOOLS/tracker:443

R3378012421 | 4759463036538773497 | Outage/RedirectOutage:0

And we've got lots of connections:

ACE30-ASBDC-1/TL-PROD-ASB# show serverfarm CTOOLS

serverfarm : CTOOLS, type: HOST

total rservers : 11

state : ACTIVE

DWS state : DISABLED

---------------------------------

----------connections-----------

real weight state current total failures

---+---------------------+------+------------+----------+----------+---------

rserver: tahoe

192.168.0.201:443 8 OPERATIONAL 93 1999973 126183

rserver: tavera

192.168.0.135:443 8 OPERATIONAL 115 1954862 115132

rserver: tempest

192.168.0.207:443 8 OPERATIONAL 94 342550 17205

rserver: terrain

192.168.0.209:443 8 OPERATIONAL 91 337468 18796

rserver: tigra

192.168.0.202:443 8 OPERATIONAL 91 2074019 125814

rserver: titan

192.168.0.155:443 8 OPERATIONAL 97 629418 36974

rserver: tornado

192.168.0.203:443 8 OPERATIONAL 111 2084020 121127

rserver: torrent

192.168.0.208:443 8 OPERATIONAL 102 357320 19392

rserver: tosca

192.168.0.136:443 8 OPERATIONAL 125 2145770 126751

rserver: townsman

192.168.0.204:443 8 OPERATIONAL 101 2111449 128722

rserver: tracker

192.168.0.205:443 8 OPERATIONAL 95 1980421 120746

But the sticky database for group CTOOLS-sticky is empty:

ACE30-ASBDC-1/TL-PROD-ASB# show sticky database group CTOOLS-sticky

ACE30-ASBDC-1/TL-PROD-ASB#

And the Apache logs on the back-end servers are showing that clients are *not* getting stuck.

What am I missing here?

No sticky database entries seen with end-to-end SSL and cookie i

KURT HILLIG — Tue, 11 Sep 2012 22:05:09 GMT

...I forgot to include this:

ACE30-ASBDC-1/TL-PROD-ASB# show stats sticky

+------------------------------------------+

+----------- Sticky statistics ------------+

+------------------------------------------+

Total sticky entries reused : 0

prior to expiry

Total active sticky entries : 21

Total active reverse sticky entries : 0

Total active sticky conns : 0

Total static sticky entries : 21

Total sticky entries from Global Pool : 0

Total insertion failures due to lack of resources : 0

No sticky database entries seen with end-to-end SSL and cookie i

Kanwaljeet Singh — Wed, 12 Sep 2012 00:41:52 GMT

Hi Kurt,

Your configuration looks good. You will not see anything in sticky database since ACE is not learning the cookies dynamically here. It is inserting it so you should see the entries in show sticky cookie-insert group.

We can see total active sticky entries as well in stats. You need to test using one client and take a pcap to see what exactly is going on. May be client is closing the browser which is deleting the cookie and hence client is going to a different server. Not sure just assuming. Pcaps should clear things here.

Regards,

Kanwal

No sticky database entries seen with end-to-end SSL and cookie i

KURT HILLIG — Wed, 12 Sep 2012 02:12:04 GMT

Interesting; sure would be nice if some of this explanation made it into the config guide... 😉

I don't have packet captures, but the apache logs (as explained to me - I'm a router monkey, not a webmaster) show traffic that looks to me like it's misdirected. For example, for one user over a 10-second period we saw this (anonymized - sorry about the lengthy excerpt) on the server "tigra":

141.211.14.186 - USERNAME [09/Sep/2012:21:50:39 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 66466718-3e38-4997-8394-ad9875fa280e.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 6933ba51-14a9-4483-8255-b7687453a1f5.tahoe Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 6933ba51-14a9-4483-8255-b7687453a1f5.tahoe Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:45 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 a76046e0-0796-47b3-94f9-96c79726ecc3.tosca Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - a76046e0-0796-47b3-94f9-96c79726ecc3.tosca Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 53dd9763-0ef2-47b0-9de2-fb28b26dbba5.townsman Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 53dd9763-0ef2-47b0-9de2-fb28b26dbba5.townsman Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

If you look at the text right before the "Mozilla" on each line, you'll see the names of several other servers - townsman, tahoe, tosca, maybe a few others. Also, in none of these do I see anything that looks like one of the cookies that the ACE should be inserting ("Rnnnnnnnnn"); but as I said this isn't my area of expertise.

The folks running this application have had a number of complaints about this ever since classes started and load picked way up; it may be that it was just not noticed before, or wasn't reproducible enough to be diagnosed. But they say that they haven't identified any obvious commonalities among the users having this problem. Which doesn't mean that it isn't a client/browser problem, just that I don't know either way...

And how do we extract cookies from a pcap when the traffic is encrypted?

No sticky database entries seen with end-to-end SSL and cookie i

Kanwaljeet Singh — Wed, 12 Sep 2012 03:22:48 GMT

Hi Kurt,

To check the pcaps you will have to decrypt them using private keys and that is normally done to troubleshoot issues where SSL is involved.

You need to check if client came with cookie or not since after the cookie has been inserted by ACE, the client should come up with that cookie in next request and ACE will use that cookie value to stick it to the same server. I just verified in lab that ACE will forward the cookie as it is to the server which makes me believe that in above requests client didn't come with cookie.

Regards,

Kanwal