cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2357
Views
0
Helpful
4
Replies

No sticky database entries seen with end-to-end SSL and cookie insertion

KURT HILLIG
Level 1
Level 1

We've got ACE30s (active/standby) running A5(1.2), and a context that's front-ending one of our major applications, doing SSL termination on the client side and SSL initiation on the back side:

parameter-map type ssl FrontEndSSL-Param

  rehandshake enabled

parameter-map type ssl BackendSSL-param

  authentication-failure ignore

parameter-map type http UP-ContentParseLength

  persistence-rebalance

  set header-maxparse-length 32767

  set content-maxparse-length 32767

parameter-map type connection WEBAPPS_TCP_P

  set timeout inactivity 180

  set tcp timeout half-closed 180

ssl-proxy service SSL_PSRVICE_CTOOLS

  key ctools.key

  cert ctools.crt

  chaingroup COMODO-INSTANTSSL

  ssl advanced-options FrontEndSSL-Param

ssl-proxy service backend-ssl

  ssl advanced-options BackendSSL-param

class-map match-any ctools-https

  3 match virtual-address 192.168.234.10 tcp eq https

policy-map type loadbalance first-match ctools_l7_policy

  class class-default

     :

    ssl-proxy client backend-ssl

policy-map multi-match ctools_ssl_l4_policy

  class ctools-https

    loadbalance vip inservice

     :

    nat dynamic 72 vlan 72

    appl-parameter http advanced-options UP-ContentParseLength

    ssl-proxy server SSL_PSRVICE_CTOOLS

    connection advanced-options WEBAPPS_TCP_P


(It's running in one-arm mode with source NAT, and using RHI to inject a host route for the VIP so we can do IP anycast across multiple data centers for high availability; but we do this for other services that are working OK so I don't think this is the problem...)

We need to stick clients to the back-end server that they initially connect to, but don't want to do it based on client IP address (we sometimes have many clients sitting behind a single NAT address); and we know that stickiness based on SSL session ID isn't a good idea.  So we've set this up to use cookie insertion:

serverfarm host CTOOLS

  predictor response app-req-to-resp

  probe HTTPS_PROBE

  rserver tahoe 443

    conn-limit max 500 min 490

    inservice

  rserver tavera 443

    conn-limit max 500 min 490

    inservice

     :

     :


sticky http-cookie CTcookie CTOOLS-sticky

  cookie insert browser-expire

  replicate sticky

  serverfarm CTOOLS backup Outage

policy-map type loadbalance first-match ctools_l7_policy

  class class-default

    sticky-serverfarm CTOOLS-sticky

    insert-http DEST_Port header-value "%pd"

    insert-http DEST_IP header-value "%id"

    insert-http SRC_Port header-value "%ps"

    insert-http I_AM header-value "SSL_INIT"

    insert-http X-Client-IP header-value "%is"

    ssl-proxy client backend-ssl

policy-map multi-match ctools_ssl_l4_policy

  class ctools-https

    loadbalance vip inservice

    loadbalance policy ctools_l7_policy

    loadbalance vip icmp-reply active

    loadbalance vip advertise active

    nat dynamic 72 vlan 72

    appl-parameter http advanced-options UP-ContentParseLength

    ssl-proxy server SSL_PSRVICE_CTOOLS

    connection advanced-options WEBAPPS_TCP_P

The ACE shows cookies for this group:

ACE30-ASBDC-1/TL-PROD-ASB# show sticky cookie-insert group CTOOLS-sticky

     Cookie   |        HashKey       |           rserver-instance  

  ------------+----------------------+----------------------------------------+

  R2536245497 | 5183849923197467535  | CTOOLS/tahoe:443

  R2274919243 | 1819252748455616984  | CTOOLS/tavera:443

  R3735812906 | 4189234711381892064  | CTOOLS/tempest:443

  R2985606557 | 13173125335060717162 | CTOOLS/terrain:443

  R3002567455 | 9439662962898967148  | CTOOLS/tigra:443

  R1961897160 | 14710369893090908424 | CTOOLS/titan:443

  R499081919  | 6217826711974382744  | CTOOLS/tornado:443

  R442350102  | 16043823351255228140 | CTOOLS/torrent:443

  R3040870402 | 15912262023226993721 | CTOOLS/tosca:443

  R508933855  | 4513105838401809319  | CTOOLS/townsman:443

  R1259946260 | 3311717765801371676  | CTOOLS/tracker:443

  R3378012421 | 4759463036538773497  | Outage/RedirectOutage:0

And we've got lots of connections:

ACE30-ASBDC-1/TL-PROD-ASB# show serverfarm CTOOLS

serverfarm     : CTOOLS, type: HOST

total rservers : 11

state          : ACTIVE

DWS state      : DISABLED

---------------------------------

                                                ----------connections-----------

       real                  weight state        current    total      failures

   ---+---------------------+------+------------+----------+----------+---------

   rserver: tahoe

       192.168.0.201:443     8   OPERATIONAL     93         1999973    126183

   rserver: tavera

       192.168.0.135:443     8   OPERATIONAL     115        1954862    115132

   rserver: tempest

       192.168.0.207:443     8   OPERATIONAL     94         342550     17205

   rserver: terrain

       192.168.0.209:443     8   OPERATIONAL     91         337468     18796

   rserver: tigra

       192.168.0.202:443     8   OPERATIONAL     91         2074019    125814

   rserver: titan

       192.168.0.155:443     8   OPERATIONAL     97         629418     36974

   rserver: tornado

       192.168.0.203:443     8   OPERATIONAL     111        2084020    121127

   rserver: torrent

       192.168.0.208:443     8   OPERATIONAL     102        357320     19392

   rserver: tosca

       192.168.0.136:443     8   OPERATIONAL     125        2145770    126751

   rserver: townsman

       192.168.0.204:443     8   OPERATIONAL     101        2111449    128722

   rserver: tracker

       192.168.0.205:443     8   OPERATIONAL     95         1980421    120746

But the sticky database for group CTOOLS-sticky is empty:

ACE30-ASBDC-1/TL-PROD-ASB# show sticky database group CTOOLS-sticky

ACE30-ASBDC-1/TL-PROD-ASB#

And the Apache logs on the back-end servers are showing that clients are *not* getting stuck.

What am I missing here?

4 Replies 4

KURT HILLIG
Level 1
Level 1

...I forgot to include this:

ACE30-ASBDC-1/TL-PROD-ASB# show stats sticky

+------------------------------------------+

+----------- Sticky statistics ------------+

+------------------------------------------+

Total sticky entries reused                           : 0

prior to expiry

Total active sticky entries                           : 21

Total active reverse sticky entries                   : 0

Total active sticky conns                             : 0

Total static sticky entries                           : 21

Total sticky entries from Global Pool                 : 0

Total insertion failures due to lack of resources     : 0

Kanwaljeet Singh
Cisco Employee
Cisco Employee

Hi Kurt,

Your configuration looks good. You will not see anything in sticky database since ACE is not learning the cookies dynamically here. It is inserting it so you should see the entries in show sticky cookie-insert group.

We can see total active sticky entries as well in stats. You need to test using one client and take a pcap to see what exactly is going on. May be client is closing the browser which is deleting the cookie and hence client is going to a different server. Not sure just assuming. Pcaps should clear things here.

Regards,

Kanwal

Interesting; sure would be nice if some of this explanation made it into the config guide...  ;-)

I don't have packet captures, but the apache logs (as explained to me - I'm a router monkey, not a webmaster) show traffic that looks to me like it's misdirected.  For example, for one user over a 10-second period we saw this (anonymized - sorry about the lengthy excerpt) on the server "tigra":

141.211.14.186 - USERNAME [09/Sep/2012:21:50:39 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 66466718-3e38-4997-8394-ad9875fa280e.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 6933ba51-14a9-4483-8255-b7687453a1f5.tahoe Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 6933ba51-14a9-4483-8255-b7687453a1f5.tahoe Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:42 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 63c77009-9ced-493c-ac88-02f776cbc503.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:45 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 a76046e0-0796-47b3-94f9-96c79726ecc3.tosca Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - a76046e0-0796-47b3-94f9-96c79726ecc3.tosca Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:46 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 bc76df11-0446-499c-b58a-cf975d8950cb.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 53dd9763-0ef2-47b0-9de2-fb28b26dbba5.townsman Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 53dd9763-0ef2-47b0-9de2-fb28b26dbba5.townsman Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /sakai-login-tool/container HTTP/1.1" 302 - 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 302 - 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

141.211.14.186 - USERNAME [09/Sep/2012:21:50:49 -0400] "GET /access/require?ref=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf&url=/content/group/594942ee-15be-4705-9e90-e5252110fd0e/Lecture%20Slides/F12-Chem-260-L03_Classical_Waves.pdf HTTP/1.1" 416 391 875a9afe-e830-4811-95d9-a074aed71217.tigra Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 71.238.68.166

If you look at the text right before the "Mozilla" on each line, you'll see the names of several other servers - townsman, tahoe, tosca, maybe a few others.  Also, in none of these do I see anything that looks like one of the cookies that the ACE should be inserting ("Rnnnnnnnnn"); but as I said this isn't my area of expertise.

The folks running this application have had a number of complaints about this ever since classes started and load picked way up; it may be that it was just not noticed before, or wasn't reproducible enough to be diagnosed.  But they say that they haven't identified any obvious commonalities among the users having this problem.  Which doesn't mean that it isn't a client/browser problem, just that I don't know either way...

And how do we extract cookies from a pcap when the traffic is encrypted?

Hi Kurt,

To check the pcaps you will have to decrypt them using private keys and that is normally done to troubleshoot issues where SSL is involved.

You need to check if client came with cookie or not since after the cookie has been inserted by ACE, the client should come up with that cookie in next request and ACE will use that cookie value to stick it to the same server. I just verified in lab that ACE will forward the cookie as it is to the server which makes me believe that in above requests client didn't come with cookie.

Regards,

Kanwal

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: