CSM with SSL Mods - Issue With Clients not Sticking

[carlsond]

I have a CSM with two SSL modules. My clients come in through a proxy, the traffic is sent from the CSM and balanced to the SSL modules for decryption, back to the CSM for cookie insertion for sticky to the servers. The traffic is then sent back to the SSL modules for re-encrytion.

First I could not get any balancing across the three servers. All traffic was stuck to one server.

After the first TAC case changes, traffic, while still almost always connecting to one server was was not sticking and the sessions were being disconnected, requiring a re-login. Once the clients logged in again they were back on the original server. I have called in two TAC cases with no resolution. This is the configuration I am down to with TAC assistance.

serverfarm P-SVC-CF8-443

  nat server

  nat client NAT-VLAN163

  real name SERVER1 443

   inservice

  real name SERVER2 443

   no inservice

  real name SERVER3 443

   no inservice

  probe KA-CF-CFM

serverfarm P-SVC-CF8-BE

  nat server

  no nat client

  real 10.6.1.33 80

   inservice

  real 10.6.1.34 80

   no inservice

  probe PING

serverfarm P-SVC-CF8-E

  nat server

  no nat client

  real 10.6.1.30 443

   inservice

  real 10.6.1.31 443

   no inservice

  probe PING

sticky 52 cookie P-SVC-COOKIE1 insert timeout 240

sticky 55 netmask 255.255.255.255 timeout 240

sticky 56 netmask 255.255.255.255 timeout 240

vserver P-SVC-CF8-443

  virtual 10.62.21.58 tcp https

  serverfarm P-SVC-CF8-E

  sticky 240 group 55

no persistent rebalance

  inservice

vserver P-SVC-CF8-BE

  virtual 10.6.1.35 tcp https

  serverfarm P-SVC-CF8-443

  sticky 240 group 56

  no persistent rebalance

  inservice

vserver P-SVC-CF8-D

  virtual 10.6.1.32 tcp www

  serverfarm P-SVC-CF8-BE

  sticky 240 group 52

  persistent rebalance

  inservice

On vserver P-SVC-CF8-443, I was originally using SSL-Sticky with the offset 20 length 6 to stick to the SSL mod but was told to remove that because the SSL session ID was probably being renegotiated.

Any ideas would be greatly appreciated. Thank you,

Dave

[Daniel Arrondo Ostiz]

Hi Dave,

Without seeing also the configuration of your SSL modules, it's hard to confirm what I'm going to say, so feel free to make any corrections if something doesn't make sense.

From what I understand, you are doing several layers of load-balancing, first of all you load-balance the SSL traffic between several SSL modules doing source-ip stickiness. Once traffic is decrypted, it arrives back on the CSM and it gets load-balanced again, this time using cookie stickiness. From your configuration, this second load-balancing is done with vserver "P-SVC-CF8-D", which is then load-balancing between the servers in serverfarm "P-SVC-CF8-BE". If I'm not mistaken, this serverfarm is just pointing to the two SSL modules, which, after encrypting will send it back to the CSM on the remaining vserver, which will do load-balancing with source ip stickiness.

On this setup, cookie-stickiness is basically going to be ignored, because, the vserver that is actually balancing between the reals is using source-ip stickiness.

The proper way to configure this would be creating a different ssl-proxy for each of the reals in each of the SSLM and then, configure the vserver for the decrypted traffic to load-balance between the 6 ssl-proxies (3 per module x 2 modules) using cookie stickiness. The third load-balancing will no longer be needed because each ssl-proxy will already be associated with a single real.

I don't have a sample configuration right now, but, if there is something you don't understand about what I just said, let me know and I will try to prepare something quickly.

Regards

Daniel

[carlsond]

The current SSL configs are as follows:

SSL-1:

service prod-service-cf8
  virtual ipaddr 10.6.1.30 protocol tcp port 443
  server ipaddr 10.6.1.32 protocol tcp port 80
  certificate rsa general-purpose trustpoint prod-service-cf8-tp-11
  inservice
!
service prod-service-cf8-be client
  virtual ipaddr 10.6.1.33 protocol tcp port 80
  server ipaddr 10.6.1.35 protocol tcp port 443
  log-auth-failures
  trusted-ca root-ca
  authenticate verify signature-only
  inservice

SSL-2:

service prod-service-cf8
  virtual ipaddr 10.6.1.31 protocol tcp port 443
  server ipaddr 10.6.1.32 protocol tcp port 80
  certificate rsa general-purpose trustpoint prod-service-cf8-tp-11
  inservice
!
service prod-service-cf8-be client
  virtual ipaddr 10.6.1.34 protocol tcp port 80
  server ipaddr 10.6.1.35 protocol tcp port 443
  trusted-ca root-ca
  authenticate verify all
  inservice

I'm not sure I understand. Some background: The point of the this configuration is to decrypt the traffic, insert a cookie for load balancing, re-encrypt and forward to the servers. The traffic is sent to the SSL modules for decryption, comes back to the CSM for a cookie insert, back to the SSL-modules for re-encyption and then to the CSM to be forwarded to the servers. At least that's the goal, the reason we are trying to accomplish this is that the bulk of the traffic comes in through a proxy and has a single source IP address.

Are you saying that I should configure a SSL-Client for each server and forward the re-encrypted traffic directly from the SSL module to the servers? Currently we don't have the server VLAN's enabled on the SSL modules. It was a security feature that Cisco suggested when we first set these up. Currently traffic can only get to the server subnets by going back through the CSM. Thanks..

[Daniel Arrondo Ostiz · Accepted solution]

Are you saying that I should configure a SSL-Client for each server and 
forward the re-encrypted traffic directly from the SSL module to the 
servers? Currently we don't have the server VLAN's enabled on the SSL 
modules. It was a security feature that Cisco suggested when we first 
set these up. Currently traffic can only get to the server subnets by 
going back through the CSM. Thanks..

That's precisely what I was suggesting. If you don't as I explained, when the traffic comes back to the CSM for the final load-balancing, it will arrive encrypted, so, the CSM cannot check the cookie. This cookie can only be checked when the traffic arrives in plain text, which only happens in the second vserver.

I don't see the added security value of not having the server vlan in the SSLM, but this should not be a problem, as long as the SSLM is sending the traffic to the IP address of the server, the CSM will route it to the servers (I assume the CSM is acting as default gateway for the SSLM and that the vlan connecting both devices is defined as a server vlan on the CSM)

Daniel

[carlsond]

Ok thanks. I will give this a try in our test environment and see how it goes. I agree with your opinion on the security aspect of this configuration but to be honest with you I have not gotten the best advice on the CSM's from Cisco over the years and support has gotten worse rather than better. That's why I came here, I think the best Cisco CSM support is located on the support community forum. Thank you and I'll let you know how the testing goes.

Dave