CSM-S - HTTP fine, HTTPS termination gets "lost" ?

[afsheen.bigdeli]

I'm having issues setting up HTTP load balancing and HTTPS load balancing / termination on a 6509-E with CSM-S module installed. While HTTP works fine, HTTPS termination only partially works - I can get the SSL daughter card to deliver the certificate, but I can't move on to establish an HTTP session with the real server.

Symptoms:

Two real servers, .50 and .51; one vip, .54; CSM gateway, .249, and SSL vlan IP is 10.11.12.2.

All servers, reals, etc. are on vlan 8.

The HTTP vserver .54 sends all HTTP requests directly to the HTTP serverfarm containing .50 and .51; the HTTPS vserver sends all HTTPS requests to the SSL daughter card, decrypts them and then forwards them to the gateway address on the CSM-S, which should direct the requests to the .54 vserver and then the HTTP serverfarm.

HTTP load balancing works without issue - requests directed to the .54 VIP are passed along properly to the real servers. Each real server has a .54 loopback interface for the VIP. (NAT isn't used at any point.)

HTTPS requests are supposed to be terminated on the SSL daughter card and then passed in to the clear to the .54 vserver, which will send the (now HTTP) request to the serverfarm IMG. but this isn't working...

In testing, "openssl s_client -connect 10.11.12.54:443" gets the cert served from the SSL card, and "show ssl-proxy conns" on the SSL card shows that the connection is established and then forwarded off the the CSM's gateway (which is specified in the server line, as per the config examples in the online documents.). After the client receives the cert from the SSL daughter card, the client connection hangs.

Configuration:

ssl-proxy:

ssl-proxy service pixel_ssl 
virtual ipaddr 10.11.12.54 protocol tcp port 443 secondary
server ipaddr 10.11.12.249 protocol tcp port 80
certificate rsa general-purpose trustpoint thawte
policy http-header session-info
inservice

CSM:

module ContentSwitchingModule 7
vlan 8 client
  ip address 10.11.12.249 255.255.255.0
  route 0.0.0.0 0.0.0.0 gateway 10.11.12.1
!        
serverfarm IMG
  no nat server
  no nat client
  real 10.11.12.50
   inservice
  real 10.11.12.51
   no inservice
!        
serverfarm IMG_SSL
  no nat server
  no nat client
  real 10.11.12.2 local
   inservice
!        
vserver IMG
  virtual 10.11.12.54 tcp www
  serverfarm IMG
  persistent rebalance
  inservice
!        
vserver IMG_SSL
  virtual 10.11.12.54 tcp https
  serverfarm IMG_SSL
  persistent rebalance
  inservice
!       

Any pointers on this would be greatly appreciated.

Cheers!

[afsheen.bigdeli]

As an update (edit: updated again):

tcpdumps on client workstation between client and VIP show that the SSL handshake succeeds, and then eventually times out with a RST from the VIP.

Looking at tcpdump on the real server with VIP loopback, and looking at the connections on both the CSM-S and the ssl-proxy card, I see that the connection comes in to CSM-S on 443, is passed to SSL card, makes it back out as port 80 decrypted, is passed to a virtual IP in the vserver farm and lands on the real server, and then real server sends ack packet to client IP that goes nowhere.

I think this ACK needs to go back to SSL card (where client connection is stuck in SYNSENT mode), but I'm not certain how to accomplish this.

I've also found this in Content Networking Fundamentals:

"Because the SSL devices preserve the source IP address of the packet as the client's IP address, the CSM-S must use special routing for real-server return traffic. If the CSM-S uses normal routing, the cleartext real server return packets are routed to the client directly, bypassing the SSL daughter card. Therefore, the CSM-S routes the back-end real server return packets to the SSL daughter card by tracking the Layer 2 MAC address of the SSL daughter card during the connection flow."

....except this is apparently not working.

[cumar.chan]

will be interesting to see the config on the SSL daughter card.

the "local" keyword on the CSM serverfarm will direct the traffic towards the SSL daughtercard where there will be config for 10.11.12.2 on the SSL daughter card with associated SSL key!

Once the SSL is terminated on the SSL D card, the traffic will need to be pointed back to the CSM towards a VIP. Does this exist on the SSL D config?

[martin.foote]

Hello

Firstly I've got to admit we have always ran our CSM and CSM-S in bridged mode rather than routed, but from your configuration there seems to be some missing vlans. I don't know if that's down to the way you have extracted the configuration or if they are really missing.

Here is an example of configuration in routed mode

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/csms/2.1.1/configuration/guide/SSLxple.html#wp1212539

Notice the requirement for three vlans (this applies to both routed and bridged mode) - one client vlan and two server vlans (one for SSL). If you are running in bridged mode you need just two IP subnets - one for SSL and the other two (so 8 and y below) can be the same. If you running routed you need three IP subnets one for each.

CSM (amendments I think are required in bold):

module ContentSwitchingModule 7

vlan 8 client

  ip address 10.11.12.249 255.255.255.0

  route 0.0.0.0 0.0.0.0 gateway 10.11.12.1

!

! Define SSL offload VLAN

vlan x server

description SSL offload VLAN

ip address 10.1x.12.249 255.255.255.0

!

! Define Server VLAN

vlan y server

description Server VLAN

ip address 10.1y.12.249 255.255.255.0

!

! Reals in this server farm need to come from server vlan

serverfarm IMG

  no nat server

  no nat client

  real 10.1y.12.50 << from server vlan

   inservice

  real 10.1y.12.51 << from server vlan

   no inservice

!

! Reals in this server farm need to come from SSL vlan

serverfarm IMG_SSL

  no nat server

  no nat client

  real 10.1x.12.2 local << from SSL vlan

   inservice

!       

! Then you need to decide if you want to let http through to your webservers from the outside, if so leave this in place, if not suggest you change it to a redirect

vserver IMG

  virtual 10.11.12.54 tcp www

  serverfarm IMG

  persistent rebalance

  inservice

!       

! This will terminate the connection in from the client ok

vserver IMG_SSL

  virtual 10.11.12.54 tcp https

  serverfarm IMG_SSL

  persistent rebalance

  inservice

!

! You then need another server farm to receieve the traffic back from the SSL in the clear

vserver IMG_SSL

  virtual 10.1x.12.54 tcp www

  serverfarm IMG

  persistent rebalance

  inservice

You'll need to update the SSL proxy to send traffic back to this new VIP

ssl-proxy service pixel_ssl 
virtual ipaddr 10.11.12.54 protocol tcp port 443 secondary
server ipaddr 10.1x.12.54 protocol tcp port 80
certificate rsa general-purpose trustpoint thawte
policy http-header session-info
inservice

Hope this makes sense and good luck

Cheers

Martin