topic Re: CSS restore sticky ssl in Application Networking

CSS restore sticky ssl

dalmada — Fri, 29 Jun 2007 14:37:37 GMT

Hi everyone,

I am facing a strange issue. I have a SSLv3 service running on two servers behind a pair of(active-standby) CSS 11501, wich do advance-balance ssl. Few days ago I noticed that the CSS stopped the LB and start sending he request to only one server( both alive). I think that is hapening due to the SSL-layer 4 fallback. How do I verify it , and restore stickyness with session ID?

Thanks in advance

David

Re: CSS restore sticky ssl

RODRGUTI — Sun, 01 Jul 2007 00:42:06 GMT

Hello David,

What you can do is get simultaneous captures, span the content rules vlan, and the servers vlan, but you need to get the captures on the switch(s) that the CSS is directly connected to. As soon as you get the captures you will see the connection incoming into the CSS and you will see the SSL ID, on the back end side connection you will see the CSS doing the load balancing based on the SSL ID.

Also please add the application SSL command, but the best way to confirm what the CSS is doing is taking the captures.

Hope this help.

- Rodrigo.

Re: CSS restore sticky ssl

skumar1969 — Mon, 02 Jul 2007 03:08:15 GMT

The issue could be more serious than what you are seeing. When you say you rely on ssl-id for LB, I recall a incident that happened some 6 months ago. Internet Explorer (IE) 5.0/5.5 SSL SID changes approximately every two minutes. if you use ssl id to do load balance you may be in trouble, one reason why we started using alternate methods like arrowpoint cookies to do load balancing ssl sessions. Arrowpoint cookie works fine. See below.

http://www.cisco.com/en/US/products/hw/contnetw/ps789/products_tech_note09186a00801c65b4.shtml#client

See the Microsoft version of the IE SSL-id issue. MS recommends a registry hack which I never tried as it is hypothetical to change everywhere on all the IE client browsers.

http://support.microsoft.com/default.aspx?scid=KB;en-us;q265369

Note: I have no information how a IE 6.0 or its later versions behaves. You may want to talk to TAC.

Re: CSS restore sticky ssl

dalmada — Mon, 02 Jul 2007 09:41:55 GMT

Hi Rodrigo,

I did some captures on the server side and I can see the SSL ID exchanged with the client, which is different for every new connection as expected. Using the "show sticky-table ssl-sticky sid" I can see the flow details associated. So I think it did not l4-fallback, but I do not understand why it does not load balance. I also use the application SSL command.

Thank you,

David

Re: CSS restore sticky ssl

dalmada — Mon, 02 Jul 2007 09:49:57 GMT

Hi Skumar,

We are not using IE in this case. There is a custom client application, which does a request and get the answer and then the server closes the connection. The load balance worked well until few days ago, and I still see the SSL ID exchanged between client and server for each new connection, but the SSL does not load balance.

David

Re: CSS restore sticky ssl

RODRGUTI — Tue, 03 Jul 2007 00:21:37 GMT

Hello David,

How are you seeing that the CSS is not doing the correct load balancing?

Are you seeing one of the servers being overloaded?

-Rodrigo

Re: CSS restore sticky ssl

dalmada — Tue, 03 Jul 2007 09:25:29 GMT

Hi Rodrigo,

The content is configured for round-robin.

Using the flows command I can see that it sends all the requests to the same server. If I suspend the service pointing to that server then it starts sending to the other and remains on that server even when the other is reactiveted.

That's why I thought it was in SSL-l4-fallback. But I don't have any entries in L4 sticky table and I can see the flows in ssl-sticky table.

The trace on the server side shows that a new Session ID is created for a new connection as expected.

David

Re: CSS restore sticky ssl

Pavel Bykov — Tue, 03 Jul 2007 09:44:53 GMT

Do you have Sticky inactive timeout?

"sticky-inact-timeout 240" worked for us.

Re: CSS restore sticky ssl

dalmada — Tue, 03 Jul 2007 10:05:33 GMT

hi s.

No, I don't have it configured. But I don't understand the need for it, in this case. As I wrote before, every new connection ( request) is expected to be a new entry in the sticky-table. The CSS should do a roud-robin load balance for those requests.

David

Re: CSS restore sticky ssl

dalmada — Tue, 03 Jul 2007 11:29:23 GMT

Further observations:

issueing the commnd ssl-l4-fallback disable, restores the load balance. This means, I think, that the l4-fallback was activated. The questions remaiins, why, if we are using ssl v3? why the l4 sticky table has no entries. Here is a ssldump example:

New TCP connection #2: 172.16.20.1(46311) <-> 172.26.80.1(3034)

2 1 0.0011 (0.0011) C>SV3.0(95) Handshake

ClientHello

Version 3.0

random[32]=

46 8a 31 6f 87 94 35 a8 30 16 42 5e e9 6e 31 c0

d6 17 ac eb 92 9c 53 db 85 92 df 30 0e 67 90 90

cipher suites

Unknown value 0x39

Unknown value 0x38

Unknown value 0x35

SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA

SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA

SSL_RSA_WITH_3DES_EDE_CBC_SHA

Unknown value 0x33

Unknown value 0x32

Unknown value 0x2f

SSL_DHE_DSS_WITH_RC4_128_SHA

SSL_RSA_WITH_RC4_128_SHA

SSL_RSA_WITH_RC4_128_MD5

SSL_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA

SSL_RSA_EXPORT1024_WITH_DES_CBC_SHA

SSL_RSA_EXPORT1024_WITH_RC2_CBC_56_MD5

SSL_DHE_RSA_WITH_DES_CBC_SHA

SSL_DHE_DSS_WITH_DES_CBC_SHA

SSL_RSA_WITH_DES_CBC_SHA

SSL_DHE_DSS_WITH_RC2_56_CBC_SHA

SSL_RSA_EXPORT1024_WITH_RC4_56_SHA

SSL_RSA_EXPORT1024_WITH_RC4_56_MD5

SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA

SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA

SSL_RSA_EXPORT_WITH_DES40_CBC_SHA

SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5

SSL_RSA_EXPORT_WITH_RC4_40_MD5

compression methods

NULL

2 2 0.0025 (0.0014) S>CV3.0(74) Handshake

ServerHello

Version 3.0

random[32]=

46 8a 19 d6 a8 f5 88 8e d4 d5 16 06 df 1e 8e e5

0e 7c 1b f2 f8 bb b9 c8 7d c6 e2 f8 fe e6 7f b0

session_id[32]=

db 23 61 fa 35 5a ef 0e da 3d be b0 c2 85 39 9f

4d b1 75 5d 5b ef c8 46 bc 4d db ab 31 23 d6 d4

cipherSuite SSL_RSA_WITH_3DES_EDE_CBC_SHA

compressionMethod NULL

2 3 0.0025 (0.0000) S>CV3.0(1147) Handshake

Certificate

2 4 0.0025 (0.0000) S>CV3.0(4) Handshake

ServerHelloDone

2 5 0.2346 (0.2320) C>SV3.0(260) Handshake

ClientKeyExchange

EncryptedPreMasterSecret[256]=

1b a5 47 46 7b 9a 8b 84 88 d4 54 ba 23 c7 7a 31

db 8a 74 c2 02 3b 8b 50 75 c3 c8 c0 38 4c 18 e8

0a e8 c8 47 39 ff 9b 3c 6a cc d3 21 78 69 e6 50

88 55 e8 b3 d3 b1 2a 04 b3 ba 66 e4 c8 49 f4 8e

a0 bd 74 60 e9 f2 0c a1 25 47 03 4e 6c ed 96 52

de 2a 9a 60 29 c5 f6 21 c8 3e 58 ef af 3f 12 b2

ee 34 c0 70 12 d0 64 30 28 65 ed fb ff 65 f2 de

d0 bf cd e6 26 79 6f 3c 61 5f df da bf ac 4a cf

4c 0e 0c 66 44 e1 b2 a3 34 f2 27 75 f0 e4 e7 a4

48 06 76 93 73 0d 09 75 35 ea a1 91 d9 c8 ad 58

b9 1f 45 bf c6 09 61 cb 2d 75 4a ba ed 45 15 44

0f fc 9e 5b 90 e7 b2 86 15 1c 43 a5 52 0d c7 1e

d8 81 42 db 77 35 99 4d 0d 5b 20 e6 dd c5 a1 7d

64 9a 13 d2 99 b7 1d 94 a7 fe ce b6 67 7d df b9

25 fb 27 d2 6e 90 49 54 b9 c1 10 32 eb 42 df 43

b6 1c 94 4e ee 0b ca 29 27 8a 3d b8 fe 59 00 f4

2 6 0.2346 (0.0000) C>SV3.0(1) ChangeCipherSpec

2 7 0.2346 (0.0000) C>SV3.0(64) Handshake

2 8 0.2708 (0.0362) S>CV3.0(1) ChangeCipherSpec

2 9 0.2708 (0.0000) S>CV3.0(64) Handshake

2 10 0.3074 (0.0365) C>SV3.0(40) application_data

2 11 7.9036 (7.5961) S>CV3.0(24) application_data

2 12 7.9036 (0.0000) S>CV3.0(104) application_data

2 13 7.9036 (0.0000) S>CV3.0(24) Alert

2 7.9036 (0.0000) S>C TCP FIN

2 14 7.9464 (0.0427) C>SV3.0(24) Alert

2 7.9524 (0.0059) C>S TCP FIN

As you can see session ID is sent correctly.

David

Re: CSS restore sticky ssl

dalmada — Tue, 03 Jul 2007 15:36:50 GMT

Hi guys,

It is solved now.

I guess a few days ago the client or server, starts retransmiting frames activating the SSL-l4-fallback feature.

And , apparently the only way to restore the behavior, is to:

issue ssl-l4-fallback disable

suspend/activate the associated content rule

which causes the sticky table to be purged for the flows associated with the content rule.

Still don't figured out why the l4 sticky table was empty.

Thank y'all

you were very helpfull