SSL handshake Failure using CSM and SSL module 6509

[mcgbic]

We have a CSM running 4.1.3 and an SSL module with 2.1.4 code. We have two 6509s that contains each type. The active CSM load-balances SSL traffic to two SSL modules.

Our application performance testers have reported random ssl handshake failures during their loadrunner tests and java script tests. An ethereal trace from their client PC indicated that after the SSLv3 Client Hello packet, there came a FIN,ACK packet right away from the "vserver VIP" configured in the csm. We think that this is something to do with the SSL negotiation and that nothing in the backend has been established yet.

What do you think could cause this FIN,ACK reply from the CSM vserver VIP that breaks the ssl handshake process?

Please help! thanks

Amante

[Gilles Dufour]

if you do sticky ssl, which you will need if you have 2 SSL modules, the CSM will spoof the connection until it can detect the SSL ID sent in the client hello.

So, if you get a FIN after the client hello, it is most probably related to this SSL ID.

What is the value shown in the trace ?

What's the percentage of error ?

Can you highlight a difference between an error and a working case ?

To be able to solve such a problem, we will need 'sho tech' before and after the test as well as the sniffer trace.

You should better open a TAC case once you have this information.

Regards,

Gilles.

[mcgbic]

Thanks for the quick reply!

Yes we do have sticky group and ssl-sticky offset 20 and length 6 in the vserver configs. At first we didn't have the "ssl-sticky offset 20 length 6" in the configuration, we put it in and noticed a bit of performance to our test but we still saw the ssl connection failure errors in loadrunner.

When you mentioned about the FIN after the client hello, what do you mean by its relation to SSL ID?

What is the value shown in the trace? What in particular are you asking for? SSL parameters from the FIN/ACK packet sourcing from the CSM?

What's the percentage of error? I would say more than 2% from the load test we've done.

Can you highlight a difference between an error and a working case? I can only tell that when the error occured, the tcp stream will only show an abnormal "Fin/ack" from the CSM Vserver vip which causes the handshake failure alert.

To be able to solve such a problem, we will need 'sho tech' before and after the test as well as the sniffer trace. ===>>>> I work in a very secure Gov site so we can't show any instance of our IP scheme, I can maybe replace the ip scheme and send sh tech to you.

As for the sniffer trace, do you want a sniffer trace from the client to CSM traffic and do you also want a sniffer trace from the CSM to the real server?

thanks again

Amante