cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3114
Views
0
Helpful
9
Replies

ACE SSL Connections Failing

THOMAS BARRERA
Level 1
Level 1

We have a new secure site where we are using the ACE as a ssl-proxy. I see connections make it all the way to the servers, but the session eventually times out (Browser responds with "The connection has timed out"). I haven't been able to grab a packet capture yet, but I am looking for some input since I am new to the ACE. We are also set up for sticky connections using cookies.

I see connections to the server but no response back. I also see the cookie places in my browser. Once I close the browser window, the current connection drops.

sh serverfarm SECUREMAIL

serverfarm     : SECUREMAIL, type: HOST

total rservers : 2

---------------------------------

                                                ----------connections-----------

       real                  weight state        current    total      failures

   ---+---------------------+------+------------+----------+----------+---------

   rserver: E01

       10.0.0.95:8080        8      OPERATIONAL  1          4          0

   rserver: E02

       10.0.0.98:8080        8      OPERATIONAL  0          1         

I verified the cert and keys match with the verify cryto command. If I bypass https and connect via http, I am able to hit the server test page. I attached the scrubbed config.

Any info is appreciated.

9 Replies 9

venkatkr
Cisco Employee
Cisco Employee

When you go via http, are you hitting hte server directly or going via the ACE  on a  http vip ? If you are going directly then you should try the following

1.Clear service-policy  SECUREMAIL-EMG

2. try the connection and on the ace do "show conn".

3. show service-policy SECUREMAIL-EMG

With the above you will be able to tell if the connection is coming into the ACE and whether the return traffic is assymetric or not.

In show conn you should see the first leg (from client to ace VIIP) show up as ESTAB.

Also if its a SSL issue, you can run show stats cryto server. Run this before the test and one after the test and see what parameter increases.

This should give you a start

Thanks

VK

Thanks for the info,

Regarding http, it was going via ACE. I built a temp http configuration for the same vip/real servers allowing port 80.

I do see the connection Establicsh

sh serverfarm SECUREMAIL

serverfarm     : SECUREMAIL, type: HOST

total rservers : 2

---------------------------------

                                                ----------connections-----------

       real                  weight state        current    total      failures

   ---+---------------------+------+------------+----------+----------+---------

   rserver: E01

       10.0.0.95:8080        8      OPERATIONAL  1          12         0

   rserver: E02

       10.0.0.98:8080        8      OPERATIONAL  0          1          0

sh conn | inc 10.0.5.19

2104390    1  in  TCP   77   x.x.x.x:40758    10.0.5.19:443         ESTAB

Thats good. Here's what you need to do to see both legs of the show conn..

Do as you did

sh conn | inc 10.0.5.19

2104390    1  in  TCP   77   x.x.x.x:40758    10.0.5.19:443         ESTAB

Then do

show conn | b 2104390

The first two output are part of this connections. You need to do this quickly. You need to see IN and OUT.

If HTTP is working then its not asymmetric...they it might be a SSL or something related to proxy connections.

Is the certificate trusted by well know root CA? If not, then if you use firefox, you should get an alert stating do you want to continue. If you are getting that then ACE is presenting the certificate.

Also in the earlier email i asked you to run show service-policy. The command i gave was not complete.

You need to run

show service-policy SECUREMAIL-EMG class-map SECUREMAIL detail

Thanks again for the reply. I do see the established session in both directions which quickly closes. I am not presented with anything from the browser. I will likely need to get a capture of the LAN to see more details.

sh conn | b 2609203

2609203    2  in  TCP   77   x.x.x.82:45879    10.0.5.19:443         ESTAB

2609204    2  out TCP   79   10.0.0.95:8080        x.x.x.82:50373    ESTAB

sh conn | b 2609203

2609203    2  in  TCP   77   x.x.x.82:45879    10.0.5.19:443         ESTAB

2609204    2  out TCP   79   10.0.0.95:8080        x.x.x.82:50373    CLOSED

cpomeroy
Level 1
Level 1

sh conn | b 2609203

2609203    2  in  TCP   77    x.x.x.82:45879    10.0.5.19:443         ESTAB

2609204    2  out  TCP   79   10.0.0.95:8080        x.x.x.82:50373    CLOSED

The outbound connnection is in a closed state.   This would indicate that the server either reset the connection or sent a fin.  Is the server lisenting on port 8080?  You may can try to telnet to the server on port 8080 to verify that is listening on that port.

On an unrelated note, you do not need persistance rebalance in your configuration.  This will force the ACE to make an addtional loadbalancing decision for every http request.

Make sure clock on supervisor/device has correct date to avoid not before not after check of cert.

Once the configuration is complete, check to make sure the VIP address can be accessed via HTTPS in a web browser. If any certificate errors are shown, this indicates a problem with the certificate, not with the Cisco ACE configuration. The above commands can be used to verify that SSL sessions are being terminated successfully.

When a client’s web browser connects to an SSL server on any device, the browser and server negotiate which encryption cipher to use for the session. The list and order of ciphers presented by the ACE in a default configuration are as follows.

1.          CM_SSL_RSA_WITH_RC4_128_MD5

2.          CM_SSL_RSA_WITH_RC4_128_SHA

3.          CM_SSL_RSA_WITH_DES_CBC_SHA

4.          CM_SSL_RSA_WITH_3DES_EDE_CBC_SHA

5.          CM_SSL_RSA_WITH_AES_128_CBC_SHA

6.          CM_SSL_RSA_WITH_AES_256_CBC_SHA

7.          CM_SSL_RSA_EXPORT_WITH_RC4_40_MD5

8.          CM_SSL_RSA_EXPORT1024_WITH_RC4_56_MD5

9.          CM_SSL_RSA_EXPORT_WITH_DES40_CBC_SHA

10.          CM_SSL_RSA_EXPORT1024_WITH_DES_CBC_SHA

11.          CM_SSL_RSA_EXPORT1024_WITH_RC4_56_SHA

If this list is not desirable or the order needs to be changed, an SSL parameter map can be configured to make such changes.

Can you send the output of the following commands to suggest more on your config

ACE-1/routed#show crypto authgroup all

ACE-1/routed# show conn display 1000 detail

ACE-1/routed# show crypto files

ACE-1/routed# show crypto certificate all

ACE-1/routed# show crypto key all

ACE-1/routed# show crypto session

ACE-1/routed# show crypto hardware

ACE-1/routed# show service-policy detail

Please Display client SSL statistics by entering the the following command and also attach it here so that I can also see what is happening in your ace device:

ACE_module5/Admin# show stats crypto client

+----------------------------------------------+

+---- Crypto client termination statistics ----+

+----------------------------------------------+

SSLv3 negotiated protocol:                        0

TLSv1 negotiated protocol:                        0

SSLv3 full handshakes:                            0

SSLv3 resumed handshakes:                         0

SSLv3 rehandshakes:                               0

TLSv1 full handshakes:                            0

TLSv1 resumed handshakes:                         0

TLSv1 rehandshakes:                               0

SSLv3 handshake failures:                         0

SSLv3 failures during data phase:                 0

TLSv1 handshake failures:                         0

TLSv1 failures during data phase:                 0

Handshake Timeouts:                               0

total transactions:                               0

SSLv3 active connections:                         0

SSLv3 connections in handshake phase:             0

SSLv3 conns in renegotiation phase:               0

SSLv3 connections in data phase:                  0

TLSv1 active connections:                         0

TLSv1 connections in handshake phase:             0

TLSv1 conns in renegotiation phase:               0

TLSv1 connections in data phase:                  0

+----------------------------------------------+

+------- Crypto client alert statistics -------+

+----------------------------------------------+

SSL alert CLOSE_NOTIFY rcvd:                      0

SSL alert UNEXPECTED_MSG rcvd:                    0

SSL alert BAD_RECORD_MAC rcvd:                    0

SSL alert DECRYPTION_FAILED rcvd:                 0

SSL alert RECORD_OVERFLOW rcvd:                   0

SSL alert DECOMPRESSION_FAILED rcvd:              0

SSL alert HANDSHAKE_FAILED rcvd:                  0

SSL alert NO_CERTIFICATE rcvd:                    0

SSL alert BAD_CERTIFICATE rcvd:                   0

SSL alert UNSUPPORTED_CERTIFICATE rcvd:           0

SSL alert CERTIFICATE_REVOKED rcvd:               0

SSL alert CERTIFICATE_EXPIRED rcvd:               0

SSL alert CERTIFICATE_UNKNOWN rcvd:               0

SSL alert ILLEGAL_PARAMETER rcvd:                 0

SSL alert UNKNOWN_CA rcvd:                        0

SSL alert ACCESS_DENIED rcvd:                     0

SSL alert DECODE_ERROR rcvd:                      0

SSL alert DECRYPT_ERROR rcvd:                     0

SSL alert EXPORT_RESTRICTION rcvd:                0

SSL alert PROTOCOL_VERSION rcvd:                  0

SSL alert INSUFFICIENT_SECURITY rcvd:             0

SSL alert INTERNAL_ERROR rcvd:                    0

SSL alert USER_CANCELED rcvd:                     0

SSL alert NO_RENEGOTIATION rcvd:                  0

SSL alert CLOSE_NOTIFY sent:                      0

SSL alert UNEXPECTED_MSG sent:                    0

SSL alert BAD_RECORD_MAC sent:                    0

SSL alert DECRYPTION_FAILED sent:                 0

SSL alert RECORD_OVERFLOW sent:                   0

SSL alert DECOMPRESSION_FAILED sent:              0

SSL alert HANDSHAKE_FAILED sent:                  0

SSL alert NO_CERTIFICATE sent:                    0

SSL alert BAD_CERTIFICATE sent:                   0

SSL alert UNSUPPORTED_CERTIFICATE sent:           0

SSL alert CERTIFICATE_REVOKED sent:               0

SSL alert CERTIFICATE_EXPIRED sent:               0

SSL alert CERTIFICATE_UNKNOWN sent:               0

SSL alert ILLEGAL_PARAMETER sent:                 0

SSL alert UNKNOWN_CA sent:                        0

SSL alert ACCESS_DENIED sent:                     0

SSL alert DECODE_ERROR sent:                      0

SSL alert DECRYPT_ERROR sent:                     0

SSL alert EXPORT_RESTRICTION sent:                0

SSL alert PROTOCOL_VERSION sent:                  0

SSL alert INSUFFICIENT_SECURITY sent:             0

SSL alert INTERNAL_ERROR sent:                    0

SSL alert USER_CANCELED sent:                     0

SSL alert NO_RENEGOTIATION sent:                  0

+-----------------------------------------------+

+--- Crypto client authentication statistics ---+

+-----------------------------------------------+

Total SSL client authentications:                 0

Failed SSL client authentications:                0

SSL client authentication cache hits:             0

SSL static CRL lookups:                           0

SSL best effort CRL lookups:                      0

SSL CRL lookup cache hits:                        0

SSL revoked certificates:                         0

Total SSL server authentications:                 0

Failed SSL server authentications:                0

+-----------------------------------------------+

+------- Crypto client cipher statistics -------+

+-----------------------------------------------+

Cipher sslv3_rsa_rc4_128_md5:                     0

Cipher sslv3_rsa_rc4_128_sha:                     0

Cipher sslv3_rsa_des_cbc_sha:                     0

Cipher sslv3_rsa_3des_ede_cbc_sha:                0

Cipher sslv3_rsa_exp_rc4_40_md5:                  0

Cipher sslv3_rsa_exp_des40_cbc_sha:               0

Cipher sslv3_rsa_exp1024_rc4_56_md5:              0

Cipher sslv3_rsa_exp1024_des_cbc_sha:             0

Cipher sslv3_rsa_exp1024_rc4_56_sha:              0

Cipher sslv3_rsa_aes_128_cbc_sha:                 0

Cipher sslv3_rsa_aes_256_cbc_sha:                 0

Cipher tlsv1_rsa_rc4_128_md5:                     0

Cipher tlsv1_rsa_rc4_128_sha:                     0

Cipher tlsv1_rsa_des_cbc_sha:                     0

Cipher tlsv1_rsa_3des_ede_cbc_sha:                0

Cipher tlsv1_rsa_exp_rc4_40_md5:                  0

Cipher tlsv1_rsa_exp_des40_cbc_sha:               0

Cipher tlsv1_rsa_exp1024_rc4_56_md5:              0

Cipher tlsv1_rsa_exp1024_des_cbc_sha:             0

Cipher tlsv1_rsa_exp1024_rc4_56_sha:              0

Cipher tlsv1_rsa_aes_128_cbc_sha:                 0

Cipher tlsv1_rsa_aes_256_cbc_sha:                 0

To  Display SSL server statistics by entering the following command and send the results to us for further suggestions:

ACE_module5/Admin# show stats crypto server

+----------------------------------------------+

+---- Crypto server termination statistics ----+

+----------------------------------------------+

SSLv3 negotiated protocol:                        0

TLSv1 negotiated protocol:                        0

SSLv3 full handshakes:                            0

SSLv3 resumed handshakes:                         0

SSLv3 rehandshakes:                               0

TLSv1 full handshakes:                            0

TLSv1 resumed handshakes:                         0

TLSv1 rehandshakes:                               0

SSLv3 handshake failures:                         0

SSLv3 failures during data phase:                 0

TLSv1 handshake failures:                         0

TLSv1 failures during data phase:                 0

Handshake Timeouts:                               0

total transactions:                               0

SSLv3 active connections:                         0

SSLv3 connections in handshake phase:             0

SSLv3 conns in renegotiation phase:               0

SSLv3 connections in data phase:                  0

TLSv1 active connections:                         0

TLSv1 connections in handshake phase:             0

TLSv1 conns in renegotiation phase:               0

TLSv1 connections in data phase:                  0

+----------------------------------------------+

+------- Crypto server alert statistics -------+

+----------------------------------------------+

SSL alert CLOSE_NOTIFY rcvd:                      0

SSL alert UNEXPECTED_MSG rcvd:                    0

SSL alert BAD_RECORD_MAC rcvd:                    0

SSL alert DECRYPTION_FAILED rcvd:                 0

SSL alert RECORD_OVERFLOW rcvd:                   0

SSL alert DECOMPRESSION_FAILED rcvd:              0

SSL alert HANDSHAKE_FAILED rcvd:                  0

SSL alert NO_CERTIFICATE rcvd:                    0

SSL alert BAD_CERTIFICATE rcvd:                   0

SSL alert UNSUPPORTED_CERTIFICATE rcvd:           0

SSL alert CERTIFICATE_REVOKED rcvd:               0

SSL alert CERTIFICATE_EXPIRED rcvd:               0

SSL alert CERTIFICATE_UNKNOWN rcvd:               0

SSL alert ILLEGAL_PARAMETER rcvd:                 0

SSL alert UNKNOWN_CA rcvd:                        0

SSL alert ACCESS_DENIED rcvd:                     0

SSL alert DECODE_ERROR rcvd:                      0

SSL alert DECRYPT_ERROR rcvd:                     0

SSL alert EXPORT_RESTRICTION rcvd:                0

SSL alert PROTOCOL_VERSION rcvd:                  0

SSL alert INSUFFICIENT_SECURITY rcvd:             0

SSL alert INTERNAL_ERROR rcvd:                    0

SSL alert USER_CANCELED rcvd:                     0

SSL alert NO_RENEGOTIATION rcvd:                  0

SSL alert CLOSE_NOTIFY sent:                      0

SSL alert UNEXPECTED_MSG sent:                    0

SSL alert BAD_RECORD_MAC sent:                    0

SSL alert DECRYPTION_FAILED sent:                 0

SSL alert RECORD_OVERFLOW sent:                   0

SSL alert DECOMPRESSION_FAILED sent:              0

SSL alert HANDSHAKE_FAILED sent:                  0

SSL alert NO_CERTIFICATE sent:                    0

SSL alert BAD_CERTIFICATE sent:                   0

SSL alert UNSUPPORTED_CERTIFICATE sent:           0

SSL alert CERTIFICATE_REVOKED sent:               0

SSL alert CERTIFICATE_EXPIRED sent:               0

SSL alert CERTIFICATE_UNKNOWN sent:               0

SSL alert ILLEGAL_PARAMETER sent:                 0

SSL alert UNKNOWN_CA sent:                        0

SSL alert ACCESS_DENIED sent:                     0

SSL alert DECODE_ERROR sent:                      0

SSL alert DECRYPT_ERROR sent:                     0

SSL alert EXPORT_RESTRICTION sent:                0

SSL alert PROTOCOL_VERSION sent:                  0

SSL alert INSUFFICIENT_SECURITY sent:             0

SSL alert INTERNAL_ERROR sent:                    0

SSL alert USER_CANCELED sent:                     0

SSL alert NO_RENEGOTIATION sent:                  0

+-----------------------------------------------+

+--- Crypto server authentication statistics ---+

+-----------------------------------------------+

Total SSL client authentications:                 0

Failed SSL client authentications:                0

SSL client authentication cache hits:             0

SSL static CRL lookups:                           0

SSL best effort CRL lookups:                      0

SSL CRL lookup cache hits:                        0

SSL revoked certificates:                         0

Total SSL server authentications:                 0

Failed SSL server authentications:                0

+-----------------------------------------------+

+------- Crypto server cipher statistics -------+

+-----------------------------------------------+

Cipher sslv3_rsa_rc4_128_md5:                     0

Cipher sslv3_rsa_rc4_128_sha:                     0

Cipher sslv3_rsa_des_cbc_sha:                     0

Cipher sslv3_rsa_3des_ede_cbc_sha:                0

Cipher sslv3_rsa_exp_rc4_40_md5:                  0

Cipher sslv3_rsa_exp_des40_cbc_sha:               0

Cipher sslv3_rsa_exp1024_rc4_56_md5:              0

Cipher sslv3_rsa_exp1024_des_cbc_sha:             0

Cipher sslv3_rsa_exp1024_rc4_56_sha:              0

Cipher sslv3_rsa_aes_128_cbc_sha:                 0

Cipher sslv3_rsa_aes_256_cbc_sha:                 0

Cipher tlsv1_rsa_rc4_128_md5:                     0

Cipher tlsv1_rsa_rc4_128_sha:                     0

Cipher tlsv1_rsa_des_cbc_sha:                     0

Cipher tlsv1_rsa_3des_ede_cbc_sha:                0

Cipher tlsv1_rsa_exp_rc4_40_md5:                  0

Cipher tlsv1_rsa_exp_des40_cbc_sha:               0

Cipher tlsv1_rsa_exp1024_rc4_56_md5:              0

Cipher tlsv1_rsa_exp1024_des_cbc_sha:             0

Cipher tlsv1_rsa_exp1024_rc4_56_sha:              0

Cipher tlsv1_rsa_aes_128_cbc_sha:                 0

Cipher tlsv1_rsa_aes_256_cbc_sha:                 0

Also you can Display the number of SSL data messages sent and SSL FIN/RST messages sent by entering the following command and send the output from your ACE devices:

ACE_module5/Admin# show stats http

+------------------------------------------+

+-------------- HTTP statistics -----------+

+------------------------------------------+

LB parse result msgs sent : 0          , TCP data msgs sent       : 0

Inspect parse result msgs : 0          , SSL data msgs sent       : 0 <-------

                      sent

TCP fin/rst msgs sent     : 0          , Bounced fin/rst msgs sent: 0

SSL fin/rst msgs sent     : 0          , Unproxy msgs sent        : 0 <-------

Drain msgs sent           : 0          , Particles read           : 0

Reuse msgs sent           : 0          , HTTP requests            : 0

Reproxied requests        : 0          , Headers removed          : 0

Headers inserted          : 0          , HTTP redirects           : 0

HTTP chunks               : 0          , Pipelined requests       : 0

HTTP unproxy conns        : 0          , Pipeline flushes         : 0

Whitespace appends        : 0          , Second pass parsing      : 0

Response entries recycled : 0          , Analysis errors          : 0

Header insert errors      : 0          , Max parselen errors      : 0

Static parse errors       : 0          , Resource errors          : 0

Invalid path errors       : 0          , Bad HTTP version errors  : 0

Headers rewritten         : 0          , Header rewrite errors    : 0

Lastly to  Display session cache statistics for the current context by entering the following command:

switch/Admin# show crypto session

SSL Session Cache Stats for Context

------------------

Number of Client Sessions:                        0

Number of Server Sessions:                        0

Please send the output of all the commands requested to see in more detail for your issue.

HTH

Sachin

When I connect via https to the IP address, I get a "Untrusted Connection" warning...

x.x.x.20 uses an invalid security certificate.

The certificate is only valid for the following names:

  x.y.com , www.x.y.com 

(Error code: ssl_error_bad_cert_domain)

OK, if I change the port on the backend from 8080 to 80, I get a test  page with legit security cert. I also just found out this rserver is  trying to establish a session outbound to the internet before the entire connection is deemed "secure". I think this is causing the issue. How would the ace handle a rserver establishing a connection outbound?

OK, it is almost working now...i built a nat for rserver access that started working once i added a permit any ACL on the vlan facing the rservers. i can pull up the website now. last piece is getting a url redirect or rewrite done to point users to login page.

i appreciate everyones input

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: