Internal error int reply to ClientHello on ACE20 module with end

Anatoly Mazalsky · ‎06-17-2013

Hello, world!

We have weird behaviour of our Cisco ACE20 module configured for end-to-end SSL (initiation+termination) - the module from time to time replies with SSLv3/TLSv1 alert "Fatal: internal error" message to the client right after client have sent 'ClientHello' SSL message. So sometimes SSL connection works and sometimes will be immediately closed due to this fatal error. The behaviour is very similar to the one described below:

CSCtc52085—After a client sends a ClientHello message, the SSL hand shake may fail with a fatal alert internal error sent by the ACE. This behavior is intermittent and may occur under the following conditions:

1. An SSL service is configured with the session-cache timeout command (session reuse).

2. SSL connections are aborted by the client after the client sends a ClientHello message to the service in condition 1 and before an internal resource state is changed. This behavior puts the internal resource in an improper state. This error is very timing sensitive.

3. The next connection that uses the internal resource in the improper state fails with a fatal alert internal error. That connection does not have to go to the service in condition 1 to experience this error because the internal resource is shared by all the SSL services.

Workaround: None.

But we have software version A2(3.1) and this bug must have been resolved in this release (at least it's listed in resolved caveats section of release notes).

Software

loader: Version 12.2[123]

system: Version A2(3.1) [build 3.0(0)A2(3.1)]

system image file: [LCP] disk0:c6ace-t1k9-mz.A2_3_1.bin

Again, we don't have session-cache timeout configured on the ACE.

On the ACE we have following stats:

ACE1/VC_UNIX# sh stats crypto server

SSL Server Statistics:

------------------

SSL alert CLOSE_NOTIFY rcvd: 0

SSL alert UNEXPECTED_MSG rcvd: 0

SSL alert BAD_RECORD_MAC rcvd: 0

SSL alert DECRYPTION_FAILED rcvd: 0

SSL alert RECORD_OVERFLOW rcvd: 0

SSL alert DECOMPRESSION_FAILED rcvd: 0

SSL alert HANDSHAKE_FAILED rcvd: 0

SSL alert NO_CERTIFICATE rcvd: 0

SSL alert BAD_CERTIFICATE rcvd: 0

SSL alert UNSUPPORTED_CERTIFICATE rcvd: 0

SSL alert CERTIFICATE_REVOKED rcvd: 0

SSL alert CERTIFICATE_EXPIRED rcvd: 0

SSL alert CERTIFICATE_UNKNOWN rcvd: 0

SSL alert ILLEGAL_PARAMETER rcvd: 0

SSL alert UNKNOWN_CA rcvd: 0

SSL alert ACCESS_DENIED rcvd: 0

SSL alert DECODE_ERROR rcvd: 0

SSL alert DECRYPT_ERROR rcvd: 0

SSL alert EXPORT_RESTRICTION rcvd: 0

SSL alert PROTOCOL_VERSION rcvd: 0

SSL alert INSUFFICIENT_SECURITY rcvd: 0

SSL alert INTERNAL_ERROR rcvd: 0

SSL alert USER_CANCELED rcvd: 0

SSL alert NO_RENEGOTIATION rcvd: 0

SSL alert CLOSE_NOTIFY sent: 0

SSL alert UNEXPECTED_MSG sent: 1

SSL alert BAD_RECORD_MAC sent: 0

SSL alert DECRYPTION_FAILED sent: 0

SSL alert RECORD_OVERFLOW sent: 0

SSL alert DECOMPRESSION_FAILED sent: 0

SSL alert HANDSHAKE_FAILED sent: 2

SSL alert NO_CERTIFICATE sent: 0

SSL alert BAD_CERTIFICATE sent: 0

SSL alert UNSUPPORTED_CERTIFICATE sent: 0

SSL alert CERTIFICATE_REVOKED sent: 0

SSL alert CERTIFICATE_EXPIRED sent: 0

SSL alert CERTIFICATE_UNKNOWN sent: 0

SSL alert ILLEGAL_PARAMETER sent: 0

SSL alert UNKNOWN_CA sent: 0

SSL alert ACCESS_DENIED sent: 0

SSL alert DECODE_ERROR sent: 0

SSL alert DECRYPT_ERROR sent: 0

SSL alert EXPORT_RESTRICTION sent: 0

SSL alert PROTOCOL_VERSION sent: 0

SSL alert INSUFFICIENT_SECURITY sent: 0

SSL alert INTERNAL_ERROR sent: 16

SSL alert USER_CANCELED sent: 0

SSL alert NO_RENEGOTIATION sent: 0

SSLv2 client hello received: 0

SSLv3 client hello received: 0

TLSv1 client hello received: 68

SSLv3 negotiated protocol: 0

TLSv1 negotiated protocol: 68

SSLv3 full handshakes: 0

SSLv3 resumed handshakes: 0

Cipher sslv3_rsa_rc4_128_md5: 0

Cipher sslv3_rsa_rc4_128_sha: 0

Cipher sslv3_rsa_des_cbc_sha: 0

Cipher sslv3_rsa_3des_ede_cbc_sha: 0

Cipher sslv3_rsa_exp_rc4_40_md5: 0

Cipher sslv3_rsa_exp_des40_cbc_sha: 0

Cipher sslv3_rsa_exp1024_rc4_56_md5: 0

Cipher sslv3_rsa_exp1024_des_cbc_sha: 0

Cipher sslv3_rsa_exp1024_rc4_56_sha: 0

Cipher sslv3_rsa_aes_128_cbc_sha: 0

Cipher sslv3_rsa_aes_256_cbc_sha: 0

TLSv1 full handshakes: 33

TLSv1 resumed handshakes: 0

Cipher tlsv1_rsa_rc4_128_md5: 68

Cipher tlsv1_rsa_rc4_128_sha: 0

Cipher tlsv1_rsa_des_cbc_sha: 0

Cipher tlsv1_rsa_3des_ede_cbc_sha: 0

Cipher tlsv1_rsa_exp_rc4_40_md5: 0

Cipher tlsv1_rsa_exp_des40_cbc_sha: 0

Cipher tlsv1_rsa_exp1024_rc4_56_md5: 0

Cipher tlsv1_rsa_exp1024_des_cbc_sha: 0

Cipher tlsv1_rsa_exp1024_rc4_56_sha: 0

Cipher tlsv1_rsa_aes_128_cbc_sha: 0

Cipher tlsv1_rsa_aes_256_cbc_sha: 0

Total SSL client authentications: 0

Failed SSL client authentications: 0

SSL authentication cache hits: 0

SSL static CRL lookups: 0

SSL best effort CRL lookups: 0

SSL CRL lookup cache hits: 0

SSL revoked certificates: 0

Total SSL server authentications: 0

Failed SSL server authentications: 0

Session headers extracted: 0

Session headers failed: 0

Server cert headers extracted: 0

Server cert headers failed: 0

Client cert headers extracted: 0

Client cert headers failed: 0

Headers truncated: 0

Redirects due to cert not yet valid: 0

Redirects due to cert expired: 0

Redirects due to unknown issuer cert: 0

Redirects due to cert revoked: 0

Redirects due to no client cert: 0

Redirects due to no CRL available: 0

Redirects due to expired CRL: 0

Redirects due to bad cert signature: 0

Redirects due to other cert error: 0

Internal error: 27

Handshake FlushRX operations: 0

Handshake FlushTX operations: 0

Xscale messages rcvd from ME: 1313330

Xscale messages sent to ME: 2041768

Finish msg split across ssl recs: 0

Fasttx msg ring full: 0

SSL_ME tx msg ring full: 0

N2 encrypt_record: 0

N2 decrypt_record: 144433

N2 random: 439915

N2 handshake_hash: 878094

N2 hash: 0

N2 gpop_master: 291164

N2 gpop_import_master_secret: 5

N2 gpop_pkcs1v15enc: 144430

N2 gpop_pkcs1v15enc_crt: 0

N2 gpop_finish: 291140

N2 gpop_verify: 0

N2 gpop_pkcs1v15dec: 0

N2 gpop_pkcs1v15dec_crt: 146752

N2 rsa_server_full: 15

N2 resume: 12

UXP A: 24576

UXP B: 0

The "Internal error" counter increases with failed connections.

Printscreen from wireshark attached.

Maybe someone has the problem like ours? I have no idea how to troubleshoot these "internal errors"... :-(

Thanks for your replies.

Jorge Bejarano · ‎06-17-2013

Anatoly,

You may want to check this command:

# show np 1 me-stats "-shttp -v"
# show np 1 me-stats -E0

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Show_Counter_Reference_--_Command_Set_4

You may want to get a 10giga capture to see the entire communication between the client, ACE and servers.

You may need to check the logging messages of the ACE as well.

Jorge

Anatoly Mazalsky · ‎06-17-2013

Thanks for your reply.

The problem is not server-related, I have exactly the same situation if I do SSL termination only, with unencrypted connection between ACE and backend servers (another servers, with blank apache installed and listening on 443 port for non-SSL traffic) - generally speaking, it works, but often with these "internal errors" - not suitable for production.

Here is the output from the commands

# show np 1 me-stats -E0

SSL Server Statistics:

------------------

SSL alert CLOSE_NOTIFY rcvd: 0

SSL alert UNEXPECTED_MSG rcvd: 0

SSL alert BAD_RECORD_MAC rcvd: 0

SSL alert DECRYPTION_FAILED rcvd: 0

SSL alert RECORD_OVERFLOW rcvd: 0

SSL alert DECOMPRESSION_FAILED rcvd: 0

SSL alert HANDSHAKE_FAILED rcvd: 0

SSL alert NO_CERTIFICATE rcvd: 0

SSL alert BAD_CERTIFICATE rcvd: 0

SSL alert UNSUPPORTED_CERTIFICATE rcvd: 0

SSL alert CERTIFICATE_REVOKED rcvd: 0

SSL alert CERTIFICATE_EXPIRED rcvd: 0

SSL alert CERTIFICATE_UNKNOWN rcvd: 0

SSL alert ILLEGAL_PARAMETER rcvd: 0

SSL alert UNKNOWN_CA rcvd: 0

SSL alert ACCESS_DENIED rcvd: 0

SSL alert DECODE_ERROR rcvd: 0

SSL alert DECRYPT_ERROR rcvd: 0

SSL alert EXPORT_RESTRICTION rcvd: 0

SSL alert PROTOCOL_VERSION rcvd: 0

SSL alert INSUFFICIENT_SECURITY rcvd: 0

SSL alert INTERNAL_ERROR rcvd: 0

SSL alert USER_CANCELED rcvd: 0

SSL alert NO_RENEGOTIATION rcvd: 0

SSL alert CLOSE_NOTIFY sent: 0

SSL alert UNEXPECTED_MSG sent: 0

SSL alert BAD_RECORD_MAC sent: 0

SSL alert DECRYPTION_FAILED sent: 0

SSL alert RECORD_OVERFLOW sent: 0

SSL alert DECOMPRESSION_FAILED sent: 0

SSL alert HANDSHAKE_FAILED sent: 0

SSL alert NO_CERTIFICATE sent: 0

SSL alert BAD_CERTIFICATE sent: 0

SSL alert UNSUPPORTED_CERTIFICATE sent: 0

SSL alert CERTIFICATE_REVOKED sent: 0

SSL alert CERTIFICATE_EXPIRED sent: 0

SSL alert CERTIFICATE_UNKNOWN sent: 0

SSL alert ILLEGAL_PARAMETER sent: 0

SSL alert UNKNOWN_CA sent: 0

SSL alert ACCESS_DENIED sent: 0

SSL alert DECODE_ERROR sent: 0

SSL alert DECRYPT_ERROR sent: 0

SSL alert EXPORT_RESTRICTION sent: 0

SSL alert PROTOCOL_VERSION sent: 0

SSL alert INSUFFICIENT_SECURITY sent: 0

SSL alert INTERNAL_ERROR sent: 0

SSL alert USER_CANCELED sent: 0

SSL alert NO_RENEGOTIATION sent: 0

SSLv2 client hello received: 0

SSLv3 client hello received: 0

TLSv1 client hello received: 0

SSLv3 negotiated protocol: 0

TLSv1 negotiated protocol: 0

SSLv3 full handshakes: 0

SSLv3 resumed handshakes: 0

Cipher sslv3_rsa_rc4_128_md5: 0

Cipher sslv3_rsa_rc4_128_sha: 0

Cipher sslv3_rsa_des_cbc_sha: 0

Cipher sslv3_rsa_3des_ede_cbc_sha: 0

Cipher sslv3_rsa_exp_rc4_40_md5: 0

Cipher sslv3_rsa_exp_des40_cbc_sha: 0

Cipher sslv3_rsa_exp1024_rc4_56_md5: 0

Cipher sslv3_rsa_exp1024_des_cbc_sha: 0

Cipher sslv3_rsa_exp1024_rc4_56_sha: 0

Cipher sslv3_rsa_aes_128_cbc_sha: 0

Cipher sslv3_rsa_aes_256_cbc_sha: 0

TLSv1 full handshakes: 0

TLSv1 resumed handshakes: 0

Cipher tlsv1_rsa_rc4_128_md5: 0

Cipher tlsv1_rsa_rc4_128_sha: 0

Cipher tlsv1_rsa_des_cbc_sha: 0

Cipher tlsv1_rsa_3des_ede_cbc_sha: 0

Cipher tlsv1_rsa_exp_rc4_40_md5: 0

Cipher tlsv1_rsa_exp_des40_cbc_sha: 0

Cipher tlsv1_rsa_exp1024_rc4_56_md5: 0

Cipher tlsv1_rsa_exp1024_des_cbc_sha: 0

Cipher tlsv1_rsa_exp1024_rc4_56_sha: 0

Cipher tlsv1_rsa_aes_128_cbc_sha: 0

Cipher tlsv1_rsa_aes_256_cbc_sha: 0

Total SSL client authentications: 0

Failed SSL client authentications: 0

SSL authentication cache hits: 0

SSL static CRL lookups: 0

SSL best effort CRL lookups: 0

SSL CRL lookup cache hits: 0

SSL revoked certificates: 0

Total SSL server authentications: 0

Failed SSL server authentications: 0

Session headers extracted: 0

Session headers failed: 0

Server cert headers extracted: 0

Server cert headers failed: 0

Client cert headers extracted: 0

Client cert headers failed: 0

Headers truncated: 0

Redirects due to cert not yet valid: 0

Redirects due to cert expired: 0

Redirects due to unknown issuer cert: 0

Redirects due to cert revoked: 0

Redirects due to no client cert: 0

Redirects due to no CRL available: 0

Redirects due to expired CRL: 0

Redirects due to bad cert signature: 0

Redirects due to other cert error: 0

Internal error: 0

SSL Client Statistics:

------------------

SSL alert CLOSE_NOTIFY rcvd: 0

SSL alert UNEXPECTED_MSG rcvd: 0

SSL alert BAD_RECORD_MAC rcvd: 0

SSL alert DECRYPTION_FAILED rcvd: 0

SSL alert RECORD_OVERFLOW rcvd: 0

SSL alert DECOMPRESSION_FAILED rcvd: 0

SSL alert HANDSHAKE_FAILED rcvd: 0

SSL alert NO_CERTIFICATE rcvd: 0

SSL alert BAD_CERTIFICATE rcvd: 0

SSL alert UNSUPPORTED_CERTIFICATE rcvd: 0

SSL alert CERTIFICATE_REVOKED rcvd: 0

SSL alert CERTIFICATE_EXPIRED rcvd: 0

SSL alert CERTIFICATE_UNKNOWN rcvd: 0

SSL alert ILLEGAL_PARAMETER rcvd: 0

SSL alert UNKNOWN_CA rcvd: 0

SSL alert ACCESS_DENIED rcvd: 0

SSL alert DECODE_ERROR rcvd: 0

SSL alert DECRYPT_ERROR rcvd: 0

SSL alert EXPORT_RESTRICTION rcvd: 0

SSL alert PROTOCOL_VERSION rcvd: 0

SSL alert INSUFFICIENT_SECURITY rcvd: 0

SSL alert INTERNAL_ERROR rcvd: 0

SSL alert USER_CANCELED rcvd: 0

SSL alert NO_RENEGOTIATION rcvd: 0

SSL alert CLOSE_NOTIFY sent: 0

SSL alert UNEXPECTED_MSG sent: 4108

SSL alert BAD_RECORD_MAC sent: 0

SSL alert DECRYPTION_FAILED sent: 0

SSL alert RECORD_OVERFLOW sent: 0

SSL alert DECOMPRESSION_FAILED sent: 0

SSL alert HANDSHAKE_FAILED sent: 63355

SSL alert NO_CERTIFICATE sent: 0

SSL alert BAD_CERTIFICATE sent: 0

SSL alert UNSUPPORTED_CERTIFICATE sent: 0

SSL alert CERTIFICATE_REVOKED sent: 0

SSL alert CERTIFICATE_EXPIRED sent: 0

SSL alert CERTIFICATE_UNKNOWN sent: 0

SSL alert ILLEGAL_PARAMETER sent: 0

SSL alert UNKNOWN_CA sent: 0

SSL alert ACCESS_DENIED sent: 0

SSL alert DECODE_ERROR sent: 0

SSL alert DECRYPT_ERROR sent: 0

SSL alert EXPORT_RESTRICTION sent: 0

SSL alert PROTOCOL_VERSION sent: 0

SSL alert INSUFFICIENT_SECURITY sent: 0

SSL alert INTERNAL_ERROR sent: 37662

SSL alert USER_CANCELED sent: 0

SSL alert NO_RENEGOTIATION sent: 0

SSLv2 client hello received: 0

SSLv3 client hello received: 0

TLSv1 client hello received: 0

SSLv3 negotiated protocol: 0

TLSv1 negotiated protocol: 4062020

SSLv3 full handshakes: 0

SSLv3 resumed handshakes: 0

Cipher sslv3_rsa_rc4_128_md5: 0

Cipher sslv3_rsa_rc4_128_sha: 0

Cipher sslv3_rsa_des_cbc_sha: 0

Cipher sslv3_rsa_3des_ede_cbc_sha: 0

Cipher sslv3_rsa_exp_rc4_40_md5: 0

Cipher sslv3_rsa_exp_des40_cbc_sha: 0

Cipher sslv3_rsa_exp1024_rc4_56_md5: 0

Cipher sslv3_rsa_exp1024_des_cbc_sha: 0

Cipher sslv3_rsa_exp1024_rc4_56_sha: 0

Cipher sslv3_rsa_aes_128_cbc_sha: 0

Cipher sslv3_rsa_aes_256_cbc_sha: 0

TLSv1 full handshakes: 4015344

TLSv1 resumed handshakes: 0

Cipher tlsv1_rsa_rc4_128_md5: 0

Cipher tlsv1_rsa_rc4_128_sha: 0

Cipher tlsv1_rsa_des_cbc_sha: 0

Cipher tlsv1_rsa_3des_ede_cbc_sha: 0

Cipher tlsv1_rsa_exp_rc4_40_md5: 0

Cipher tlsv1_rsa_exp_des40_cbc_sha: 0

Cipher tlsv1_rsa_exp1024_rc4_56_md5: 0

Cipher tlsv1_rsa_exp1024_des_cbc_sha: 0

Cipher tlsv1_rsa_exp1024_rc4_56_sha: 0

Cipher tlsv1_rsa_aes_128_cbc_sha: 4062020

Cipher tlsv1_rsa_aes_256_cbc_sha: 0

Total SSL client authentications: 0

Failed SSL client authentications: 0

SSL authentication cache hits: 4059147

SSL static CRL lookups: 0

SSL best effort CRL lookups: 0

SSL CRL lookup cache hits: 0

SSL revoked certificates: 0

Total SSL server authentications: 4059888

Failed SSL server authentications: 0

Session headers extracted: 0

Session headers failed: 0

Server cert headers extracted: 0

Server cert headers failed: 0

Client cert headers extracted: 0

Client cert headers failed: 0

Headers truncated: 0

Redirects due to cert not yet valid: 0

Redirects due to cert expired: 0

Redirects due to unknown issuer cert: 0

Redirects due to cert revoked: 0

Redirects due to no client cert: 0

Redirects due to no CRL available: 0

Redirects due to expired CRL: 0

Redirects due to bad cert signature: 0

Redirects due to other cert error: 0

Internal error: 20380

Handshake FlushRX operations: 0

Handshake FlushTX operations: 0

Xscale messages rcvd from ME: 12092768

Xscale messages sent to ME: 0x0176adac

Finish msg split across ssl recs: 0

Fasttx msg ring full: 0

SSL_ME tx msg ring full: 0

N2 encrypt_record: 0

N2 decrypt_record: 4015344

N2 random: 8148797

N2 handshake_hash: 4322635

N2 hash: 0

N2 gpop_master: 4041700

N2 gpop_import_master_secret: 0

N2 gpop_pkcs1v15enc: 4041700

N2 gpop_pkcs1v15enc_crt: 0

N2 gpop_finish: 4031710

N2 gpop_verify: 0

N2 gpop_pkcs1v15dec: 0

N2 gpop_pkcs1v15dec_crt: 0

N2 rsa_server_full: 0

N2 resume: 0

UXP A: 24576

UXP B: 0

# show np 1 me-stats "-shttp -v"

HTTP Statistics (Current)

--------------

Unknown msgs received: 0 0

Data rx msgs received: 288293958 4

TCP proxy rx msgs received: 9816884 1

Ack trigger rx msgs received: 0 0

TCP event rx msgs received: 52961189 2

Dest decision tx msgs received: 55155089 1

LB dest decision tx msgs received: 0 0

Close tx msgs received: 83942817 0

Inspect allow tx msgs received: 0 0

Inspect drop tx msgs received: 0 0

DRAM blocks read: 577612022 16

Buffers dropped: 2702255 0

Regex states read: 38438408 25

Unproxy cancellations: 0 0

Redundant closes: 2990271 0

Internal errors: 0 0

Conn mismatch errors: 2748628 0

Exception with close: 6 0

Dest errors: 1 0

Total Packet count (Tx & Rx): 490169937 8

Stop regex: 12 0

(Context 5 Statistics)

Parse result LB msgs sent: 121180 0

Drop: LB queue full: 0 0

Parse result Inspect msgs sent: 0 0

Drop: Inspect queue full: 0 0

TCP data msgs sent: 96215 0

TCP queue full: 0 0

SSL data msgs sent: 516306 0

SSL queue full: 0 0

TCP fin msgs sent: 939 0

TCP rst msgs sent: 147 0

SSL fin msgs sent: 102907 0

SSL rst msgs sent: 38548 0

Bounced fin msgs sent: 1481 0

Bounced rst msgs sent: 2 0

Unproxy msgs sent: 25333 0

Drain msgs sent: 113966 0

Reuse msgs sent: 2304 0

Particles read: 1448314 0

HTTP requests: 121688 0

Reproxied requests: 17680 0

Headers inserted: 3825 0

Headers removed: 51 0

Headers rewritten: 0 0

HTTP redirects: 0 0

HTTP chunks: 42154 0

Unproxy conns: 25325 0

Pipelined requests: 0 0

Pipeline flushes: 0 0

Whitespace appends: 0 0

Response entries recycled: 24493 0

Second pass parsing: 0 0

Vserver mismatch errors: 5 0

Analysis errors: 0 0

Static parse errors: 20 0

Max parselen errors: 0 0

Resource errors: 75 0

Invalid path errors: 0 0

Bad HTTP version errors: 0 0

Header insert errors: 75 0

Header rewrite errors: 0 0

Invalid policy errors: 0 0

Invalid rserver errors: 0 0

Recycled requests: 0 0

SSL header insert success: 0 0

SSL header insert errors: 0 0

SSL spoof header deleted: 0 0

Drop: RST pipelined request: 0 0

There's nothing in ACE logs.

Forgot to mention - we are running ACE in one-arm mode, but I don't believe it makes a difference.

Jorge Bejarano · ‎06-18-2013

Anatoly,

and did you try to load balance 443 traffic only? meaning not to terminate the traffic on the ACE and do it under the servers to see if the behavior persists?

How often do you have this behavior?

Jorge

Anatoly Mazalsky · ‎06-18-2013

Hello Jorge,

I balance both 80 and 443 but SSL termination is only on 443.

Again, with pure L4 load balancing (without SSL-termination) ACE works fine. Directly from client to servers there's no problem with SSL either.

I see this behavior really often, about 20-40% of all SSL connections are reset with SSL Alert Fatal: Internal error.

So now I run ACE in L4 only and it works, but that's not what I want, I need some URL filtering for HTTPS too.

chrhiggi · ‎06-18-2013

Hello Anatoly!

In the stats you sent above, I only see backend SSL occuring - as there are no "SSL Server" stats. You noted this behavior occurs with 443 on the front and 80 on the back -or- with 443 on front and back (decrypted, then encrypted.)

Do you have a sniffer trace you can share? Can you gather 2 sets of stats - one before and one after a failure plus the relevant configuration involved for me?

Regards,

Chris Higgins

Anatoly Mazalsky · ‎06-19-2013

Hello Christopher,

I noted this behavior with both 443 both in end-to-end SSL and SSL termination modes.

End-to-end SSL: Server (443)---SSL----ACE(443)---SSL---Client

SSL termination: Server(443)---unencrypted---ACE(443)----SSL---Client

In SSL termination mode Apache is listening on 443 port but without mod_ssl.

The config is like that:

rserver host FSLIN

ip address 10.0.0.251

inservice

serverfarm host SF_FSLIN

predictor leastconns

rserver FSLIN 443

probe SIMPLEPING

inservice

sticky ip-netmask 255.255.255.255 address source SRC-IP-STICKY-FSLIN

timeout 3600

serverfarm SF_FSLIN

ssl-proxy service Wildcard-SSL

key Lenta_Wildcard

cert Lenta_Wildcard

chaingroup Wildcard_Chain

class-map type http loadbalance match-any OTM_ACL

4 match source-address 9.6.25.108 255.255.255.255

6 match source-address 9.6.25.102 255.255.255.255

class-map match-all OTM_L4_SIMPLE

2 match virtual-address 10.2.100.175 tcp eq https

policy-map type loadbalance http first-match OTM_L7_HTTPS

class class-default

sticky-serverfarm SRC-IP-STICKY-FSLIN

policy-map multi-match PM_ONE_ARM_MULTI_MATCH

class OTM_L4_SIMPLE

loadbalance vip inservice

loadbalance policy OTM_L7_HTTPS

loadbalance vip icmp-reply active

nat dynamic 5 vlan 240

ssl-proxy server Wildcard-SSL

interface vlan 240

description Client_Server

ip address 10.2.100.254 255.255.255.0

ip df clear

mtu 1500

no normalization

fragment chain 128

fragment min-mtu 28

no icmp-guard

access-group input INBOUND

access-group output INBOUND

nat-pool 5 10.2.100.245 10.2.100.252 netmask 255.255.255.0 pat

service-policy input PM_ONE_ARM_MULTI_MATCH

no shutdown

Sniffer trace of erroneous session will be a bit later.

What kind of stats would you like to have?

sh stats crypto server ?

Thanks for your help

Anatoly Mazalsky · ‎06-21-2013

Hello everybody!

At first I would like to thank all of you guys who participated in the discussion and tried to help me with my problem! Thanks for your time guys!

The problem has been solved be reload of Cisco ACE module, now it works fine. The problem was not SSL-related, generally all hosts configured with anything but pute TCP-balancing were affected, i.e. cookie-based sticky serverfarm and so on. After reload of one module, the problem disappeared.

So is the ACE, heh. ((( Actually it had pretty much uptime and made no troubles. The problem appeared after I tried to reassing some of module resources (SSL connection rate) - I wanted to add more SSL rate to one of virtual contexts. Not sure it's related but that's all I've done.

Jorge Bejarano · ‎06-24-2013

Anatoly,

It sounds good, please monitor it and let us know any other news!

Jorge

Internal error int reply to ClientHello on ACE20 module with end-to-end SSL