WSA S190 SSL Configuration - can't open some web-resourses

Anton84 · ‎06-22-2020

Hello all, we are using proxy S190, some sites can't open (example 220-volt.ru, onlinetrade.ru etc.)I think, SSL Configuration is not correctly. Can anybody help me with that problem?

opryluts · ‎06-22-2020

Hi Anton,

I'd start troubleshooting that by checking:

1. Access Logs for transactions towards the failing servers.

2. Proxy error logs/HTTPS logs for any error messages related to the destination. Enable debug log level temporary to see more details for the transactions.

If nothing has been found, I'd collect packet captures on WSA (both legs - client to WSA and WSA to the server at the same time). I'd be looking at TLS handshakes in captures:

1. if any alerts/errors reported

2. what are protocol versions and cipher suits and if they match from both sides

3. any certificate related issues

Also to better understand the problem I'd like to know:

1. WSA version

2. Is HTTPS proxy enabled? Is HTTPS decryption enabled for those specific destinations?

3. How does the issue look from the user perspective? Any error messages displayed in the browser?

Anton84 · ‎06-22-2020

I see in https -log that, when trying to open onlinetrade.ru:

Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: 56:B5:6A:93:3B:B9:0D:10:21:07:43:8E:FA:00:41:EE:EC:75:CD:C3 for Cert: /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.onlinetrade.ru
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: 53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB for Cert: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: 8D:8C:5E:C4:54:AD:8A:E1:77:E9:9B:F9:9B:05:E1:B8:01:8D:61:E1 for Cert: /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A for Cert: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Error Bitmap is - 00000000 00000000 0
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Generating long serial-number for expired certificate.
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Certificate sha256 fingerprint is - 0x96E31336990C7E64FF723774A78FF5BC2722977836583CC4A4A84BDC67990650
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : New serial computed is 0x96E31336990C7E64FF723774A78FF5BC2722977836583CC4A4A84BDC6799065058304C12
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : SSLVersionCallback: Invalid SSL version 0
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : New Session negotiated with SSL client
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL

opryluts · ‎06-23-2020

Hi Anton,

Thanks for the logs.

Seems the issue is caused by cross-signed certs in the chain and recently expired Add-Trust certificate. More details are here https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020

Please check the version of the trusted cert bundle on the WSA (GUI -> Network -> Certificate Management) - version 1.7 should have the fix for the issue. If it is lower than 1.7 please try updating it.

Anton84 · ‎06-23-2020

Thanks for your answer, I see 1.7 version, but stange that status is "updates in progress"

opryluts · ‎06-23-2020

Hi Anton,

Please grep updater logs for the "trusted_root" keyword and check if any error messages displayed.

Also you can try restarting updater service by CLI -> diagnostic -> services -> updater -> restart command.

Anton84 · ‎06-23-2020

when i click to update, i see in logs:

Tue Jun 23 16:12:50 2020 Info: Starting scheduled update
Tue Jun 23 16:12:51 2020 Info: Scheduled next update to occur at Tue Jun 23 16:17:51 2020
Tue Jun 23 16:15:45 2020 Info: Scheduled next wbrs update to occur at Tue Jun 23 16:20:45 2020
Tue Jun 23 16:17:51 2020 Info: Starting scheduled update
Tue Jun 23 16:17:56 2020 Info: Scheduled next update to occur at Tue Jun 23 16:22:56 2020
Tue Jun 23 16:20:45 2020 Info: Scheduled next wbrs update to occur at Tue Jun 23 16:25:45 2020
Tue Jun 23 16:22:56 2020 Info: Starting scheduled update
Tue Jun 23 16:22:58 2020 Info: Scheduled next update to occur at Tue Jun 23 16:27:58 2020
Tue Jun 23 16:23:14 2020 Info: Received remote command to signal a manual update
Tue Jun 23 16:23:14 2020 Info: Starting manual update
Tue Jun 23 16:23:15 2020 Info: Scheduled next update to occur at Tue Jun 23 16:28:15 2020
Tue Jun 23 16:25:45 2020 Info: Scheduled next wbrs update to occur at Tue Jun 23 16:30:45 2020
Tue Jun 23 16:28:15 2020 Info: Starting scheduled update
Tue Jun 23 16:28:16 2020 Info: Scheduled next update to occur at Tue Jun 23 16:33:16 2020

If i grep "trusted_root" in log, i see old logs:

Mon Jun 1 23:02:27 2020 Info: trusted_root applying file "trusted_root/1.0.0/trustedca.pem/default/1583495981"
Mon Jun 1 23:02:44 2020 Info: trusted_root verifying applied files
Mon Jun 1 23:02:44 2020 Info: trusted_root updating the client manifest
Mon Jun 1 23:02:44 2020 Info: trusted_root update completed
Mon Jun 1 23:02:44 2020 Info: trusted_root waiting for new updates
Thu Jun 4 02:46:41 2020 Info: trusted_root updater shutdown complete
Thu Jun 4 02:46:41 2020 Info: trusted_root waiting for new updates
Fri Jun 5 12:32:52 2020 Info: trusted_root updater shutdown complete
Fri Jun 5 12:32:52 2020 Info: trusted_root waiting for new updates
Fri Jun 5 16:53:10 2020 Info: trusted_root updater shutdown complete
Fri Jun 5 16:53:10 2020 Info: trusted_root waiting for new updates
Wed Jun 10 17:38:19 2020 Info: trusted_root updater shutdown complete
Wed Jun 10 17:38:19 2020 Info: trusted_root waiting for new updates

by CLI - > diagnostic I dont have any services:

diagnostic

Choose the operation you want to perform:
- NET - Network Diagnostic Utility.
- PROXY - Proxy Debugging Utility.
- REPORTING - Reporting Utilities.

opryluts · ‎06-23-2020

Thanks Anton,

Those lines indicate that the trust cert bundle has been downloaded:

Mon Jun 1 23:02:27 2020 Info: trusted_root applying file "trusted_root/1.0.0/trustedca.pem/default/1583495981"
Mon Jun 1 23:02:44 2020 Info: trusted_root verifying applied files
Mon Jun 1 23:02:44 2020 Info: trusted_root updating the client manifest
Mon Jun 1 23:02:44 2020 Info: trusted_root update completed
Mon Jun 1 23:02:44 2020 Info: trusted_root waiting for new updates

Might be it was not properly applied to the proxy services since still GUI reports that update is in progress. You can try rebooting the box out of production hours and see if that helps.

Anton84 · ‎06-28-2020

Unfortunately after reboot i look the same picture.

Anton84 · ‎06-28-2020

maybe blocked certs can prevent to open sites

opryluts · ‎06-29-2020

Hi Anton,

Thank you for the update.

I don't see any cert from the chain in the block list. Let's start from the beginning - what AsyncOS version is your WSA running?

Anton84 · ‎06-30-2020

Version: 11.5.1-124

opryluts · ‎06-30-2020

Thank you, that version supports cross-signed certificates.

Could you please describe how does the issue look from users perspective? What do they see in the browser window when accessing those sites? Any error messages or a blank screen?

Also what do you see in access logs for those failing transactions?

Anton84 · ‎07-02-2020

On all sites I see the same java-scrypt

opryluts · ‎07-09-2020

Hi Anton,

Thanks for the inputs.

Seems I see the same behaviour in my lab. I'll dig into the issue...