topic Re: WSA S190 SSL Configuration - can't open some web-resourses in Web Security

WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Mon, 22 Jun 2020 13:20:57 GMT

Hello all, we are using proxy S190, some sites can't open (example 220-volt.ru, onlinetrade.ru etc.)I think, SSL Configuration is not correctly. Can anybody help me with that problem?

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Mon, 22 Jun 2020 14:42:50 GMT

Hi Anton,

I'd start troubleshooting that by checking:

1. Access Logs for transactions towards the failing servers.

2. Proxy error logs/HTTPS logs for any error messages related to the destination. Enable debug log level temporary to see more details for the transactions.

If nothing has been found, I'd collect packet captures on WSA (both legs - client to WSA and WSA to the server at the same time). I'd be looking at TLS handshakes in captures:

1. if any alerts/errors reported

2. what are protocol versions and cipher suits and if they match from both sides

3. any certificate related issues

Also to better understand the problem I'd like to know:

1. WSA version

2. Is HTTPS proxy enabled? Is HTTPS decryption enabled for those specific destinations?

3. How does the issue look from the user perspective? Any error messages displayed in the browser?

Re: WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Tue, 23 Jun 2020 06:32:52 GMT

I see in https -log that, when trying to open onlinetrade.ru:

Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: 56:B5:6A:93:3B:B9:0D:10:21:07:43:8E:FA:00:41:EE:EC:75:CD:C3 for Cert: /OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.onlinetrade.ru
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: 53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB for Cert: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: 8D:8C:5E:C4:54:AD:8A:E1:77:E9:9B:F9:9B:05:E1:B8:01:8D:61:E1 for Cert: /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Subject Key Identifier: AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A for Cert: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Error Bitmap is - 00000000 00000000 0
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Generating long serial-number for expired certificate.
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Certificate sha256 fingerprint is - 0x96E31336990C7E64FF723774A78FF5BC2722977836583CC4A4A84BDC67990650
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : New serial computed is 0x96E31336990C7E64FF723774A78FF5BC2722977836583CC4A4A84BDC6799065058304C12
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : SSLVersionCallback: Invalid SSL version 0
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : New Session negotiated with SSL client
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL
Tue Jun 23 10:06:20 2020 Debug: HTTPS : - : Verify Cert Callback error - code = 3 : unable to get certificate CRL

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Tue, 23 Jun 2020 07:44:54 GMT

Hi Anton,

Thanks for the logs.

Seems the issue is caused by cross-signed certs in the chain and recently expired Add-Trust certificate. More details are here https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020

Please check the version of the trusted cert bundle on the WSA (GUI -> Network -> Certificate Management) - version 1.7 should have the fix for the issue. If it is lower than 1.7 please try updating it.

Re: WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Tue, 23 Jun 2020 11:56:22 GMT

Thanks for your answer, I see 1.7 version, but stange that status is "updates in progress"

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Tue, 23 Jun 2020 12:09:34 GMT

Hi Anton,

Please grep updater logs for the "trusted_root" keyword and check if any error messages displayed.

Also you can try restarting updater service by CLI -> diagnostic -> services -> updater -> restart command.

Re: WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Tue, 23 Jun 2020 12:59:21 GMT

when i click to update, i see in logs:

Tue Jun 23 16:12:50 2020 Info: Starting scheduled update
Tue Jun 23 16:12:51 2020 Info: Scheduled next update to occur at Tue Jun 23 16:17:51 2020
Tue Jun 23 16:15:45 2020 Info: Scheduled next wbrs update to occur at Tue Jun 23 16:20:45 2020
Tue Jun 23 16:17:51 2020 Info: Starting scheduled update
Tue Jun 23 16:17:56 2020 Info: Scheduled next update to occur at Tue Jun 23 16:22:56 2020
Tue Jun 23 16:20:45 2020 Info: Scheduled next wbrs update to occur at Tue Jun 23 16:25:45 2020
Tue Jun 23 16:22:56 2020 Info: Starting scheduled update
Tue Jun 23 16:22:58 2020 Info: Scheduled next update to occur at Tue Jun 23 16:27:58 2020
Tue Jun 23 16:23:14 2020 Info: Received remote command to signal a manual update
Tue Jun 23 16:23:14 2020 Info: Starting manual update
Tue Jun 23 16:23:15 2020 Info: Scheduled next update to occur at Tue Jun 23 16:28:15 2020
Tue Jun 23 16:25:45 2020 Info: Scheduled next wbrs update to occur at Tue Jun 23 16:30:45 2020
Tue Jun 23 16:28:15 2020 Info: Starting scheduled update
Tue Jun 23 16:28:16 2020 Info: Scheduled next update to occur at Tue Jun 23 16:33:16 2020

If i grep "trusted_root" in log, i see old logs:

Mon Jun 1 23:02:27 2020 Info: trusted_root applying file "trusted_root/1.0.0/trustedca.pem/default/1583495981"
Mon Jun 1 23:02:44 2020 Info: trusted_root verifying applied files
Mon Jun 1 23:02:44 2020 Info: trusted_root updating the client manifest
Mon Jun 1 23:02:44 2020 Info: trusted_root update completed
Mon Jun 1 23:02:44 2020 Info: trusted_root waiting for new updates
Thu Jun 4 02:46:41 2020 Info: trusted_root updater shutdown complete
Thu Jun 4 02:46:41 2020 Info: trusted_root waiting for new updates
Fri Jun 5 12:32:52 2020 Info: trusted_root updater shutdown complete
Fri Jun 5 12:32:52 2020 Info: trusted_root waiting for new updates
Fri Jun 5 16:53:10 2020 Info: trusted_root updater shutdown complete
Fri Jun 5 16:53:10 2020 Info: trusted_root waiting for new updates
Wed Jun 10 17:38:19 2020 Info: trusted_root updater shutdown complete
Wed Jun 10 17:38:19 2020 Info: trusted_root waiting for new updates

by CLI - > diagnostic I dont have any services:

diagnostic

Choose the operation you want to perform:
- NET - Network Diagnostic Utility.
- PROXY - Proxy Debugging Utility.
- REPORTING - Reporting Utilities.

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Tue, 23 Jun 2020 13:26:22 GMT

Thanks Anton,

Those lines indicate that the trust cert bundle has been downloaded:

Mon Jun 1 23:02:27 2020 Info: trusted_root applying file "trusted_root/1.0.0/trustedca.pem/default/1583495981"
Mon Jun 1 23:02:44 2020 Info: trusted_root verifying applied files
Mon Jun 1 23:02:44 2020 Info: trusted_root updating the client manifest
Mon Jun 1 23:02:44 2020 Info: trusted_root update completed
Mon Jun 1 23:02:44 2020 Info: trusted_root waiting for new updates

Might be it was not properly applied to the proxy services since still GUI reports that update is in progress. You can try rebooting the box out of production hours and see if that helps.

Re: WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Mon, 29 Jun 2020 05:17:58 GMT

Unfortunately after reboot i look the same picture.

Re: WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Mon, 29 Jun 2020 08:30:18 GMT

maybe blocked certs can prevent to open sites

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Mon, 29 Jun 2020 09:03:37 GMT

Hi Anton,

Thank you for the update.

I don't see any cert from the chain in the block list. Let's start from the beginning - what AsyncOS version is your WSA running?

Re: WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Tue, 30 Jun 2020 08:10:32 GMT

Version: 11.5.1-124

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Tue, 30 Jun 2020 13:28:21 GMT

Thank you, that version supports cross-signed certificates.

Could you please describe how does the issue look from users perspective? What do they see in the browser window when accessing those sites? Any error messages or a blank screen?

Also what do you see in access logs for those failing transactions?

Re: WSA S190 SSL Configuration - can't open some web-resourses

Anton84 — Thu, 02 Jul 2020 13:55:20 GMT

On all sites I see the same java-scrypt

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Thu, 09 Jul 2020 08:11:05 GMT

Hi Anton,

Thanks for the inputs.

Seems I see the same behaviour in my lab. I'll dig into the issue...

Re: WSA S190 SSL Configuration - can't open some web-resourses

opryluts — Thu, 09 Jul 2020 09:05:27 GMT

Hi Anton,

I think I see the problem...

All those websites use a protection service https://variti.com/ru-ru/

And for some reason, it blocks or tries to verify those transactions. Not sure why but to me it looks like a misbehaving from their side. I tested those 2 sites from different locations, directly and via WSA and most of times I'm getting responses with "Server: Variti/0.9.3a" HTTP header and JS in the payload.

Sometimes I was able to load the page. In those cases, Server header was set to Nginx and payload was in HTML.

I'd advise contacting Variti/Website owners to see why their protection service blocks those transactions.