Secure Web Gateway - 516 Upstream Certificate CN Mismatch (Umbrella)

[kbull]

The certificate errors related to 516 Upstream Certificate CN Mismatch are becoming very problematic for our company. Marketing emails are particularly problematic, and this is disruptive to our staff when trying to sign up for webinars and other "normal" activities. 

I can't believe Cisco Umbrella is okay with such a disruptive and non-productive workflow. Users don't even get the normal block screen allowing them to request access to the blocked page. 

Are there any plans to give administrators the control to allow users to bypass these warnings or a more user-friendly way of dealing with this issue? 

[Rob Ingram]

@kbull it sounds like you using SSL decryption with Umbrella, in which case you must import the Umbrella root certificate to your computers, so they trust the Umbrella certificate.

https://docs.umbrella.com/deployment-umbrella/docs/enable-ssl-decryption

[Ken Stieers]

This is nor the problem he's having.

What happens is that the URL is for company.com, but the server serving it is run by the bulk mailer, with their cert on it (e.g. sendgrid.com) So there is a cert/url mismatch.

Happens with a bunch of different services, like Sendgrid, Mailchimp, etc.

The company sending via needs to fix it... documented fixes exist.

As a workaround you can look at not decrypting for companies you have this issue with.

[Rob Ingram]

thanks, good to know.

[kbull]

Ken, you are correct that documented fixes exist, but getting companies to fix this is not easy or even feasible in many cases. For example, Home Depot's emails with order tracking links were breaking with this 516 Upstream error.
The other challenge is that there is no easy reporting for end users like a normal blocked page warning. Our staff aren't telling us about blocked pages until they get really frustrated or a block prevents them from doing their job, which is certainly not a good end-user experience. This will also start pushing staff to use personal devices over corporate devices, which, again, is not ideal. 

[IT-LARL-2024]

This is funny, I just found that Cisco themselves need to fix their Umbrella test site http://exampleadultsite.com/ .  It isn't possible to test bypass of that site in Chrome since there is no TLS certificate setup for it specifically.  I guess that shows how wide spread this issue is. 

[howe]

The test site you mentioned: http://www[.]exampleadultsite[.]com/ is http not https; I'm surprised you are seeing a certificate miss-match error being served from Umbrella.  Can you confirm the exact error you are seeing?

I also note that this test location is for DNS testing only:

"The following test pages apply to Umbrella DNS coverage and may not apply to users with active SIG coverage."

https://support.umbrella.com/hc/en-us/articles/115000411528-What-are-the-Umbrella-Test-Destinations

However I do note that none of the common names in the certificate match the name that was used www[.]exampleadultsite[.]com, so proxying this test location over https will likely throw an error in some cases.

[IT-LARL-2024]

Hello, here is a step by step example of what I'm seeing... using incognito mode in chrome.

1. Using Umbrella dns based protection.
2. Visit http://exampleadultsite.com
3. In incognito mode... I'm seeing a redirect to https at this point.
4. 
5. Get Umbrella block page. (https://bpb.opendns.com/b/https/exampleadultsite.com/) (notice changed to https)
6. Bypass using account or code.
7. Get bypass confirmation page.
8. Click "Continue browsing at exampleadultsite.com, which is proxied to be able to bypass the block.  URL starts with https://exampleadultsite.com/_bpb/1/[lots of extra args]
9. Umbrella servers see the mismatch and return "516 Upstream Certificate CN Mismatch" error page
10. 

I then tried it in a new chrome profile... and it worked fine.  No https redirect for some reason???

Maybe incognito mode is more enthusiastic about forcing to https?  Maybe http links are treated differently than manually typed in URLs?  Maybe I'm hitting chrome caching the redirects which I remember being a PITA in some situations.

In any case, it seems like Umbrella's test site should be setup to work in https mode.

[IT-LARL-2024]

Ah, found this note about google future plans for chrome HTTPS first settings from a year ago.

Currently, HTTPS-First mode is enabled for users who are logged into their accounts and have agreed to participate in the Google Advanced Protection program. In future Chrome releases, HTTPS-First will be enabled by default for pages opened in incognito mode. Experiments are also being conducted to automatically enable HTTPS-First for sites known to support HTTPS and for users who rarely use HTTP in their browser.

My first test was from my main chrome account, and I am opted into Advanced protection....

Next test was incognito.

Last was a generic profile.

Needing to use http sites is going to be a pain to deal with.

[IT-LARL-2024]

One last note, I opened a ticket about this yesterday and just received this, so hopefully the next time the certs get issued they will have the test site SANs.

Thanks for bringing this to our attention. After verifying the certificate errors, it turns out the engineering team is aware of this issue with test sites and they should update the certificates for them soon.

[howe]

"In any case, it seems like Umbrella's test site should be setup to work in https mode."

Oh yes, I do agree- the cert is broken for this site, CN mismatch. Either the webserver shouldn't respond on https or the cert should be fixed. 

Whats interesting here is the browser behaviour which appears to be enforcing https. Which is a good thing really, for most users, most of the time. Looks liek I need to do some research...:-) 

[kbull]

We are using SSL decryption, and we have the Umbrella root certificate deployed otherwise, every site would break. The issue I am speaking of is documented here: Error 516 Upstream Certificate CN Mismatch – Cisco Umbrella