Options for certifcates for REST server

[Jonathan King]

I have a UCCX script which places REST calls to an HTTPS server. The certificate on the HTTPS server changes frequently. I am wondering what my options are, other than manually uploading the new Tomcat-Trust certificate each time it changes.

1) Is it possible to push new certificates to the Tomcat-Trust store via the REST API or some other way, automatically?

2) Is it possible to disable certificate checking for an HTTPS REST call? I realize that defeats half the purpose of using HTTPS to begin with. But half is still better than nothing. Per the documentation for the Make Rest Call step, setting the cc_accept_all_hosts session variable will override the hostname check, but still requires the certificate be in the trust store.

3) Any way to get UCCX to only need the Intermediate / Root certs for an HTTPS server, and not the endpoint cert itself?

4) Any other ideas I haven't thought of?

[Roger Kallberg]

Isn’t it enough to have the certificate(s) of the CA that signed the certificate used by the REST api call recipient? Or is it using self signed certificate?

[Jonathan King]

Hi Roger,

Unfortunately not. Per TAC: "UCCX requires the entire certificate chain (including the endpoint certificate) to be uploaded to the tomcat-trust keystore because the Tomcat keystore does not contain root certificates by default. Uploading only the intermediate or root certificates without the endpoint certificate is insufficient for trust validation in HTTPS REST calls."

[Nithin Eluvathingal]

I have a working script on UCCX that makes a REST call to a cloud-based SMS gateway for sending messages via an HTTPS connection. I haven’t uploaded the server certificates and only the CA certificate who signed it got uploaded. Is it mandatory to upload the server certificates to UCCX for the connection to work? AFAIK its not, recheck with TAC.

[Jonathan King]

I'm not sure how you got yours working! Mine errors until the endpoint cert is uploaded.

[Jonathan Schulenberg]

My experience matches @Nithin Eluvathingal here. I have never needed to upload the individual server certificate, only the issuing CA chain.

What version of CCX?

And you’re certain the correct root & intermediate CA certs are uploaded?

Can TAC back that statement up with unambiguous product documentation? Because that sure sounds like nonsense to avoid finding root cause and a defect to me. It is not how certificate validation has worked on any Tomcat-trust store of any Cisco collab app on VOS (e.g., CUCM, IM&P, CUC, CCX).

[Elliot Dierksen · Accepted solution]

I will second @Jonathan Schulenberg on this. If you have the Root CA and any applicable intermediate CA certificates loaded as tomcat-trust, that has worked for me. As Jonathan points out, it would break the whole premise of certificates and CA's if it didn't.

[Roger Kallberg · Accepted solution]

I'm with both @Jonathan Schulenberg and @Elliot Dierksen on this. We do REST calls from CCX to ServiceNow and we have from what I can recall and also see in the Tomcat-trust store not had to upload any server certificates, all we have is a bunch of intermediate or root CA certificates. As Jonathan wrote I'd say go back to TAC and request them to show you the factual evidence to the claim that you'll need to have the server cert in this trust store.

[Jonathan King]

TAC is pointing to documentation saying the "entire chain" must be uploaded, which I agree defeats part of the purpose of PKI to begin with.

When I attempt the REST call with only the intermediate/root certs trusted, I get the following response: "java.io.IOException: HTTPS hostname wrong: should be <hostname redacted>"

The hostname it mentions is the same as the server URL I am pointing to. The REST server is in Azure. The server URL I am pointing to is a CNAME in front of many A records.

I attempted to set the session behavior to skip hostname validation. Not quite sure if I did it correctly. How I have it today seems to make no difference to the behavior.

Adding the endpoint cert does resolve the issue - even without restarting the cluster, as the system indicates should be done.

UCCX version 12.5.1.11003-511 

[Jonathan Schulenberg · Accepted solution]

---

@Jonathan King wrote:

Adding the endpoint cert does resolve the issue - even without restarting the cluster, as the system indicates should be done.

---

Please actually restart the CCX Engine so we're not chasing our tail here. Also be sure you upload to the publisher and that node is active for testing to keep this scope as narrow as possible. (That doesn't rule out the mess of how certificates can get out of sync between Informix and the file system though.)

I agree that it's worth confirming the cert chain in the PCAP matches what you uploaded though. Good idea @Igor-Lukic. TAC should also be able to see certificate detail in the Tomcat security logs IIRC.

Which document is TAC citing? If it's Configure UCCX Solution Certificate Management and this sentence specifically, it's pretty clear that the author meant all issuing certificate authorities - root & intermediate(s) - when read in context, not the server identity certificate when CA-signed. The wording is less than ideal though and allows for this sort of misunderstanding. It also should state "Tomcat-trust" instead of "Tomcat keystore"; those aren't the same thing.

The entire certificate chain must be uploaded to the platform Tomcat keystore, accessible via the OS Administration page, as the Tomcat keystore contains no root certificates by default.

---

That TAC engineer needs to pause for a moment and do some basic critical thinking about the implications of what they're claiming. Escalate to your Cisco account team if you hit a brick wall with the engineer and their manager. Anyone claiming this is expected behavior should be laughed out of the room. Certs are already capped at 13 month validity periods (90 days if using Let's Encrypt) and will drop to 47 days by 2029. It would be operationally impossible to support if CCX required a monthly maintenance window. Even now that would mean that you have to accept an outage and emergency change request every time whatever CCX is connecting to (e.g. any SaaS product that you can't control) renews their server cert! The business will never tolerate that; IT will be instructed to replace CCX with something else.

[Igor-Lukic · Accepted solution]

Hi @Jonathan King,

where did you get the certificate chain from?

Have you tried capturing the network traffic when making the REST request? I have come across an issue with publicly signed certificates, where automatically provided CA chain had incorrect certificates that were not part of the certificate chain, when you open the service certificate under Windows (for example). This became evident when comparing the certificate chain as seen in the capture with the provided certificate chain.

As @Jonathan Schulenberg , @Elliot Dierksen and @Roger Kallberg pointed out, you should not require the full chain.

Best regards

Igor

[Jonathan King]

Igor,

I downloaded the chain via web browser, just navigating to the API URL and grabbing the certificate from Firefox. I will do a packet capture and see what I can see.

[Igor Lukic]

Hi @Jonathan King ,

in case that you are using Microsoft Windows based client, then it can be the case that Windows downloads any missing intermediate CA certificates automatically (if the group policy allows it).

To be on the safe side, I open (double-click) on the actual service certificate, then click on the tab "Certification Path", then click on on the first intermediate certificate, followed by "View Certificate" and then note the serial number / fingerprint. I then repeat the process for any outstanding intermediate / root CA certifcates. I then use the collected data (serial numbers / fingerprints) to compare it with collected data from the capture.

If there is a mismatch, I then export the missing certificates using the above procedure by viewing the respective CA certificate and then clicking on the "Copy to File" button.

Best regards

Igor

[Jonathan King]

Update to this - per Igor's recommendation, I pulled the certs from a packet capture, as opposed to getting them from the browser, and sure enough, they were different (and missing). I uploaded the missing intermediate CA cert and that fixed the problem. 

So - oddly - when I did have the endpoint cert directly trusted, even though I was missing an intermediate cert, it worked. But, once I got the correct CA chain trusted, I no longer needed the endpoint cert, as many of you have correctly stated.

Thank you all for your input!