Jabber Invalid Certificate

[seangolyer]

I have an issue with a customer's Jabber deployment having ongoing invalid certificate messages. I'm fairly positive it's related to the two domain names they have and some mismatched configurations, but I'm lost on where to start. Here are a bunch of configurations I've poured over to try and piece together what's happening. All customer names have been replaced with "example".

example.local = Internal server domain. This is the domain in which all servers reside (including IM&P and CUCM).
example.com = Presence domain/the domain users want to login with.

CUCM Config:

• All nodes have been changed to FQDN's (IE: imp-sub.example.local)
• Certificate Exchange between CUCM and IM&P has occurred
• SIP Publish Trunk is active
• Enterprise Parameters: Top Level Domain = example.com // Cluster FQDN's include *.example.com *.example.local *.example
• UC Service Profiles have UC Services with correct server FQDN's

IM&P Config:

• Presence Domain is example.com
• "tomcat" and "cup-xmpp" certs have been signed by their internal CA. Certs' CN are their FQDNs ( CN=imp-sub.example.local )
• "cup-xmpp" cert has a SAN of the Presence domain ( example.com )

Users' local Trusted Root stores contain their Root and Intermediate CA's (and the chain, for good measure) that signed all the server certs. Users' workstation are in the "example.local" domain. User logins for Jabber use the "example.com" domain.

I've checked Jabber logs and I've noticed some discovery issues in the environment which are easy enough to fix, but I'm not sure if they have any bearing on the Invalid Certficate issue. Their auto-discovery SRV record is in the "example.com" domain, however I notice Jabber attempts to search the "example.local" domain first. I'm not sure why this is, could be cached somehow or was incorrectly put in the bootstrap, or maybe Jabber looks at the local machine's domain first. I haven't found a lot of solid documentation on which Service Domains it will try first (or I've found documentation to be inaccurate). It does eventually search the "example.com" domain (based on the domain used in the user login ID) after a long period of failure with the "example.local" domain and returns the correct servers.

Afterwards, however, it I get the Certificate Invalid messages. It's attempting to verify literally "example.com" and cannot. I'm partially confused as to why (example.com is in the SAN of the XMPP cert). But I'm also partially confused why it's not using "example.local", though that's more of a question of "what domain does Jabber determine it wants to authenticate with" in this type of situation. Is it just trying to authenticate based off the Default Presence Domain? And if so, why doesn't the SAN work (which is auto-populated in the CSR)?

Not even sure I'm on the right track here, wanted to come here and hopefully find some direction on how to resolve. Uploaded the edited Jabber log.

[Mark Swanson]

You should replace the self-signed certs with the CA signed certs. If you SSH into your server and run the following cmd; show cert list trust... do you see duplicate tomcat, cup-xmpp and cup-xmpp-trust certs? If so, delete the old certs. Most likely, you're prompted to accept these certs because the validity date hasn't expired yet.

If you were using self-signed certs, which you clearly stated you don't. Then Cisco Jabber requires the user to accept the tomcat cert for whatever server they're assigned to. If you submitted all of the tomcat certs for CA signed certs, then you need to upload the CA signed tomcat certs as Tomcat-Trust certs on CUCM and restart Cisco Tomcat on CUCM. Call Manager should distribute the Tomcat-Trust certs throughout the cluster (i.e. Call Managers, IM/Presence, Unity Connection, etc.).

Again, if you were using self-signed certs, these self-signed certs would be automatically uploaded to the Enterprise Trust store; certmgr.msc. But since you have CA signed certs, these type of certs should be automatically uploaded to the Trusted Root store. If not, then you can manually upload them to the Trusted Root store... and make sure you delete all of the old certs.

[seangolyer]

Noted, thank you. There was never much emphasis placed on deleting old certs and my past experience with them on CUCM has just been for tomcat to stop getting cert warnings on the splash page. Having old self-signed with CA-signed wasn't typically an issue for that (though I tended to delete them anyway out of habit).

I will delete the old duplicates (and reset services I would assume?) and attempt again.

[Mark Swanson]

I believe, if you have valid self-signed tomcat certs, then Cisco Jabber attempts to use them and prompts users to accept these certs to the Enterprise Trust store. Once you upload all of the CA signed tomcat certs to CUCM as Tomcat-Trust certs... it should auto-populate these trusted tomcat certs throughout the domain. You might need to restart the Cisco Tomcat service to help this process along. Obviously, you need to install the CA root certs as well... which it looks like you already did. Just make sure the CA root cert is the correct cert, as you stated, they recently changed something.

[Mark Swanson]

Take a look at this Cisco doc;

http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-presence/116917-technote-certificate-00.html

It talks about self signed certs vs CA signed certs, and where these certs reside. This doc basically states;

Import root certificates into the MS Windows certificate store if:

• The certificates are signed by a CA that does not already exist in the trust store, such as a private CA. If so, you must import the private CA certificate to the Trusted Root Certification Authorities store.
• The certificates are self-signed. If so, you must import self-signed certificates to the Enterprise Trust store.

[seangolyer]

This was already addressed in my OP.

"Users' local Trusted Root stores contain their Root and Intermediate CA's (and the chain, for good measure) that signed all the server certs."


They are using a private CA, but that root/chain CA cert is already in all users' Trusted Root Certificate stores. Additionally, my "cup", "cup-xmpp", and "tomcat" certs are already signed by this very same CA.

[afmmanicke]

I came across a somewhat similar issue on an install. *I Think*

The cluster was signed with an internally validated Multi Server certificate per Cisco documentation - http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/118731-configure-san-00.html

After running through the steps, including restarting Cisco Tomcat, my web browser received the correct certificate and I am not prompted to accept an untrusted cert, because I've imported the Trusted Root from the internal CA to Trusted Root Certification Authorities. 

Jabber on the other hand prompted me to accept certificates from the UCM servers in the cluster.  After reviewing these certificates I determined that they were the self signed certs from the original installation based on their "issued by" information.

This resulted in a TAC case for me where I found out about this: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy13916/?reffering_site=dumpcr

Basically stopping, waiting, and starting the TFTP service via the service activation page was required - simply restarting the service was not good enough.

[seangolyer]

Nice find.

As an update to this issue, I was able to resolve 4 out of 5 certificate warnings in the environment by correcting some things they glossed over on their initial deployment. Primarily: not deploying a "jabber-config.xml" with declared FQDN's of various services, fixing their auto-discovery SRV records, and deleting old, self-signed certificates. I think I ran into a similar issue that you describe where we were getting certificate warnings from self-signed certificates that no longer existed on the TFTP nodes in the cluster. A reboot of the TFTP servers resolved the issue.

The only outstanding issue is a certificate warning for a CA-signed certificate, but the request is either being initiated or responded to/from an IP Address rather than a FQDN. I have advised the client to open a TAC case on the issue as I've scoured all known places where FQDN's should be declared and have turned up empty.

If I had a screenshot I would share, but the message is effectively: "An invalid certificate <cert name> when connecting to <IP Address> has been rejected." IP Addresses shouldn't be turning up when connecting to either IM&P or CUCM services, but they are. There must be something in the config of one of the clusters that is causing this but I have so far been able to find the root cause.

[Mark Swanson]

Check the following;

Login to IM&Presence Admin page. Under System, select Presence Topology.

Do the nodes display an IP Address, Hostname or FQDN... they should display FQDN.

*If you need to fix this, you can do so via CUCM Admin page > System > Server but I highly recommend you search for Cisco documentation explaining how to do this. It's not extremely difficult but you need to reboot services/servers and regenerate certificates. It becomes slightly more difficult when you introduce Persistent Chat, MFT and Intercluster Peers into the picture.

IF you have Intercluster Peers, then make sure ALL of the connections and certs are valid before generating the CSR. The Intercluster Peers are located under Presence > Inter-Clustering. IF you have any, go through all of them and make sure the Certificate Status displays "Connection is Secure". If needed, you can perform a 'Force Manual Sync' and on the next page, check the box "Also resync peer's Tomcat Certs" and click OK.

If that doesn't work, then you can always compare the contents of the CSR vs the self-signed cert via CSR Decorder or OpenSSL. If they look different, then that's most likely your problem. However, if you performed the resync process for the Intercluster Peers and everything looks OK... then you really don't need to do this unless you wanna play Dick Tracy.

[seangolyer]

No intercluster peers. Been there, done that with the server list. I've replaced IPs with FQDNs in nearly every instance they occur in the environment. No self-signed certs exist for Tomcat on any node in the entire cluster. Ditto for CUP, CUP-XMPP, and CallManager certs, all are CA-signed by a trusted internal CA.

There's nothing wrong with the cert itself that is being presented to users. It's either how the cert is being requested or returned, since it's coming from an attempted connection to an IP rather than FQDN. Since the IP isn't (and can't) be the Subject or SAN, it's requesting users to manually accept the cert. So, something weird is going on with a particular connection between the Jabber clients and CUCM, just haven't been able to find where the client is getting an IP rather than FQDN for this connection. It's likely specific to this customer's environment, have not been able to recreate in my own labs.

[Mark Swanson]

By default, the domain is added as a SAN when you generate the CSR. By any chance, did you delete/edit the SANs before you generated the CSR?

There's one more thing you can verify...

Under the Cisco Jabber login prompt, click on the "Advanced Settings" link.

What do you see here? Account Type is what? Login Server is what?

My guess is, DNS is not performing the SRV lookup. Check out this doc (a little out dated but still valid);

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/jabber/Windows/9_7/CJAB_BK_C606D8A9_00_cisco-jabber-dns-configuration-guide/CJAB_BK_C606D8A9_00_cisco-jabber-dns-configuration-guide_chapter_010.html

If your clients are trying to connect to 192.168.1.100 but IM&P and your certs are using FQDN, then you need to ensure there's a DNS SRV or you can manually configure your 'Advanced Settings' as;

Select your account type: Cisco IM & Presence

Login server: Use the following server [Server address: myIMPserver.domain.com]

Manually changing the 'Advanced Settings' is obviously not acceptable or desirable, but it would confirm if this was the problem or not.