topic JID should be "domain.com". in Collaboration Applications

Jabber Invalid Certificate

seangolyer — Mon, 18 Mar 2019 01:23:36 GMT

I have an issue with a customer's Jabber deployment having ongoing invalid certificate messages. I'm fairly positive it's related to the two domain names they have and some mismatched configurations, but I'm lost on where to start. Here are a bunch of configurations I've poured over to try and piece together what's happening. All customer names have been replaced with "example".

example.local = Internal server domain. This is the domain in which all servers reside (including IM&P and CUCM).
example.com = Presence domain/the domain users want to login with.

CUCM Config:

All nodes have been changed to FQDN's (IE: imp-sub.example.local)
Certificate Exchange between CUCM and IM&P has occurred
SIP Publish Trunk is active
Enterprise Parameters: Top Level Domain = example.com // Cluster FQDN's include *.example.com *.example.local *.example
UC Service Profiles have UC Services with correct server FQDN's

IM&P Config:

Presence Domain is example.com
"tomcat" and "cup-xmpp" certs have been signed by their internal CA. Certs' CN are their FQDNs ( CN=imp-sub.example.local )
"cup-xmpp" cert has a SAN of the Presence domain ( example.com )

Users' local Trusted Root stores contain their Root and Intermediate CA's (and the chain, for good measure) that signed all the server certs. Users' workstation are in the "example.local" domain. User logins for Jabber use the "example.com" domain.

I've checked Jabber logs and I've noticed some discovery issues in the environment which are easy enough to fix, but I'm not sure if they have any bearing on the Invalid Certficate issue. Their auto-discovery SRV record is in the "example.com" domain, however I notice Jabber attempts to search the "example.local" domain first. I'm not sure why this is, could be cached somehow or was incorrectly put in the bootstrap, or maybe Jabber looks at the local machine's domain first. I haven't found a lot of solid documentation on which Service Domains it will try first (or I've found documentation to be inaccurate). It does eventually search the "example.com" domain (based on the domain used in the user login ID) after a long period of failure with the "example.local" domain and returns the correct servers.

Afterwards, however, it I get the Certificate Invalid messages. It's attempting to verify literally "example.com" and cannot. I'm partially confused as to why (example.com is in the SAN of the XMPP cert). But I'm also partially confused why it's not using "example.local", though that's more of a question of "what domain does Jabber determine it wants to authenticate with" in this type of situation. Is it just trying to authenticate based off the Default Presence Domain? And if so, why doesn't the SAN work (which is auto-populated in the CSR)?

Not even sure I'm on the right track here, wanted to come here and hopefully find some direction on how to resolve. Uploaded the edited Jabber log.

I don't see you mentioned

Jaime Valencia — Mon, 26 Sep 2016 21:51:27 GMT

I don't see you mentioned anything about changing the JID schema, so, right now what is the JID for the users???

@domain.local??

@domain.com??

Notice that if you have not installed the certificates in the user's machine, before trying to use Jabber, it's the expected behavior to be prompted to accept or decline the certificates. If that's all you're worried about, the certificates need to be distributed to the clients, before they attempt to login, to prevent that pop-up from showing up, the Jabber documentation explains this. If not, they will be prompted, and need to accept the cert themselves (just the first time)

JID should be "domain.com".

seangolyer — Mon, 26 Sep 2016 22:15:52 GMT

JID *should* be "domain.com". This is what they use to successfully login.

The issue is users are getting invalid certificate errors every time they login, not just the first time.

If you did not change the JID

Jaime Valencia — Mon, 26 Sep 2016 23:47:53 GMT

If you did not change the JID schema, the JID will be userID@default.domain, which according to what you've said. is domain.local, which is where your IM&P server is.

You should know from your config exactly how your JIDs look like.

EDIT: also, if you accept the certificate, do you see in the certificate management?? and if you reboot, is it still there??

I did not say my default

seangolyer — Tue, 27 Sep 2016 12:14:29 GMT

I did not say my default domain is "domain.local", I said it is "domain.com". I called that out in my first bullet under "IM&P Config" in my OP.

Other users are experiencing the issue, I have not been able to replicate it on my machine (though Invalid Certs are showing up in my "Error Notifications" under the "Help" menu in my client). I'm still gathering details and am attempting to replicate on my end and will report back with any findings.

Got additional clarification.

seangolyer — Tue, 27 Sep 2016 19:05:12 GMT

Got additional clarification. So, the certificate warning is only happening upon first login. They want this to go away. We already have the CA certs that signed all the server certs in the users' Trusted Root Certificate stores. What other certs do they need to have in there? The "cup" cert? "cup-xmpp"? "Call-Manager"? And if we do need one (or multiple) of those, do we put those in the Root, Intermediate, or Personal stores?

As a test, I removed my IMP cert from my local store to recreate the issue. I notice that the cert I get when I log back into Jabber and manually "Accept" matches the "cup-xmpp-trust" cert that was self-signed by the server and not one of the CA-signed certs... very confused now.

To eliminate the pop-ups, you

Mark Swanson — Tue, 27 Sep 2016 20:02:27 GMT

To eliminate the pop-ups, you need to obtain CA-signed certs for the following; Tomcat, CUP-XMPP and potentially, CUP-XMPP-S2S if you need to federate with external domains. If you're running in full UC mode; meaning, you want to control your phone through Cisco Jabber, then you need to obtain CA-singed certs for CUCM. Likewise with Unity Connection, if you want to integrate this service into Cisco Jabber.

When you accept self-signed certs, they should be automatically uploaded as Enterprise Trust Certificates... not as Trusted Root Certificates. Once you obtain CA-signed certs, then you can advertise these certs as Trusted Root Certificates.

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/jabber/10_6/planning-guide/CJAB_BK_CD3376A0_00_cisco-jabber-106-planning-guide/CJAB_BK_CD3376A0_00_cisco-jabber-planning-guide_chapter_0110.html

In addition to this, Cisco recommends using the FQDN as the server's name. If you're currently using the Hostnames or IP Addresses, then you might continue to experience cert related problems. The server name should match the certificate's CN value.

All node names are FQDN's, no

seangolyer — Tue, 27 Sep 2016 20:34:57 GMT

All node names are FQDN's, no problems there.

I added the Tomcat certs from all CUCM and IMP servers as well as the CUP-XMPP cert to my Trusted Root store. I reset Jabber and am still being prompted to accept a different self-signed cert from IMP (cup-xmpp-trust) that gets added to my Enterprise Trust store.

Actually, there is no need to

Jaime Valencia — Tue, 27 Sep 2016 21:09:01 GMT

Actually, there is no need to have CA signed certs, you can do this as well with self-signed, it's just easier to do with CA signed.

I suggest you do some self-study on certificates to understand how they work (they have become part of every deployment), you're placing them all in the wrong place, those are NOT trusted root certificates. A trusted root, is the guy that will be recognized as the authenticator for someone else, the guy that will vouch for them Verisign, GoDaddy, .etc. You should have seen all those in that folder. If you're using CA signed, you need the root and intermediate certificates there.

What you have, are server certificates, not root or intermediate certificates.

Mark already pointed out where they should be placed, and you should have found them in that same place.

When you accept self-signed certs, they should be automatically uploaded as Enterprise Trust Certificates... not as Trusted Root Certificates

You're prompted to accept them, because they're not in the right place.

Take a look at this Cisco doc

Mark Swanson — Tue, 27 Sep 2016 21:24:30 GMT

Take a look at this Cisco doc;

http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-presence/116917-technote-certificate-00.html

It talks about self signed certs vs CA signed certs, and where these certs reside. This doc basically states;

Import root certificates into the MS Windows certificate store if:

The certificates are signed by a CA that does not already exist in the trust store, such as a private CA. If so, you must import the private CA certificate to the Trusted Root Certification Authorities store.
The certificates are self-signed. If so, you must import self-signed certificates to the Enterprise Trust store.

I'm not sure anybody is

seangolyer — Tue, 27 Sep 2016 21:38:29 GMT

I'm not sure anybody is actually reading what I've sent. If my server certs are CA-signed (which they are), what store should they go in to stop "Accept" messages?

From Mark's last message he said: "Once you obtain CA-signed certs, then you can advertise these certs as Trusted Root Certificates." I'm not sure how else to interpret that other than to put the CA-signed server certs in the Trusted Root store.

All of my server certs (cup, cup-xmpp, and tomcat) are already CA-signed. I know that the Root CA certs that signed these go into the Root store. Those are there already. But where then do the CA-signed server certs go?

Where do you suggest I self-study? Because I've read multiple Jabber Deployment guides and CUCM/IM&P Security Guides. I've had no issues with Tomcat certs in the past, but when it comes to Jabber certs, there is often missing or disconnected information across multiple sources.

I've already read through these:

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/jabber/11_0/CJAB_BK_D657A25F_00_deployment-installation-guide-jabber-110/CJAB_BK_D657A25F_00_deployment-installation-guide-jabber-110_chapter_011.html
http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-presence/116917-technote-certificate-00.html
http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/admin/11_5_1_SU1/Administration/cucm_b_administration-guide-1151su1/cucm_b_administration-guide-1151su1_chapter_01110.html

Jabber Deployment guide only makes mention of the CA certs themselves (the certs from the CA that signed your server certs). No mention of placing server certs in user stores. The TechNote also makes no mention of what to do with server certs once they are CA-signed (and contains already outdated information on XMPP Domains). Admin guide is very generic about the various tasks to perform in regards to certs.

This was already addressed in

seangolyer — Tue, 27 Sep 2016 21:42:51 GMT

This was already addressed in my OP.

"Users' local Trusted Root stores contain their Root and Intermediate CA's (and the chain, for good measure) that signed all the server certs."

They are using a private CA, but that root/chain CA cert is already in all users' Trusted Root Certificate stores. Additionally, my "cup", "cup-xmpp", and "tomcat" certs are already signed by this very same CA.

Root cert for your CA,

Jaime Valencia — Tue, 27 Sep 2016 21:49:05 GMT

Root cert for your CA, whoever that is, goes to Trusted Root CA

Server certificates, go to Enterprise Trust.

However, something of what you're saying doesn't add up, CUCM and IM&P can only have one of two options, either they have self-signed certs, or they have CA certs for each kind of cert, you cannot have both for a specific certificate, lets says Tomcat for example.

I reset Jabber and am still being prompted to accept a different self-signed cert from IMP (cup-xmpp-trust) that gets added to my Enterprise Trust store.

You just said all of them are CA signed, so, something of what you've said, cannot be. Unless you have some of the required certs CA signed, and some self-signed.

Yes, that is the discrepancy

seangolyer — Tue, 27 Sep 2016 22:02:29 GMT

Yes, that is the discrepancy I've observed as well, which is why I'm here for help, haha.

My cup, cup-xmpp, and tomcat certs are all CA-signed. However, when I clear my local certs and reset my login, the Jabber client is grabbing a cert that matches a "cup-xmpp-trust" cert on the server. I'm completely lost as to why it's grabbing an old self-signed "cup-xmpp-trust" cert and not the "cup-xmpp" cert, and whether or not it's safe to remove it from the server (and restart Cisco XCP Router I assume).

The certs currently associated to "cup-xmpp-trust" are the old self-signed cert as well as the Root CA that signed the new "cup-xmpp" cert. Do I need to remove the self-signed cert from this "-trust" store in IM&P? Will it regenerate a new one after service(s) are restarted (I've noted this happens in my lab, but that cluster is not CA-signed).

I'd probably start by

Jaime Valencia — Tue, 27 Sep 2016 22:07:10 GMT

I'd probably start by something as simple as trying a reboot to make sure nothing is there, but I've never seen that once you upload CA certs, you still get the old self-signed certs. Can you post a screenshot of your certs?? And I mean from CUCM/IM&P OS admin pages.

You should replace the self

Mark Swanson — Tue, 27 Sep 2016 22:25:55 GMT

You should replace the self-signed certs with the CA signed certs. If you SSH into your server and run the following cmd; show cert list trust... do you see duplicate tomcat, cup-xmpp and cup-xmpp-trust certs? If so, delete the old certs. Most likely, you're prompted to accept these certs because the validity date hasn't expired yet.

If you were using self-signed certs, which you clearly stated you don't. Then Cisco Jabber requires the user to accept the tomcat cert for whatever server they're assigned to. If you submitted all of the tomcat certs for CA signed certs, then you need to upload the CA signed tomcat certs as Tomcat-Trust certs on CUCM and restart Cisco Tomcat on CUCM. Call Manager should distribute the Tomcat-Trust certs throughout the cluster (i.e. Call Managers, IM/Presence, Unity Connection, etc.).

Again, if you were using self-signed certs, these self-signed certs would be automatically uploaded to the Enterprise Trust store; certmgr.msc. But since you have CA signed certs, these type of certs should be automatically uploaded to the Trusted Root store. If not, then you can manually upload them to the Trusted Root store... and make sure you delete all of the old certs.

Absolutely.I would also like

seangolyer — Tue, 27 Sep 2016 22:27:57 GMT

Absolutely.

I would also like to note that most of these certs were here before I walked into this situation, I'm here to get things working and clean up! But if I need to clean-up before I can get things working... that's good information to know. They also recently changed/updated their internal CA, so that's why there's a few old Root/Sub CAs in there.

The cert with the green box around it in the "cup-xmpp" image is the cert that seems to be the one Jabber is grabbing.

Noted, thank you. There was

seangolyer — Tue, 27 Sep 2016 22:32:36 GMT

Noted, thank you. There was never much emphasis placed on deleting old certs and my past experience with them on CUCM has just been for tomcat to stop getting cert warnings on the splash page. Having old self-signed with CA-signed wasn't typically an issue for that (though I tended to delete them anyway out of habit).

I will delete the old duplicates (and reset services I would assume?) and attempt again.

I believe, if you have valid

Mark Swanson — Tue, 27 Sep 2016 22:42:17 GMT

I believe, if you have valid self-signed tomcat certs, then Cisco Jabber attempts to use them and prompts users to accept these certs to the Enterprise Trust store. Once you upload all of the CA signed tomcat certs to CUCM as Tomcat-Trust certs... it should auto-populate these trusted tomcat certs throughout the domain. You might need to restart the Cisco Tomcat service to help this process along. Obviously, you need to install the CA root certs as well... which it looks like you already did. Just make sure the CA root cert is the correct cert, as you stated, they recently changed something.

But that is a -trust

Jaime Valencia — Tue, 27 Sep 2016 22:51:53 GMT

But that is a -trust certificate, those are the equivalent of the trusted root CA store on windows, the certificate you should be presented with, is the one that doesn't have -trust, and you should only have one of each kind. You should be presented with the server certificate, not the trust certificate.

Once you have the cert that you accepted on your Jabber machine, you can open it, what does it say on Issued by:??

Does it have the CN of your CA?? and issued to: the FQDN??

If they're indeed self-signed, you should see the same value in issued to and issued by, the hostname or FQDN, depending on what you had upon installation.

topic JID *should* be "domain.com". in Collaboration Applications