Jabber Invalid Certificate

[seangolyer]

I have an issue with a customer's Jabber deployment having ongoing invalid certificate messages. I'm fairly positive it's related to the two domain names they have and some mismatched configurations, but I'm lost on where to start. Here are a bunch of configurations I've poured over to try and piece together what's happening. All customer names have been replaced with "example".

example.local = Internal server domain. This is the domain in which all servers reside (including IM&P and CUCM).
example.com = Presence domain/the domain users want to login with.

CUCM Config:

• All nodes have been changed to FQDN's (IE: imp-sub.example.local)
• Certificate Exchange between CUCM and IM&P has occurred
• SIP Publish Trunk is active
• Enterprise Parameters: Top Level Domain = example.com // Cluster FQDN's include *.example.com *.example.local *.example
• UC Service Profiles have UC Services with correct server FQDN's

IM&P Config:

• Presence Domain is example.com
• "tomcat" and "cup-xmpp" certs have been signed by their internal CA. Certs' CN are their FQDNs ( CN=imp-sub.example.local )
• "cup-xmpp" cert has a SAN of the Presence domain ( example.com )

Users' local Trusted Root stores contain their Root and Intermediate CA's (and the chain, for good measure) that signed all the server certs. Users' workstation are in the "example.local" domain. User logins for Jabber use the "example.com" domain.

I've checked Jabber logs and I've noticed some discovery issues in the environment which are easy enough to fix, but I'm not sure if they have any bearing on the Invalid Certficate issue. Their auto-discovery SRV record is in the "example.com" domain, however I notice Jabber attempts to search the "example.local" domain first. I'm not sure why this is, could be cached somehow or was incorrectly put in the bootstrap, or maybe Jabber looks at the local machine's domain first. I haven't found a lot of solid documentation on which Service Domains it will try first (or I've found documentation to be inaccurate). It does eventually search the "example.com" domain (based on the domain used in the user login ID) after a long period of failure with the "example.local" domain and returns the correct servers.

Afterwards, however, it I get the Certificate Invalid messages. It's attempting to verify literally "example.com" and cannot. I'm partially confused as to why (example.com is in the SAN of the XMPP cert). But I'm also partially confused why it's not using "example.local", though that's more of a question of "what domain does Jabber determine it wants to authenticate with" in this type of situation. Is it just trying to authenticate based off the Default Presence Domain? And if so, why doesn't the SAN work (which is auto-populated in the CSR)?

Not even sure I'm on the right track here, wanted to come here and hopefully find some direction on how to resolve. Uploaded the edited Jabber log.

[Jaime Valencia]

I don't see you mentioned anything about changing the JID schema, so, right now what is the JID for the users???

@domain.local??

@domain.com??

Notice that if you have not installed the certificates in the user's machine, before trying to use Jabber, it's the expected behavior to be prompted to accept or decline the certificates. If that's all you're worried about, the certificates need to be distributed to the clients, before they attempt to login, to prevent that pop-up from showing up, the Jabber documentation explains this. If not, they will be prompted, and need to accept the cert themselves (just the first time)

[seangolyer]

JID *should* be "domain.com". This is what they use to successfully login.

The issue is users are getting invalid certificate errors every time they login, not just the first time.

[Jaime Valencia]

If you did not change the JID schema, the JID will be userID@default.domain, which according to what you've said. is domain.local, which is where your IM&P server is.

You should know from your config exactly how your JIDs look like.

EDIT: also, if you accept the certificate, do you see in the certificate management?? and if you reboot, is it still there??

[seangolyer]

I did not say my default domain is "domain.local", I said it is "domain.com". I called that out in my first bullet under "IM&P Config" in my OP.

Other users are experiencing the issue, I have not been able to replicate it on my machine (though Invalid Certs are showing up in my "Error Notifications" under the "Help" menu in my client). I'm still gathering details and am attempting to replicate on my end and will report back with any findings.

[seangolyer]

Got additional clarification. So, the certificate warning is only happening upon first login. They want this to go away. We already have the CA certs that signed all the server certs in the users' Trusted Root Certificate stores. What other certs do they need to have in there? The "cup" cert? "cup-xmpp"? "Call-Manager"? And if we do need one (or multiple) of those, do we put those in the Root, Intermediate, or Personal stores?

As a test, I removed my IMP cert from my local store to recreate the issue. I notice that the cert I get when I log back into Jabber and manually "Accept" matches the "cup-xmpp-trust" cert that was self-signed by the server and not one of the CA-signed certs... very confused now.

[Mark Swanson]

To eliminate the pop-ups, you need to obtain CA-signed certs for the following; Tomcat, CUP-XMPP and potentially, CUP-XMPP-S2S if you need to federate with external domains. If you're running in full UC mode; meaning, you want to control your phone through Cisco Jabber, then you need to obtain CA-singed certs for CUCM. Likewise with Unity Connection, if you want to integrate this service into Cisco Jabber.

When you accept self-signed certs, they should be automatically uploaded as Enterprise Trust Certificates... not as Trusted Root Certificates. Once you obtain CA-signed certs, then you can advertise these certs as Trusted Root Certificates.

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/jabber/10_6/planning-guide/CJAB_BK_CD3376A0_00_cisco-jabber-106-planning-guide/CJAB_BK_CD3376A0_00_cisco-jabber-planning-guide_chapter_0110.html

In addition to this, Cisco recommends using the FQDN as the server's name. If you're currently using the Hostnames or IP Addresses, then you might continue to experience cert related problems. The server name should match the certificate's CN value.     

[seangolyer]

All node names are FQDN's, no problems there.

I added the Tomcat certs from all CUCM and IMP servers as well as the CUP-XMPP cert to my Trusted Root store. I reset Jabber and am still being prompted to accept a different self-signed cert from IMP (cup-xmpp-trust) that gets added to my Enterprise Trust store.

[Jaime Valencia]

Actually, there is no need to have CA signed certs, you can do this as well with self-signed, it's just easier to do with CA signed.

I suggest you do some self-study on certificates to understand how they work (they have become part of every deployment), you're placing them all in the wrong place, those are NOT trusted root certificates. A trusted root, is the guy that will be recognized as the authenticator for someone else, the guy that will vouch for them Verisign, GoDaddy, .etc. You should have seen all those in that folder. If you're using CA signed, you need the root and intermediate certificates there.

What you have, are server certificates, not root or intermediate certificates.

Mark already pointed out where they should be placed, and you should have found them in that same place.

When you accept self-signed certs, they should be automatically uploaded as Enterprise Trust Certificates... not as Trusted Root Certificates

You're prompted to accept them, because they're not in the right place.

[seangolyer]

I'm not sure anybody is actually reading what I've sent. If my server certs are CA-signed (which they are), what store should they go in to stop "Accept" messages?

From Mark's last message he said: "Once you obtain CA-signed certs, then you can advertise these certs as Trusted Root Certificates." I'm not sure how else to interpret that other than to put the CA-signed server certs in the Trusted Root store.

All of my server certs (cup, cup-xmpp, and tomcat) are already CA-signed. I know that the Root CA certs that signed these go into the Root store. Those are there already. But where then do the CA-signed server certs go?

Where do you suggest I self-study? Because I've read multiple Jabber Deployment guides and CUCM/IM&P Security Guides. I've had no issues with Tomcat certs in the past, but when it comes to Jabber certs, there is often missing or disconnected information across multiple sources.

I've already read through these:

• http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/jabber/11_0/CJAB_BK_D657A25F_00_deployment-installation-guide-jabber-110/CJAB_BK_D657A25F_00_deployment-installation-guide-jabber-110_chapter_011.html
• http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-presence/116917-technote-certificate-00.html
• http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/admin/11_5_1_SU1/Administration/cucm_b_administration-guide-1151su1/cucm_b_administration-guide-1151su1_chapter_01110.html

Jabber Deployment guide only makes mention of the CA certs themselves (the certs from the CA that signed your server certs). No mention of placing server certs in user stores. The TechNote also makes no mention of what to do with server certs once they are CA-signed (and contains already outdated information on XMPP Domains). Admin guide is very generic about the various tasks to perform in regards to certs.

[Jaime Valencia]

Root cert for your CA, whoever that is, goes to Trusted Root CA

Server certificates, go to Enterprise Trust.

However, something of what you're saying doesn't add up, CUCM and IM&P can only have one of two options, either they have self-signed certs, or they have CA certs for each kind of cert, you cannot have both for a specific certificate, lets says Tomcat for example.

I reset Jabber and am still being prompted to accept a different self-signed cert from IMP (cup-xmpp-trust) that gets added to my Enterprise Trust store.

You just said all of them are CA signed, so, something of what you've said, cannot be. Unless you have some of the required certs CA signed, and some self-signed.

[seangolyer]

Yes, that is the discrepancy I've observed as well, which is why I'm here for help, haha.

My cup, cup-xmpp, and tomcat certs are all CA-signed. However, when I clear my local certs and reset my login, the Jabber client is grabbing a cert that matches a "cup-xmpp-trust" cert on the server. I'm completely lost as to why it's grabbing an old self-signed "cup-xmpp-trust" cert and not the "cup-xmpp" cert, and whether or not it's safe to remove it from the server (and restart Cisco XCP Router I assume).

The certs currently associated to "cup-xmpp-trust" are the old self-signed cert as well as the Root CA that signed the new "cup-xmpp" cert. Do I need to remove the self-signed cert from this "-trust" store in IM&P? Will it regenerate a new one after service(s) are restarted (I've noted this happens in my lab, but that cluster is not CA-signed).

[Jaime Valencia]

I'd probably start by something as simple as trying a reboot to make sure nothing is there, but I've never seen that once you upload CA certs, you still get the old self-signed certs. Can you post a screenshot of your certs?? And I mean from CUCM/IM&P OS admin pages.

[seangolyer]

Absolutely.

I would also like to note that most of these certs were here before I walked into this situation, I'm here to get things working and clean up! But if I need to clean-up before I can get things working... that's good information to know. They also recently changed/updated their internal CA, so that's why there's a few old Root/Sub CAs in there.

The cert with the green box around it in the "cup-xmpp" image is the cert that seems to be the one Jabber is grabbing.

[Jaime Valencia]

But that is a -trust certificate, those are the equivalent of the trusted root CA store on windows, the certificate you should be presented with, is the one that doesn't have -trust, and you should only have one of each kind. You should be presented with the server certificate, not the trust certificate.

Once you have the cert that you accepted on your Jabber machine, you can open it, what does it say on Issued by:??

Does it have the CN of your CA?? and issued to: the FQDN??

If they're indeed self-signed, you should see the same value in issued to and issued by, the hostname or FQDN, depending on what you had upon installation.