Jabber phone services are not working since renewing certificates

[Djeten]

Last week, we renewed the certificates on our expressways, cucm and im&p. Since then Jabber can't register to the phone services from outside the network. Inside  the network, jabber works fine.

We see the same with the phone service in Webex Teams. In the Teams Settings I can see I am logged in to CUCM, but the phone service says 'Phone Service disconnected'.

I ran the collab edge validator and there I got the error '407 Proxy Authentication Required'

I analysed the diagnostics logs with the TAC Tool and there I got this reply: Delegated credential checking error - There is a communication problem with the traversal client zone Traversal Zone to EWE used to receive delegated credential checking requests

I have checked the configuration of the Traversal Zone, but all seems fine. The configuration of the traversal zones hasn't been changed. Only change is the renewal of certificates. All devices have been rebooted.

Has anyone seen this behavior before and knows how to resolve this?

[Nithin Eluvathingal]

Hope your renewed certificates has DNS entries as before. 

Have a look on below link.

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/expressway/config_guide/X12-5/exwy_b_mra-expressway-deployment-guide/exwy_b_mra-expressway-deployment-guide_chapter_01101.html#reference_8A6377297196DDFBE2E74674BA77FD21

[Djeten]

Yes, the SAN's are identical in the new certificate.

[Adam Pawlowski]

If you're using SSO, renewing the certificate may require you to exchange new metadata with the IDP.

The zone status should be up, but, in the event logs you'll want to look and make sure there are not SSL errors anyway.

If the certificates were renewed, then the trust stores should contain valid CA certificates, and it should just be a matter of looking at the SANs, and that everything that serves a certificate has been restarted/refreshed - refresh all the UC servers, make sure Tomcat has been restarted, etc.

[Nithin Eluvathingal]

Hope the Zones are UP. 

Troubleshooting steps are mentioned for 407 Proxy Authentication Required on the document which I shared previously. have you tried this ?

[Djeten]

I see no SSL errors in the logs, Authentication Policy is set on 'Do not check credentials', certificates seem ok on all devices...

When I try to log in with Webex Teams, I can see that authentication works, but the registration of the softphone fails...

[Adam Pawlowski]

Next step I would undertake, if not TAC, is reviewing logging from Jabber. I have no idea how logging from Teams is reviewed to say if it’s similar or where it is. 

[Djeten]

I see these errors in the expressway E logs. 

When I google this error, I find that the FQDN of the E must match the result of the collab-edge SRV record. Well it does... This has always matched and hasn't changed with the renewal of the certificate...

[Adam Pawlowski]

If I had to guess that red error is just from a client that thinks it had authenticated prior to you replacing the certificate and is trying to register, but the proxy doesn't know about it so it's being rejected.

The bottom two lines appear to show a Jabber client that is probing for SSO to want to sign in, and being told that SSO is not supported. That could be normal if you don't use SSO, but if you do then your SSO is not operating.

[Djeten]

We don't use SSO...

[Nithin Eluvathingal]

I am not aware of your  Expressway versions. but I see a bug related  to the error which you described above.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt74089/?rfs=iqvred

[Djeten]

I have upgraded to firmware 12.6.1, but the issue is still there...

[Adam Pawlowski]

I have to go back to suggest that you look at the client logs to see if that can help pinpoint what the fault is, or if there's additional messaging there to narrow it down.

[Djeten]

This is the jabber.log file from my cellphone.

Logging in to jabber works, but the phoneservice can't register to our CUCM.

[Djeten]

When I analyse the logs with the Cisco TAC Tool, I get this message:

• Server failed to verify certificate causing TLS issues
• • Description

    TLS handshake could not complete because 34.197.146.115 could not verify the certificate of the local server.
    The reason of the TLS handshake failure is : Got EOF on socket.
    

    Further information

    More info can be found in the packet capture by filtering with "tcp.port == 9657 and ip.addr==34.197.146.115".

    Action

    - If the local server is using a self signed-certificate, generate a CSR for the server, get it signed by a CA and upload it to the server.
    - Make sure that the certificates of the intermediate CA's are in the local server's trust store

However all certificates are in the trust store.