04-13-2021 03:57 PM - edited 04-14-2021 10:10 AM
The purpose of this document is to provide troubleshooting insight into a well-known issue for the IM&P nodes when going to the Presence Topology Page and seeing all the services as Unknown, even though they are started as shown on the CLI via utils service list
When we go to the IM&P Administration webpage -> System -> Presence Topology to verify the health status of the server, it is possible that we might encounter that our Server is in a bad state, i.e., the server will display a white cross within a red circle.
Here we will note the most common reasons we see those errors on the Presence Topology webpage.
By clicking on view of one of the affected nodes, we can see the following errors on the webpage: the status of the services is unknown:
However, by going to the SSH/CLI Session of the IM&P Server and running the command: utils service list we will see that all those services are actually running (Started state).
The error on the GUI is associated with a Tomcat certificate issue, and what requires to be verified is the following:
Here is an example of a missing Certificate in the Tomcat chain. In this case, the Tomcat certificate chain consists of only 2 certificates: Root -> Leaf, however, there are scenarios where more than 2 or 3 certificates build the chain.
In the image example, the Issuer: mexrus-TENOCHTITLAN-CA is the certificate missing.
Navigate to IM and Presence Serviceability > Trace > Trace Configuration > Server to select: IM&P Publisher > Service Group > Database and Admin Services > Service: Cisco IM and Presence Admin > Apply to all Nodes > Debug level: Debug > Check the Enable All Trace Checkbox > Save
Navigate to IM and Presence Administration > System > Presence Topology > Select the node that is affected by the unknown services (And note the timestamp)
Go to RTMT and gather the following logs:
From the cupadminX.log
Accessing the Presence Topology > Node panel
2021-01-23 17:54:57,036 DEBUG [Thread-137] logging.IMPCommonLogger - IMPSocketFactory: Create socket called with host tenochtitlanIMP.mexrus.ru and port 8443
2021-01-23 17:54:57,040 DEBUG [Thread-137] logging.IMPCommonLogger - Enabled protocols: [TLSv1.1, TLSv1, TLSv1.2]
Exception got because a certificate was not verified.
2021-01-23 17:54:57,087 ERROR [Thread-137] services.ServiceUtil - Got an exception setting up the HTTPS connection.
javax.net.ssl.SSLException: Certificate not verified.
at com.rsa.sslj.x.aH.b(Unknown Source)
at com.rsa.sslj.x.aH.a(Unknown Source)
at com.rsa.sslj.x.aH.a(Unknown Source)
at com.rsa.sslj.x.ap.c(Unknown Source)
at com.rsa.sslj.x.ap.a(Unknown Source)
at com.rsa.sslj.x.ap.j(Unknown Source)
at com.rsa.sslj.x.ap.i(Unknown Source)
at com.rsa.sslj.x.ap.h(Unknown Source)
at com.rsa.sslj.x.aS.startHandshake(Unknown Source)
at com.cisco.cup.services.ServiceUtil.init(ServiceUtil.java:118)
at com.cisco.cup.services.ServiceUtil.getServiceInfo(ServiceUtil.java:197)
at com.cisco.cup.services.ServiceUtil.getServiceInfo(ServiceUtil.java:182)
Attempting to retrieve the Node Status for the Topology
at com.cisco.cup.admin.actions.TopologyNodeStatusAction$ServiceRunner.run(TopologyNodeStatusAction.java:358)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.rsa.sslj.x.aK: Certificate not verified.
at com.rsa.sslj.x.bg.a(Unknown Source)
at com.rsa.sslj.x.bg.a(Unknown Source)
at com.rsa.sslj.x.bg.a(Unknown Source)
... 13 more
The exception is caused due to the missing issuer of the Tomcat Certificate
Caused by: java.security.cert.CertificateException: Issuer for signed certificate [CN=tenochtitlanCM-ms.mexrus.ru,OU=Collab,O=Cisco,L=Mexico,ST=Mexico City,C=MX] not found: CN=mexrus-TENOCHTITLAN-CA,DC=mexrus,DC=ru
at com.cisco.cup.security.TLSTrustManager.checkServerTrusted(TLSTrustManager.java:309)
at com.rsa.sslj.x.aE.a(Unknown Source)
... 16 more
2021-01-23 17:54:57,087 DEBUG [Thread-137] actions.TopologyNodeStatusAction$ServiceRunner - Retrieved service status for node tenochtitlanIMP.mexrus.ru
2021-01-23 17:54:57,088 DEBUG [http-bio-443-exec-8] actions.TopologyNodeStatusAction - [Topology] VerifyNodeServices - Complete.
Another possibility is that can be found on the same CUPADMIN logs is the following:
Caused by: java.security.cert.CertificateException: Incorrect issuer for server cert
at com.cisco.cup.security.TLSTrustManager.checkServerTrusted(TLSTrustManager.java:226)
at com.rsa.sslj.x.aE.a(Unknown Source)
... 16 more
2017-10-14 09:04:01,667 ERROR [Thread-125] services.ServiceUtil - Failed to retrieve service status. Reason: Certificate not verified.
javax.net.ssl.SSLException: Certificate not verified.
In this case, the IM&P is not recognizing the Issuer certificate for the Tomcat as a valid Issuer certificate, which most probably was caused due to a corrupted certificate, options here are:
Lastly, be aware of the following defect: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvu78005
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: