ISE 2.1 Joining to the Cluster Issue

[Ali]

Hello Community,

We have Two node deployment and currently running with 2.1 with Patch 7 (Physical Appliance)

Primary  - Policy Service (Primary) , PRI(A), PRI(M)

Secondary - Policy Service (Primary), SEC(A), SEC(M)

Let me brief you about our issue, last week on our Primary node the Application Service got stopped running(Don't know exactly what happened) we open a support case the after troubleshooting, TAC suggested to go for latest patch 7.

After applying the patch we got one more issue, the secondary node is not joining to the cluster, we tried many time its getting failed.

From GUI :

Sync Status:

Node Registration or Sync failed. Please deregister and register the node again

Below is the error message from the CLI......

2018-10-07 21:25:10,441 INFO   [admin-http-pool2930][] cpm.admin.infra.action.DeploymentEditAction -:admin:Secondary.node:registerNode:- HostConfig hostId of registering node Secondary.node at time of local cert save: aff7bf20-ca00-11e8-b228-843dc6e
c40ec. All local certs and CSRs reference the HostConfig using this hostId.
2018-10-07 21:15:11,028 ERROR  [admin-http-pool2930][] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -:admin:Secondary.node.com:registerNode:- An error occurred while importing deployment shared system certificates to the regis
tering node
2018-10-07 21:25:11,029 ERROR  [admin-http-pool2930][] cpm.admin.infra.action.DeploymentEditAction -:admin:Secondary.node.com:- Error occurred while replicating deployment shared certificates
com.cisco.cpm.infrastructure.certmgmt.api.CertMgmtException: An error occurred while importing deployment shared system certificates to the registering node
        at com.cisco.cpm.infrastructure.deployment.client.DeploymentRegistrationClient.importDeploymentSharedCertificates(DeploymentRegistrationClient.java:1608)
        at com.cisco.cpm.admin.infra.action.DeploymentEditAction.replicateDeploymentSharedCertificates(DeploymentEditAction.java:1512)
        at com.cisco.cpm.admin.infra.action.DeploymentEditAction.createSubmit(DeploymentEditAction.java:1205)
        at com.cisco.cpm.admin.infra.action.DeploymentEditAction.createSubmit(DeploymentEditAction.java:1035)
       

Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2328)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
        at com.cisco.cpm.infrastructure.deployment.client.DeploymentRegistrationClient.importDeploymentSharedCertificates(DeploymentRegistrationClient.java:1598)
        ... 89 more

2018-10-07 21:25:11,029 ERROR  [admin-http-pool2930][] cpm.admin.infra.action.DeploymentEditAction -:admin:Secondary.node.com:registerNode:- Replicating the deployment shareable certificates to the registering node failed:
com.cisco.cpm.infrastructure.certmgmt.api.CertMgmtException: Error occurred while replicating deployment shared certificates
        at com.cisco.cpm.admin.infra.action.DeploymentEditAction.replicateDeploymentSharedCertificates(DeploymentEditAction.java:1527)
        at com.cisco.cpm.admin.infra.action.DeploymentEditAction.createSubmit(DeploymentEditAction.java:1205)
        at com.cisco.cpm.admin.infra.action.DeploymentEditAction.createSubmit(DeploymentEditAction.java:1035)

2018-10-07 21:27:06,220 WARN   [Thread-114][] com.cisco.epm.util.NodeCheckHelper -::::- Unable to retrieve the host config from standby pap java.io.IOException: Server returned HTTP response code: 401 for URL:  WARNING : This URL stripped from the email as per Company policy
f.com/deployment-rpc/getNodeConfig?hostname=Secondary.Node
3dd70-ca00-11e8-b228-843ertchcec::- Setting credentials on http client

Does any one has faced the above issue while joining to the cluster or any suggestion for the above Problem would be appreciated.

[Mohammed al Baqari]

Did you backup your certificates before the update. If yes, go ahead and
reimport them.

Another option is trying to promote the secondary node as primary then go
ahead and de-register PSNs then register them again. That might fix
the problem

[Ali]

Hello Mohammed,

Thank you for the reply, i re-imported the certificates still Secondary is getting failed to join.

Below are the logs .......

2018-10-05 16:51:19,443 ERROR  [pool-46-thread-1][] cisco.cpm.deployment.replication.ReplayerImpl -::::- Could not get all the missing change data records from 4172912 to 4172921

2018-10-05 16:51:19,443 ERROR  [pool-46-thread-1][] cisco.cpm.deployment.replication.ReplayerImpl -::::- Could not apply all missing change records. Expected to apply upto 4227056 but applied only upto 4172911

2018-10-05 16:51:34,238 INFO   [localhost-startStop-2][] cisco.cpm.cluster.impl.ClusterManagerImpl -::::- LocalCluster - Deregistering servicecom.cisco.profiler.api.ProfilerEpRemoteInterface

2018-10-05 16:51:34,238 INFO   [localhost-startStop-2][] cisco.cpm.cluster.impl.ClusterManagerImpl -::::- GlobalCluster - Deregistering servicecom.cisco.profiler.api.ProfilerEpRemoteInterface

2018-10-05 16:52:09,044 INFO   [main][] cisco.epm.fullsync.secondary.SecondarySyncManager -::c7e55690-c868-11e8-b228-843dc6ec40ec:FullSync:- Sending Transient Sync status:DBIMPORT_INITIATED, Node Sync Status: SYNC_INPROGRESS to PAP. Sync req id : c7e55690-c868-11e8-b228-843dc6ec40ec

2018-10-05 16:52:09,054 INFO   [main][] class com.cisco.epm.fullsync.FileUtil -::c7e55690-c868-11e8-b228-843dc6ec40ec:FullSync:- Reading primary node info from : /opt/oracle/base/admin/cpm10/dpdump/config_c7e55690-c868-11e8-b228-843dc6ec40ec.properties

2018-10-05 16:52:09,054 INFO   [main][] class com.cisco.epm.fullsync.HttpClientHelper -::c7e55690-c868-11e8-b228-843dc6ec40ec:FullSync:- syncRequestIdentifier..local value : c7e55690-c868-11e8-b228-843dc6ec40ec & syncRequestIdentifier in config file : c7e55690-c868-11e8-b228-843dc6ec40ec

2018-10-05 16:52:09,054 INFO   [main][] class com.cisco.epm.fullsync.HttpClientHelper -::c7e55690-c868-11e8-b228-843dc6ec40ec:FullSync:- Sending sync status to [https[:]//PrimaryNode.com/deployment-rpc/updateSyncStatus] for syncRequestIdentifier c7e55690-c868-11e8-b228-843dc6ec40ec

2018-10-05 16:52:09,054 INFO   [main][] class com.cisco.epm.fullsync.HttpClientHelper -::c7e55690-c868-11e8-b228-843dc6ec40ec:FullSync:- Creating http connection manager

2018-10-05 16:53:35,161 INFO   [main][] class com.cisco.epm.fullsync.HttpClientHelper -::c7e55690-c868-11e8-b228-843dc6ec40ec::- Setting credentials on http client

after this i am not getting any logs.....

For the 2nd option our customer is not agreeing.

[Damien Miller · Accepted solution]

I would recommend continuing to work with TAC and sharing their findings when it is resolved.  If it is impacting production you should ask to escalate the case above a severity 3.