Node not reachable

[stephan.l.martin1]

Version: 3.1

Patch: 8

Old cluster: SNS-3655

New Nodes: SNS-3755

I am working on integrating three new ISE servers into our existing cluster in preparation for lifecycle replacements. Prior to pulling the node into the cluster I perform the following:

Setup ISE and install system certificate and all associated CA certs (root and CA including the CA for the current cluster certificates) The node successfully joins to the cluster and the deployment health shows green for all nodes. We attempted to join the nodes to the domain which failed as ISE indicated that it cannot communicate with the node.

When attempting to view system certificates ISE indicates that it cannot communicate with the node.

The device statistics (CPU, Memory, Authentication latency) all show unavailable.

I have disabled the ISE messaging logging feature which resolves the CPU/Memory issues but results in a critical error complaining about ISE API Gateway service not running (not necessarily a problem as we aren't actively using APIs at the moment)

Steps taken thus far:

-Toggled ISE messaging on/off

-Stop/Start ISE application on PAN

-Stop/Start problematic nodes

-Dropped node from cluster, reinstalled ISE (reset config did not work as I was unable to access the web gui)

Errors: No queue link errors, replication errors, or other faults that would indicate an immediate issue.

Edit: There are no firewalls, access-lists or other restricting policies that would prevent the communication of these nodes in either direction. 

Any assistance is greatly appreciated.

[Arne Bier]

That does sound a bit esoteric. A few more things to add to your checklist (especially regarding the inability to Join new nodes to the AD)

• Is NTP configured on the new nodes and is it sync'd ?
• Do you have DNS PTR records for all the new nodes?

If the above is all true, then I would also try rebuilding the ISE Internal CA (I know you mentioned that you don't see any Queue Link errors, but it doesn't harm)

There should be no need to disable the ISE Secure Logging feature.

And you have no issues with the existing 3 nodes (on older SNS servers) ?

What personas did you assign to the new nodes (e.g. PAN/MNT/Services etc.) ?

[stephan.l.martin1]

@Arne Bier all DNS records were created. I can resolve to host and resolve to IP. As far as NTP all devices are synced and operational. The nodes in question are operational (2 nodes are being replaced and a 3rd node is being added in preparation for enterprise posture assessment)

The nodes were added to the cluster as PSNs only at this time.

For rebuilding the ISE internal CA- should I expect any service impacts or is this a relatively benign task? 

[Arne Bier]

Just to be clear, you have DNS PTR records and can resolve the hostname by looking up its IP address?

Rebuilding the internal ISE Root CA is benign, if you're not using it (e.g. for ISE BYOD feature). It will regenerate the Root CA, and then all the CA certs under the root. It's done under the Certificate Signing Request section (pull down menu to find Root CA) - it's not service affecting. It does leave the old certs in place, and you can optionally delete them (by deleting the child certs, and working your way up to the old Root CA ... in that order)

Regarding the ISE API Gateway service, that should be in RUNNING state for a dedicated PSN node (even if this feature is disabled in the GUI for that node).

In my experience, when you can't browse the System certs of a node added to a deployment, it indicates that the sync up is not done yet.  if the above doesn't work, have you tried doing a manual sync up?

I have to be honest, all of these issues should not be happening for such simple steps. It is ISE 3.1 though and I don't recall if there are issues with this release.  TAC would have a better memory of issues than me.

[stephan.l.martin1]

Unfortunately, I think I will need to open a TAC. I regenerated the Root CA and the ISE Messaging cert for all nodes with no results. When attempting to conduct a manual sync-up I receive an error "unable to sync node". I also agree that the issues don't really make sense. 

[Arne Bier]

My comment about ISE 3.1 and idiotic software defects relates to things like this - you can't see this coming as a regular user and then TAC can get you out of the mess.