ISE Distributed deployment - Expired certificates

[pHz]

Hello,

I have a 4 node deployment, 2x PSN, 1x PAN, 1x Secondary Administration Node.

I did a failover of the PAN, and, following the promotion, the former primary could not communicate with the newly-elected primary. The replication does not work anymore. A manual sync up won't work.

After troubleshooting, this is because the Admin certificate of both the primary & secondary are expired.

I have tried updating the certificate on my secondary admin node (from the GUI of the primary admin), but obviously the configuration is not received since the config replication has problems.

Do I have to remove the current secondary admin node from the deployment to be able to update its Admin certificate, and add it back to the deployment ?

I'm afraid that this would not be sufficient, as my current primary also has an expired certificate, which would also need updating for the secondary node to accept joining the deployment.

But at that point, I'm afraid that updating my primary would break the distributed deployment with my PSNs as well, who of course also have expired Admin certificates.

What's the best course of action here ?

[Greg Gibbs · Accepted solution]

That's a pretty sticky situation. If all the Admin certs are expired, you don't really have a distributed deployment and each node is acting on stale information from the last time it was able to sync/replicate with the Primary PAN. I don't believe there's a way to install a new Admin cert unless the node is either part of an active cluster (managed from the PAN) or it's in Standalone mode.

Depending on how long the P-PAN and S-PAN certs have been expired, the S-PAN (now acting Primary) might not have the current configuration database that the P-PAN (now acting Secondary) has. You should check the policy configuration on the acting Primary to see if it is the most current. If not, you might want to promote the P-PAN (acting Secondary) back to the Primary role first.

Either way, you'll need to fix the Admin cert on the acting Primary first. I don't recall if you can change a dedicated PSN back to a Standalone node. If so, you'll likely need to make the PSNs standalone (one at a time if you need to retain service), fix the certificates and rejoin them to the Primary. If not, you'll likely need to default them using the "application reset-config ise" CLI command, install the new Admin cert, then rejoin them to the Primary.

Whichever PAN is remaining as acting Secondary will also likely need to be defaulted and joined back to the cluster.

[pHz]

Greg,

Thank you for your answer. I've been going over what you said and think I will proceed slightly differently.

I'll be updating the PSN certificates from the GUI, which I believe should work as the replication is still established, then update the current Primary PAN. Once it has rebooted, I believe all 3 nodes should be part of the cluster again, possibly with a manual sync up required.

The secondary PAN is more touchy. I will be running the application reset-config ise command, but I have read that this would remove the license from the node. At this point I'm unsure if joining this node back to my deployment will re-enable the licenses for this node or not. I believe it should, as it should be one set of license for the deployment, and this license should be hosted on the current primary PAN.

Am I wrong ? 

[Greg Gibbs]

The license is tied to the Serial Number of the original Primary and Secondary PAN. Defaulting the configuration will not change the Serial Number, so I would not expect any issues with the licensing.