ISE upgrade failure

[torstensson]

Hi all! 

I got a fun one today  

I was updating our ISE cluster with two admin nodes and two PSN nodes. The primary admin node and one of the PSN nodes was updated from 2.4 to 2.7.9.356 but the update halted when the remaining PSN node had low disk space. The picture attached displays the current status. So Radius live logs shows that both admin nodes are working and clients can authenticate with no issues, at the moment at least.

So what to do now? Can I somehow clear up some space on the remaining PSN node and try again or should I simply remove it from the old cluster and reinstall in to the new deployment. But do I need to reinstall the admin node from the old deployment also? Or can I somehow update it even when the old PSN node is removed? I think it's possible to update the old admin node through CLI and then connect it to the new cluster as well, never tried it though. 

Does I risk some kind of split brain? And what happens if the primary node fails and the secondary is in another deployment?

Br

Lars

[hslai]

I agree with your plan:

• Remove the failed-upgrade PSN from the old deployment, install fresh the new release, and join to the new deployment.
• For ise01, you have the option to either also install fresh the new release or to upgrade, and then join to the new deployment. It's usually faster and cleaner to install fresh.

If not already done, read Cisco Identity Services Engine Upgrade Journey, Release 2.7 

[Peter Koltl · Accepted solution]

Split brain: not an issue until you make a configuration change. The new PAN and the old PAN are independent and do not fight over master role or configuration precedence.

---

---

> And what happens if the primary node fails and the secondary is in another deployment?

Any PAN is suitable to take over the whole deployment assuming its version is fresh (matches PSN) and its configuration is fresh . To take over you have to register the PSNs to the new PAN . But this is tiring.

Just make sure you have a good backup from the new PAN. After a failure, you can just reinstall the PAN and restore the configuration. 

Even better if you have two PANs. In case of a failure you just promote the secondary PAN.