ISE upgrade timed out

[a.maldonado]

Hi, I wonder if someone can help me. I will appreciate it very much.

I started the process to upgrade our ISE deployment consisting of 2 SNS-3615-K9 servers. These have AD-OS version 3.0 (with no patches) and I wanted to upgrade it to AD-OS 3.1.

I read I had to install patch 2 of version 3.0 first, which after a few issues I managed to do but that was because an issue with smart accounts.

I went through the Checklist OK twice, once before the download and installation of patch 2 and another after I installed it. The URT tool indicated no problems and every test it did was successful. On the day of the upgrade the health checks (from the Administration à System menu) were also OK apart from DNS resolvability and the Trust Store Certificate Validation which were highlighted yellow. The Trust Store Certificate Validation had been highlighted yellow before and after I installed patch 2 but DNS resolvability

had been green before the installation of patch 2 and then yellow. It remined yellow even after I ran health checks again. I decide to go ahead with it since there had not been network changes that could have impacted this connectivity.

The system calculated 600 mins to upgrade both servers so I left it and monitored it from time to time. The first server to be upgraded was the Secondary (Secondary PAN and MnT) The progress bar indicated 80% of the process had been completed of this Secondary and that is where I left it. The day after I logged on to see if the servers had been upgraded and found the message Upgrade timed out in the status column of the secondary server, and the Primary server displayed Upgrade cancelled in the status column. I understand the system didn’t go ahead with upgrading the Primary server because the Secondary was not completed.

Please note that the servers subnet and the ftp server (where the repositories are) are both connected via Gigabit interfaces using different SVI interfaces of the same core switch, hence I do not believe bandwidth is an issue here. But I am happy to consider your thoughts.

The situation now is, I still have service form the Primary server but there is no backup as I have no connectivity with the Secondary server (cannot even ping it) and I cannot do anything in the Upgrade section in the GUI as everything is greyed out, I cannot deselect the nodes or click Continue or Download the upgrade file to the servers, etc.

Can someone suggest the best way to recover from this? I would like the option of reimaging the secondary server to be the last resort. I saved all the logs (bundles) and backed up the Operational and Configuration data.

Can someone suggest what I the best thing to do in these circumstances?

Thank you in advance.

[Leo Laohoo]

---

@a.maldonado wrote:

Can someone suggest the best way to recover from this?

Can someone suggest what I the best thing to do in these circumstances?

---

Always raise a TAC Case before every upgrade.

[Aref Alsouqi]

Personally I always use the CLI to do ISE upgrades to remove any dependency on the browsers that would cause the sessions to timeout. What do you see on the stuck node's screen? does it show anything?

[ahollifield]

What do you mean by "These have AD-OS version 3.0 (with no patches) and I wanted to upgrade it to AD-OS 3.1."  Do you mean an ISE 3.0 to ISE 3.1 upgrade?  

At this point I would just rebuild the secondary server from scratch and just re-add to the deployment.  All of the configuration is stored on the PAN anyways.  You will need certificates re-generated and the secondary node joined back to the AD domain (if there is an AD join point).

[Marcelo Morais · Accepted solution]

Hi @a.maldonado ,

1st ISE 3.1.0 parity with 3.0 P2

2nd ISE 3.1 supports restore from backups obtained from ISE 2.6+.

3rd upgrade ISE using BACKUP & RESTORE is RECOMMENDED, because it helps to reinstate the ISE Deployment settings and prevent data loss in case of any breakage during the upgrade process.

4th when upgrading ISE using the GUI, note that the timeout for the process is 4 hours. If the process takes more than 4 hours, the UPGRADE FAILS !!!

5th you have the option to Purge M&T Operation Data to speed up the process (your case < 600 min) via the following command:

ise/admin# application configure ise
Selection configuration option
...
[3]Purge M&T Operational Data
...

Putting ALL together

Node A

1. Install ISE 3.1 from scratch

2. Update to ISE 3.1 P3

3. Use the Backup from 3.0 and Restore (with ADE 0S) to ISE 3.1 P3

Node B

1. Install ISE 3.1 from scratch

2. Update to ISE 3.1 P3

3. Register to Node A Cluster

Hope this helps !!!

[thomas]

If you are not sure what to do, please call TAC when you have a production system down!

As @Leo Laohoo said, you may create a pre-emptive TAC case just in case.