ISE 2.4 upgrade issues

[Elly Bornstein]

Hey guys,

We have a 7 node deployment (3PSN, 2PAN, 2MNT) currently on 2.2p10. We ran the URT and everything came out fine.

However, upon upgrading the secondary PAN first, we encountered an issue that after the VM rebooted it ended up in the grub rescue prompt. TAC said we have to rebuild this node, instead we restored the LUN from before the upgrade. We believe this issue is due to all the relics in the system from all the bugs/patches we hit/applied in 2.2. In the end of the day, upgrading from 2.2 to 2.4 is too disruptive and requires WAY to much knowledge of 'under the hood' of ISE to make it happen (it's not easy). Plus we want to start with a clean DB.

Is there a document on how to build a parallel ISE deployment using config backups in a manner that it will not trample on our production deployment?

[Surendra · Accepted solution]

I doubt you will find any document on this. If I had to do this (assuming same IP addresses being used the original deployment) with almost zero downtime, I would follow this :

Let’s say my existing nodes are P, S , M, SM, PS1, PS2, PS3 where they are PAN, SPAN, MnT, SMnT , PSN1, PSN2, PSN3 respectively.
Let’s assume that the new nodes be p, s, m, sm, ps1, ps2, ps3 following the same nomenclature as above.

Export the system certificates and CA certificates (not Trusted Certificates) from all the nodes. Take a configuration data backup and an operational data backup.


1. Shut down S. Build s first as a standalone node and restore the configuration data backup.
2. Make it a Primary Admin and Primary Monitoring node.
3. Check if the CA certificates match (by checking serial numbers).
4. If not, import them to the Trusted Certificate store of s.
5. Shut down M. Build m as a standalone node and import M’s certificate to m. Register it to s as a Primary Monitoring Node.
6. Shut down PS1. Build ps1 as a standalone node and import PS1 certificate onto this node. Register it to s as a PSN.
7. Shut down PS2. Build ps2 as a standalone node and import PS2 certificate onto this node. Register it to s as a PSN.
8. Shut down PS3. Build ps3 as a standalone node and import PS3 certificate onto this node. Register it to s as a PSN.
9. Shut down P and SM. Build p and sm as a standalone nodes and import their respective certificates exported from the old deployment.
10. Register sm to s as Secondary Monitoring node.
11. Restore operational data backup.
12. Register p to s as a Secondary Admin node.
13. Promote p to Primary Admin node.

During this process, the new deployment will not have logs from the time this process started till ps1 comes up.

If this is complicated than simply clicking an upgrade button on the GUI of the PAN of the existing deployment, I would suggest you provide the configuration backup to TAC and test a real upgrade in the lab and see if they see any issues. If they don’t see any issues, give the normal upgrade process another try.

P.S : I have not tried this myself and would highly recommend you to try this out in a lab first and then try to do the same in production. In theory, this should work.