ISE 3.0p3 Distributed: Restore from backup secondary nodes not in sync

[mattw]

Hi,

I'm doing some lab testing for a customer who want me to prove on their new distributed deployment (2x PAN/MNT, 3x PSNs for TACACS only) that it is possible to restore from a backup.

I've done the backup successfully.

I've done the restore successfully... to a point.

The primary PAN/MNT is happy with a green tick in the deployment screen.

The secondary PAN/MNT and the 3 PSNs are all in the "Not in Sync" state with a yellow exclamation mark.

I've tried selecting the nodes and clicking Syncup. This causes the ISE application on the nodes to restart but they remain in the "Not in Sync" state even after quite some time.

I've never actually tried restoring from a backup before. Is this normal behaviour? What is the best way to get all the nodes fully synced?

MTIA,

Matt.

[Milos_Jovanovic]

Hi @mattw,

As per admin guide, you can only restore PAN or standalone node, and you must synchronize all other nodes manually. Have you used same IP/Hostname while restoring, as there is section in same document for multiple scenarios?

Fortunately for me, I never had to actually restore backup in a production environment, in a distributed deployment.

I would consider restoration only if both PAN nodes died, as I would rather promote secondary PAN to primary then restore it from backup. For PSNs it is easy - you can always rebuild them, and register them back to the deployment.

BR,

Milos

[mattw]

Hi @Milos_Jovanovic ,

You're absolutely right and I agree with everything you say.

I am simply doing a backup and then a restore straight back onto the same primary PAN node so literally everything is the same.

Some observations I have have made:

• Doing a restore (without checking the "restore ADE-OS - needs a reboot" option does not bring the secondary nodes back into sync
• After a restore, selecting all secondary nodes and doing a "syncup" does not bring the secondary nodes back into sync (weird?!)
• Reloading all secondary nodes (but not the primary PAN) does not bring the secondary nodes back into sync
• Reloading ALL nodes (including the primary PAN) brings everything into sync

I am currently doing another restore with the "restore ADE-OS - needs a reboot" option checked. This will force a reboot of the primary PAN. I'm thinking this might be the fix. We shall see....

[mattw]

Bad form to reply to my own post but... That did it.

Looks like the primary PAN has to be reloaded after a restore for the secondary nodes to come back into sync.

I achieved this in my latest test by having the "restore ADE-OS (needs a reboot)" option checked when I did the restore.

I guess if I did not check this box, I could wait for the restore to complete and then reload the primary PAN and possibly force a manual syncup if the secondary nodes did not sync on their own.

Hope this helps someone.

Matt.

[Marcelo Morais]

Hi @mattw ,

  just to add one thing ...

 Please take a look at: CSCvy71406 Update Admin Guide to include Manual Sync as part of patching process.
"...

Symptom:

Per ISE Engineering, we should update the ISE Admin Guide to include a Manual Sync as a part of the patching process. This needs to be documented as a best practice step in the admin guide.
Known Affected Releases:
2.6(0.908) - 2.7(0.904) - 3.0(0.902) - 3.1(0.901)

..."

 Although it's for "patching process", I started doing a Manual Sync not only after a patching process but also after a ISE restore.

Hope this helps !!!