ISE cluster proper shutdown order

[Greg Gibbs]

Hi team,

I've been doing some regression testing with a small VM-based ISE 2.0 p5 cluster in a customer environment. The cluster has 5 nodes; 2x PAN/MnT nodes, 2x PSN, 1x pxGrid. I've found that when I shutdown all of the VMs for maintenance (taking snapshots, modifying VM settings, etc), and power them all back up, the 2x PSNs come back as Replication Stopped and I have to do a manual resync.

I've tried shutting them down in different sequences (PSNs first then PAN/MnT and vice-versa, halting from CLI, stopping ISE application then halting, etc) but have not found a way to do this without having to resync the PSNs.

Is there a proper sequence for shutting down the cluster so as to prevent this Replication Stopped issue?

I have not found anything documented, so this might be helpful info for future documentation.

Thanks

[Arne Bier]

What order do you restart the VM's?  Although, that shouldn't theoretically make a difference, since we can't always control the order in which VMs restart.  But maybe try starting PANs, then MnT and then PSN.  A lot of waiting in between though :-(

[hslai · Accepted solution]

Please note that taking VM snapshots is not a supported means for ISE backup/restore and may drag VM performance down considerably in some cases. When shutting down ISE VMs, please explicitly stop ISE services first as the VM stop option does not seem to do that in ISE 2.0. The sequences I would suggest to try are:

stop: PAN, PSNs, and MnT

start: MnT, PSNs and then PAN

[Jason Maynard]

Just want to confirm full order of execution - examples below. Please confirm 

The correct way to shut down an ISE node

1. Application stop ISE
2. Halt

Order of shutting down ISE Environment

- Stop: PAN, PSNs, and MnT

- Start: MnT, PSNs and then PAN

If HA then shutdown standby first.

- Stop: PAN (Standby), PAN (Active), PSNs (Standby), PSNs (Active), and MnT (Standby), MnT (Active) 

- Start: MnT (Active), MnT (Standby), PSNs (Active), PSNs (Standby)  and then PAN (Active), PAN (Standby)

[hslai]

Yes, that looks to me.

[Jason Maynard]

Thanks for confirming.