topic Re: ISE cluster proper shutdown order in Network Access Control

ISE cluster proper shutdown order

Greg Gibbs — Tue, 08 Aug 2017 22:01:21 GMT

Hi team,

I've been doing some regression testing with a small VM-based ISE 2.0 p5 cluster in a customer environment. The cluster has 5 nodes; 2x PAN/MnT nodes, 2x PSN, 1x pxGrid. I've found that when I shutdown all of the VMs for maintenance (taking snapshots, modifying VM settings, etc), and power them all back up, the 2x PSNs come back as Replication Stopped and I have to do a manual resync.

I've tried shutting them down in different sequences (PSNs first then PAN/MnT and vice-versa, halting from CLI, stopping ISE application then halting, etc) but have not found a way to do this without having to resync the PSNs.

Is there a proper sequence for shutting down the cluster so as to prevent this Replication Stopped issue?

I have not found anything documented, so this might be helpful info for future documentation.

Thanks

Re: ISE cluster proper shutdown order

Arne Bier — Tue, 08 Aug 2017 23:47:15 GMT

What order do you restart the VM's? Although, that shouldn't theoretically make a difference, since we can't always control the order in which VMs restart. But maybe try starting PANs, then MnT and then PSN. A lot of waiting in between though 😞

Re: ISE cluster proper shutdown order

hslai — Sat, 12 Aug 2017 02:15:16 GMT

Please note that taking VM snapshots is not a supported means for ISE backup/restore and may drag VM performance down considerably in some cases. When shutting down ISE VMs, please explicitly stop ISE services first as the VM stop option does not seem to do that in ISE 2.0. The sequences I would suggest to try are:

stop: PAN, PSNs, and MnT

start: MnT, PSNs and then PAN

Re: ISE cluster proper shutdown order

Jason Maynard — Thu, 28 Mar 2019 18:26:36 GMT

Just want to confirm full order of execution - examples below. Please confirm

The correct way to shut down an ISE node

Application stop ISE
Halt

Order of shutting down ISE Environment

- Stop: PAN, PSNs, and MnT

- Start: MnT, PSNs and then PAN

If HA then shutdown standby first.

- Stop: PAN (Standby), PAN (Active), PSNs (Standby), PSNs (Active), and MnT (Standby), MnT (Active)

- Start: MnT (Active), MnT (Standby), PSNs (Active), PSNs (Standby) and then PAN (Active), PAN (Standby)

Re: ISE cluster proper shutdown order

hslai — Thu, 28 Mar 2019 20:27:33 GMT

Yes, that looks to me.

Re: ISE cluster proper shutdown order

Jason Maynard — Thu, 28 Mar 2019 20:35:53 GMT

Thanks for confirming.