Rebooting multiple ISE nodes - what's the quickest yet safe sequence?

[SanahGrat]

I had to reboot our 2 ISE VMs recently. TAC has said to just be sure I leave one node up so there are still services available. They always say this. I never get a really clear answer to my question.

I have been rebooting the secondary node, making sure there are no pending sync operations, then promoting secondary to primary, then rebooting the now secondary node. This way I am always rebooting the secondary node. This takes a LONG time but seems like the safest way to go. My question is, is this really necessary? Is it safe to reboot the primary without doing the promotion routine? Do the nodes sort out the details/sync if I reboot the primary leaving only the secondary up?

[balaji.bandi]

May something miss understood here :

what TAC saying, when you reboot Secondary - the Primary still service and Active. when the Secondary come back - promote that as Primary that means (secondary become active here)

here is promote example :

https://bluenetsec.com/promote-ise-secondary-pan-to-become-the-primary/

when the Secondary become primary,  (primary become secondary, so you rebooting secondary for safe )

this will have no impact on services. Once Secondary (original Primary back only) you can promote this as Primary to leave as it is all up to the business decision.

(in other way what you said correct, you reboot secondary all time - but you rebooting both ISE- with out any service impact)

not sure what reason you rebooting by TAC suggestion.

[thomas · Accepted solution]

With only 2 ISE nodes, you only want to reboot only 1 at a time because otherwise you will lose all ISE services (network access outage!) for 15-20 minutes while both nodes reboot simultaneously. This is why TAC always says this. It is Good Advice.

To answer your question, you need to think about the multiple personas that each node performs: PAN+MNT+PSN. When you reboot the Primary, you are losing half of your PSN (RADIUS/TACACS) capacity but you are also losing your Primary PAN which performs some critical functions beside configuration. Read the ISE Administration Guide section High Availability for the Administrative Node to understand what functions are down while the Primary PAN reboots:

If you cannot live without these services for the 15-20 minutes that the Primary PAN reboots, you will want to take the necessary steps and time to perform the Primary node election! This is the recommended and "safest way to go" as you said.

That leaves the MNT persona. Read the ISE Administration Guide section Automatic Failover in MnT Nodes . You will effectively lose the logging data from the PSNs while the MNT was down/rebooting - which is also a PSN in your small deployment. To get that period of logging data back, do a backup from the secondary:

When the primary node comes back up after a failover, obtain a backup of the secondary and restore the data to update the primary node.

[Marcelo Morais]

Hi @SanahGrat ,

 just to add one piece of information beyond what @thomas said ...

 About Posture ... take a look at: CSCvu62938 Posture fails when primary PSN/PAN are unreachable. (solved on ISE 3.0 P3, ISE 2.7 P4 and ISE 2.6 P9).

Hope this helps !!!