Best practice: how to upgrade a ww ISE cluster

[Marc Bosecki]

Dear team,

my customer is running a ww ISE cluster, consisting of 14 PSNs, plus Admin and Monitoring nodes.

In the past, problems often occurred not directly after an upgrade, but a few days or weeks later,

and then we had an ww incident, or even more worse had to roll-back the whole cluster.

If we now want to install a minor upgrade (just a patch or from .x to .y) we would like to make it step-by-step.

So we came up with the following questions:

a) Can we run the ISE cluster mixed with consecutive minor releases?

b) Do we test interoperability of new minor release to the predecessor  release?

c) do we have a best practice, how to make a smooth upgrade of a large cluster?

Thanks team

Marc

[Timothy Abbott]

Marc,

We recommend that the entire deployment run the same version of code as a best practice.  We do test our patches, which consists of bug fixes, internally before releasing them to the public.  Our admin guide outlines how to install these patches which consist of loading them through the WebUI and allowing the system to install them automatically or (if you would like finer control of the installation) you can install them via the CLI one node at a time.

Regards,

-Tim

[Marc Bosecki]

Hi Tim,

thank you for your answer. But you're covering the technical point only.

As I mentioned, are the questions more related to operation management.

The customer does not want to make a bulk update of the whole cluster, because if this update fails or creates any other issue, that then the whole cluster has to be rolled-back.

Therefor he wants to start the update process for a part of the cluster, check functionality/stability for a while (3-4 weeks) and then upgrade the rest of the cluster.

I understand and fully agree that under normal conditions the entire cluster run the same code.

But is has to be possible (and supported), that a large ISE cluster is upgrade over 2-4 weeks, and is running 2 consecutive releases during this timeframe.

[paul]

I wouldn't do this for patches, but if you update the admin/monitoring nodes manually to the new patches you could probably update the PSNs slower and let a few of them run on the new patch for a while before committing the patch to all the PSNs.

If you are doing an ISE version upgrade you can definitely go as slow as you want if you are using the manual upgrade method (which I always do).  I usually do a fresh install/restore.  So essentially you have two independent deployments running, one on old version and one on new version. You control what PSN is in what deployment.

[kthiruve]

Thanks Paul. If you are wondering on the steps from the old deployment you move

Secondary PAN, Secondary MnT, PSNs in the same order. Please look at CCO guide for details under the ISE version for specific steps.

Cisco Identity Services Engine Upgrade Guide, Release 2.2 - Upgrade a Cisco ISE Deployment from the CLI [Cisco Identit…

Again doing upgrade or backup/restore is a choice.

Also if upgrading PSN's is going to be a problem. You can install PSNs from scratch and replicate instead of upgrading.

It is important to move one PSN's at a time, test it and move to another. During this process, make sure the user is not impacted. Move the PSN's less used first and so on.

Also try to purge MnT logs if you can for faster upgrade. Once you move these to new deployment, then you can run this to test it and move the rest of the deployment from old release to new release.

Thanks

Krishnan

[Marc Bosecki]

So looking into the Upgrade Guide, there is not mentioned how fast the steps 1 to 5 has to be done.

So it is ok to do steps 1, 2 and 3 for a first PSN and wait for a week to check stability? And then proceed, or?

Thanks

Marc

[hslai]

Fine, other than more chances for any interim changes of the configurations (e.g. endpoint updates) in the source deployment not copied to the target deployment.

[Marc Bosecki]

Ok, so no need to upgrade the cluster in one shoot.

But I'm wondering when we make instead of an upgrade a fresh install, we have to migrate the license, or?

Thx

Marc

[paul]

Yes you will need to rehost the license. As someone that has done this many times, working with licensing@cisco.com<mailto:licensing@cisco.com> is great. If you have the license files saved you can just send them the files and they will rehost. If you have the CCO account the license files were registered under originally you can simply rehost from there. If you only have the serial number of the current admin node they probably can fine the license files and rehost.