Upgrade ISE from 2.7 to 3.1

[BoomShakaLak]

I am going to upgrade virtual ISE in HA running all personas to 3.1, and in regard to this I have a few questions.  I have done quite a bit of reading on the subject but would like to get the community's thoughts and recommendations on this topic.

My thoughts on the upgrade process is the following:

1. migrate licenses to to the new standard in CSSM
2. upload the upgrade package to the repository
3. take a current backup of ISE configuration
4. run the command application upgrade prepare on both primary and secondary ISE nodes
5. when ready, run command application upgrade proceed on the secondary node
6. wait for show application status ise to show all processes running
7. run application upgrade proceed on the primary node
8. wait for show application status ise to show all processes running
9. install the log4j patch on both nodes
10. install latest patch on both nodes.
11. test network connectivity

• How would you recommend performing the upgrade for little to no network outage? Are my thoughts correct or should I be doing something else?
• Does anyone know the total time it takes to upgrade all the way from 2.7 to 3.1 latest patch?
• I know I need to migrate the licenses to the new standard first, will this affect operation until the devices are migrated?
• What is the preferred way to perform the upgrade, via GUI or CLI? 
• If upgrading via GUI is it possible to download the upgrade package to the nodes one day and complete the upgrade another day?
• I assume that the application upgrade prepare in the CLI uploads the upgrade package and then run application upgrade proceed when ready to install a day or more later?
• many upgrade guides recommend exporting the system certificates is this a best practice?  What about the trusted root certificates should they be exported as well? There are quite a few root certs and if something should go wrong tracking down them all could be time-consuming.
• Any tips or gotcha's I need to be aware about?

[marce1000]

                       >...How would you recommend performing the upgrade for little to no network outage

 - Well that's the central on here , note that I think that ISE is marvelous and very advanced. But the multi-node model with sometimes complex policies make upgrading a real adventure and sometimes a hazard. That is why I used to prepare a new environment (offline) , upgrade on it and test it , when it was ready (the PSN's and the rest). I used to have a script based on the CISCO-COPY-CONFIG-MIB that could switch radius  servers in switches on the fly pointing to the new ISE(PSN's)  environment or back (if needed). It has a double advantage. 1) You can observe the behavior of the migrated switches in a contained environment and check if still all is working 2) You don't need to go to the new ISE environment 'all at once' with the risks included.

 M.

[BoomShakaLak]

thanks for your reply.

Setting up two new ISE out of band or deploying with new IPs is currently not an option unfortunately, though I agree this would be a best case scenario.  So with that in mind is the procedure I posted correct?  Have you had any experience doing a similar upgrade?  Or how would you go about the upgrade given these restrictions?

[marce1000]

 - As stated due to my handling and procedures followed  I can't give further fundamental feedback and advise , check reply from Marvin too.

 M.

[Marvin Rhoads · Accepted solution]

Are you running ISE on VMs or hardware appliances?

Either way you should run the Upgrade Readiness Tools on the secondary PAN. Depending on how much log data you have, you may want to purge old logs first. The URT will give you an upgrade time estimate that I have found to be pretty accurate.

The only certificates you generally need to worry about exporting are the system certificates. Any expired trusted certificates should be deleted prior to upgrading.

GUI or CLI work equally well. GUI will do both nodes for you. CLI you have to manually do each one.

Licensing should at least be Smart prior to upgrade. Get all required licenses (Essentials, Advantage, Premier, TACACS and VM as applicable) provisioned into the Smart account before upgrading.

[BoomShakaLak]

thank you for the reply.

Through GUI, will I be able to push the software to the nodes and then proceed with the upgrade at a later time?

How do you usually perform the upgrade?

[Mike.Cifelli · Accepted solution]

• How would you recommend performing the upgrade for little to no network outage? Are my thoughts correct or should I be doing something else?

-Increase reauth timers in authz profile & do split upgrade approach.  Purge logs & upgrade to latest 2.7 patch as this will save time.

• Does anyone know the total time it takes to upgrade all the way from 2.7 to 3.1 latest patch?

-As @Marvin Rhoads already mentioned, run the URT tool on 2nd PAN.  Not only will it give you an estimate, but it will also highlight any potential errors since it clones the db and runs upgrade checks against it.  Purging logs and applying the latest 2.7 past will help save upgrade time.

• I know I need to migrate the licenses to the new standard first, will this affect operation until the devices are migrated?

-I recently migrated a 2.7 cluster to 3.0p5 and actually migrated the licenses post upgrade.  No operations were impacted.  You will get out of compliance warnings etc. You will need TAC to assist with migrating to the new licensing model so your best bet is to open two TAC cases; 1 for the upgrade so you have that in case of any issues and needing to escalate; 2 for the licensing migration.  In my most recent experience TAC actually suggested to do the licensing after the upgrade.  I can tell you that the license migration was pretty smooth.  Note though that this was for a 3.0 migration;  There may be something I am missing in regard to 3.1 migration so I would ask TAC to be sure.

• What is the preferred way to perform the upgrade, via GUI or CLI? 

-This is a preference thing.  You definitely have more control over the upgrade process if using CLI.  I typically use GUI for patching and CLI for bundle upgrades.  

• If upgrading via GUI is it possible to download the upgrade package to the nodes one day and complete the upgrade another day?

-AFAIK this would work.  Reason being is you download bundle to nodes from repository.  Once the downloads are completed your Status column will depict 'Ready for Upgrade'.  From there you have to hit Continue which takes you to the final step in GUI which is settings up upgrade sequence order (if doing split).  Then the last step is to proceed with the upgrade which actually kicks it off

• I assume that the application upgrade prepare in the CLI uploads the upgrade package and then run application upgrade proceed when ready to install a day or more later?

-Prepare extracts it; upgrade proceed begins upgrade;

• many upgrade guides recommend exporting the system certificates is this a best practice?  What about the trusted root certificates should they be exported as well? There are quite a few root certs and if something should go wrong tracking down them all could be time-consuming.

-Best practice is to export system certs + priv keys;  As far as the trusted root ca store IMO is totally up to you;  Definitely make sure you have at a minimum a configuration backup;

• Any tips or gotcha's I need to be aware about?

-Highly recommend opening those TAC tickets because you never know what you will run into during a bundle upgrade.  Unless you have a large ASI window I typically recommend the split upgrade approach.  Also (mentioned earlier), for onboarding purposes you can always the day before upgrade increase reauth timers in authz profiles in case you have PSN issues users wont be re-auth'ing so they remain online with service.  I hit a few bugs along the way that made things interesting (not sure if still present in 3.1pX, but still sharing):

ISE 3.0 PAN GUI Certificate Auth Lockout/Auth fail bug:

-Fixed by applying patch
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx22594

Queue Link Bug still seen in ISE 3.0p5:
The workaround is:
1.- Regenerate ISE Root CA
2.- Regenerate ISE Messaging service Certificate.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr40715

Good luck & HTH!

[marce1000 · Accepted solution]

   >...-Highly recommend opening those TAC tickets because you never know what you will run into during a bundle upgrade.

 - Adding to that is the possibility to have on site assistance from your reseller (pay for it) and let them co-execute the whole process. This will depend on the manner in which business is considered critical. But delegating responsibility has benefits (...sorry guys , they did it guys not me...) You know what I mean.

 M.

[thomas]

You should take a look at our ISE Webinars on YouTube for general upgrade routine.

Patch then HotPatch (log4j).

Upgrading your Cisco ISE Deployment

Learn the options to migrate your ISE deployment to the latest version. This will prepare you with strategies to upgrade your deployment to take advantage of the latest features.
You will learn :

• what changes went into the latest patches and previous releases
• how upgrades are more reliable
• your available upgrade options
• pre-upgrades, methods for upgrade as well as post-upgrade tasks
• best practices for migrating your ISE deployments

00:20 Agenda
01:44 ISE Versions and Suggested Releases
02:57 ISE 3.1 Release Features
03:40 Supported Releases for Upgrading to ISE 3.1
06:02 Upgrade Workflow
07:04 Upgrade Methods: Backup & Restore, GUI, or CLI
11:00 Upgrade Preparations: Backup, Certs, Health Check
14:32 Upgrade Options Overview: Split and Full Upgrades
17:40 Full Upgrade Process Details
21:28 Upgrade Pre-Checks
27:09 Demo of Full Upgrade from ISE 2.7 to 3.1
38:45 Split Upgrade Process Details
44:11 Demo of Split Upgrade from ISE 2.6 to 3.1
52:38 Split vs Full Upgrade Process Comparison
54:38 Post Upgrade Tasks
56:13 ISE Upgrade Resources

• https://cs.co/ise-3-1-upgrade
• https://cs.co/ise-rn
• https://cs.co/ise-license-migration
• https://cs.co/URT-overview