02-07-2022 01:24 PM
I am going to upgrade virtual ISE in HA running all personas to 3.1, and in regard to this I have a few questions. I have done quite a bit of reading on the subject but would like to get the community's thoughts and recommendations on this topic.
My thoughts on the upgrade process is the following:
Solved! Go to Solution.
02-08-2022 12:33 AM
Are you running ISE on VMs or hardware appliances?
Either way you should run the Upgrade Readiness Tools on the secondary PAN. Depending on how much log data you have, you may want to purge old logs first. The URT will give you an upgrade time estimate that I have found to be pretty accurate.
The only certificates you generally need to worry about exporting are the system certificates. Any expired trusted certificates should be deleted prior to upgrading.
GUI or CLI work equally well. GUI will do both nodes for you. CLI you have to manually do each one.
Licensing should at least be Smart prior to upgrade. Get all required licenses (Essentials, Advantage, Premier, TACACS and VM as applicable) provisioned into the Smart account before upgrading.
02-08-2022 03:52 AM
-Increase reauth timers in authz profile & do split upgrade approach. Purge logs & upgrade to latest 2.7 patch as this will save time.
-As @Marvin Rhoads already mentioned, run the URT tool on 2nd PAN. Not only will it give you an estimate, but it will also highlight any potential errors since it clones the db and runs upgrade checks against it. Purging logs and applying the latest 2.7 past will help save upgrade time.
-I recently migrated a 2.7 cluster to 3.0p5 and actually migrated the licenses post upgrade. No operations were impacted. You will get out of compliance warnings etc. You will need TAC to assist with migrating to the new licensing model so your best bet is to open two TAC cases; 1 for the upgrade so you have that in case of any issues and needing to escalate; 2 for the licensing migration. In my most recent experience TAC actually suggested to do the licensing after the upgrade. I can tell you that the license migration was pretty smooth. Note though that this was for a 3.0 migration; There may be something I am missing in regard to 3.1 migration so I would ask TAC to be sure.
-This is a preference thing. You definitely have more control over the upgrade process if using CLI. I typically use GUI for patching and CLI for bundle upgrades.
-AFAIK this would work. Reason being is you download bundle to nodes from repository. Once the downloads are completed your Status column will depict 'Ready for Upgrade'. From there you have to hit Continue which takes you to the final step in GUI which is settings up upgrade sequence order (if doing split). Then the last step is to proceed with the upgrade which actually kicks it off
-Prepare extracts it; upgrade proceed begins upgrade;
-Best practice is to export system certs + priv keys; As far as the trusted root ca store IMO is totally up to you; Definitely make sure you have at a minimum a configuration backup;
-Highly recommend opening those TAC tickets because you never know what you will run into during a bundle upgrade. Unless you have a large ASI window I typically recommend the split upgrade approach. Also (mentioned earlier), for onboarding purposes you can always the day before upgrade increase reauth timers in authz profiles in case you have PSN issues users wont be re-auth'ing so they remain online with service. I hit a few bugs along the way that made things interesting (not sure if still present in 3.1pX, but still sharing):
ISE 3.0 PAN GUI Certificate Auth Lockout/Auth fail bug:
-Fixed by applying patch
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx22594
Queue Link Bug still seen in ISE 3.0p5:
The workaround is:
1.- Regenerate ISE Root CA
2.- Regenerate ISE Messaging service Certificate.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr40715
Good luck & HTH!
02-08-2022 05:23 AM
>...-Highly recommend opening those TAC tickets because you never know what you will run into during a bundle upgrade.
- Adding to that is the possibility to have on site assistance from your reseller (pay for it) and let them co-execute the whole process. This will depend on the manner in which business is considered critical. But delegating responsibility has benefits (...sorry guys , they did it guys not me...) You know what I mean.
M.
02-08-2022 12:10 AM
>...How would you recommend performing the upgrade for little to no network outage
- Well that's the central on here , note that I think that ISE is marvelous and very advanced. But the multi-node model with sometimes complex policies make upgrading a real adventure and sometimes a hazard. That is why I used to prepare a new environment (offline) , upgrade on it and test it , when it was ready (the PSN's and the rest). I used to have a script based on the CISCO-COPY-CONFIG-MIB that could switch radius servers in switches on the fly pointing to the new ISE(PSN's) environment or back (if needed). It has a double advantage. 1) You can observe the behavior of the migrated switches in a contained environment and check if still all is working 2) You don't need to go to the new ISE environment 'all at once' with the risks included.
M.
02-08-2022 12:27 AM
thanks for your reply.
Setting up two new ISE out of band or deploying with new IPs is currently not an option unfortunately, though I agree this would be a best case scenario. So with that in mind is the procedure I posted correct? Have you had any experience doing a similar upgrade? Or how would you go about the upgrade given these restrictions?
02-08-2022 01:41 AM
- As stated due to my handling and procedures followed I can't give further fundamental feedback and advise , check reply from Marvin too.
M.
02-08-2022 12:33 AM
Are you running ISE on VMs or hardware appliances?
Either way you should run the Upgrade Readiness Tools on the secondary PAN. Depending on how much log data you have, you may want to purge old logs first. The URT will give you an upgrade time estimate that I have found to be pretty accurate.
The only certificates you generally need to worry about exporting are the system certificates. Any expired trusted certificates should be deleted prior to upgrading.
GUI or CLI work equally well. GUI will do both nodes for you. CLI you have to manually do each one.
Licensing should at least be Smart prior to upgrade. Get all required licenses (Essentials, Advantage, Premier, TACACS and VM as applicable) provisioned into the Smart account before upgrading.
02-08-2022 02:35 AM
thank you for the reply.
Through GUI, will I be able to push the software to the nodes and then proceed with the upgrade at a later time?
How do you usually perform the upgrade?
02-08-2022 03:52 AM
-Increase reauth timers in authz profile & do split upgrade approach. Purge logs & upgrade to latest 2.7 patch as this will save time.
-As @Marvin Rhoads already mentioned, run the URT tool on 2nd PAN. Not only will it give you an estimate, but it will also highlight any potential errors since it clones the db and runs upgrade checks against it. Purging logs and applying the latest 2.7 past will help save upgrade time.
-I recently migrated a 2.7 cluster to 3.0p5 and actually migrated the licenses post upgrade. No operations were impacted. You will get out of compliance warnings etc. You will need TAC to assist with migrating to the new licensing model so your best bet is to open two TAC cases; 1 for the upgrade so you have that in case of any issues and needing to escalate; 2 for the licensing migration. In my most recent experience TAC actually suggested to do the licensing after the upgrade. I can tell you that the license migration was pretty smooth. Note though that this was for a 3.0 migration; There may be something I am missing in regard to 3.1 migration so I would ask TAC to be sure.
-This is a preference thing. You definitely have more control over the upgrade process if using CLI. I typically use GUI for patching and CLI for bundle upgrades.
-AFAIK this would work. Reason being is you download bundle to nodes from repository. Once the downloads are completed your Status column will depict 'Ready for Upgrade'. From there you have to hit Continue which takes you to the final step in GUI which is settings up upgrade sequence order (if doing split). Then the last step is to proceed with the upgrade which actually kicks it off
-Prepare extracts it; upgrade proceed begins upgrade;
-Best practice is to export system certs + priv keys; As far as the trusted root ca store IMO is totally up to you; Definitely make sure you have at a minimum a configuration backup;
-Highly recommend opening those TAC tickets because you never know what you will run into during a bundle upgrade. Unless you have a large ASI window I typically recommend the split upgrade approach. Also (mentioned earlier), for onboarding purposes you can always the day before upgrade increase reauth timers in authz profiles in case you have PSN issues users wont be re-auth'ing so they remain online with service. I hit a few bugs along the way that made things interesting (not sure if still present in 3.1pX, but still sharing):
ISE 3.0 PAN GUI Certificate Auth Lockout/Auth fail bug:
-Fixed by applying patch
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx22594
Queue Link Bug still seen in ISE 3.0p5:
The workaround is:
1.- Regenerate ISE Root CA
2.- Regenerate ISE Messaging service Certificate.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr40715
Good luck & HTH!
02-08-2022 05:23 AM
>...-Highly recommend opening those TAC tickets because you never know what you will run into during a bundle upgrade.
- Adding to that is the possibility to have on site assistance from your reseller (pay for it) and let them co-execute the whole process. This will depend on the manner in which business is considered critical. But delegating responsibility has benefits (...sorry guys , they did it guys not me...) You know what I mean.
M.
02-28-2022 10:51 AM
You should take a look at our ISE Webinars on YouTube for general upgrade routine.
Patch then HotPatch (log4j).
Upgrading your Cisco ISE Deployment
Learn the options to migrate your ISE deployment to the latest version. This will prepare you with strategies to upgrade your deployment to take advantage of the latest features.
You will learn :
00:20 Agenda
01:44 ISE Versions and Suggested Releases
02:57 ISE 3.1 Release Features
03:40 Supported Releases for Upgrading to ISE 3.1
06:02 Upgrade Workflow
07:04 Upgrade Methods: Backup & Restore, GUI, or CLI
11:00 Upgrade Preparations: Backup, Certs, Health Check
14:32 Upgrade Options Overview: Split and Full Upgrades
17:40 Full Upgrade Process Details
21:28 Upgrade Pre-Checks
27:09 Demo of Full Upgrade from ISE 2.7 to 3.1
38:45 Split Upgrade Process Details
44:11 Demo of Split Upgrade from ISE 2.6 to 3.1
52:38 Split vs Full Upgrade Process Comparison
54:38 Post Upgrade Tasks
56:13 ISE Upgrade Resources
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide