Cisco ISE upgrade from 1.1.4 to 1.2.1 (Case Study)
Table of Contents
- Brief of case study.
- Topology/Deployment for Cisco ISE 1.1.4.
- Pre-requisites.
- Performing Cisco ISE cluster upgrade in live production setup.
- Post-upgrade verification.
- Troubleshooting and Issues encountered.
- Brief of case study:
This document is based out from the real time upgrade performed for Cisco ISE 1.1.4 to 1.2.1 for one of the Cisco’s large account. The aim of this document is to share the experience of performing such critical upgrade wherein you don’t have any maintenance window and all the upgrade is performed in production setup.
- Topology/Deployment for Cisco ISE 1.1.4.
The topology has 8 Cisco ISE nodes, 6 of the node in the deployment have policy node persona and other two have admin & MnT persona.
- Pre-requisites.
- Ensure that you uncheck the Disable user account after <60> days if password was not changed (valid range 1 to 3650) option here: Administration > Identity Management > Settings > User Password Policy page.
Reason: Users are disabled, if the password expires after the default setting (60 days) when you upgrade to Cisco ISE, Release 1.2 and restore the Cisco ISE, Release 1.1.x backup.
- Validate the Persona of the node.
Reason: You can upgrade only Administration, Policy Service, and Monitoring nodes. Upgrades are not supported for Inline Posture Nodes (IPNs). For IPNs, you must reimage your appliance and perform a fresh installation.
- Setup a repository and copy the upgrade bundle to a local repository on all nodes.
Recommendations:
- Create a local repository for disk:/ from the Cisco ISE UI.
- Copy the upgrade bundle to the local disk using the copy command from the Cisco ISE
CLI: copy ftp-filepath ise-upgradebundle-1.1.x-to-1.2.0.899.i386.tar.gz disk:/
Again, after you copy the upgrade bundle to the local disk, check to ensure that the size of the upgrade bundle in your local disk is the same as it is in the repository. Use the dir command to verify the size of the upgrade bundle in the local disk.
- Perform a backup of Cisco ISE configuration data from the primary Administration node, which includes the Cisco Application Deployment Engine (ADE) configuration data.
http://www.cisco.com/c/en/us/td/docs/security/ise/
Take on-demand backup of “Admin node”, as per below steps:
Backup can be taken via CLI or GUI:
- Backup via CLI:
AdMon/admin# backup ISE_Backup repository SFTP encryption-key plain Lab12345
% Creating backup with timestamped filename: ISE_Backup-150117-1306.tar.gpg
- Backup via GUI:
To perform an on-demand backup, complete the following steps:
Step 1 Choose Administration > System > Maintenance .
Step 2 From the Operations navigation pane on the left, choose Data Management > Administration Node > Full Backup On Demand .
The Backup On Demand page appears.
Step 3 Enter the name of your backup file.
Step 4 Select the repository where your backup file should be saved.
You cannot enter a repository name here. You can only choose an available repository from the drop-down list. Ensure that you create the repository before you run a backup.
Step 5 Check the Application-Only Backup, Excludes OS System Data check box to obtain a Cisco ISE application data backup. Uncheck this check box if you want the Cisco ADE operating system data as well.
Step 6 Enter the Encryption Key. This key is used to encrypt and decrypt the backup file.
Step 7 Click Backup Now to run your backup.
Note In a distributed deployment, do not change the role of a node or promote a node when the backup is running. Changing node roles will shut down all the processes and might cause some inconsistency in data if backup is running concurrently. Wait for the backup to complete before you make any node role changes.
Step 8 Your screen will be refreshed and the following message will appear in the lower right corner of your screen, if you are viewing the Backup On Demand page:
Backup is done successfully.
- Performing Cisco ISE cluster upgrade.
- Upgrade the secondary Administration node, before upgrading add MnT persona to Primary admin. And make sure Secondary Admin isn’t MnT node as sometimes the MnT database can be huge which can take a lot of time for upgrade.
Note: You do not have to manually deregister the node before an upgrade. Use the application upgrade command to upgrade nodes to Release 1.2. The upgrade process deregisters the node automatically and moves it to the new deployment. If you manually deregister the node before an upgrade, ensure that you have the license file for the Primary Administration node before beginning the upgrade process.
- Upgrade policy nodes:
Note: You can upgrade several PSN nodes in parallel, but if you upgrade all the PSNs concurrently, your network will experience a downtime.
- Finally, upgrade the primary Administration node.
Note: This node will be upgraded to Release 1.2 and added to the new deployment as a secondary Administration node. You can promote the secondary Administration node to be the primary node in the new deployment. If you want to retain Node your primary node, you must obtain a license that includes the UDI of both the primary and secondary Administration nodes.
After the upgrade is complete, if the Monitoring nodes that were upgraded to Release 1.2 contain old logs, ensure that you run the application configure ise command and choose 11 (Refresh M&T Database Statistics) on the Monitoring nodes.
- Post upgrade verification:
- Ensure active Directory connection is up and running.
- Reconfigure the alarm, e-mail settings, report customization, favorite reports, monitoring data backup schedules, and data purge settings after upgrade.
- Manually perform posture update:
- Verify machine authentications are working as expected.
- Install the latest Cisco ISE 1.2 patch.
- Troubleshooting and Issue encountered:
- MnT node upgrade stuck:
We may see logs like below when MnT node is upgrading, basically all the MnT database converts to 64 bit OS once we upgrade the node hence it may take a lot of time.
During our case we waited for about 12 hours we could still see similar logs. Hence we went ahead and built the node from the scratch.
Note - You can track/monitor the Cisco ISE upgrade by opening another SSH session and checking logs via “sh logging system tail count 200”.
Feb 6 14:31:21 BEISEA01 logger: PL/SQL procedure successfully completed.
Feb 6 14:31:21 BEISEA01 logger:
Feb 6 14:31:21 BEISEA01 logger: Elapsed: 04:40:12.39
Feb 6 14:31:23 BEISEA01 logger:
Feb 6 14:31:23 BEISEA01 logger: Table dropped.
Feb 6 14:31:23 BEISEA01 logger:
Feb 6 14:31:23 BEISEA01 logger: Elapsed: 00:00:02.11
Feb 6 14:31:24 BEISEA01 logger:
Feb 6 14:31:24 BEISEA01 logger: Table altered.
Feb 6 14:31:24 BEISEA01 logger:
Feb 6 14:31:24 BEISEA01 logger: Elapsed: 00:00:00.73
Feb 6 14:52:40 BEISEA01 logger:
Feb 6 14:52:40 BEISEA01 logger: Index created.
Feb 6 14:52:40 BEISEA01 logger:
Feb 6 14:52:40 BEISEA01 logger: Elapsed: 00:21:15.86
Feb 6 15:01:02 BEISEA01 logger: info:[isehourlycron.sh] Retaining latest core file: core.BEISEA01.adclient.21216.gz
Feb 6 15:01:02 BEISEA01 logger: info:[isehourlycron.sh] Retaining latest core file: core.BEISEA01.adclient.6994.gz
Feb 6 15:01:05 BEISEA01 logger: info:[dbcleanup-timely.sh] The ISE data filesystem is 40 percent full. No ISE DB cleanup required for disk
space.
Feb 6 15:01:05 BEISEA01 logger: info:[dbcleanup-timely.sh] The ISE database transaction log dir is only 16543 MB. No ISE DB cleanup requir
ed for transaction log size.
Feb 6 15:01:08 BEISEA01 logger: info:[isehourlycron.sh] Checking m&t tablespace size...:57
Feb 6 15:01:08 BEISEA01 logger: info:[isehourlycron.sh] Undo tablespace size: 3587512 Kb. Disk available in 1Kb blocks: 309247188
Feb 6 15:01:08 BEISEA01 logger: info:[isehourlycron.sh] DB undo tablespace size is at 3587512 1K blocks, no cleanup required
Feb 6 15:01:08 BEISEA01 logger: info:[isehourlycron.sh] Start monitoring the Aq for pending messages on secondary...
Feb 6 15:01:08 BEISEA01 logger: info:[isehourlycron.sh] End of monitoring the AQ for pending messages on secondary
Feb 6 16:01:01 BEISEA01 logger: info:[isehourlycron.sh] Retaining latest core file: core.BEISEA01.adclient.21216.gz
Feb 6 16:01:01 BEISEA01 logger: info:[isehourlycron.sh] Retaining latest core file: core.BEISEA01.adclient.6994.gz
Feb 6 16:01:04 BEISEA01 logger: info:[dbcleanup-timely.sh] The ISE data filesystem is 40 percent full. No ISE DB cleanup required for disk
space.
Feb 6 16:01:04 BEISEA01 logger: info:[dbcleanup-timely.sh] The ISE database transaction log dir is only 16708 MB. No ISE DB cleanup requir
ed for transaction log size.
Feb 6 16:01:07 BEISEA01 logger: info:[isehourlycron.sh] Checking m&t tablespace size...:57
Feb 6 16:01:07 BEISEA01 logger: info:[isehourlycron.sh] Undo tablespace size: 3587512 Kb. Disk available in 1Kb blocks: 309078504
Feb 6 16:01:07 BEISEA01 logger: info:[isehourlycron.sh] DB undo tablespace size is at 3587512 1K blocks, no cleanup required
Feb 6 16:01:07 BEISEA01 logger: info:[isehourlycron.sh] Start monitoring the Aq for pending messages on secondary...
Feb 6 16:01:07 BEISEA01 logger: info:[isehourlycron.sh] End of monitoring the AQ for pending messages on secondary
Feb 6 17:01:01 BEISEA01 logger: info:[isehourlycron.sh] Retaining latest core file: core.BEISEA01.adclient.21216.gz
Feb 6 17:01:01 BEISEA01 logger: info:[isehourlycron.sh] Retaining latest core file: core.BEISEA01.adclient.6994.gz
Feb 6 17:01:05 BEISEA01 logger: info:[dbcleanup-timely.sh] The ISE data filesystem is 40 percent full. No ISE DB cleanup required for disk
space.
Feb 6 17:01:05 BEISEA01 logger: info:[dbcleanup-timely.sh] The ISE database transaction log dir is only
- Wrong Endpoints on the homepage:
You may encounter an error wrt to active endpoint on Cisco ISE post upgrade.
There could be multiple reasons for the dashboard count to get struck.
Check this article and see if customer is hitting this issue.
Homepage :
Licensing tab:
Note – In my case we upgraded to latest patch of Cisco ISE 1.2.1 which fixed the issue.
- Licensing queries:
Primary Admin node of old deployment for version 1.1 would join the new deployment of version 1.2.1 as secondary node. And if Cu has license provisioned with the UDI of Primary Admin node of old deployment which is Secondary node of new deployment than it should not be an issue.
You can make the secondary admin node as Primary admin node without worrying about license issue.
