ise distributed deployment upgrade

[descalante2007]

My customer has an ISE deployment with 4 nodes: Admin/Monitor Primary and Secondary plus 2 Policy Server. The Admin nodes are VMs, the Policy nodes are 3315 appliances.

The system was installed almost three years ago with the version 1.1.0 ... It appears the system never had issues so never was patched or upgraded. Why fix something that is working fine?

Today there was an issue because the certificates expired, so in the review to get the system up and running again, the update issue bring on to the conversation. We like to do an upgrade to the last supported version. So I wonder for some tips and ideas to take care for planning the upgrade.

I have some doubts:

Can the 3315 appliance support the release 1.3 without issues?

I know the upgrade procedure is basically installing a .tar file, but I'm not clear how the process in a distributed deployment should be. I had run upgrades in standalone systems, but never in a distributed deployment. So, I need to upgrade the Primary Admin only and the other nodes would upgrade automatically?

I would need to upgrade 1.1 to 1.2 first and then 1.2 to 1.3?

I undertand release 1.1 was in 32 bits, and the version 1.2 and 1.3 are in 64 bits, so I guess the process would take a long time (perhaps a couple of hours), so a maintenance window would need 3 or 4 hours until the full system became stable.

Can you give me some advice and suggestions to avoid major issues?

Regards.

Daniel Escalante.

[Venkatesh Attuluri · Accepted solution]

Supported Hardware and Personas for ISE 1.3 include 3315

http://www.cisco.com/c/en/us/td/docs/security/ise/1-3/release_notes/ise13_rn.html#pgfId-42971

You can upgrade to ISE 1.3 from 1.2 or 1.2.1

http://www.cisco.com/c/en/us/td/docs/security/ise/1-3/upgrade_guide/b_ise_upgrade_guide_13/b_ise_upgrade_guide_13_chapter_011.html#ID7

[cciesec2011]

I am NOT an ISE expert but I've done a few upgrade from 1.1 to 1.2 and from 1.2 to 1.3 and your situation
is actually quite common.  Here is what I would do if I were you:

0- ***** BACKUP THE PRIMARY ADMIN/MNT NODE BEFORE YOU BEGIN ****

1- Delete one of the PSN nodes from the cluster,
2- Delete the Secondary Admin/MNT node from the cluster

At this point your cluster should consist a single Primary Admin/MNT and 1 PSN.  So you don't have redundancy
but so what.

3- re-image the previous Sedondary Admin/MNT node with ISE 1.3 with patch 2 (patch 2 will be released at the end of March)
and make it part of a new cluster and call it "new_cluster".  This node will be the new Primary Admin/MNT node,

4- perform a restore on the new Primary Admin/MNT with the backup performed in step 0,
5- Make it part of AD if you have one,
6- apply the new certificate if you have in-house CA,
7- reimage the PSN node in step 1 with ISE 1.3 patch 2 and make it PSN node
8- Make the PSN node part of AD if you have one,
9- apply the new certificate to the new PSN node if you have one,
10- Add the new PSN node into the "new cluster",

At this point, you will have two separate cluster ISE 1.1 and ISE 1.3; however, they share the same database, everything will
be the same.  

11- test and validate that your new cluster works,

12- re-image the 1.1 PSN node with ISE 1.3 patch 2
13- add it into AD,
14- add certificate to the new PSN node in step 11,
15- now add the new PSN node into the "new cluster",

16- test and validate that you have redundancy with PSN node in the "new cluster"

Last step,

17- re-image the previous 1.1 Admin/MNT node with ISE 1.3 patch 2
18- add it into AD
19- add certificate to it if you have one,
20- Add this node into the "new cluster" as Secondary Admin/MNT.

21- validate that everything in the new cluster is working.

Easy right/

[mohanak · Accepted solution]

Can you give me some advice and suggestions to avoid major issues?

Documents related to upgarde were given by Venkatesh refer those. Along with that additional information.

Can the 3315 appliance support the release 1.3 without issues?

Cisco ISE-3315-K9 (small) 3 Supports ISE 1.3

Any

1x Xeon 2.66-GHz quad-core processor 4 GB RAM 2 x 250 GB SATA4 HDD5 4x 1 GB NIC6

I know the upgrade procedure is basically installing a .tar file, but I'm not clear how the process in a distributed deployment should be. I had run upgrades in standalone systems, but never in a distributed deployment. So, I need to upgrade the Primary Admin only and the other nodes would upgrade automatically?

When upgrading to Cisco ISE, Release 1.2, first upgrade the secondary Administration node to Release 1.2. You do not have to manually deregister the node before an upgrade. Use the application upgrade command to upgrade nodes to Release 1.2. The upgrade process deregisters the node automatically and moves it to the new deployment. If you manually deregister the node before an upgrade, ensure that you have the license file for the Primary Administration node before beginning the upgrade process. If you do not have the file on hand (if your license was installed by a Cisco partner vendor, for example), contact the Cisco Technical Assistance Center for assistance.

I would need to upgrade 1.1 to 1.2 first and then 1.2 to 1.3? I undertand release 1.1 was in 32 bits, and the version 1.2 and 1.3 are in 64 bits, so I guess the process would take a long time (perhaps a couple of hours), so a maintenance window would need 3 or 4 hours until the full system became stable

If you are on a version earlier than Cisco ISE, Release 1.2, you must first upgrade to 1.2 and then to 1.3.

You can upgrade to Cisco ISE, Release 1.2, from any of the following releases:

• Cisco ISE, Release 1.1.0.665 (or 1.1.0 with the latest patch applied)
  Cisco ISE, Release 1.1.1.268 (or 1.1.1 with the latest patch applied)
• Cisco ISE, Release 1.1.2, with the latest patch applied
• Cisco ISE, Release 1.1.3, with the latest patch applied

• Cisco ISE, Release 1.1.4, with the latest patch applied

Type of Deployment	Node Persona	Time Taken for Upgrade
Standalone (2000 endpoints)	Administration, Policy Service, Monitoring	1 hour 20 minutes
Distributed (25,000 users and 250,000 endpoints)	Secondary Administration	2 hours
	Monitoring	1.5 hours

After upgrading to ISE 1.2, upgrade to ISE 1.3

Type of Deployment	Node Persona	Time Taken for Upgrade
Standalone (2000 endpoints)	Administration, Policy Service, Monitoring	1 hour 20 minutes
Distributed (25,000 users and 250,000 endpoints)	Secondary Administration	2 hours
	Monitoring	1.5 hours

Factors That Affect Upgrade Time

Number of endpoints in your network

Number of users and guest users in your network

Profiling service, if enabled

[descalante2007]

Thank you for your detailed answer, but still I'm not clear, what would happen during the process ...

I should start by issuing application upgrade in every node, starting by the secondary Admin, and then every other node (MN, PSN)?

After the upgrade every node would keep its role, or they will back to standalone and I will need to set roles again after all the boxes are upgraded?

The nodes would be deregistered until the primary Admin is upgraded?

I would need to register the nodes again or it would happen automatically?

What about the IPNs? I understand the upgrade is not applicable to them.

Regards.

[Tushar Bangia]

Hi,

There are lot of detailed documentation such as below available for upgrade path however the moving from Cisco ISE version 1.1 to 1.3 is a major upgrade which require meticulous planning.

http://www.cisco.com/c/en/us/td/docs/security/ise/1-2/upgrade_guide/b_ise_upgrade_guide/b_ise_upgrade_guide_chapter_01.html#reference_6623307F8CB14DE09C3234FE43091591

Please do make sure you adhere to upgrade path i.e. 1.1 to 1.2/1.2.1 and than 1.3.

There is a change in OS level from 32-bit for Cisco ISE version 1.1 to 64-bit for Cisco ISE version 1.2 hence it takes a lot of time for Monitoring node to upgrade as all the monitoring DB will be migrated 64 bit OS.

Please do review your monitoring DB size as it may take about 8-10 hours for whole of your DB to convert for new OS.

I have a handled a lot for Cisco ISE migration from 1.1 to 1.2 and above and I have jotted down below best practices apart from Cisco document:

1. Make sure you upgrade the secondary Admin node first.

2. Secondary Admin node should not be monitoring node for original deployment.

3. Dont try to kill the upgrade by pressing "ctrl + c" if the upgrade is taking long.

4. Always prefer automatic upgrade, i.e. dont deregister the node from deployment and than upgrade the node.

5. Monitor the upgrade process via:

show logging system:

You will see logs as below:

ISE/Admin# sh logging system tail count 200

ADEOS Platform log:

-----------------

Feb  5 09:00:45 BEISEM01 logger: creating table p_nad_aaa_status using CTAS

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: PL/SQL procedure successfully completed.

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Elapsed: 00:00:00.19

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Table dropped.

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Elapsed: 00:00:00.04

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Table altered.

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Elapsed: 00:00:00.03

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Table altered.

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Elapsed: 00:00:00.04

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Index created.

Feb  5 09:00:45 BEISEM01 logger:

Feb  5 09:00:45 BEISEM01 logger: Elapsed: 00:00:00.21

Feb  5 09:00:45 BEISEM01 logger: creating table p_netflow_aggregation using CTAS

Feb  5 09:00:45 BEISEM01 logger: 

6. Make sure you have the upgrade bundle in the local repository to avoid network related issues.

Regards,

Tushar Bangia

Note: Please do rate the post if you find it helpful!!