ISE 2.0 distributed deployment upgrade experiences

Ben.Levin · ‎03-17-2016

I'm getting ready to upgrade our 8 node ISE 1.3 deployment to 2.0. I've followed the upgrade documentation to prepare for this but I was wondering if anyone has experience doing the 2.0 upgrade on a similar setup. Do you have any experiences, issues, etc, you can share? I'm particularly interested in how long it took. We did set up a 2 node deployment in our lab but the upgrade was pretty quick, about 1 hour per server.

Thank you.

Marvin Rhoads · ‎03-17-2016

I've done several upgrades, from 1.2 up to and including to 2.0. Though the two upgrades I did to 2.0 were were basic 2-node deployments, they all went well.

The Release Notes and Upgrade Guides are quite thorough and accurate in my experience. Just be careful to follow them exactly. Have a full backup before beginning and be prepared to rejoin the deployment to AD (if you're using an AD identity store).

For 2.0, be sure to upgrade all the way to the current patch version (Cumulative Patch 2 as of right now).

joevimal01 · ‎08-14-2016

Hi Marvin,

I am planning to upgrade 1.4 to 2.1 version. we have distributed environment running on VMware setup. Could you please share the detailed steps pre-upgrade, upgrade and rollback in case of any issues. I have gone through many documents but, i don't find document to rollback to old version in case of any issues.

Thanks in advance.

Marvin Rhoads · ‎08-14-2016

The 2.1 upgrade guide is quite meticulous. Follow it and you will be well-served.

http://www.cisco.com/c/en/us/td/docs/security/ise/2-1/upgrade_guide/b_ise_upgrade_guide_21.html

As far as rollback, there is no "downgrade" that can be done in place. What Cisco recommends is to have a fresh full backup prior to starting the upgrade process. You can then use that on a fresh VM (of the old version) to restore to the previous state.

(Depending on your environment, you may also be able to use VMware snapshots.)

By the way, Cisco's current most stable release recommendation is 2.0 Patch 3 is you are not using Device Admin. 2.1 is recommended if you are using Device Admin. However, I find 2.1 compelling due to the numerous improvements to the UI. Watch for 2.1 Patch 1 to come out (hopefully in the next several weeks) to address some of the bugs identified in 2.1. We've seen a few, but generally not show stoppers.

joevimal01 · ‎08-15-2016

I have gone through the document. can you please confirm if below steps are correct to execute the upgrade process?

I skipped the commands: application upgrade prepare and application upgrade proceed.

The upgrade sequence is

secondary admin node
PSN Node
Primary admin node:

step1: # copy repository_url/path/ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz disk:/

step2: # show repository upgrade (to ensure the file is in the local repository)

step3: # application upgrade ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz upgrade

I am not understanding what you mean Device admin (you said; 2.1 is recommended if you are using Device Admin). could you please brief about this.

Rajat Gupta · ‎08-15-2016

Hello Joe,

The upgrade sequence is correct.

1. You should upgrade your Primary Admin Node first.

2. If there are separate Monitoring Nodes, then Primary Monitoring Node should be upgraded next.

3. Secondary Monitoring Node ( if there is one)

3. If not, then you can upgrade PSN's and then your Secondary Admin Nodes..

Also, by Device Admin Marvin is referring to TACACS+ support on ISE.

TACACS+ support came from ISE 2.0. You can read more about in the 2.0 release notes:

https://tools.cisco.com/squish/665F1

Please refer to TACACS+ Device Administration Section.

Also, if the upgrade fails, the rollback will happen automatically on the devices. In case you do have any issues after upgrade, you will need to re-image the box. There is no command for rollback.

Regards,

Rajat

Marvin Rhoads · ‎08-15-2016

Rajat,

I think you meant upgrade secondary PAN first.

Joe,

Rajat is correct regarding what I was talking about regarding Device Admin.

From how you laid it out, it appears you have a 4-5 node deployment.

Primary PAN/MnT

Secondary PAN/MnT

Several PSNs

Is that correct?

Rajat Gupta · ‎08-15-2016

Hello Marvin,

Yes. Listed it the other way round. Secondary PAN and then Primary Admin in the end.

Regards,

Rj

joevimal01 · ‎08-15-2016

First, i would like thank you all for your help.

Hi Marvin,

Yes, you are correct, I have a deployment as follows.

Primary PAN/MnT

Secondary PAN/MnT

Several PSNs

In that case, the upgrade sequence should be. First; Secondary PAN/MnT, 2nd; PSNs and finally Primary PAN/MnT. please correct me if I am wrong.

One more clarification since I am going to perform this in live environment.

1. after Secondary PAN/MnT upgrade, it will be deregisters from Primary PAN/MnT and it will become itself as primary admin/monitoring node. In that case, I hope this node will have all configuration and certificates remains and works as Primary PAN/MnT.(in this stage we will have two Primary PAN/MnT but no communication between them)

2. After one PSN node upgrade, will this PSN node will register with Secondary PAN/MnT(currently it is also primary PAN/MnT)? If yes, any request comes to this PSN will be authenticated through Secondary PAN/MnT(currently it is also primary PAN/MnT) ?

3. after all the PSN are upgraded (in this stage only we left primary PAN/MnT upgrade), all this this PSN node will be registered with Secondary PAN/MnT(currently it is also primary PAN/MnT)? If yes, all request comes to this PSN will be authenticated through Secondary PAN/MnT(currently it is also primary PAN/MnT) ?

Marvin Rhoads · ‎08-15-2016

Your sequence is correct.

1. Correct. Technically they are two deployments during that interim point of some but not all nodes upgraded.

2. Correct.

3. Correct.

Also, when your upgrade the last node (Primary PAN/MnT of the original deployment) it will join the new deployment as Secondary PAN/MnT. If you want it te reclaim the primary role, you need to log into it and, under the deployment menu, "Promote to Primary". This is an optional step.

joevimal01 · ‎08-15-2016

Thank you so much Marvin. I hope the upgrade sequence is still same for our deployment:

here is our deployment :

Node A:Primary PAN/secondary MnT

Node B: Secondary PAN/primary MnT

Node C,D,E and F: PSNs

The upgrade sequence is:

secondary admin node/primary montitoring node
PSN Node
Primary admin node/secondary admin node

Marvin Rhoads · ‎08-15-2016

Yes, still correct.

joevimal01 · ‎08-15-2016

Thank you. i will share my experience soon.

Ben.Levin · ‎08-16-2016

We're in the middle of the lab upgrade from 2.0 to 2.1, using the GUI upgrade method. It seems better than using the CLI so far. However, my lab deployment only has 2 servers and production has 8, so I'm really not looking forward to how long that is going to take. I'm planning to open a TAC case to make sure we have the procedure down and try to get a better idea of how long it will take to finish all 8 servers. When we upgraded from 1.3 to 2.0, it took us 13 hours.

joevimal01 · ‎09-13-2016

Hi Marvin,

Need your help to clarify below query.

when we use below command to copy the file to ISE(local disk). what is the setup do i need to do on remote PC(where IOS bundle is available), do we need to setup FTP server?

copy repository_url/path/ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz disk:/

Repository_url = Repository Name

Path = server IP address.

Please clarify in detail. Thanks