topic When you move to the next in Network Access Control

ISE 2.0 distributed deployment upgrade experiences

Ben.Levin — Mon, 11 Mar 2019 06:35:30 GMT

I'm getting ready to upgrade our 8 node ISE 1.3 deployment to 2.0. I've followed the upgrade documentation to prepare for this but I was wondering if anyone has experience doing the 2.0 upgrade on a similar setup. Do you have any experiences, issues, etc, you can share? I'm particularly interested in how long it took. We did set up a 2 node deployment in our lab but the upgrade was pretty quick, about 1 hour per server.

Thank you.

I've done several upgrades,

Marvin Rhoads — Fri, 18 Mar 2016 01:23:22 GMT

I've done several upgrades, from 1.2 up to and including to 2.0. Though the two upgrades I did to 2.0 were were basic 2-node deployments, they all went well.

The Release Notes and Upgrade Guides are quite thorough and accurate in my experience. Just be careful to follow them exactly. Have a full backup before beginning and be prepared to rejoin the deployment to AD (if you're using an AD identity store).

For 2.0, be sure to upgrade all the way to the current patch version (Cumulative Patch 2 as of right now).

Hi Marvin,

joevimal01 — Mon, 15 Aug 2016 01:02:23 GMT

Hi Marvin,

I am planning to upgrade 1.4 to 2.1 version. we have distributed environment running on VMware setup. Could you please share the detailed steps pre-upgrade, upgrade and rollback in case of any issues. I have gone through many documents but, i don't find document to rollback to old version in case of any issues.

Thanks in advance.

The 2.1 upgrade guide is

Marvin Rhoads — Mon, 15 Aug 2016 01:55:06 GMT

The 2.1 upgrade guide is quite meticulous. Follow it and you will be well-served.

http://www.cisco.com/c/en/us/td/docs/security/ise/2-1/upgrade_guide/b_ise_upgrade_guide_21.html

As far as rollback, there is no "downgrade" that can be done in place. What Cisco recommends is to have a fresh full backup prior to starting the upgrade process. You can then use that on a fresh VM (of the old version) to restore to the previous state.

(Depending on your environment, you may also be able to use VMware snapshots.)

By the way, Cisco's current most stable release recommendation is 2.0 Patch 3 is you are not using Device Admin. 2.1 is recommended if you are using Device Admin. However, I find 2.1 compelling due to the numerous improvements to the UI. Watch for 2.1 Patch 1 to come out (hopefully in the next several weeks) to address some of the bugs identified in 2.1. We've seen a few, but generally not show stoppers.

I have gone through the

joevimal01 — Mon, 15 Aug 2016 19:01:09 GMT

I have gone through the document. can you please confirm if below steps are correct to execute the upgrade process?

I skipped the commands: application upgrade prepare and application upgrade proceed.

The upgrade sequence is

secondary admin node
PSN Node
Primary admin node:

step1: # copy repository_url/path/ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz disk:/

step2: # show repository upgrade (to ensure the file is in the local repository)

step3: # application upgrade ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz upgrade

I am not understanding what you mean Device admin (you said; 2.1 is recommended if you are using Device Admin). could you please brief about this.

Hello Joe,

Rajat Gupta — Mon, 15 Aug 2016 21:27:00 GMT

Hello Joe,

The upgrade sequence is correct.

1. You should upgrade your Primary Admin Node first.

2. If there are separate Monitoring Nodes, then Primary Monitoring Node should be upgraded next.

3. Secondary Monitoring Node ( if there is one)

3. If not, then you can upgrade PSN's and then your Secondary Admin Nodes..

Also, by Device Admin Marvin is referring to TACACS+ support on ISE.

TACACS+ support came from ISE 2.0. You can read more about in the 2.0 release notes:

https://tools.cisco.com/squish/665F1

Please refer to TACACS+ Device Administration Section.

Also, if the upgrade fails, the rollback will happen automatically on the devices. In case you do have any issues after upgrade, you will need to re-image the box. There is no command for rollback.

Regards,

Rajat

Rajat,

Marvin Rhoads — Mon, 15 Aug 2016 21:49:47 GMT

Rajat,

I think you meant upgrade secondary PAN first.

Joe,

Rajat is correct regarding what I was talking about regarding Device Admin.

From how you laid it out, it appears you have a 4-5 node deployment.

Primary PAN/MnT

Secondary PAN/MnT

Several PSNs

Is that correct?

Hello Marvin,

Rajat Gupta — Mon, 15 Aug 2016 22:05:22 GMT

Hello Marvin,

Yes. Listed it the other way round. Secondary PAN and then Primary Admin in the end.

Regards,

First, i would like thank you

joevimal01 — Tue, 16 Aug 2016 02:07:12 GMT

First, i would like thank you all for your help.

Hi Marvin,

Yes, you are correct, I have a deployment as follows.

Primary PAN/MnT

Secondary PAN/MnT

Several PSNs

In that case, the upgrade sequence should be. First; Secondary PAN/MnT, 2nd; PSNs and finally Primary PAN/MnT. please correct me if I am wrong.

One more clarification since I am going to perform this in live environment.

1. after Secondary PAN/MnT upgrade, it will be deregisters from Primary PAN/MnT and it will become itself as primary admin/monitoring node. In that case, I hope this node will have all configuration and certificates remains and works as Primary PAN/MnT.(in this stage we will have two Primary PAN/MnT but no communication between them)

2. After one PSN node upgrade, will this PSN node will register with Secondary PAN/MnT(currently it is also primary PAN/MnT)? If yes, any request comes to this PSN will be authenticated through Secondary PAN/MnT(currently it is also primary PAN/MnT) ?

3. after all the PSN are upgraded (in this stage only we left primary PAN/MnT upgrade), all this this PSN node will be registered with Secondary PAN/MnT(currently it is also primary PAN/MnT)? If yes, all request comes to this PSN will be authenticated through Secondary PAN/MnT(currently it is also primary PAN/MnT) ?

Your sequence is correct.

Marvin Rhoads — Tue, 16 Aug 2016 02:22:43 GMT

Your sequence is correct.

1. Correct. Technically they are two deployments during that interim point of some but not all nodes upgraded.

2. Correct.

3. Correct.

Also, when your upgrade the last node (Primary PAN/MnT of the original deployment) it will join the new deployment as Secondary PAN/MnT. If you want it te reclaim the primary role, you need to log into it and, under the deployment menu, "Promote to Primary". This is an optional step.

Sorry, here is our deployment

joevimal01 — Tue, 16 Aug 2016 02:23:22 GMT

Sorry, here is our deployment :

Node A:Primary PAN/secondary MnT

Node B: Secondary PAN/primary MnT

Node C,D,E and F: PSNs

Thank you so much Marvin. I

joevimal01 — Tue, 16 Aug 2016 02:30:20 GMT

Thank you so much Marvin. I hope the upgrade sequence is still same for our deployment:

here is our deployment :

Node A:Primary PAN/secondary MnT

Node B: Secondary PAN/primary MnT

Node C,D,E and F: PSNs

The upgrade sequence is:

secondary admin node/primary montitoring node
PSN Node
Primary admin node/secondary admin node

Yes, still correct.

Marvin Rhoads — Tue, 16 Aug 2016 02:32:43 GMT

Yes, still correct.

Thank you. i will share my

joevimal01 — Tue, 16 Aug 2016 02:35:47 GMT

Thank you. i will share my experience soon.

We're in the middle of the

Ben.Levin — Tue, 16 Aug 2016 18:34:09 GMT

We're in the middle of the lab upgrade from 2.0 to 2.1, using the GUI upgrade method. It seems better than using the CLI so far. However, my lab deployment only has 2 servers and production has 8, so I'm really not looking forward to how long that is going to take. I'm planning to open a TAC case to make sure we have the procedure down and try to get a better idea of how long it will take to finish all 8 servers. When we upgraded from 1.3 to 2.0, it took us 13 hours.

Hi Marvin,do you know which

Maximilian Usinger — Wed, 17 Aug 2016 07:28:11 GMT

Hi Marvin,

do you know which bugs will be addressed with the upcoming patch for ISE 2.1?
By the way, where can I get some "insider" information about when a update will be released?
As I have some minor bugs (and a missing deny profile for TACACS authorization) I have to deal with, I am waiting desperately for any new update 🙂

@Maximilian Usinger ,

Marvin Rhoads — Wed, 17 Aug 2016 12:47:23 GMT

[@usi.usinger] ,

The missing TACACS Deny profile was fixed in ISE 2.1. That was the following bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy46322

If you have questions about specific bugs that are impacting your environment, you can inquire with your Cisco SE or via the TAC.

Sometimes you can tell that a bug fix is imminent when the Bug search tool shows it as "fixed" but the most recent public release doesn't include the fix. That usually indicates it's been fixed by Development Engineering and is pending release.

@Marvin Rhoads ,Thanks for

Maximilian Usinger — Fri, 19 Aug 2016 08:16:24 GMT

[@mrhoads-cco] ,
Thanks for the reply, but I just found out that I am suffering from this fresh bug here:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva04654/?referring_site=bugquickviewredir

(In case you run in a similiar problem)

I was starting to get crazy, as everyone said this bug should have been fixed, but I couldn't find any deny shell profile. 🙂

Hi Marvin,

joevimal01 — Tue, 13 Sep 2016 10:59:10 GMT

Hi Marvin,

Need your help to clarify below query.

when we use below command to copy the file to ISE(local disk). what is the setup do i need to do on remote PC(where IOS bundle is available), do we need to setup FTP server?

copy repository_url/path/ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz disk:/

Repository_url = Repository Name

Path = server IP address.

Please clarify in detail. Thanks

In the last 2 weeks I have

Martin Bosch — Tue, 13 Sep 2016 14:59:57 GMT

In the last 2 weeks I have upgraded from ISE 1.2 -> 1.3

then 1.3 -> 2.0. finally 2.0 - 2.1.

I used the same repository where I keep my backups.

So in mycase CiscoISE is my repository

"application upgrade prepare ise-upgradebundle-1.3.x-and-1.4.x-to-2.0.0.306.x86_64.tar.gz CiscoISE"

once uploaded

"application upgrade proceed"

but if you have done a upgrade in between remember to use

"application upgrade cleanup"

Make sure you have a backup before you start, including your cert's.