Re: Stability of ISE version 3.2?

adamscottmaster2013 · ‎05-12-2023

I am testing out the upgrade on a single node ISE 3.0 patch-7 to ISE 3.2 patch-2. When I run the preparation using ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz, it fails at "Configuration at Data Upgrade". To be clear, this node is up and running just fine with ISE 3.0 patch-7 with no issue when using the ISE 3.1 upgrade bundle ise-upgradebundle-2.6.x-3.0.x-to-3.1.0.518b.SPA.x86_64.tar.gz.

I also rebooted the ISE several times prior to running the preparation check with ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz but it fails every time. This is what I am seeing in the log:

Fri May 12 10:03:13 UTC 2023 : Changing host config entry to standalone...
Fri May 12 10:03:13 UTC 2023 : ORACLE_SID : cpm11
Fri May 12 10:03:13 UTC 2023 : NODECNT :
Fri May 12 10:03:13 UTC 2023 : - Successful
Fri May 12 10:03:13 UTC 2023 : - Successful
Fri May 12 10:03:13 UTC 2023 :
Fri May 12 10:03:13 UTC 2023 : runDBClone method finished executing
Fri May 12 10:03:13 UTC 2023 : triggerUpgradeOnClonedInstance method started executing
Fri May 12 10:03:14 UTC 2023 : Modifying upgrade scripts to run on cloned database
Fri May 12 10:03:14 UTC 2023 : - Successful
Fri May 12 10:03:36 UTC 2023 :
Fri May 12 10:03:36 UTC 2023 : Running schema upgrade on cloned database
Fri May 12 10:06:59 UTC 2023 : - Successful
Fri May 12 10:06:59 UTC 2023 :
Fri May 12 10:06:59 UTC 2023 : Running data upgrade on cloned database
Fri May 12 10:07:52 UTC 2023 : - Failed
Fri May 12 10:07:52 UTC 2023 : ConfigDBUpgrade : Performing Clean-up
Fri May 12 10:08:11 UTC 2023 : ConfigDBUpgrade : Clean-up Completed

really don't want to open a TAC case because it will take forever to get the case to a TAC engineer to help me with this issue, weeks and months rather days.

This makes me question the stability of ISE 3.2. Thoughts?

Nancy Saini · ‎05-12-2023

Get the output of "show logging system ade/ADE.log tail" while doing the upgrade. Also, the logs you have shared seems to be of application upgrade proceed and not application upgrade prepare.

Arne Bier · ‎05-15-2023

Please let us know the outcome of this - I was looking at a 3.0 to 3.2 - but now I am inclined to rather rebuild the entire deployment from scratch - get rid of years of corruption and also a TAC case that has been open for over a year to sort out why the Profiler Feed doesn't update my Policies. A rebuild gets rid of a lot of junk. And I also tested in the lab that you can export and import Profiling Policies - the nice thing is, that this also includes all the dependencies - will save some time.

adamscottmaster2013 · ‎06-02-2023

@Arne Bier : Unfortunately I have no good news to report. TAC support is still looking into the issue. Today marks the 21 days since I opened the first ticket and still has no solution from Cisco TAC.

Just want to give you an update to this. ISE version 3.2, even with the latest patch-2,
is full of bugs and not definitely for prime time. I have multiple TAC cases open with Cisco
and they are to reproduce the issue and they are currently working through them. I really don't
want to use ISE 3.1 witch patches because the life cycle of ISE 3.1 will be a lot shorter than
version 3.2. I would rather use version 3.2 now so that I don't have to deal with another upgrade
for another three years. I don't think there are many ISE 3.2 deployment so there are not that
many feedbacks to Cisco, and because of that, Cisco can't fix what it doesn't know.

Here are some of the issues I ran into with version 3.2:

- can no longer ssh into the ISE. Had to toggle/untoggle the ACL for UI and CLI,

- ISE application crashes whenever I need to edit the network devices such as changing the network
device name or pre-share key. As soon as I hit the save button, the application server crashes,

- Integration with Active Directory does not work,

- Integration with External radius server does not work. It works if I clone the existing external radius
but as soon the application crashes, it stops working,

- Radius service is not responding. This is a big red flag,

I am sure there are many other things that are not working but I have not been able to test
due to all of the above issues I am having with.

Arne Bier · ‎06-02-2023

Oh dear that is not good feedback. Did the URT eventually run clean?

See, this is why I plan to build 3.2 from scratch. No upgrade. I still don’t trust them.
I have had 3.2 running in lab which was built from scratch and no issues.

marco.merlo · ‎09-05-2023

Hi,

Actually 3.1and 3.2 life cycle should have the same duration , I am pretty sure that there no longer difference between odd and even subversions life cycle.
Now it seems that 3.2 version is the recommended one (the "star" is apperead near the name) but personally I am struggling with a 2.4 -> 3.1 migration since November 2022. I chose to built a parallel deployment: being our PSNs served by a load balancer the actual migration would have been just a matter of NLB configuration
When I started 3.1 was the reccomanded version but during tests I met lots of bugs resulting in at least 5 cases opened with TAC. I think the Cisco should "reset" the beginning of 3.1 lyfe cycle to April 2023 because at least one bug was destructive and IMHO that version was not ready for production in complex environment before patch 6. Anyway I have still an opened case that gets back to January 2023 so I am afraid I am going to upgrade my parallel deployment to 3.2. I am quite upset because I met very similar bugs in version 2.x and my project is almost one year late ....

Regards

Marco

p.s urt run successfully on my ISE 3.1 deployment

Tiroyaone72926925 · ‎09-17-2023

I have been using 3.2 for the past 2 months, unfortunately i dont have good news about. I even patched to the latest patch 3, but it did not improve anything. A week cant pass without the application service for the primary admin node restarting. and when it restarts during production, it takes for over an hour initializing, and you cant ssh to it when its initializing. i have opened a tac case with ISE engineers, i keep on sending logs but with no solution. the main problem is application services, and when you reboot the node, it takes up to an hour to restart. i hope they could resolve it, because going back to 3.1 is not an option

Marcelo Morais · ‎05-15-2023

Hi @adamscottmaster2013 ,

Cisco released on May 9th a new ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64, please double check if you are using this new Upgrade Bundle:

ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.png

What is the result when you use URT (ise-urtbundle-3.2.0.542a-1.0.0.SPA.x86_64.tar.gz) ?

Note: I tested the upgrade from ISE 2.7 P8 to 3.2 P1 without issues (using the upgrade bundle and also using backup/restore method) ... I totally agree with @Arne Bier about " ... A rebuild gets rid of a lot of junk ... " !!!

Hope this helps !!!

adamscottmaster2013 · ‎05-15-2023

Hi @Marcelo Morais,

Yes, I am using this bundle ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

md5sum ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz
ac89dee5fa86377836a497c981d7480c ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

I also have a TAC case open with Cisco on another issue after importing 3.0 configuration into 3.2 patch-2 system. Things just stops working after that for some of the most basic functions. I suspect that ISE 3.2 is not widely used yet so there are lot of bugs that are still in the system.

Marcelo Morais · ‎05-15-2023

Hi @adamscottmaster2013 ,

did you upgrade via CLI ? If the answer is no, could you please try via CLI ?

Just an example:

ise/admin# application upgrade prepare ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542a.SPA.x86_64.tar.gz <REPOSITORY>
Be sure that all your software is working stable, check your system on UI page (Administration > System > Health Checks) 
Type yes once confirmed that health of the system is good to proceed: (yes/no) [yes] ? yes

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...

Application upgrade preparation successful

Hope this helps !!!

adamscottmaster2013 · ‎05-16-2023

@Marcelo Morais: Ofcourse, I tried both the CLI and UI. Both failed.

Marcelo Morais · ‎05-16-2023

@adamscottmaster2013 ,

good to know ... two last questions:

your version was ISE 3.0 P7, but P7 was the only patch installed (show version command) ?
was there any Hot Patch (show logging application hotpatch.log or show version history command) ?

Thanks

adamscottmaster2013 · ‎05-16-2023

@Marcelo Morais: I originally had the ISE 3.0 patch-3 and everything was working fine. I patched the box with patch-5 a few months ago to fix some ssh issue. Then I patched it with P7 a few days ago before upgrading it to 3.2. No hot patch on the system.

patherton · ‎09-19-2023

I also have been having a lot of issues with 3.2 I was on 3.0, and 3.1 previously without issues. I've even rebuilt my nodes, which I did previously for 3.0. I am disappointed that it shows up as a recommended version.

adamscottmaster2013 · ‎09-19-2023

@patherton & @Tiroyaone72926925: Can you provide specifically issues you're facing with? I am about to deploy ISE 3.2 patch-3 in my production in replacing the legacy ISE 3.0 and you guys are making me nervous.

TIA