topic Re: Stability of ISE version 3.2? in Network Access Control

Stability of ISE version 3.2?

adamscottmaster2013 — Fri, 12 May 2023 10:19:18 GMT

I am testing out the upgrade on a single node ISE 3.0 patch-7 to ISE 3.2 patch-2. When I run the preparation using ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz, it fails at "Configuration at Data Upgrade". To be clear, this node is up and running just fine with ISE 3.0 patch-7 with no issue when using the ISE 3.1 upgrade bundle ise-upgradebundle-2.6.x-3.0.x-to-3.1.0.518b.SPA.x86_64.tar.gz.

I also rebooted the ISE several times prior to running the preparation check with ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz but it fails every time. This is what I am seeing in the log:

Fri May 12 10:03:13 UTC 2023 : Changing host config entry to standalone...
Fri May 12 10:03:13 UTC 2023 : ORACLE_SID : cpm11
Fri May 12 10:03:13 UTC 2023 : NODECNT :
Fri May 12 10:03:13 UTC 2023 : - Successful
Fri May 12 10:03:13 UTC 2023 : - Successful
Fri May 12 10:03:13 UTC 2023 :
Fri May 12 10:03:13 UTC 2023 : runDBClone method finished executing
Fri May 12 10:03:13 UTC 2023 : triggerUpgradeOnClonedInstance method started executing
Fri May 12 10:03:14 UTC 2023 : Modifying upgrade scripts to run on cloned database
Fri May 12 10:03:14 UTC 2023 : - Successful
Fri May 12 10:03:36 UTC 2023 :
Fri May 12 10:03:36 UTC 2023 : Running schema upgrade on cloned database
Fri May 12 10:06:59 UTC 2023 : - Successful
Fri May 12 10:06:59 UTC 2023 :
Fri May 12 10:06:59 UTC 2023 : Running data upgrade on cloned database
Fri May 12 10:07:52 UTC 2023 : - Failed
Fri May 12 10:07:52 UTC 2023 : ConfigDBUpgrade : Performing Clean-up
Fri May 12 10:08:11 UTC 2023 : ConfigDBUpgrade : Clean-up Completed

really don't want to open a TAC case because it will take forever to get the case to a TAC engineer to help me with this issue, weeks and months rather days.

This makes me question the stability of ISE 3.2. Thoughts?

Re: Stability of ISE version 3.2?

Nancy Saini — Fri, 12 May 2023 13:59:45 GMT

Get the output of "show logging system ade/ADE.log tail" while doing the upgrade. Also, the logs you have shared seems to be of application upgrade proceed and not application upgrade prepare.

Re: Stability of ISE version 3.2?

Arne Bier — Mon, 15 May 2023 20:26:02 GMT

Please let us know the outcome of this - I was looking at a 3.0 to 3.2 - but now I am inclined to rather rebuild the entire deployment from scratch - get rid of years of corruption and also a TAC case that has been open for over a year to sort out why the Profiler Feed doesn't update my Policies. A rebuild gets rid of a lot of junk. And I also tested in the lab that you can export and import Profiling Policies - the nice thing is, that this also includes all the dependencies - will save some time.

Re: Stability of ISE version 3.2?

Marcelo Morais — Mon, 15 May 2023 23:05:00 GMT

Hi @adamscottmaster2013 ,

Cisco released on May 9th a new ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64, please double check if you are using this new Upgrade Bundle:

What is the result when you use URT (ise-urtbundle-3.2.0.542a-1.0.0.SPA.x86_64.tar.gz) ?

Note: I tested the upgrade from ISE 2.7 P8 to 3.2 P1 without issues (using the upgrade bundle and also using backup/restore method) ... I totally agree with @Arne Bier about " ... A rebuild gets rid of a lot of junk ... " !!!

Hope this helps !!!

Re: Stability of ISE version 3.2?

adamscottmaster2013 — Tue, 16 May 2023 01:11:27 GMT

Hi @Marcelo Morais,

Yes, I am using this bundle ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

md5sum ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz
ac89dee5fa86377836a497c981d7480c ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

I also have a TAC case open with Cisco on another issue after importing 3.0 configuration into 3.2 patch-2 system. Things just stops working after that for some of the most basic functions. I suspect that ISE 3.2 is not widely used yet so there are lot of bugs that are still in the system.

Re: Stability of ISE version 3.2?

Marcelo Morais — Tue, 16 May 2023 01:59:57 GMT

Hi @adamscottmaster2013 ,

did you upgrade via CLI ? If the answer is no, could you please try via CLI ?

Just an example:

ise/admin# application upgrade prepare ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542a.SPA.x86_64.tar.gz <REPOSITORY>
Be sure that all your software is working stable, check your system on UI page (Administration > System > Health Checks) 
Type yes once confirmed that health of the system is good to proceed: (yes/no) [yes] ? yes

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...

Application upgrade preparation successful

Hope this helps !!!

Re: Stability of ISE version 3.2?

adamscottmaster2013 — Tue, 16 May 2023 10:37:33 GMT

@Marcelo Morais: Ofcourse, I tried both the CLI and UI. Both failed.

Re: Stability of ISE version 3.2?

Marcelo Morais — Tue, 16 May 2023 12:40:16 GMT

@adamscottmaster2013 ,

good to know ... two last questions:

your version was ISE 3.0 P7, but P7 was the only patch installed (show version command) ?
was there any Hot Patch (show logging application hotpatch.log or show version history command) ?

Thanks

Re: Stability of ISE version 3.2?

adamscottmaster2013 — Tue, 16 May 2023 12:55:30 GMT

@Marcelo Morais: I originally had the ISE 3.0 patch-3 and everything was working fine. I patched the box with patch-5 a few months ago to fix some ssh issue. Then I patched it with P7 a few days ago before upgrading it to 3.2. No hot patch on the system.

Re: Stability of ISE version 3.2?

adamscottmaster2013 — Fri, 02 Jun 2023 13:35:37 GMT

@Arne Bier : Unfortunately I have no good news to report. TAC support is still looking into the issue. Today marks the 21 days since I opened the first ticket and still has no solution from Cisco TAC.

Just want to give you an update to this. ISE version 3.2, even with the latest patch-2,
is full of bugs and not definitely for prime time. I have multiple TAC cases open with Cisco
and they are to reproduce the issue and they are currently working through them. I really don't
want to use ISE 3.1 witch patches because the life cycle of ISE 3.1 will be a lot shorter than
version 3.2. I would rather use version 3.2 now so that I don't have to deal with another upgrade
for another three years. I don't think there are many ISE 3.2 deployment so there are not that
many feedbacks to Cisco, and because of that, Cisco can't fix what it doesn't know.

Here are some of the issues I ran into with version 3.2:

- can no longer ssh into the ISE. Had to toggle/untoggle the ACL for UI and CLI,

- ISE application crashes whenever I need to edit the network devices such as changing the network
device name or pre-share key. As soon as I hit the save button, the application server crashes,

- Integration with Active Directory does not work,

- Integration with External radius server does not work. It works if I clone the existing external radius
but as soon the application crashes, it stops working,

- Radius service is not responding. This is a big red flag,

I am sure there are many other things that are not working but I have not been able to test
due to all of the above issues I am having with.

Re: Stability of ISE version 3.2?

Arne Bier — Fri, 02 Jun 2023 21:26:11 GMT

Oh dear that is not good feedback. Did the URT eventually run clean?

See, this is why I plan to build 3.2 from scratch. No upgrade. I still don’t trust them.
I have had 3.2 running in lab which was built from scratch and no issues.

Re: Stability of ISE version 3.2?

marco.merlo — Tue, 05 Sep 2023 07:55:24 GMT

Hi,

Actually 3.1and 3.2 life cycle should have the same duration , I am pretty sure that there no longer difference between odd and even subversions life cycle.
Now it seems that 3.2 version is the recommended one (the "star" is apperead near the name) but personally I am struggling with a 2.4 -> 3.1 migration since November 2022. I chose to built a parallel deployment: being our PSNs served by a load balancer the actual migration would have been just a matter of NLB configuration
When I started 3.1 was the reccomanded version but during tests I met lots of bugs resulting in at least 5 cases opened with TAC. I think the Cisco should "reset" the beginning of 3.1 lyfe cycle to April 2023 because at least one bug was destructive and IMHO that version was not ready for production in complex environment before patch 6. Anyway I have still an opened case that gets back to January 2023 so I am afraid I am going to upgrade my parallel deployment to 3.2. I am quite upset because I met very similar bugs in version 2.x and my project is almost one year late ....

Regards

Marco

p.s urt run successfully on my ISE 3.1 deployment

Re: Stability of ISE version 3.2?

Tiroyaone72926925 — Sun, 17 Sep 2023 22:57:48 GMT

I have been using 3.2 for the past 2 months, unfortunately i dont have good news about. I even patched to the latest patch 3, but it did not improve anything. A week cant pass without the application service for the primary admin node restarting. and when it restarts during production, it takes for over an hour initializing, and you cant ssh to it when its initializing. i have opened a tac case with ISE engineers, i keep on sending logs but with no solution. the main problem is application services, and when you reboot the node, it takes up to an hour to restart. i hope they could resolve it, because going back to 3.1 is not an option

Re: Stability of ISE version 3.2?

patherton — Tue, 19 Sep 2023 17:27:39 GMT

I also have been having a lot of issues with 3.2 I was on 3.0, and 3.1 previously without issues. I've even rebuilt my nodes, which I did previously for 3.0. I am disappointed that it shows up as a recommended version.

Re: Stability of ISE version 3.2?

adamscottmaster2013 — Tue, 19 Sep 2023 17:53:00 GMT

@patherton & @Tiroyaone72926925: Can you provide specifically issues you're facing with? I am about to deploy ISE 3.2 patch-3 in my production in replacing the legacy ISE 3.0 and you guys are making me nervous.

TIA

Re: Stability of ISE version 3.2?

Arne Bier — Tue, 19 Sep 2023 20:30:17 GMT

Everyone will have their own horror stories but that doesn't mean there is a fundamental problem with that version of ISE. I have good news so far (touch wood).

I built a customer two node guest wifi solution recently, and granted, it's just RADIUS and Portals, but it's working beautifully. The only issue I had so far has been with SNMPv3, which is a real developer hack-job. It seems to be a bit better in ISE 3.3.

I see a real use-case for the new API in ISE 3.3 now. In an ideal world, I would like to have the entire ISE VM build process, cert creation and deploy, and config in something like Ansible. The Cisco TMEs (Charlie and Thomas) have great YouTube videos on that - it's an eye-opener! The dream seems to be possible now. I think the initial creation of such a playbook would be hard work, but in the end, we could blow away any version of ISE, and rebuild from ground zero. Never again having to restore from a config backup (which I believe is partly to blame for issues seen in ISE deployments). The other culprit in VM deployments is potentially VM Snapshots (happening without our knowledge or consent) and live vMotions (again, happening without our knowledge). Throw in the odd power failure here and there (disk corruption) and you have a nice mess in the database.

How do I know this? 99% of the time when I have gotten to the end of a TAC case, and the engineers run the CLI root access, they invariably fix something in the Oracle database.

My gut feeling is also that, under the hood, ISE is a pile of cobbled together Java code that is haemorrhaging errors at an alarming rate. I am amazed it even works. And it's disappointing to thinks how inefficient the whole thing is (why does it take 10 min to start application services with all that horsepower??). Look at the logs of a system that is "running smoothly" (as far as you're concerned) and you'll be amazed at how many ERROR messages you see logged 24/7.

Let's not forget that despite all that, when we're working in the Policy Set and making new Profiler Policies, etc, ISE is a great tool. When it does the job, it does it well. And if we find issues then we should report them to TAC to allow them to fix it.

Re: Stability of ISE version 3.2?

marco.merlo — Thu, 21 Sep 2023 08:22:03 GMT

Hi Aren,

personally I had been able to migrate our AAA servers from ACS 4.x to ACS 5.3 then to ISE 2.3, ISE 2.4 . Every time headaches seem to increase along with the release number. I am stuck with 3.x migration since last December. First I tried with at the time reccomended version that was 3.1, I had to open 6 SR to TAC, two of which were blocking ones. At the beginning of September, after waiting for the TAC 8 months to solve a minor issue with 3.1, I decided to upgrade to the new reccomended release before actual migration from 2.4 . The very next day I had to open other 2 SRs , one of which was identical to one I had opened for version 3.1 . Please note that we are using ISE just as radius server to perform basic wired/wireless 802.1x authentication and wireless guest authentication. I can't even imagine how many SRs I would have had to open if we used advanced feature as PIXGRID or profiling. Yes ISE is a good product and actually we did not meet any critical issue with 2.4 but maybe It would be the same with freeradius or Microsoft NPS ...

Re: Stability of ISE version 3.2?

marco.merlo — Thu, 21 Sep 2023 08:48:35 GMT

Hi,

I upgrade from 3.1 last patch to 3.2 patch 3 with "full upgrade procedure" since the NADs are still pointing to our 2.4 deployment . So far I meet three issues:

1)mnt reports stopped to work, I am not able to say whether it was because of the upgrade or because the restore of the previous MNT database but reports show just pre upgrade data

2)live logs seem to register just session events due to accounting packets but not auth events even if some fail attempt seems to be logged randomly

3) For sponsor portal we hit this bug https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd97582 we alredy met in 3.1 and that was actually solved.

Of course if your nodes are not dual homed you have not to worry about 3).

Regards

Re: Stability of ISE version 3.2?

Arne Bier — Thu, 21 Sep 2023 20:50:42 GMT

I feel that the more we discuss these upgrade issues, the more we're becoming desensitized to this topic, and eventually so worn down by it that we just accept that it will happen. That's how I feel anyway. When I upgrade my Windows or iPhone I don't give it a second thought. Imagine having to worry if you can login after a patch/upgrade or whether your phone can still send an SMS after upgrading to iOS 17? Seriously.

There is no excuse for Cisco to keep breaking the fundamentals of ISE (RADIUS/802.1X/TACACS+) - but this is an ongoing theme. The customer can't get access to the Linux CLI to even help themselves when things go wrong. We are beholden to the TAC and their slow processes to fix issues.

My solution to this is to never upgrade ISE in the traditional way (using the upgrade bundle). Sad but true, I have chosen to always build new VMs (or re-image the SNS) because the upgrade process is such a dumpster fire. Maybe it will work better in ISE 5.x - but by then we'll all have our Ansible Playbooks ready to deploy our ISE as "Infrastructure as code" - no need to upgrade.

My personal view is that ISE has become a sprawl of spaghetti code that is now so hard to maintain, that even Cisco have no way to keep quality control over it. One small change here breaks something else over there. Maybe they should consider re-working the architecture from scratch instead of piling onto that house of cards. Customers are the test bed, and these forums are proof of that. Many of us don't deploy cutting edge ISE features - we just want the basic stuff from version 1.0 - it should be taken for granted that this is rock-solid and maintained to the highest quality assurance testing in Cisco.

Re: Stability of ISE version 3.2?

adamscottmaster2013 — Fri, 22 Sep 2023 14:39:13 GMT

The worst mistake Cisco did to the ISE is to run Oracle database in the backend. mySQL, MongoDB or Postgres would have been a much better choice than Oracle.