topic Re: ISE v3.4 Patch 1 - Exercise caution in Network Access Control

ISE v3.4 Patch 1 - Exercise caution

dal — Sat, 28 Dec 2024 11:40:57 GMT

Take a backup before you do an upgrade to the new Patch 1 for ISE v3.4

My installation did not start up afterwards.

I even did a fresh install of v3.4 and applied the patch.. and the installation did not start even then.

I also tried to roll back the patch, and that failed too

Cannot see anything sticking out in the logs, but then again; searching the ISE logs are not easy

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Sat, 28 Dec 2024 13:33:27 GMT

Hi @dal ,

I successfully upgrade an ISE 3.3 P4 to ISE 3.4 P1 without a problem:

ise/admin# show version history 
---------------------------------------------
Install Date: Fri Sep 27 13:58:06 -03 2024
Application: ise
Version: 3.3.0.430
Install type: Application Install
Bundle filename: ise.tar.gz
Repository: SystemDefaultPkgRepos

---------------------------------------------
Install Date: Fri Sep 27 17:27:56 -03 2024
Application: ise
Version: 2
Install type: Patch Install
Bundle filename: ise-patchbundle-3.3.0.430-Patch2-24041511.SPA.x86_64.tar.gz
Repository: LOCAL

---------------------------------------------
Install Date: Thu Nov 7 11:30:03 -03 2024
Application: ise
Version: 4
Install type: Patch Install
Bundle filename: ise-patchbundle-3.3.0.430-Patch4-24102504.SPA.x86_64.tar.gz
Repository: LOCAL

---------------------------------------------
Install Date: Sun Dec 22 12:02:13 -03 2024
Application: urt
Version: 1.0.0
Install type: Application Install
Bundle filename: ise-urtbundle-3.4.0.608-1.0.0.SPA.x86_64.tar.gz
Repository: LOCAL

---------------------------------------------
Install Date: Sun Dec 22 12:04:27 -03 2024
Application: urt
Version: 1.0.0
Install type: Application Remove
Bundle filename: NA
Repository: NA

---------------------------------------------
Install Date: Sun Dec 22 14:39:46 -03 2024
Application: ise
Version: 3.4.0.608
Install type: Application Upgrade
Bundle filename: ise-upgradebundle-3.1.x-3.3.x-to-3.4.0.608a.SPA.x86_64.tar.gz

---------------------------------------------
Install Date: Sun Dec 22 15:37:02 -03 2024
Application: ise
Version: 1
Install type: Patch Install
Bundle filename: ise-patchbundle-3.4.0.608-Patch1-24121602.SPA.x86_64.tar.gz
Repository: LOCAL

What was the Version & Patch of your ISE before the 1st installation for ISE 3.4 P1 ?

A fresh install to ISE 3.4 and an update to P1 without Data Restore and ISE didn't Start, is my understanding correct ?

Best regards

Re: ISE v3.4 Patch 1 - Exercise caution

dal — Sat, 28 Dec 2024 14:10:41 GMT

I have tried several times, same result.

First time I tried to patch my v3.4 installation
When that failed, I have tried several times (installation from OVA and ISO) to patch a clean v3.4 install, and it fails each time.

If I were Cisco, I would pull this patch

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Sat, 28 Dec 2024 15:17:25 GMT

Hi @dal ,

in other words, you are able to do the following:

ise/admin# patch install ise-patchbundle-3.4.0.608-Patch1-24121602.SPA.x86_64.tar.gz LOCAL

% Warning: Patch will be installed only on this node. Install using Primary Administration node GUI to install on all nodes in deployment.
Continue? (yes/no) [yes] ? yes

Initiating Application Patch installation...
Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Patch successfully installed

% This application Install or Upgrade requires reboot, rebooting now...

am I correct ?

Note: new ISE 3.4 Software was released on 18-Dec-2024 with ISE 3.4 P1:

Best regards.

Re: ISE v3.4 Patch 1 - Exercise caution

dal — Sat, 28 Dec 2024 17:24:34 GMT

That is correct, I was able to install the patch, and got the successfull message

Re: ISE v3.4 Patch 1 - Exercise caution

dal — Sun, 29 Dec 2024 00:16:43 GMT

I can also confirm that the new 608a image does NOT include P1:

Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version : 3.4.0.608
Build Date : Wed Jul 31 09:25:40 2024
Install Date : Sun Dec 29 00:20:43 2024

Can't speak for the upgrade bundle, though

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Sun, 29 Dec 2024 00:55:11 GMT

Hi @dal ,

this is "By Design", in other words, no Cisco ISE ISO image or Upgrade Bundle has a Patch included in it .... but ... it's important to use the latest ISO or Upgrade Bundle whenever possible !!!

Re: ISE v3.4 Patch 1 - Exercise caution

Marcus Hunold — Sat, 25 Jan 2025 18:11:17 GMT

Unfortunately I have to confirm that post.
I got the same sh...t. Saw this post aftwards of course 😉 but it is Cisco...take a snapshot in advance meanwhile always...

FYI my Server was a fresh install of 3.4 with a restore of a 3.2 backup. The 3.4 NoPatch ISE worked in production the last 3 months.

Patch via GUI failed, services don't come up after reload (see screenshots before you ask questions)
Patch via CLI failed, services don't come up after reload (see screenshots before you ask questions)

Result: Revert to Snapshot without Patch.

Re: ISE v3.4 Patch 1 - Exercise caution

Scott Fella — Sat, 25 Jan 2025 22:42:40 GMT

Interesting, because I patched my v3.4 to patch 1 and didn't run into any issues. I'm going to spin up a new VM and test it again.

wa-red-ise-01/admin#show version history
---------------------------------------------
Install Date: Sun Nov 24 16:40:23 UTC 2024
Application: ise
Version: 3.4.0.608
Install type: Application Install
Bundle filename: ise.tar.gz
Repository: SystemDefaultPkgRepos

---------------------------------------------
Install Date: Tue Dec 24 04:06:19 UTC 2024
Application: ise
Version: 1
Install type: Patch Install
Bundle filename: ise-patchbundle-3.4.0.608-Patch1-24121602.SPA.x86_64.tar.gz
Repository: tmplocalpatchinstallrepo

**** Update ****

Well I spun up a new ISE v3.4 and patched it with no issues.

(view in My Videos)

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Sat, 25 Jan 2025 20:27:39 GMT

Hi @Marcus Hunold ,

I'm testing Cisco ISE 3.4 P1 since Jan 6th, so far so good ...

<ISE Hostname>/admin# show version
 Cisco Application Deployment Engine OS Release: 3.4
 ADE-OS Build Version: 3.4.P.010
 ADE-OS System Architecture: x86_64

 Copyright (c) 2005-2023 by Cisco Systems, Inc.
 All rights reserved.
 Hostname: <ISE Hostname>

 Version information of installed applications
 ---------------------------------------------
 
 Cisco Identity Services Engine
 ---------------------------------------------
 Version : 3.4.0.608
 Build Date : Wed Jul 31 04:25:40 2024
 Install Date : Mon Jan 6 13:29:43 2025

 Cisco Identity Services Engine Patch 
 ---------------------------------------------
 Version : 1
 Install Date : Mon Jan 06 14:08:36 2025

Since Cisco ISE 3.4 have parity with Cisco ISE: 3.3 P3, 3.2 P6, and 3.1 P9, I prefer to:

update my ISE 3.2 to ISE 3.2 P6
backup my ISE 3.2 P6
install from scratch an ISE 3.4
restore the ISE 3.2 P6 backup to ISE 3.4
update my ISE 3.4 to ISE 3.4 P1

Note: I also upgrade from ISE 3.3 P4 to ISE 3.4 P1, similar procedure.

Hope this helps !!!

Re: ISE v3.4 Patch 1 - Exercise caution

Pawel Przybyszewski — Sun, 26 Jan 2025 04:58:32 GMT

I also can confirm that after installing P1 application does't start. We will raise request to TAC.

Many Oracle errors in logs, even ADE OS functions don't work correctly.

BTW in the past we had many troubles after upgrade to 3.4 version. This version is unfinished :(.

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Sun, 26 Jan 2025 12:52:23 GMT

Hi @Pawel Przybyszewski , @dal , and @Marcus Hunold ,

could you please tell us more about the Upgrade Procedure ?

1. What was your Cisco ISE Version & Patch before the upgrade ?

2. Did you have any Hotpatch installed ?

Best regards !!!

Re: ISE v3.4 Patch 1 - Exercise caution

Scott Fella — Sun, 26 Jan 2025 18:39:50 GMT

I'm still curious on this and would like to also hear your response on the questions @Marcelo Morais posted.

In my case, I used the ova (Cisco-vISE-300-3.4.0.608.ova) and then patched using ise-patchbundle-3.4.0.608-Patch1-24121602.SPA.x86_64.tar.gz. I was able to patch and rollback multiple times with no issue. Now my testing was a fresh install and patching, I have an ISE v3.4 in my home network that I spun up and did a restore from a previous ISE v3.3 and was able to patch that with no issues. Both are running in VMWare ESXi 7.x hosts.

Re: ISE v3.4 Patch 1 - Exercise caution

Pawel Przybyszewski — Mon, 27 Jan 2025 06:36:34 GMT

In my case:

1. What was your Cisco ISE Version & Patch before the upgrade ? Upgrade from 3.1P9 to 3.4 in August 2024.

2. Did you have any Hotpatch installed ? No patch available other than 1 for 3.4, but before upgrade to 3.4 patch ise-apply-CSCwk61938_3.1_patchall-SPA.tar.gz was installed in end of July 2024.

3. Did you use the ise-3.4.0.608a.SPA.x86_64.iso or ise-upgradebundle-3.1.x-3.3.x-to-3.4.0.608a.SPA.x86_64.tar.gz (both released on Dec 18th) for the upgrade (or did you use an older file, e.g.: ise-3.4.0.608.SPA.x86_64.iso, released on Aug 1st) ? I used older file because upgrade to 3.4 was in August. After upgrade we had some troubles (HA flapped for RADIUS dot1.x and user authenticated on one instances denied on other during reauthentication, but a month all was fine).

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Mon, 27 Jan 2025 12:42:54 GMT

Hi @Pawel Przybyszewski ,

thanks ... in other words:

HOTPATCH: ise-apply-CSCwk61938_3.1_patchall-SPA.tar.gz (CSCwk61938 ISE Evaluate OpenSSH CVE-2024-6387 "regreSSHion")
OLD ISE ISO: ise-3.4.0.608.SPA.x86_64.iso (released on Aug 1st)

Since last month, everything has been fine, correct ?

If the answer is Yes, what did you do "to become fine" ?

Regards

Re: ISE v3.4 Patch 1 - Exercise caution

Pawel Przybyszewski — Mon, 27 Jan 2025 12:52:49 GMT

From about 20th September all was fine. After upgrade RADIUS service fapped betwee 2 ISE instances (timeouts). We increased RAM from 32 to 64GB, configured cache DNS, few times reload and ISE fixed self. Cisco TAC verified configuration and all was right.

In show tech-support I saw some Oracle errors both after upgrade and now after installed P1 also see Oracle and permission errors. TAC SR has been raised.

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Mon, 27 Jan 2025 15:13:05 GMT

Hi @Pawel Przybyszewski ,

interesting ...

You increased the RAM from 32 GB to 64 GB, so your ISE Profile was a SNS 3615 and now is a SNS 3595, correct ?

In Operations > Reports > Reports > Endpoints and Users > Authentication Summary > please take a look at Authentication by Day and Quick Link, what is you "Total Number of Authentications" (Pass & Fail) in a day ?

Regards.

Re: ISE v3.4 Patch 1 - Exercise caution

Pawel Przybyszewski — Tue, 28 Jan 2025 08:13:49 GMT

Endpoints - Pass 2748 Fail 23; device admin 300 Fail 0 - it's small deployment in HA (2 ISE VM in different localization).

At this moment 16CPU/64GB RAM/600GB HDD VM on ESXi - additional 32GB probably doesn't matter because small instance should use 32, medium 96. 64GB is unsupported (EoL) SNS 3595.

Re: ISE v3.4 Patch 1 - Exercise caution

Pawel Przybyszewski — Tue, 28 Jan 2025 08:17:55 GMT

It seems you tested on empty application (no authentications in Dashboard).

TAC found bug in Patch 1 with database credentials in existing HA deployment. Issue is under investigation.

Re: ISE v3.4 Patch 1 - Exercise caution

Marcelo Morais — Tue, 28 Jan 2025 10:11:32 GMT

Hi @Pawel Przybyszewski ,

thanks ...

I asked about the Total Number of Authentications per Day, because even a Small Deployment can generate many Authentications per Day ... it's not your case (only 3K Authentication per Day)

Your Hardware was compatible with a SNS 3615 (16vCPU & 32GB RAM) but you need to add more RAM (32 GB more) to everything goes fine ... interesting ...

Since you have 2x Nodes, can you test both Nodes as Standalone ? I'm wondering if the Replications of a Distributed Node has any impact on your Deployment.

Note: please take a look at ISE - Slow Replication, maybe you can find useful info.

Regards