topic Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing in Network Access Control

ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Thu, 31 Aug 2023 10:14:22 GMT

We are trying to patch our ISE nodes with patch number 3: - ise-patchbundle-3.2.0.542-Patch3-23071904.SPA.x86_64.tar.gz

we have a 2 node environment

i logged onto the primary node
administration
selected maintenance
patch management
selected the patch and clicked install.... at this point after about 60s i got logged out of the web UI.

i decided to log onto the node via cli
ran the following
show application status ise - all processes/services running as expected
show version - Patch 3 not showing up
show uptime - node has not been rebooted

I logged onto the 2nd node and under the upgrade section i saw this.

so i thought ok something is happening.
unfortunately this has now been like this for close to 24hrs and ive also restarted the 2nd node just to see if this would clear.

i ran the same cli commands on node 2 and they are exactly the same as node 1.
services are running, patch 3 not installed and node has not been rebooted.

something is definitely not playing ball and im not sure what the best way forward is or what our options are.

1. should i manually patch node 2 and then failover the services to node 2 making it the primary
2. once node 2 has been patched, reboot and patch node 1
3. retry patch install via web UI
4. bin patch 3 and try patch 2
5. a different approach?

i know that patching via cli will not patch both nodes and each node has to be done individually.
my concern is that because node 2 is showing this "upgrade progress bar", if I failover services to node 2 it could break ISE.

has anybody experienced something like this with patching?
thoughts on what we can do?

I have a case with TAC but waiting for a reply... but its slow going

cheers!

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

ahollifield — Thu, 31 Aug 2023 13:41:00 GMT

Yeah not sure why you are seeing that progress bar on the secondary node. That page in the UI is only for major version upgrades, not patch installs. I would personally open a TAC case as well as planning on attempting the patch installation from the CLI.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

Mark Elsen — Thu, 31 Aug 2023 16:20:52 GMT

- Check disk space on all involved nodes with (CLI) : show disks
Also check outputs from : show logging system ade/ADE.log
show logging application ise-psc.log

Also read : https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/220195-troubleshoot-identity-services-engine-i.html

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

adamscottmaster2013 — Thu, 31 Aug 2023 16:55:32 GMT

@TedB123: Welcome to patch-3 patching misery. I have a similar issue with patch-3 and the case has been opened for almost three weeks.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

adamscottmaster2013 — Thu, 31 Aug 2023 16:56:50 GMT

@ahollifield: Patch installation from the CLI is a BAD idea. You will definitely create more problems down the road for yourself.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Fri, 01 Sep 2023 09:50:11 GMT

TAC advised me to select option 27 from cli to clear the upgrade bar
this option here...

[27]Reset Upgrade Tables

but this hasnt done anything, the progress bar is still there... so i will need to get back to them and let them know.

they also advised to update each node individually via cli... maybe once the 2nd node is patched that progress bar will clear.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Fri, 01 Sep 2023 09:51:40 GMT

@Mark Elsen wrote:

- Check disk space on all involved nodes with (CLI) : show disks
Also check outputs from : show logging system ade/ADE.log
show logging application ise-psc.log
Also read : https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/220195-troubleshoot-identity-services-engine-i.html
M.

yeah ive done all that.. nothing in the logs to indicate that any patches have been installed

wouldnt be surprised if ive hit a bug thats prevented the patches from being installed via the web UI.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Fri, 01 Sep 2023 09:53:06 GMT

@adamscottmaster2013 wrote:
@ahollifield: Patch installation from the CLI is a BAD idea. You will definitely create more problems down the road for yourself.

can you provide more detail as to why installing the patch via CLI is a BAD idea?

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

Mark Elsen — Fri, 01 Sep 2023 09:59:43 GMT

>..also advised to update each node individually via cli..
- I prefer that too , to get a better overview on what is happening

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

ahollifield — Fri, 01 Sep 2023 13:15:39 GMT

WHAT???? Please elaborate?

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

hslai — Fri, 01 Sep 2023 22:00:18 GMT

@TedB123 If possible, please share your TAC case number.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

adamscottmaster2013 — Sat, 02 Sep 2023 13:23:45 GMT

@TedB123: It is a BAD idea because doing it from the CLI is that it might be masking other issues that you might have between nodes themselves. That's why TAC always tells you the "preferred" method is to do it from the UI.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

ahollifield — Sat, 02 Sep 2023 14:37:32 GMT

? But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues, or overall progress. Also you cannot control the order of the patch install. What if you wanted to update certain PSNs before others? You do not have this option in the UI….

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

Mark Elsen — Sat, 02 Sep 2023 16:50:51 GMT

@adamscottmaster2013 >...It is a BAD idea because doing it from the CLI is that it might be masking other issues that you might have between nodes themselves.
@ahollifield >... But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues,

- Indeed patching with the CLI is much better ; actually it's better to get rid of potential issues between nodes and focus on a single node being patched ; if that goes wrong it can then still be removed from the cluster (e.g : depending on topologies). Also on the CLI you have potential feedback such as errors immediately returned , but with the GUI , you are working in the dark and if 'everything gets messed up' then it will take more time to get ISE functional again

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Mon, 04 Sep 2023 08:05:45 GMT

@hslai wrote:
@TedB123 If possible, please share your TAC case number.

SR 696081805

the engineer assigned has been very helpful but i have a few more pending questions about this whole process.

cheers,

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Mon, 04 Sep 2023 08:08:16 GMT

@ahollifield wrote:
? But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues, or overall progress. Also you cannot control the order of the patch install. What if you wanted to update certain PSNs before others? You do not have this option in the UI….

agree... the UI patching system is drastically lacking in any information.
apparently a progress bar has been provided in ISE 3.3, but i cant confirm this as i havent seen it.. theres mention of it in the release notes.

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Mon, 04 Sep 2023 08:09:48 GMT

@Mark Elsen wrote:

@adamscottmaster2013 >...It is a BAD idea because doing it from the CLI is that it might be masking other issues that you might have between nodes themselves.
@ahollifield >... But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues,
- Indeed patching with the CLI is much better ; actually it's better to get rid of potential issues between nodes and focus on a single node being patched ; if that goes wrong it can then still be removed from the cluster (e.g : depending on topologies). Also on the CLI you have potential feedback such as errors immediately returned , but with the GUI , you are working in the dark and if 'everything gets messed up' then it will take more time to get ISE functional again
M.

appreciate the feedback from you guys.. im going to proceed with the patch install via cli as i just dont trust the web UI.

always nice to see some sort of progress during any install right

Re: ISE 3.2 - trying to install patch 3, 24hrs later still nothing

TedB123 — Mon, 04 Sep 2023 14:36:26 GMT

final update for you all...

i ended up patching both nodes via CLI and had no issues what so ever... if anything i was very surprised at how fast the patch was applied... no more than 10 minutes... maybe even 5.

once the first node was patched and rebooted i checked the version number and confirmed that all services were running.
make this node the primary and then proceeded to patch the other node.

ISE is working as expected so we are all good now.

thanks for all your input, it was very helpful.