ISE 3.2 - trying to install patch 3, 24hrs later still nothing

[TedB123]

hi

We are trying to patch our ISE nodes with patch number 3: - ise-patchbundle-3.2.0.542-Patch3-23071904.SPA.x86_64.tar.gz

we have a 2 node environment

i logged onto the primary node
administration
selected maintenance 
patch management
selected the patch and clicked install.... at this point after about 60s i got logged out of the web UI. 

i decided to log onto the node via cli
ran the following
show application status ise - all processes/services running as expected
show version - Patch 3 not showing up
show uptime - node has not been rebooted

I logged onto the 2nd node and under the upgrade section i saw this.

so i thought ok something is happening.
unfortunately this has now been like this for close to 24hrs and ive also restarted the 2nd node just to see if this would clear.

i ran the same cli commands on node 2 and they are exactly the same as node 1.
services are running, patch 3 not installed and node has not been rebooted.

something is definitely not playing ball and im not sure what the best way forward is or what our options are.

1. should i manually patch node 2 and then failover the services to node 2 making it the primary
2. once node 2 has been patched, reboot and patch node 1
3. retry patch install via web UI
4. bin patch 3 and try patch 2
5. a different approach?

i know that patching via cli will not patch both nodes and each node has to be done individually.
my concern is that because node 2 is showing this "upgrade progress bar", if I failover services to node 2 it could break ISE.

has anybody experienced something like this with patching?
thoughts on what we can do?

I have a case with TAC but waiting for a reply... but its slow going

cheers!

[ahollifield]

Yeah not sure why you are seeing that progress bar on the secondary node.  That page in the UI is only for major version upgrades, not patch installs.  I would personally open a TAC case as well as planning on attempting the patch installation from the CLI.

[adamscottmaster2013]

@ahollifield:  Patch installation from the CLI is a BAD idea.  You will definitely create more problems down the road for yourself.

[TedB123]

---

@adamscottmaster2013 wrote:

@ahollifield:  Patch installation from the CLI is a BAD idea.  You will definitely create more problems down the road for yourself.

---

can you provide more detail as to why installing the patch via CLI is a BAD idea?

[adamscottmaster2013]

@TedB123:  It is a BAD idea because doing it from the CLI is that it might be masking other issues that you might have between nodes themselves.  That's why TAC always tells you the "preferred" method is to do it from the UI.

[ahollifield]

? But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues, or overall progress. Also you cannot control the order of the patch install. What if you wanted to update certain PSNs before others? You do not have this option in the UI….

[marce1000]

@adamscottmaster2013 >...It is a BAD idea because doing it from the CLI is that it might be masking other issues that you might have between nodes themselves. 
 @ahollifield >... But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues,

  - Indeed patching with the CLI is  much better ; actually it's better to get rid of potential issues between nodes and focus on a single node being patched ; if that goes wrong it can then still be removed from the cluster (e.g : depending on topologies). Also on the CLI you have potential feedback such as errors immediately returned , but with the GUI  , you are working in the dark and if 'everything gets messed up'  then it will take more time to get ISE functional again

 M.

[TedB123]

---

@marce1000 wrote:

@adamscottmaster2013 >...It is a BAD idea because doing it from the CLI is that it might be masking other issues that you might have between nodes themselves. 
 @ahollifield >... But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues,

  - Indeed patching with the CLI is  much better ; actually it's better to get rid of potential issues between nodes and focus on a single node being patched ; if that goes wrong it can then still be removed from the cluster (e.g : depending on topologies). Also on the CLI you have potential feedback such as errors immediately returned , but with the GUI  , you are working in the dark and if 'everything gets messed up'  then it will take more time to get ISE functional again

 M.

---

appreciate the feedback from you guys.. im going to proceed with the patch install via cli as i just dont trust the web UI.

always nice to see some sort of progress during any install right

[TedB123]

---

@ahollifield wrote:
? But you get ZERO feedback from the UI on the progress of the patch installation… no feedback on errors, issues, or overall progress. Also you cannot control the order of the patch install. What if you wanted to update certain PSNs before others? You do not have this option in the UI….

---

agree... the UI patching system is drastically lacking in any information. 
apparently a progress bar has been provided in ISE 3.3, but i cant confirm this as i havent seen it.. theres mention of it in the release notes.

[ahollifield]

WHAT???? Please elaborate?

[marce1000]

 - Check disk space on all involved nodes with (CLI) : show disks
    Also check outputs from : show logging system ade/ADE.log
                                            show logging application ise-psc.log

                Also read : https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/220195-troubleshoot-identity-services-engine-i.html

 M.

[TedB123]

---

@marce1000 wrote:

 - Check disk space on all involved nodes with (CLI) : show disks
    Also check outputs from : show logging system ade/ADE.log
                                            show logging application ise-psc.log

                Also read : https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/220195-troubleshoot-identity-services-engine-i.html

 M.

---

yeah ive done all that.. nothing in the logs to indicate that any patches have been installed

wouldnt be surprised if ive hit a bug thats prevented the patches from being installed via the web UI.

[adamscottmaster2013]

@TedB123:  Welcome to patch-3 patching misery.  I have a similar issue with patch-3 and the case has been opened for almost three weeks.

[TedB123]

TAC advised me to select option 27 from cli to clear the upgrade bar
this option here... 

[27]Reset Upgrade Tables

but this hasnt done anything, the progress bar is still there... so i will need to get back to them and let them know.

they also advised to update each node individually via cli... maybe once the 2nd node is patched that progress bar will clear.

[marce1000]

             >..also advised to update each node individually via cli..
 - I prefer that too , to get a  better overview on what is happening 

 M.