iSE 2.0 Patch 3 issue

dporod · ‎09-15-2016

Hi, I am patching our 2.0.0.306 patch 2 ISE deployment to patch 3 using the GUI. Yesterday I kicked off the process and have been unable to access the primary admin node via the web page since then. I can access the CLI. I think the patch process is hung up some how. The rest of the deployment has not been affected.

I tried stopping the ISE application but cannot:

apppdiseadmin01/admin# app stop ise

Waiting up to 20 seconds for lock: APP_PATCH to complete
Database is still locked by lock: APP_PATCH. Aborting. Please try it later
% Error: Another ISE DB process (APP_PATCH) is in progress, cannot perform Application Stop at this time

I'd really like to cancel the patch process if possible.

apppdiseadmin01/admin# sh ver

Cisco Application Deployment Engine OS Release: 2.3
ADE-OS Build Version: 2.3.0.187
ADE-OS System Architecture: x86_64

Copyright (c) 2005-2014 by Cisco Systems, Inc.
All rights reserved.
Hostname: apppdiseadmin01

Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version : 2.0.0.306
Build Date : Thu Oct 8 02:55:23 2015
Install Date : Wed Jan 20 11:37:02 2016

Cisco Identity Services Engine Patch
---------------------------------------------
Version : 2
Install Date : Thu Jan 21 07:38:01 2016

apppdiseadmin01/admin# sh app sta ise

ISE PROCESS NAME STATE PROCESS ID
--------------------------------------------------------------------
Database Listener running 3560
Database Server running 31 PROCESSES
Application Server not running
Profiler Database running
AD Connector running 6771
M&T Session Database running 3104
M&T Log Collector not running
M&T Log Processor not running
Certificate Authority Service disabled
SXP Engine Service disabled
pxGrid Infrastructure Service disabled
pxGrid Publisher Subscriber Service disabled
pxGrid Connection Manager disabled
pxGrid Controller disabled
Identity Mapping Service disabled

Gagandeep Singh · ‎09-15-2016

Hi,

You have to reload the node instead on restarting of services.

If still it doesn't help then probably need to take root access in to the node. Remove the installing patch file which got hung. Need to contact TAC for root access.

Regards

Gagan

PS: rate if it helps!!!

Joseph Johnson · ‎09-18-2016

I just ran into this exact same issue. ISE 2.0 with patch 2 trying to upgrade to patch 3. I let it run for over an hour and it was stuck with only the Database Listener, Database Server, AD Connector, and M&T Session Database services running.

I reloaded the node (primary admin in two node deployment) and the Application Server service wouldn't start. Instead, it went back through the patch install process and finished successfully. The secondary node never installed the patch automatically. So I did the following:

I used the CLI to remove patch 3 from the primary node.
Rolled back Patch 2 from the deployment using the Primary Admin GUI.
Installed Patch 3 using the Primary Admin GUI.

Everything appears to be working okay for now.

dporod · ‎09-19-2016

I have tried a reload but end up in the same status.

Joseph Johnson · ‎09-19-2016

That will require a TAC call because, like gagsing said, you will need root access in order to delete the offending patch. Once that's deleted, you should be able to boot the system and start back in the previous state. I would rollback patch 2 and install patch 3 once it's all up and running.

dporod · ‎09-19-2016

If I have to reimage would the process be: ?

build a new VM
deploy the ise-2.0.0.306.SPA.x86_64.iso
run setup with the same IP and hostname
install patch 2
rejoin deployment

Couple of licensing questions. Will the UDI be the same on the new server if the IP and hostname are the same? Will the license files need to be installed on the new server manually or will they be installed as part of joining the deployment?

Joseph Johnson · ‎09-19-2016

You shouldn't have to build a new VM. You should have the option to install when booting from the ISO and that will overwrite the existing installation because one of the steps is provisioning/formatting the drives. If you don't get that option, then you will need to delete the VM and recreate it.

Yes, use the same set up information. If you have another node in the deployment as a secondary admin, go ahead promote it to primary admin while recreating the failed node. Once the failed node is recreated and patched to the same level, join it to the deployment.

Yes, the UDI information will change so you will need to rehost the license(s). The licenses do not need to be reinstalled if you have a secondary admin node in place because they are already replicated to it from the primary admin when you added a secondary admin to the deployment originally.

Gagandeep Singh · ‎09-19-2016

Hi,

In order to proceed further on this issue, potentially need root access and get that patch removed.

Regards

Gagan

PS: rate if it helps!!!

dporod · ‎09-21-2016

Ended up reimaging. It was actually pretty easy.

Thanks for your help.