CSPC upgrade from 2.7.3 to 2.7.3.1 stuck in Apply-In-Progress

[rdruhland]

I have been working on upgrading to CSPC Version 2.7.3.1 over the course of a week.  First it was stuck in "Downloading" so I logged on to the CLI as ADMIN and re-downloaded use download <latest patch>.   After, I checked show download until the new version was downloaded:

Version : sp-30.0.3-1-0-lnx64
Download Location : /opt/LCM/tmp/sp-30.0.3-1-0-lnx64-package.zip
Status : Downloaded
Start Time : May/25/50060 05:43:55
End Time : Feb/06/2018 14:08:50
Reason For Failure : -

Next I did an apply sp-30.0.3-1-0-lnx64 and let that run overnight.  When I came back in the morning I did a show apply and the status was "Apply-in-Progress"  I let it run over the weekend and when I came back it was in the same state.  So I tried to restart the patch by running the apply command again.  The appliance rebooted once but the /opt/LCM/logs/apply have been stuck in the following state for over 12 hours again:

empty
Local DB and SchedulerDB are not in sycn so jobs scheduled will be lost

============================
Rebooting the appliance now
============================

Any ideas on what I can do next to complete this upgrade?  I'm looking for once that doesn't involve deploying a new VM because I have multiple CSPC to upgrade so I am hoping to not have to redeploy multiple VMs.  I also ran into a similar issue going from 2.7.2 to 2.7.3 but eventually it completed about 36 hours after I initiated the upgrade.

[Cheri Fairbrother]

Hi Ryan,

We'll have an expert reply just as soon as possible.

Cheers,

Cheri

[adias]

Hello rdruhland,

Yes I do believe we are experiencing  a problem with the latest patch 2.7.3.1.  I would like to try a workaround with you since you do have all Packages downloaded.

But first on this one in particular let me verify the version first please send the screen shot of the

help---> about--->view versions

[rdruhland]

Here is the screen shot....thank you

[adias]

Communicating privately with customer

[rdruhland]

Output is below:

[collectorlogin@cspcollector ~]$ su -
Password:
[root@cspcollector ~]# cp /opt/LCM/tmp/sp-30.0.3-1-0-lnx64-package.zip /home/collectorlogin/sp-30.0.3-1-0-lnx64-package.zip
[root@cspcollector ~]# cd /home/collectorlogin/
[root@cspcollector collectorlogin]# ls -al
total 160436
drwx------. 3 collectorlogin collectorlogin 4096 Feb 12 11:25 .
drwxr-xr-x. 6 root root 4096 Sep 11 07:14 ..
-rw-------. 1 collectorlogin collectorlogin 652 Feb 6 15:04 .bash_history
-rwx------. 1 collectorlogin collectorlogin 18 May 10 2016 .bash_logout
-rwx------. 1 collectorlogin collectorlogin 176 May 10 2016 .bash_profile
-rwx------. 1 collectorlogin collectorlogin 124 May 10 2016 .bashrc
-rw-r-----. 1 collectorlogin tcpdump 8871 Aug 15 13:51 sntc.pcap
-rw-r-----. 1 root root 164225387 Feb 12 11:25 sp-30.0.3-1-0-lnx64-package.zip
drwx------. 2 collectorlogin collectorlogin 4096 Apr 7 2017 .ssh
-rw-------. 1 root root 12288 Aug 9 2017 .testsntconnectivity.swp
[root@cspcollector collectorlogin]# unzip /home/collectorlogin/sp-30.0.3-1-0-lnx64-package.zip
Archive: /home/collectorlogin/sp-30.0.3-1-0-lnx64-package.zip
extracting: jeos-30.0.3-1-lnx64.zip
extracting: sp-30.0.3-1-0-lnx64.zip
[root@cspcollector collectorlogin]# ls -lart
total 320816
-rwx------. 1 collectorlogin collectorlogin 124 May 10 2016 .bashrc
-rwx------. 1 collectorlogin collectorlogin 176 May 10 2016 .bash_profile
-rwx------. 1 collectorlogin collectorlogin 18 May 10 2016 .bash_logout
drwx------. 2 collectorlogin collectorlogin 4096 Apr 7 2017 .ssh
-rw-------. 1 root root 12288 Aug 9 2017 .testsntconnectivity.swp
-rw-r-----. 1 collectorlogin tcpdump 8871 Aug 15 13:51 sntc.pcap
drwxr-xr-x. 6 root root 4096 Sep 11 07:14 ..
-rw-r--r--. 1 root root 707093 Feb 6 13:58 sp-30.0.3-1-0-lnx64.zip
-rw-r--r--. 1 root root 163517924 Feb 6 14:03 jeos-30.0.3-1-lnx64.zip
-rw-------. 1 collectorlogin collectorlogin 652 Feb 6 15:04 .bash_history
-rw-r-----. 1 root root 164225387 Feb 12 11:25 sp-30.0.3-1-0-lnx64-package.zip
drwx------. 3 collectorlogin collectorlogin 4096 Feb 12 11:25 .
[root@cspcollector collectorlogin]# tail -100 /opt/LCM/logs/install
Verifying : kernel-headers-2.6.32-696.18.7.el6.x86_64 1/2
Verifying : kernel-headers-2.6.32-696.16.1.el6.x86_64 2/2

Updated:
kernel-headers.x86_64 0:2.6.32-696.18.7.el6

Complete!

============================================================================
Started execution of post-install scripts in package jeos-30.0.3-1-lnx64 ...
============================================================================

============================================================================
Finished execution of post-install scripts in package jeos-30.0.3-1-lnx64 ...
============================================================================

ts
keytool error: java.lang.Exception: Certificate not imported, alias <concsoweb-lt> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <concsoweb-prd> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <ca:1> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <ca:0> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <concso> already exists

Service Name : COLLECTOR
empty
Local DB and SchedulerDB are not in sycn so jobs scheduled will be lost

============================
Rebooting the appliance now
============================
[Wed Feb 7 08:44:36 CST 2018] Called with argument "sp-30.0.3-1-0-lnx64" and "false"
[Wed Feb 7 08:44:43 CST 2018] Restart required for package "jeos-30.0.3-1-lnx64"
[Wed Feb 7 08:44:43 CST 2018] Apply-check result for version "sp-30.0.3-1-0-lnx64": false, false, false, false, false, true, none, none

============================================================================
Started execution of pre-install scripts in package jeos-30.0.3-1-lnx64 ...
============================================================================

============================================================================
Finished execution of pre-install scripts in package jeos-30.0.3-1-lnx64 ...
============================================================================

============================================================================
Started installation of component CASP in package jeos-30.0.3-1-lnx64 ...
============================================================================

============================================================================
Finished installation of component CASP in package jeos-30.0.3-1-lnx64 ...
============================================================================

ng fastest mirrors
Package kernel-2.6.32-696.18.7.el6.x86_64 already installed and latest version
Nothing to do
Loaded plugins: fastestmirror
Setting up Upgrade Process
Loading mirror speeds from cached hostfile
No Packages marked for Update

============================================================================
Started execution of post-install scripts in package jeos-30.0.3-1-lnx64 ...
============================================================================

============================================================================
Finished execution of post-install scripts in package jeos-30.0.3-1-lnx64 ...
============================================================================

ts
keytool error: java.lang.Exception: Certificate not imported, alias <concsoweb-lt> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <concsoweb-prd> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <ca:1> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <ca:0> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <concso> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <concsoweb3-prd> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <concsoweb3-prd> already exists
keytool error: java.lang.Exception: Certificate not imported, alias <concsoweb3-prd> already exists

Service Name : COLLECTOR
empty
Local DB and SchedulerDB are not in sycn so jobs scheduled will be lost

============================
Rebooting the appliance now
============================
[root@cspcollector collectorlogin]#

[adias · Accepted solution]

 

Customer has been affected by the following bug  --> CSCvh44113

Workaraound provided on the BUG description but if unfortunetely workaround provided does not resolve the problem standard recomendation of the Forum is to redeploy the Virtual Image

with the newest ova available

We have recovered and upgraded this particular customer to 2.7.3.1

[JER78]

FYI,

I worked with TAC on this issue and they were able to manually re-run the upgrade from within CentOS (Linux) and resolve the issue without requiring a re-install.

There may, of course, be multiple causes/resolutions but they did link our issue to this defect.

Replying here for posterity...