Upgrade to 2.8.0.1 from 2.8 stuck in Apply-in-progress

[miralpatel91]

I have CSPC running 2.8 version which got stuck in apply- in process while upgrading to 2.8.0.1.

I am following process from troubleshooting guide to resolve the issue but not working out.

[jumin]

Hey miralpatel91,

Which method of upgrading have you tried to get to version 2.8.0.1?  Please provide any screenshots that show upgrading issues.

Thanks,
Justin

[miralpatel91]

I have scheduled upgrade process that if new version available then download and upgrade. I have attached screenshot from GUI.

[jumin]

Hey miralpatel91,

Please run the 'show apply' and 'show apply progress' commands in the CSPC GUI from the admin login.  Let me know what you see here.

Thanks,
Justin

[miralpatel91]

Find following.

admin# show apply
Version Number : sp-30.1.0-1-0-lnx64
Status : Apply-in-progress
Start Time : -
End Time : -

!

admin# show apply progress
execute % Error occured while executing command

!

admin# sho apply *
---------------------------------------
admin# show apply [version|progress]
Eg:
admin# show apply
admin# show apply progress
admin# show apply sp-1.11.1-2-0-lnx64
admin# show apply jeos-1.11.1-2-0-lnx64
---------------------------------------

admin# show apply sp-1.11.1-2-0-lnx64

Apply information is not available.

[jumin]

Hey miralpatel91,

I will message you with what other methods to proceed with upgrade.

Thanks,

[BrianV]

We are having the same exact issue here. Can you please advise on any potential fixes?

admin# show apply
Version Number : sp-30.1.0-1-0-lnx64
Status : Apply-in-progress
Start Time : -
End Time : -

admin# show apply progress
execute % Error occured while executing command
admin#

[miralpatel91]

At this time there are no fixes. Found out that, There has been issues seen with upgrading to 2.8.0.1 and DEV will need to troubleshoot this. So I am standby until Cisco suggest any solutions.

[miralpatel91]

I have followed following steps to upgrade CSPC.

1) Reboot CSPC.

2) Follow process for Apply-in-progress Hunged from troubleshooting guide.

3) Follow process for Apply Failed from  from troubleshooting guide.

4) Manually upgrade JeOS.

5) Manually upgrade SP.

6) Reboot.

Hope this is helpful.

[Bharath S]

Hi MiralPatel91

I went through the logs which Justin shared with me . 

Below are the observations  - 

1.  Looks like upgrade was tried in Dec 11 first to 2.8.0.1 using LCM command "download" , "apply" or Using LCM UI. Though the download is successful , i see the apply is not proceeded. 

               Probable causes - Box reboot during apply (unknowingly). 

2. After that above case , customer has tried to upgrade manually using "/opt/LCM/bin/lcmagent-apply.tcl JeOS jeos-30.1.0-1-lnx64.zip &", but seems like the above command is not triggered with "&" . In the case if & is missed, the session should be kept alive till the installation finishes and box goes for reboot. 

The Best guess is apply is not triggered with "&" and since Harend CentOS component takes more time than 5 mins , session expires and the process will be killed. Came to this conclusion because of the below log entries - 

                   a. Session timed out case due to & miss . 

                       [Tue Jan 8 10:53:49 EST 2019] Executing install script for "Hardened CentOS"
                       [Tue Jan 8 13:02:36 EST 2019] Apply TCL Called with arguments "JeOS" and "jeos-30.1.0-1-lnx64.zip"

                   b. Triggered SP patch installation as soon as jeos is started. (Need to wait for JeOS patch installation to complete)

                       [Tue Jan 8 13:03:10 EST 2019] Executing install script for "Hardened CentOS"
                       [Tue Jan 8 13:09:33 EST 2019] Apply TCL Called with arguments "SP" and "sp-30.1.0-1-0-lnx64.zip"

                       [Tue Jan 8 13:09:33 EST 2019] Inside existence of status db
                       [Tue Jan 8 13:09:36 EST 2019] Parsing package info
                       [Tue Jan 8 13:09:36 EST 2019] Validating package details 
                       [Tue Jan 8 13:09:38 EST 2019] Parsing Component info 
                       [Tue Jan 8 13:09:38 EST 2019] Validating Component info
                       [Tue Jan 8 13:09:38 EST 2019] Checking component dependency
                       [Tue Jan 8 13:09:38 EST 2019] Apply TCL returned 1
                       [Tue Jan 8 13:09:38 EST 2019] Executing pre-install scripts
                       [Tue Jan 8 13:12:39 EST 2019] Inside preinstall script
                       Fetching Versions info
                       current CSPC version is : 2.8 
                       current CASP version is : 2.8 
                       Current CASP version is 2.8 
                     . Please update the jeos to 2.8.0.1 and above

         Note - No damage though in this case as the version check stops it from getting installed. 

Solution - 

                  1.  Set the box to a proper state by following the instructions in "Apply-in-progress Hunged" section in trouble shoot guide - https://www.cisco.com/c/dam/en/us/support/docs/cloud-systems-management/common-services-platform-collector-cspc/CSPC_Troubleshooting_Guide.pdf 

                    Note - Upgrade can be retried from LCM UI as part of last step in the  "Apply-in-progress Hunged" section or follow step 2. 

                   2. Re-trigger upgrade from UI or follow the below manual steps - 

                                a. Box is to be set to proper state i.e, box should not show any patch with "apply-in-progress" state which can be achieved by step 1. 

                                b.  Trigger installation of JeOS patch with - /opt/LCM/bin/lcmagent-apply.tcl JeOS <path to jeos patch>/jeos-30.1.0-1-lnx64.zip &

                                c.  It will take nearly 30-40 mins maximum to finish the installation , customer can tail install logs with - tailf /opt/LCM/logs/install to see the installation process

                                d. Once box goes for reboot and comes back and once adminshell service is up , login to adminshell and "sh ver -d" command to verify that JeOS versions are updated properly to jeos-30.1.0-1-lnx64 . After this SP patch installation can be triggered with -  /opt/LCM/bin/lcmagent-apply.tcl SP <path to jeos>/sp-30.1.0-1-0-lnx64.zip &

                                e. Installation will take time depending on the data in the collector. 

If there are any confusions in the above information do let me know will try my best to clarify.

Thanks & Regards,

Bharath S

[BrianV]

Thanks. Trying to follow the process in the troubleshooting guide. Tells us to:

3. Run below commend to check apply process is running Ex:- #ps –aef | grep apply

When I try and run it I get  

[root@ciscocspcprd01 ~]# Ex:- #ps –aef | grep apply
-bash: Ex:-: command not found

Thoughts?

[jofrumki]

The '#' indicates a command prompt, please try again without it.

[Bharath S]

Hi Brian , 

Any update on this upgrade ? 

Thanks & Regards,

Bharath S

[BrianV]

Upgrade was successful. For those that may hit this in the future, here are my step by steps.

Run below commend to check apply process is running
ps –aef | grep apply

If any apply process is running, then kill that process using below command.
Kill -9 <PID> from above output

Download updateDBScript.s file as indicated in troubleshooting guide:

https://www.cisco.com/c/dam/en/us/support/docs/cloud-systems-management/common-services-platform-collector-cspc/CSPC_Troubleshooting_Guide.pdf

[root@ciscocspcprd01 ~]# ftp
ftp> open 10.92.65.245
Connected to 10.92.65.245 (10.92.65.245).
220 3Com 3CDaemon FTP Server Version 2.0
Name (10.92.65.245:root): cisco
331 User name ok, need password
Password:
230 User logged in
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir
227 Entering passive mode (10,92,65,245,205,98)
125 Using existing data connection
<snip>
-rwxrwxrwx 1 owner group 3703 Jan 17 11:37 updateDBScript.sh
226 Closing data connection
ftp> get updateDBScript.sh
local: updateDBScript.sh remote: updateDBScript.sh
227 Entering passive mode (10,92,65,245,205,109)
125 Using existing data connection
226 Closing data connection; File transfer successful.
3703 bytes received in 0.00288 secs (1286.21 Kbytes/sec)
ftp> exit
221 Service closing control connection

[root@ciscocspcprd01 ~]# chmod 755 updateDBScript.sh
[root@ciscocspcprd01 ~]# ./updateDBScript.sh

reboot appliance

Now go to GUI>>Software Updates, it should say update failed
Click "Upgrade Now"
Confirm "yes" it will reboot after update - note, window can take several minutes to come up
Update will change to initiated
Update will change to Downloading
Update will change to Apply-in-progress

[Bharath S]

Thanks a lot Brian for the detailed steps .

Good to hear the upgrade was successful . 

Regards,

Bharath S