Solved: CSPC upgrade from 2.7.3 to 2.7.3.2 stuck in downloading

evanswee · ‎03-15-2018

I've been trying to upgrade our CSPC collector since months now. Initially I did the 2.7.3.1 update from the web console but it remained in status downloading. restarting the server didn't help. Same now with the 2.7.3.2 update.

I've followed the upgrade instructions and also tried from CLI (enabled the collectorlogin and root user, added the required entries in the /etc/hosts file.). When I look at the download progress, it is always downloading the current version and the previous version, both at exactly the same percentage. In the beginning it moves slowly, but after a while it gets stuck and never completes. Reloading or restarting the server or the download doesn't help.

admin# show download
        Version             : sp-30.0.3-0-0-lnx64
        Download Location   : /opt/LCM/tmp/sp-30.0.3-0-0-lnx64-package.zip
        Status              : Downloaded
        Start Time          : Jan/12/2018 16:28:26
        End Time            : Jan/12/2018 17:12:30
        Reason For Failure : -

        Version             : sp-30.0.3-1-0-lnx64
        Download Location   : -
        Status              : Downloading
        Start Time          : Nov/26/50078 00:39:07
        End Time            : -
        Reason For Failure : -

        Version             : sp-30.0.3-2-0-lnx64
        Download Location   : -
        Status              : Downloading
        Start Time          : Jun/07/50169 23:35:52
        End Time            : -
        Reason For Failure : -

admin# show download progress
Package Name                    Package Status          Downloaded Percentage
-----------------------------------------------------------------------------
sp-30.0.3-1-0-lnx64             Downloading             18
sp-30.0.3-2-0-lnx64             Downloading             18

adias · ‎03-22-2018

the above folder should have been downloads

# cp /opt/LCM/tmp/downloads/sp-30.0.3-2-0-lnx64.zip /home/collectorlogin
# cp /opt/LCM/tmp/downloads/jeos-30.0.3-2-lnx64.zip /home/collectorlogin
# cd /home/collectorlogin

To confirm the files are there:

#ls -lart

View solution in original post

Cheri Fairbrother · ‎03-16-2018

Hi Erik,

Please excuse our delay in responding. An expert will reply further as soon possible.

Cheers,

Cheri

Jarrett Pomeroy · ‎03-19-2018

Hello evanswee,

It looks like the download may have timed out and never completed successfully. Can you please try setting the last download attempt to failed and start a new request?

The steps can be found in the latest release notes here, in the section "Known Issue". Once completed, you can try the "download " command again from the admin CLI shell.

Thank you,

Jarrett

evanswee · ‎03-20-2018

I executed the steps described. The upgrade button in the webinterface is now available again, but it always says apply-failed.

also from CLI it doesn't do anything, see below

admin# check update

Patch versions
------------------------------------------------------
    Version Number:         sp-30.0.3-2-0-lnx64
        Description:        Collector patch to upgrade to CSPC 2.7.3.2
        Required Versions: sp-30.0.3-0-0-lnx64
        Optimal JeOS:       jeos-30.0.3-2-lnx64
        Reboot Needed:      Yes
        Restart Needed:     Yes
        Package Size:       9158909 bytes
        Contents:
          Component Name:   CSPC Base
            Version:        2.7.3.2
            Description:    CSPC Base 2.7.3.2
        Installable:        Yes

admin# upgrade

Version                 Description
-------------------------------------------------------
sp-30.0.3-2-0-lnx64     Collector patch to upgrade to CSPC 2.7.3.2

Download and apply of a package can take a while.
It is recommended to schedule the upgrade.

Do you want to continue [y|n] ?y
Please enter the version to upgrade :sp-30.0.3-2-0-lnx64

Do you want to continue with download and apply [y|n] ?y
Connection to Software Download Server will use User Id 'erxx.xxxxxxxxxxt@gxxxf.com'.

In order to download software, please indicate that you have read and agree
to be bound by the Cisco End User License Agreement which can be viewed at
http://www.cisco.com/en/US/docs/general/warranty/English/EU1KEN_.html

Accept License Agreement (y|n)? y

Attempting to download the requested package.....sp-30.0.3-2-0-lnx64
Download of package has been initated.
admin# show download progress

Download information is not available.

admin# show download
        Version             : sp-30.0.3-0-0-lnx64
        Download Location   : /opt/LCM/tmp/sp-30.0.3-0-0-lnx64-package.zip
        Status              : Downloaded
        Start Time          : Jan/12/2018 16:28:26
        End Time            : Jan/12/2018 17:12:30
        Reason For Failure : -

        Version             : sp-30.0.3-1-0-lnx64
        Download Location   : -
        Status              : Downloaded
        Start Time          : Nov/26/50078 00:39:07
        End Time            : -
        Reason For Failure : -

        Version             : sp-30.0.3-2-0-lnx64
        Download Location   : -
        Status              : Downloaded
        Start Time          : Jun/07/50169 23:35:52
        End Time            : -
        Reason For Failure : -

adias · ‎03-22-2018

It is possible that you may have the necessary package to have the installation of the patch 2.7.3.2 and likely you nay already have installed the Jeos for it as well let me confirm this with you and then I will forward additional commands to finalize your upgrade.

Please send me the output of the below command on adminshell

admin# show version -d

and the below command output on root shell ( note : you cannot connect directly to root --> connect first to collectorlogin and then su root) if you do not have the access credentials for collectorlogin and root let me know and I will send the steps to reset it

once you have it do the following from the root shell

# ls -lart /opt/LCM/tmp/

evanswee · ‎03-22-2018

admin# show version -d
        Build-name   : Collection Platform Software 2.7.3

        Package-type : ServicePack
        Version      : sp-30.0.3-0-0-lnx64
             Component : CSPC Base
             Version   : 2.7.3
        Package-type : JeOS
        Version      : jeos-30.0.3-0-lnx64
             Component : AdminShell
             Version   : 1.6
             Component : LCM
             Version   : 1.6
             Component : CASP
             Version   : 2.7.3
             Component : ConcsoTgw
             Version   : 1.7.0
             Component : Hardened CentOS
             Version   : 6.8 patch#0

[root@localhost collectorlogin]# ls -lart /opt/LCM/tmp/
total 977560
-rw-r--r--. 1 casuser casusers 1001002317 Jan 12 17:12 sp-30.0.3-0-0-lnx64-package.zip
drwxr-xr-x. 9 casuser casusers       4096 Jan 12 17:38 ..
drwxr-xr-x. 3 casuser casusers       4096 Feb 9 09:59 .
drwxr-x---. 2 casuser casusers       4096 Mar 20 14:58 downloads

adias · ‎03-22-2018

At this point your problem is not similar to the one on the other post you still need the Jeos and SP packages to be downloaded. Please send me the following output

# tail -100 /opt/LCM/logs/install

and

# env | grep proxy

and

# ls -lart /opt/LCM/tmp/downloads

Additionally save or create the attached file in your /home/collectorlogin name it toreviewstatus.txt and then do the following to make it executable

# chmod 755 toreviewstatus.txt

then run it as below it should give you a prompt like ij>

./toreviewstatus.txt

on the prompt type the following line

ij> connect 'jdbc:derby://localhost:1527/opt/LCM/info/updates;create=true;';

ij> select * from updates;

send me the output of the above

evanswee · ‎03-22-2018

[root@localhost collectorlogin]# tail -100 /opt/LCM/logs/install

ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcUw30rmt6tKY2sLMyNc3qCOGS30J2v5YrMogB-bYq4u_nA_JAG8AzzFi8l0Nb6S4RB2rhqnSyOI7cswr7UPNNrUvxzsT199nKXPE-uFYZuJIoHv3LJnEgM4AgZPv9adR_kFuRgGT0pmphar...>>>>>
/opt/LCM/tmp/downloads/jeos-30.0.3-2-lnx64.zip~~~~~~~~~~~~~~~~~~
sp-30.0.3-2-0-lnx64.zip ==========
[Tue Mar 20 13:02:27 UTC 2018] Starting download operation
sp-30.0.3-2-0-lnx64.zip ++++++++++++
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0conn-teg-http.gen ~~~~~~~~
0     0    0     0    0     0      0      0 --:--:-- 0:00:01 --:--:--     0
ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcU4KLZ9Evon8ZHL2fFe3Wrfx9QVX_uAXwtnXudi5imsBhBVmviYck9Ywx6-MSMv6WVwxvR6xWTkU_ykyBvpUQ6DcGo2oIBTdIy8uUNxtmZpbOAGBc1TftZPWFZy4gAcrO_W1hsJlgQPNcOu...>>>>>
/opt/LCM/tmp/downloads/sp-30.0.3-2-0-lnx64.zip~~~~~~~~~~~~~~~~~~
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
100 8944k 100 8944k    0     0   216k      0 0:00:41 0:00:41 --:--:-- 252k    2 3 8 k0 0:18:27 0:00:32 0:17:55 236k

ASD-DOWNLOAD: Removing temporary files...
100 226M 100 226M    0     0   240k      0 0:16:04 0:16:04 --:--:-- 246k

ASD-DOWNLOAD: Removing temporary files...
jeos-30.0.3-2-lnx64.zip ==========
[Tue Mar 20 14:01:54 UTC 2018] Starting download operation
jeos-30.0.3-2-lnx64.zip ++++++++++++
conn-teg-http.gen ~~~~~~~~

ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcUw30rmt6tKY2sLMyNc3qCOGS30J2v5YrMogB-bYq4u_nA_JAG8AzzFi8l0Nb6S4RB2rhqnSyOI7csnLgHpOLvPYeFNtS4GlhNlNFpOwMat97R9Pe7hXWxVui-Q4ZQGwJcCEOliMTVOz1Lr...>>>>>
/opt/LCM/tmp/downloads/jeos-30.0.3-2-lnx64.zip~~~~~~~~~~~~~~~~~~
sp-30.0.3-2-0-lnx64.zip ==========
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0[Tue Mar 20 14:01:56 UTC 2018] Starting download operation
sp-30.0.3-2-0-lnx64.zip ++++++++++++
conn-teg-http.gen ~~~~~~~~
0     0    0     0    0     0      0      0 --:--:-- 0:00:01 --:--:--     0
ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcU4KLZ9Evon8ZHL2fFe3Wrfx9QVX_uAXwtnXudi5imsBhBVmviYck9Ywx6-MSMv6WVwxvR6xWTkU_kaSisso2SrDaEwKoFKF0cHxeMSjaCHHfsaBJYJMQQTDUtyrWz-Uwrrf-TLdrUqvCm3...>>>>>
/opt/LCM/tmp/downloads/sp-30.0.3-2-0-lnx64.zip~~~~~~~~~~~~~~~~~~
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
100 8944k 100 8944k    0     0   195k      0 0:00:45 0:00:45 --:--:-- 240k
4 226M    4 9464k    0     0   198k      0 0:19:28 0:00:47 0:18:41 239k
ASD-DOWNLOAD: Removing temporary files...
24 226M   24 55.7M    0     0   220k      0 0:17:30 0:04:18 0:13:12 238kjeos-30.0.3-2-lnx64.zip ==========
[Tue Mar 20 14:06:15 UTC 2018] Starting download operation
jeos-30.0.3-2-lnx64.zip ++++++++++++
24 226M   24 55.9M    0     0   220k      0 0:17:29 0:04:19 0:13:10 241kconn-teg-http.gen ~~~~~~~~
24 226M   24 56.1M    0     0   220k      0 0:17:29 0:04:20 0:13:09 238k
ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcUw30rmt6tKY2sLMyNc3qCOGS30J2v5YrMogB-bYq4u_nA_JAG8AzzFi8l0Nb6S4RB2rhqnSyOI7clYfkpy6Iu7QBrrfR-Ke-r2qtC7TwYpGosRTd6BSV1gt7wRvO9Lr1CHvuruArq_Rwkm...>>>>>
/opt/LCM/tmp/downloads/jeos-30.0.3-2-lnx64.zip~~~~~~~~~~~~~~~~~~
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
24 226M   24 56.4M    0     0   220k      0 0:17:29 0:04:21 0:13:08 238ksp-30.0.3-2-0-lnx64.zip ==========
[Tue Mar 20 14:06:18 UTC 2018] Starting download operation
sp-30.0.3-2-0-lnx64.zip ++++++++++++
25 226M   25 56.6M    0     0   220k      0 0:17:29 0:04:22 0:13:07 233kconn-teg-http.gen ~~~~~~~~

ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcU4KLZ9Evon8ZHL2fFe3Wrfx9QVX_uAXwtnXudi5imsBhBVmviYck9Ywx6-MSMv6WVwxvR6xWTkU_9KLr2K1geaaBAq8NNGRjdnyql64IkjMi4BfMEaNvGaQtvyVxrD1bXGQO_lIoS3eyox...>>>>>
/opt/LCM/tmp/downloads/sp-30.0.3-2-0-lnx64.zip~~~~~~~~~~~~~~~~~~
0     0    0     0    0     0      0      0 --:--:-- 0:00:02 --:--:--     0 % Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
100 8944k 100 8944k    0     0   206k      0 0:00:43 0:00:43 --:--:-- 224k3 k    0 0:00:44 0:00:30 0:00:14 235kk
4 226M    4 9457k    0     0   207k      0 0:18:36 0:00:45 0:17:51 222k
ASD-DOWNLOAD: Removing temporary files...
6 226M3 2     262 61M5 . 5 M3 2   7 20. 8 M      0 0    2 1 6 k0       2 2 20k 0:1 7 : 5 1    00 : 001::1173: 2 00 : 106::0358: 3 42 3 1k0:11:46     3282 6 M2 2 6 M1 3   2398. 68M7 . 0 M 0      0    0      02 2 0 k2 2 3 k      0     00:17:28 0:02:17 0:15:11 222k 0:17:15 0:06:38 0:10:37 2243 226 M1 7    4226M   173 4908..60MM        00          00      222244kk            00    00::1177::131 0 0 :00:30:70:52 6 0 :01:4:08 20394:k44    3539    222266MM      3539 7 61.303MM        00          00   224k      0 0:17:09 0:10: 0 7 2 204:k0 7 : 0 2    02 4 00k:17:11 0:05:46 0:11:25 7 34 7 2 2262M6 M    7 34 7 1 6150M7 M      0 0        0 0    2 2252k5 k          0 0 0 :01:71:70:60 5 0 :01:20:82:90 8 0 :00:40:83:75 7 2 325 7917    222266MM      7917    126129MM        00          00      222277kk           0 0 0 :01:61:75:90 0 0 :01:21:61:03 1 0 :00:40:04:92 9 2 32100 226M 100 226M    0     0   227k      0 0:16:59 0:16:59 --:--:-- 241k    0:16:41 0   0 : 000::1169: 5 92 2 40k:12:20 0:04:39 224k

ASD-DOWNLOAD: Removing temporary files...
100 226M 100 226M    0     0   226k      0 0:17:01 0:17:01 --:--:-- 217k

ASD-DOWNLOAD: Removing temporary files...
jeos-30.0.3-2-lnx64.zip ==========
[Tue Mar 20 14:58:14 UTC 2018] Starting download operation
jeos-30.0.3-2-lnx64.zip ++++++++++++
conn-teg-http.gen ~~~~~~~~

ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcUw30rmt6tKY2sLMyNc3qCOGS30J2v5YrMogB-bYq4u_nA_JAG8AzzFi8l0Nb6S4RB2rhqnSyOI7cNSxa8kSs_kIr2SOqSZlyiBERL9_2Ityjpxu1RW3KausSV9FAKF8KLA5oUrNaL1xJJS...>>>>>
/opt/LCM/tmp/downloads/jeos-30.0.3-2-lnx64.zip~~~~~~~~~~~~~~~~~~
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0sp-30.0.3-2-0-lnx64.zip ==========
[Tue Mar 20 14:58:15 UTC 2018] Starting download operation
sp-30.0.3-2-0-lnx64.zip ++++++++++++
conn-teg-http.gen ~~~~~~~~
0     0    0     0    0     0      0      0 --:--:-- 0:00:01 --:--:--     0
ASD-DOWNLOAD: Getting File details from the server
-==https://concsoweb-prd.cisco.com/lcmdownload/?dwnld_code=xhMnkw8Z-oGUCa0SF-5fpsoPHQD-_fY4L1TguMo0sQIuV3FwE5HcU4KLZ9Evon8ZHL2fFe3Wrfx9QVX_uAXwtnXudi5imsBhBVmviYck9Ywx6-MSMv6WVwxvR6xWTkU_0CFX2dd6arT3Dwvf6PvOmSWBMus4HZELVEA9LzBzwOl8jRazASdw4ulXrSTwGWo4sn...>>>>>
/opt/LCM/tmp/downloads/sp-30.0.3-2-0-lnx64.zip~~~~~~~~~~~~~~~~~~
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
100 8944k 100 8944k    0     0   199k      0 0:00:44 0:00:44 --:--:-- 174k

ASD-DOWNLOAD: Removing temporary files...
100 226M 100 226M    0     0   229k      0 0:16:49 0:16:49 --:--:-- 233k

ASD-DOWNLOAD: Removing temporary files...

[root@localhost collectorlogin]# env | grep proxy

[root@localhost collectorlogin]# ls -lart /opt/LCM/tmp/downloads
total 240632
drwxr-xr-x. 3 casuser casusers      4096 Feb 9 09:59 ..
-rw-r-----. 1 casuser casusers       903 Feb 9 09:59 jeos-30.0.3-1-lnx64.zip
-rw-r-----. 1 casuser casusers       903 Feb 9 09:59 sp-30.0.3-1-0-lnx64.zip
drwxr-x---. 2 casuser casusers      4096 Mar 20 14:58 .
-rw-r--r--. 1 root    root       9158909 Mar 20 14:59 sp-30.0.3-2-0-lnx64.zip
-rw-r--r--. 1 root    root     237221619 Mar 20 15:15 jeos-30.0.3-2-lnx64.zip

[root@localhost collectorlogin]# ./toreviewstatus.txt
ij version 10.13
ij> connect 'jdbc:derby://localhost:1527/optLCM/info/updates;create=true;';
ij> select * from updates;
ERROR 42X05: Table/View 'UPDATES' does not exist.
ij>

adias · ‎03-22-2018

I believe you do have the necessary files now for the upgrade

# cp /opt/LCM/tmp/downloads/sp-30.0.3-2-0-lnx64.zip /home/collectorlogin
# cp /opt/LCM/tmp/downloads/jeos-30.0.3-2-lnx64.zip /home/collectorlogin
# cd /home/collectorlogin

To confirm the files are there:

#ls -lart
once you confirm the files are there run the commands I send you in private for Jeos and SP installation

evanswee · ‎03-22-2018

It seems nothing is there

[collectorlogin@localhost ~]$ cp /opt/LCM/tmp/download/sp-30.0.3-2-0-lnx64.zip /home/collectorlogin
cp: cannot stat `/opt/LCM/tmp/download/sp-30.0.3-2-0-lnx64.zip': No such file or directory
[collectorlogin@localhost ~]$ cp /opt/LCM/tmp/download/jeos-30.0.3-2-lnx64.zip /home/collectorlogin
cp: cannot stat `/opt/LCM/tmp/download/jeos-30.0.3-2-lnx64.zip': No such file or directory
[collectorlogin@localhost ~]$ cd /home/collectorlogin
[collectorlogin@localhost ~]$ ls -lart
total 32
-rwx------. 1 collectorlogin collectorlogin 124 May 10 2016 .bashrc
-rwx------. 1 collectorlogin collectorlogin 176 May 10 2016 .bash_profile
-rwx------. 1 collectorlogin collectorlogin 18 May 10 2016 .bash_logout
drwxr-xr-x. 5 root root 4096 Jan 12 17:52 ..
-rwxr-xr-x. 1 collectorlogin collectorlogin 410 Mar 22 13:55 toreviewstatus.txt
drwxr-x---. 2 collectorlogin collectorlogin 4096 Mar 22 13:55 .oracle_jre_usage
drwx------. 3 collectorlogin collectorlogin 4096 Mar 22 13:55 .
-rw-------. 1 collectorlogin collectorlogin 275 Mar 22 14:02 .bash_history

adias · ‎03-22-2018

the above folder should have been downloads

# cp /opt/LCM/tmp/downloads/sp-30.0.3-2-0-lnx64.zip /home/collectorlogin
# cp /opt/LCM/tmp/downloads/jeos-30.0.3-2-lnx64.zip /home/collectorlogin
# cd /home/collectorlogin

To confirm the files are there:

#ls -lart

evanswee · ‎03-23-2018

Copying these files and running the manual upgrade with the commends send via IM worked ok, running version 2.7.3.2 now, thanks! I hope upgrade to 2.7.4 will go smoother :).

adias · ‎03-23-2018

I hope so too, perhaps there were other steps within the admin shell we could have taken that was going to get the same result like mentioned by my colleague Taylor which was

download the patch again from the admin shell

and then reapply it seems that the GUI downloading was placing it on the wrong place. but just wanted to ensure the steps we have already established which was the manual upgrade as show on the CSPC upgrade Guide from the release notes.

Cisco Service · ‎03-22-2018

adias · ‎03-22-2018

Here was the full suggestion from taywelch

Please run the download sp-30.0.3-2-0-lnx64 command again from adminshell if the download is successful.

admin# download sp-30.0.3-2-0-lnx64

admin# apply sp-30.0.3-2-0-lnx64

taywelch has seen a few cases where the download never completes and this command when executed again restarts the download process. If the download is successful run the apply sp-30.0.3-2-0-lnx64 command. as shown above