Update to 2.7 stuck at Apply-in-Progress

[Cisco SmartNet]

Hi there,

I was updating our collector from 2.5 to the newest 2.7. I followed the documentation and patched it via linux shell to add the "Software Updates" on the GUI.

After that I started the update and it kept stuck at the "Apply in Progress" status. Tried to log in through SSH but as soon as I log I got the messages:

java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:244)
at com.cisco.ca.ss.lcm.adminshell.client.ShellClient.main(ShellClient.java:51)
Error occured: could not connect to the database server.
Please make sure mysql database server is up and running.

I'm able to log in on the GUI but the version still 2.5 and I can't restart the update procedure because it's stuck as Apply in Progress, also there is no way to gain access over SSH using the admin credential. 

Also tried to restart the VM but nothing changed...

[CC Cisco]

Hello, if you have root access, can you please extract and share the following files:

/opt/LCM/logs/install

/opt/LCM/logs/apply

These will help for analysis of the problem.

Regards,

Cesar

[Cisco SmartNet]

Hi Cesar,

Yeah, I still have root access and I'm attaching the logs you requested.

Thanks

[CC Cisco]

Thank you,

I've shared this with the CSPC team. Will let you know any feedback.

Regards,

Cesar

[CC Cisco]

Hello,

I received a few extra steps for you to try. I will be sending them to you via private message.

Regards,

Cesar

[Cisco SmartNet]

Hi there,

I did the extra steps you asked me to do, and on the GUI I was able to restart the update process. Unfortunately, the process now is stuck on the "Apply in progress" status again.

I'm also unable to login using SSH, the same messages from the 1st post are there.

What a troublesome process. I'm glad I tried this update process over our own lan and collector first, I can only imagine what our clients would say to me if this happens to their collectors.

I really hope Cisco set this thing right or else people will start to abandon this plataform.

Best regards.

[john.chavez1]

Would you happen to have the extra steps they sent you still?  I am having this issue.  Thank you.

[trapasso]

I happened to have my ssh console output to a file when this issue was
resolved. I cannot guarantee that this procedure will fix the problem you
are using, so use with care and at your own risk.

[collectorlogin@ ~]$ su


[root@sp-30.0.3-2-0-lnx64]# cd /opt/LCM/apply/tmp/
[root@tmp]# ls
jeos-30.0.3-2-lnx64 jeos-30.0.3-2-lnx64.zip sp-30.0.3-2-0-lnx64
sp-30.0.3-2-0-lnx64.zip
[root@ tmp]# cp sp-30.0.3-2-0-lnx64.zip /home/collectorlogin
[root@ tmp]# cd /home/collectorlogin/
[root@ collectorlogin]# ls

index.html index.html.1 index.html.2 index.html.3
jeos-20.1.0-0-lnx64.zip sp-2.0.6-0-0-lnx64.zip sp-30.0.3-2-0-lnx64.zip

[root@collectorlogin]# unzip sp-30.0.3-2-0-lnx64.zip

[root@ collectorlogin]# ^C
[root@ collectorlogin]# cp sp-30.0.3-2-0-lnx64.zip /home/collectorlogin

[root@collectorlogin]# /opt/LCM/bin/lcmagent-apply.tcl SP
sp-30.0.3-2-0-lnx64.zip &

[root@collectorlogin]# tail -f /opt/LCM/logs/install



Hope it helps.

[Cisco SmartNet]

Hi there,

Just a follow up, I gave up on the update process, deleted the 2.6 CSCP and made a fresh install of 2.7.

Guys, avoid at ALL COST to upgrade yours 2.x collectors, there is a good chance to get your perfectly fine and working collector ruined.

[CC Cisco]

Hello,

I've got an update from the CSPC team, can you run these two commands for me as well and provide the output?

*must run them as root

sqlite3 /opt/LCM/info/status.db    

select * from status;

Regards,

Cesar

[Cisco SmartNet]

Hi there,

Done.

sqlite> select * from status;
ConcsoTgw|Component|JeOS|1.6.4|Connectivity TGW component
AdminShell|Component|JeOS|1.3|AdminShell component
Hardened CentOS|Component|JeOS|6.8 patch#0|Hardened CentOS
JeOS|JeOS||jeos-20.1.2-0-lnx64|Appliance Base Package patch update
Data Pack|Component|SP|RP3.22|RP3.22 fix for SNTC
CSPC Base|Component|SP|2.5.2.1|CSPC Base component
SP|SP||sp-2.0.4-0-0-lnx64|This software pack contains components for SNTC1.x market-offer
CASP|Component|JeOS|1.10|CASP component
LCM|Component|JeOS|1.3|LCM component
sqlite>

You need anything else?

Thanks!

[trapasso]

Anyone has a procedure to fix this issue?

There is a known bug, https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvh79086.  And the workaround is:

Workaround:
Unzip the downloaded sp-30.0.x patch and manually install the jeos & sp patches separately
or
install "jeos-20.1.2-0-lnx64" before initiating the 2.7 upgrade

But I would like to know if this works and have an actual procedure.

Thanks

[adias]

.

Can you please see if running the steps in the section "Known Issue" of the latest Release notes helps to resolve the error you're seeing?  After running the steps, we will need to restart the patching process.

The Release notes are available here.

Thank you,

[adias]

Customer performed the manual install for the missing SP patch similar to noted on the guide above session 6 -guide cspc upgrade BUT instead of Jeos he did for the SP patch.