CSPC 2.9.1.3 Device collection runs on long time and upload fails

Nike · ‎02-01-2022

We are having two issues on our CSPC after installing the 2.9.1.3 patch:

1. The collection process runs for days and during it runs, the web interface is no more available. When I checked the logs, I found the error message: "Error during discovery process. Error message: DCSO-1000:Discovery live job queue size exceeded maximum limit {0}.".

2. We are no more able to upload to Cisco server, as the job always fail with the error: "Upload failed at 02/01/2022 10 14 38 EST with the response Connectivity service is not running".

Thanks in advance for your assistance.

shipand2 · ‎02-01-2022

Hi,

How did you upgrade your 2.9 server to 2.9.1.3?

Regards,
Shivam Pandey

Nike · ‎02-01-2022

Hi Shivam,

here is the picture of the installed updates:

We upgraded from version 2.9.1.1 to version 2.9.1.3.

Every time I log in from web browser, I receive this notification:

"Software Update Last job triggerd was Upgrade Now and the result of the job - Failed - Download of the package is failed. Please check the logs for more details."

Kindly let me know if you need more information.

shipand2 · ‎02-01-2022

Hey,

It looks like you have upgraded using gui , this might be the reason of such errors you can repatch the cspc using cli or we can schedule a webex meeting to do it

We work from 8am - 5pm EST

Regards,
Shivam Pandey

Nike · ‎02-01-2022

Thanks for your quick reply.

Can you send me the procedure to repatch using CLI, if not too complex, I can try, otherwise, yes, we can schedule a webex meeting to do it together.

Regards,

shipand2 · ‎02-01-2022

Hey,

To upgrade from CSPC CLI we must have login as collectorlogin and root user.
I would like to share steps to upgrade from CLI:

1. Download the Collector patch to the local system (Zip file) from https://software.cisco.com/download/home/286312935/type/286312958/release/2.9.1.3
2. Transfer the patch to the CSPC using the Collectorlogin home directory via WinSCP software. below are the steps to use WinSCP.

Download WinSCP or any other File transfer tool. WinSCP is recommended.
Once download then open WinSCP. The login page will pop-up. Enter the hostname (IP address) of CSPC and login as collector login by entering the username and password.
After login the left side will show your local machine's data and the right side would be CSPC.
Transfer the downloaded CSPC upgrade patch file from local machine to CSPC by drag and drop. Then continue with the step no. 3 in upgrade steps.

3. Log in to the console of CSPC as a root user.
4. Unzip the patch file using the command unzip <Filename> and cd to the unzipped directory as a root user
5. There will be 2 files under the unzipped directory which are JEOS and SP patch.
6. First, we will execute JEOS using the command below:

./install.sh JeOS <JeOS filename>
Then cd to the directory /opt/LCM/logs/ to check the logs using the commands tail -f install and tail -f apply
Once a script is executed successfully, then continue with the step no. 7.

7. Then follow the below steps for the SP patch as well:

./install.sh SP <SP filename>
Monitor the installation by executing tail -f /opt/LCM/logs/apply
After successfully applying the patch in /opt/LCM/logs/apply

The upgrade process will be done and check the version in CSPC GUI.
To know more about this upgrade, you can follow the CISCO CSPC upgrade guide as well.
CSPC Upgrade Guide (cisco.com)

If you have any queries we can schedule the webex

Regards,
Shivam Pandey

Nike · ‎02-01-2022

Thanks,

I will try this and will let you know the result.

If there is any issue, we will schedule a webex meeting.

Regards,

Nike · ‎02-02-2022

Hey Shivam,

I tried the procedure you gave me and couldn't iinstall JEOS and SP as they were already installed.

Find below the output.

Let me know if we can schedule a webex meeting so you can have a look too.

[root@localhost collectorlogin]# ls

collector_2.9.1.3_Build-03.zip

[root@localhost collectorlogin]# unzip collector_2.9.1.3_Build-03.zip

Archive: collector_2.9.1.3_Build-03.zip

creating: collector_2.9.1.3_Build-03/

extracting: collector_2.9.1.3_Build-03/jeos-30.2.1-3-lnx64.zip

extracting: collector_2.9.1.3_Build-03/sp-30.2.1-3-0-lnx64.zip

inflating: collector_2.9.1.3_Build-03/install.sh

[root@localhost collectorlogin]# cd collector_2.9.1.3_Build-03

[root@localhost collector_2.9.1.3_Build-03]# ls

install.sh jeos-30.2.1-3-lnx64.zip sp-30.2.1-3-0-lnx64.zip

[root@localhost collector_2.9.1.3_Build-03]# ./install.sh jeos-30.2.1-3-lnx64.zip

Patch path is not provided. Checking available patches in present working directory

Usage: This script installs the JeOS and SP patch for upgrading the Collector appliance to latest available version

./install.sh <patch-type> <patch-path>

[root@localhost collector_2.9.1.3_Build-03]# ./install.sh JeOS jeos-30.2.1-3-lnx64.zip

Appliance will auto reboot once the patch installation is completed. Press [y/Y] to proceed for installation, any other key to exit :y

Installing JeOS patch...

Patch installation in-progress, system will auto reboot once the patch installation is completed.

Kindly refer the logs in /opt/LCM/logs/ directory for more information.

[root@localhost collector_2.9.1.3_Build-03]# Logging installation - CSPC Home exists

1643794350

[root@localhost collector_2.9.1.3_Build-03]# cd /opt/LCM/logs/

[root@localhost logs]# tail -f install

--------------Removing Rules from icmp_packets------------------

============================================================================

Started execution of pre-install scripts in package jeos-30.2.1-3-lnx64 ...

============================================================================

Latest or 'jeos-30.2.1-3-lnx64' JeOS patch is already installed , so exiting from installation....

^C

[root@localhost logs]# tail -f apply

[Wed Feb 2 04:33:18 EST 2022] Cyclic dependency check

[Wed Feb 2 04:33:18 EST 2022] Apply TCL returned 1

[Wed Feb 2 04:33:18 EST 2022] Executing pre-install scripts

[Wed Feb 2 04:33:19 EST 2022] Latest or 'jeos-30.2.1-3-lnx64' JeOS patch is already installed , so exiting from installation....

child process exited abnormally

^C

[root@localhost logs]# ls

apply install

[root@localhost logs]# cd /

[root@localhost /]# ls

bin derby.log etc lib log4j.log lost+found mnt proc run srv tmp var

boot dev home lib64 logs media opt root sbin sys usr

[root@localhost /]# cd home

[root@localhost home]# ls

admin collectorlogin

[root@localhost home]# cd collectorlogin/

[root@localhost collectorlogin]# ls

collector_2.9.1.3_Build-03 collector_2.9.1.3_Build-03.zip

[root@localhost collectorlogin]# cd collector_2.9.1.3_Build-03

[root@localhost collector_2.9.1.3_Build-03]# ls

install.sh jeos-30.2.1-3-lnx64.zip sp-30.2.1-3-0-lnx64.zip

[root@localhost collector_2.9.1.3_Build-03]# ./install.sh SP sp-30.2.1-3-0-lnx64.zip

Appliance will auto reboot once the patch installation is completed. Press [y/Y] to proceed for installation, any other key to exit :y

Installing SP patch...

Patch installation in-progress, system will auto reboot once the patch installation is completed.

Kindly refer the logs in /opt/LCM/logs/ directory for more information.

[root@localhost collector_2.9.1.3_Build-03]# Logging installation - CSPC Home exists

1643794614

login as: collectorlogin

Pre-authentication banner message from server:

|

| ############################################################

|

| Please use below url to access CSP Collector appliance GUI

| IPv4 URL : https://10.117.0.20:8001

|

| ############################################################

End of banner message from server

Keyboard-interactive authentication prompts from server:

| Password:

End of keyboard-interactive prompts from server

Warning: your password will expire in 43 days

############################## WARNING!!! #################################

################## READ THIS BEFORE ATTEMPTING TO LOGON ###################

# #

# This System is for the use of authorized users only. Individuals #

# using this computer without authority, or in excess of their #

# authority, are subject to having all of their activities on this #

# system monitored and recorded by system personnel. In the course #

# of monitoring individuals improperly using this system, or in the #

# course of system maintenance, the activities of authorized users #

# may also be monitored. Anyone using this system expressly #

# consents to such monitoring and is advised that if such #

# monitoring reveals possible criminal activity, system personnel #

# may provide the evidence of such monitoring to law enforcement #

# officials. You cannot copy, disclose, display or otherwise #

# communicate the contents of this server except to other Cisco #

# employees who have been authorized to access this server. #

# #

######################### Confidential Information ########################

Last login: Wed Feb 2 04:29:19 2022 from 172.16.252.97

[collectorlogin@localhost ~]$ su

Password:

Warning: your password will expire in 43 days

[root@localhost collectorlogin]# tail -f /opt/LCM/logs/apply

[Wed Feb 2 04:37:04 EST 2022] Executing pre-install scripts

[Wed Feb 2 04:40:44 EST 2022] Removing old Kernels

Inside preinstall script

Fetching Version info

Initializing SP Patch installation .....

current CSPC version is : 2.9.1.3 and casp version is : 2.9.1.3

Current CSPC version 2.9.1.3

SP 2.9.1.3 can install on 2.9.1 and above only

child process exited abnormally

^C

[root@localhost collectorlogin]#

Nike · ‎02-03-2022

Hi Sivam,

I replied yesterday, but it seems you didn't get it.

I was not able to patch by CLI. It failed because it found the version was already there.

[root@localhost collector_2.9.1.3_Build-03]# ./install.sh JeOS jeos-30.2.1-3-lnx 64.zip

Appliance will auto reboot once the patch installation is completed. Press [y/Y] to proceed for installation, any other key to exit :y

Installing JeOS patch...

Patch installation in-progress, system will auto reboot once the patch installat ion is completed.

Kindly refer the logs in /opt/LCM/logs/ directory for more information.

[root@localhost collector_2.9.1.3_Build-03]# Logging installation - CSPC Home ex ists

1643794350

[root@localhost collector_2.9.1.3_Build-03]# cd /opt/LCM/logs/

[root@localhost logs]# tail -f install

--------------Removing Rules from icmp_packets------------------

============================================================================

Started execution of pre-install scripts in package jeos-30.2.1-3-lnx64 ...

============================================================================

Latest or 'jeos-30.2.1-3-lnx64' JeOS patch is already installed , so exiting from installation....

^C

[root@localhost logs]# tail -f apply

[Wed Feb 2 04:33:18 EST 2022] Cyclic dependency check

[Wed Feb 2 04:33:18 EST 2022] Apply TCL returned 1

[Wed Feb 2 04:33:18 EST 2022] Executing pre-install scripts

[Wed Feb 2 04:33:19 EST 2022] Latest or 'jeos-30.2.1-3-lnx64' JeOS patch is already installed , so exiting from installation....

child process exited abnormally

[root@localhost collector_2.9.1.3_Build-03]# ./install.sh SP sp-30.2.1-3-0-lnx64.zip

Appliance will auto reboot once the patch installation is completed. Press [y/Y] to proceed for installation, any other key to exit :y

Installing SP patch...

Patch installation in-progress, system will auto reboot once the patch installation is completed.

Kindly refer the logs in /opt/LCM/logs/ directory for more information.

[root@localhost collector_2.9.1.3_Build-03]# Logging installation - CSPC Home exists

1643794614

login as: collectorlogin

Warning: your password will expire in 43 days

[root@localhost collectorlogin]# tail -f /opt/LCM/logs/apply

[Wed Feb 2 04:37:04 EST 2022] Executing pre-install scripts

[Wed Feb 2 04:40:44 EST 2022] Removing old Kernels

Inside preinstall script

Fetching Version info

Initializing SP Patch installation .....

current CSPC version is : 2.9.1.3 and casp version is : 2.9.1.3

Current CSPC version 2.9.1.3

SP 2.9.1.3 can install on 2.9.1 and above only

child process exited abnormally

shipand2 · ‎02-03-2022

Hey Nike,

Can we schedule a webex as per your availability?

We work from 8am - 5pm est

Regards,

Shivam Pandey

Nike · ‎02-03-2022

Hi Shivam,
What about Monday next week at 11:00 Am EST?

Thanks!

shipand2 · ‎02-03-2022

Hey Nike,

Yaa, that works for me please share your email id so I can share you the invite

Regards,

Shivam Pandey

Nike · ‎02-03-2022

My email address is eric.tankeo@concordia.ca

Thanks for your assistance!

shipand2 · ‎02-07-2022

Hey Eric,

As discussed over webex please deploy the latest cspc ova 2.10.0.1 and if you have any query please feel free to reach us.

Regards,

Shivam Pandey