Solved: Discovery problem with 2.9.1.2

Charlie_Z · ‎11-22-2021

I upgraded from 2.9.1.1 to 2.9.1.2. Everything seemed fine. I went in via the GUI and verified the upgraded version. I then tried to run a collection. The web page hung. I closed the browser and reopened. The login dialog would not show. I got a blank screen. Decided to just reboot the device. ssh into CSPC as admin successful. Typed reboot and waited. Tried ssh again. Now I get the error "Adminshell service is not running. Please check the status using 'service adminshell status' command.". Then I get disconnected.

Looks like the Discovery is causing the problem. I ran a separate discovery from the scheduled discovery and it fails.

Charlie_Z · ‎12-03-2021

Since the base OS for 2.9.1.2 goes end of life this month, instead of continued troubleshooting, I decided to build a 2.10 server and recreate the settings.

View solution in original post

vivekcha@cisco.com · ‎11-22-2021

Hi Charlie,

Hope you are doing well!

We are looking into the issue and get back to you with updates.

Regards,

Vivek Chauhan

vivekcha@cisco.com · ‎11-22-2021

Hi Charlie,

Could you please let us know that have you upgraded the CSPC from CLI or GUI?
Please login as collectorlogin then go to root user in cspc cli:
To connect to root first connect to collectorlogin prompt and then issue the command su root as shown below
$ su root

a. Could you please run the following in CSPC cli
i. Run these commands in Root
ii. Run the following commands to restart the services:
iii. systemctl restart tomcat
iv. systemctl restart adminshell
v. systemctl restart cspc
vi. Then run to check the status:
vii. systemctl status tomcat
viii. systemctl status adminshell
ix. systemctl status cspc
b. Once you do the following please go back to the GUI then run the sntc collection profile again.
> It is recommended to clear cache before login to CSPC GUI

If you face any issue then please share the error.

Regards,

Vivek Chauhan

Charlie_Z · ‎11-23-2021

Upgraded from CLI.

I restarted the 3 services, tomcat, adminshell and cspc.

Tomcat and adminshell returned to prompt.

After restarting cspc and returning to prompt:

[root@localhost collectorlogin]# [SSH] ERROR: Read from remote host 172.30.46.20: An existing connection was forcibly closed by the remote host. [SSH] FAIL: client_loop: send disconnect: An existing connection was forcibly closed by the remote host.

Logged back into CLI

[root@localhost collectorlogin]# systemctl status tomcat
— tomcat.service - SYSV: Tomcat Server basic start/shutdown script
Loaded: loaded (/etc/rc.d/init.d/tomcat; generated)
Active: active (exited) since Tue 2021-11-23 08:47:34 EST; 6min ago
Docs: man:systemd-sysv-generator(8)
Process: 65850 ExecStop=/etc/rc.d/init.d/tomcat stop (code=exited, status=1/FAILURE)
Process: 67494 ExecStart=/etc/rc.d/init.d/tomcat start (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 23637)
Memory: 0B
CGroup: /system.slice/tomcat.service

Nov 23 08:45:34 localhost systemd[1]: Starting SYSV: Tomcat Server basic start/shutdown script...
Nov 23 08:45:34 localhost su[67524]: (to casuser) root on none
Nov 23 08:45:34 localhost su[67524]: pam_unix(su-l:session): session opened for user casuser by (uid=0)
Nov 23 08:45:34 localhost tomcat[67494]: Starting Tomcat... su: warning: cannot change directory to /home/casuser: No s>
Nov 23 08:45:34 localhost tomcat[67494]: -sh: line 0: ulimit: max user processes: cannot modify limit: Operation not pe>
Nov 23 08:45:34 localhost tomcat[67494]: Tomcat started.
Nov 23 08:45:34 localhost su[67524]: pam_unix(su-l:session): session closed for user casuser
Nov 23 08:47:34 localhost tomcat[67494]: [ OK ]
Nov 23 08:47:34 localhost tomcat[67494]: chmod: cannot access '/opt/cisco/ss/adminshell/applications/CSPC/cspc_tomcat_s>
Nov 23 08:47:34 localhost systemd[1]: Started SYSV: Tomcat Server basic start/shutdown script.

[root@localhost collectorlogin]# systemctl status adminshell
— adminshell.service - SYSV: none
Loaded: loaded (/etc/rc.d/init.d/adminshell; generated)
Active: active (running) since Tue 2021-11-23 08:42:45 EST; 12min ago
Docs: man:systemd-sysv-generator(8)
Process: 65445 ExecStop=/etc/rc.d/init.d/adminshell stop (code=exited, status=0/SUCCESS)
Process: 65484 ExecStart=/etc/rc.d/init.d/adminshell start (code=exited, status=0/SUCCESS)
Tasks: 69 (limit: 23637)
Memory: 599.0M
CGroup: /system.slice/adminshell.service
”œ”€65552 /bin/sh /etc/rc.d/init.d/adminshell start
”œ”€65556 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/bin/nohup /opt/cisco/ss/adminshell/derby/bin/star>
”œ”€65557 /opt/java/jre/bin/java -classpath /opt/cisco/ss/adminshell/derby/lib/derby.jar:/opt/cisco/ss/adminsh>
”œ”€65584 /bin/sh /etc/rc.d/init.d/adminshell start
”œ”€65588 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/bin/nohup /opt/cisco/ss/adminshell/bin/shellserver
”œ”€65589 /bin/sh /opt/cisco/ss/adminshell/bin/shellserver
”””€65595 java -Djsse.enableSNIExtension=false -cp /opt/cisco/ss/adminshell/properties:/opt/cisco/ss/adminshel>

Nov 23 08:42:14 localhost adminshell[65484]: ]
Nov 23 08:42:24 localhost adminshell[65484]: Starting Derby DB: [ OKtput: No valu>
Nov 23 08:42:24 localhost adminshell[65484]: ]
Nov 23 08:42:25 localhost adminshell[65484]: Stopping Derby DB: [ OKtput: No valu>
Nov 23 08:42:25 localhost adminshell[65484]: ]
Nov 23 08:42:35 localhost adminshell[65484]: Starting Derby DB: [ OKtput: No valu>
Nov 23 08:42:35 localhost adminshell[65484]: ]
Nov 23 08:42:45 localhost adminshell[65484]: Starting Adminshell server: [ OKtput>
Nov 23 08:42:45 localhost adminshell[65484]: ]
Nov 23 08:42:45 localhost systemd[1]: Started SYSV: none.

[root@localhost collectorlogin]# systemctl status cspc
— cspc.service - SYSV: CSPC Start/Stop Script
Loaded: loaded (/etc/rc.d/init.d/cspc; generated)
Active: active (exited) since Tue 2021-11-23 08:47:35 EST; 8min ago
Docs: man:systemd-sysv-generator(8)
Process: 65791 ExecStop=/etc/rc.d/init.d/cspc stop (code=exited, status=0/SUCCESS)
Process: 66205 ExecStart=/etc/rc.d/init.d/cspc start (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 23637)
Memory: 0B
CGroup: /system.slice/cspc.service

Nov 23 08:43:59 localhost su[66684]: pam_unix(su-l:session): session opened for user casuser by (uid=0)
Nov 23 08:43:59 localhost cspc[66205]: [89B blob data]
Nov 23 08:43:59 localhost su[66684]: pam_unix(su-l:session): session closed for user casuser
Nov 23 08:45:21 localhost cspc[66205]: [770B blob data]
Nov 23 08:45:33 localhost su[67441]: (to casuser) root on none
Nov 23 08:45:33 localhost su[67441]: pam_unix(su-l:session): session opened for user casuser by (uid=0)
Nov 23 08:45:33 localhost cspc[66205]: [267B blob data]
Nov 23 08:47:34 localhost cspc[66205]: Starting tomcat (via systemctl): [ OK ]
Nov 23 08:47:35 localhost cspc[66205]: [54B blob data]
Nov 23 08:47:35 localhost systemd[1]: Started SYSV: CSPC Start/Stop Script.

Cleared cache in browser prior to going into GUI.

Requires authentication twice. First time returns Invalid credentials even though i know it was typed correctly.

Clicked Collect Button to run collection profile. No response. Clicked again. No response. Canceled operation.

Clicked Job Run Status. Stuck on loading. No progress.

vivekcha@cisco.com · ‎11-23-2021

Hi Charlie,

Thank you for sharing the details. Please provide the below details in PM:

Company name:

CCO ID of user:

Regards,

Vivek Chauhan

sjames · ‎12-02-2021

I see the same issues after upgrading to 2.9.1.2. The GUI dies after you request a discovery. I also had it die after trying to change a collection setting. My log files look similar to the ones provided by Charlie_Z.

Charlie_Z · ‎12-03-2021

Since the base OS for 2.9.1.2 goes end of life this month, instead of continued troubleshooting, I decided to build a 2.10 server and recreate the settings.