Admin unable to sign in to CSP Collector CLI or GUI; Hard drive filling up quickly

[cmking]

I initially thought the CSP Collector’s full hard drive (%100 Full) was preventing me from accessing the application (cli access).  After additional troubleshooting, I am not sure the full hard drive was preventing cli access functionality as after remembering the additional Collector accounts that were initially setup, I was able to get logged into the OS of the appliance as collectorlogin, free up hard drive space (removed some log files), and add 100GB to the OS partition.  I then tried logging into the appliance (cli) and GUI as the admin application user and received the same results.  The admin application user is unable to log in to the GUI or CLI.  At the cli, the admin user login attempt experiences a quick flash and then immediately closes the connection to the session.  The application GUI just fails to load. 

I read in the Cisco forums that this may be a bug in the appliance and may be addressed in later versions of the appliance?  In addition, I was unable to get the web application working.  When trying to navigate to the appliance IP address web interface, I can never get a web page to load.  When viewing all of the applications that are listening on the appliance, I am not seeing port 8001 listening so am thinking that the web application is not even running?  I am not sure how to use the collectorlogin account to get the application functioning again or apply a possible patch to regain Admin user account access to the device.  I am thinking I may be further ahead by just starting over??  This is our second attempt at deploying this appliance and noticed both times the hard drive filled up rather quickly so prefer to find a fix to the hard drive becoming full before trying another fresh deployment of the appliance.

Thanks in Advance for any participation!!

[npicard]

Hello,

An engineer is reviewing your request. We will post a response shortly.

[CC Cisco]

Hello!

This is a test!

[CC Cisco]

Hello,


To help pinpoint your problem, could you please share the following:

* Names of the log files that were filling up your hard drive.
* Which CSPC version you have installed, and which virtualization platform you are using. Or if you are using a hardware server.

A few things I recommend checking would be the following:


* Please make sure that your /etc/hosts file is not empty. The default entries are the following:

72.163.9.100 nettools-upload.cisco.com
192.135.250.12 nettools-upload.cisco.com
64.103.36.134 nettools-upload.cisco.com
72.163.7.113 concsoweb-prd.cisco.com
173.37.186.17 concsoweb-prd.cisco.com
72.163.4.161 www.cisco.com
72.163.7.60 dl.cisco.com
72.163.7.60 dl1.cisco.com
173.37.146.12 dl2.cisco.com
173.37.144.208 sso.cisco.com
127.0.0.1  localhost.localdom localhost localhost
::1     localhost.localdom localhost localhost ip6-localhost ip6-loopback

* Delete the file /etc/ptmp if it exists.
* Reboot the server.

After the reboot, please wait about 10-15 minutes and see if you are able to log into the admin shell and/or Web GUI.


If you still have issues, please start an SSH session and run the below command as root. Please paste here any output from that command. (This restart should take 7-8 minutes).

# service cspc restart

Please reply back with any feedback to see if there are further things we can suggest.


Regards,

Cesar

[cmking]

Thanks for the Reply Cesar!!  To answer your questions:

'Names of the log files that were filling up your hard drive:'

After logging in as collectorlogin, I saw that the hard drive was %100 full.  In order to free up some space, I found that a messages file was approximately 70GB in size.  I copied that off to give me room to extend the drive.  After the drive was extended, I was going to copy the file back to the location but seen that a new file was created as I expected the OS to do.

'Which CSPC version you have installed, and which virtualization platform you are using. Or if you are using a hardware server'

The CSPC version is 2.4 and it was deployed as a VMware appliance downloaded from Cisco's Site.  Since the appliance has become unusable, I have tried upgrading to version 2.5 by applying patch: casp1.6.0.4_security and upgrade: jeos-3.0.6-0-lnx64andsp-3.0.6-0-0-lnx64

We use a proxy server so am not sure the /etc/hosts file still needs to contain all of those entries??  I did peak into the file and only see 127.0.0.1 listed

'Delete the file /etc/ptmp if it exists.'

I did not see a /etc/ptmp file to delete.

I was unable to successfully restart the service.  Here is the error that I am receiving when trying to do this:

[root@localhost etc]# service cspc restart
Stopping tomcat: /etc/init.d/tomcat: line 48: cd: /webui/tomcat: No such file or directory

Stopped tomcat:
Stopping Common Services Platform Collector:
Common Services Platform Collector: is not running.
Starting Common Services Platform Collector:
-sh: /etc/init.d/mysql: Permission denied
Starting CSPC Helper Servers:
touch: cannot touch `/run/cspc_utils.pid': No such file or directory
chown: cannot access `/run/cspc_utils.pid': No such file or directory
-bash: /bin/cspc_helper_servers.sh: No such file or directory

Starting Base Collector and add-ons touch: cannot touch `/run/cspc.pid': No such file or directory
chown: cannot access `/run/cspc.pid': No such file or directory
-sh: /bin/cspc_server.sh: No such file or directory
chown: cannot access `/run/subsys/cspc': No such file or directory
nohup: ignoring input and appending output to `nohup.out'
Starting tomcat: /etc/init.d/tomcat: line 28: cd: /webui/tomcat/bin: No such file or directory
/etc/init.d/tomcat: line 29: /cspc_tomcat_start.log: Permission denied

Started tomcat:
chmod: cannot access `/cspc_tomcat_start.log': No such file or directory

I am thinking I will just start over??

Thanks Again for the help Cesar!

[CC Cisco]

Hello, thanks for the response.

Based on this error, it seems that at some point during the time where the hard disk was full some corruption took place.

Starting Base Collector and add-ons touch: cannot touch `/run/cspc.pid': No such file or directory
chown: cannot access `/run/cspc.pid': No such file or directory
-sh: /bin/cspc_server.sh: No such file or directory
chown: cannot access `/run/subsys/cspc': No such file or directory
nohup: ignoring input and appending output to `nohup.out'
Starting tomcat: /etc/init.d/tomcat: line 28: cd: /webui/tomcat/bin: No such file or directory
/etc/init.d/tomcat: line 29: /cspc_tomcat_start.log: Permission denied

My suggestion at this time would be to if possible, start from scratch and deploy a new VM. As this would probably be easier than try to troubleshoot via this forum.

A few things to keep in mind:

Even though a proxy server is in place, the hosts file still needs to be configured with the entries posted above.

The /var/log/messages file is not required, therefore we can clear it as needed. The default I believe is to have it rotate.

If your file is not rotating or it it's filling up too quickly, we can try to troubleshoot that. Please share the contents of your logrotate settings:

/etc/logrotate.conf

/etc/logrotate.d/syslog

Please run the following command to make sure the SNMP Traps server is disabled which could also be a cause for the hard drive to fill up. (Run command as root).

touch $CSPCHOME/donotStartSyslog $CSPCHOME/donotStartSnmpTrap $CSPCHOME/donotStartFtp $CSPCHOME/donotStartTftp; chown casuser:casusers $CSPCHOME/donot*; chmod 710 $CSPCHOME/donot*; service cspc restart

Please reply back if you run into any other issues or have other questions!

Regards,

Cesar