Cannot SSH / console to CSPC

[krinaldo-wab]

Suddenly I am unable to gain console access to the CSPC appliance via SSH nor local console.  When I log in, credentials for the "admin" account are accepted and then the following is displayed:

#########################################################################
#     This system is hardened and for the use of authorized users only. #
#     Individuals using this computer system without authority, or in   #
#     excess of their authority, are subject to having all of their     #
#     activities on this system monitored and recorded by system        #
#     personnel.                                                        #
#                                                                       #
#     In the course of monitoring individuals improperly using this     #
#     system, or in the course of system maintenance, the activities    #
#     of authorized users may also be monitored.                        #
#                                                                       #
#     Anyone using this system expressly consents to such monitoring    #
#     and is advised that if such monitoring reveals possible           #
#     evidence of criminal activity, system personnel may provide the   #
#     evidence of such monitoring to law enforcement officials.         #
#########################################################################
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at java.net.Socket.<init>(Unknown Source)
        at java.net.Socket.<init>(Unknown Source)
        at com.cisco.ca.ss.adminshell.client.ShellClient.main(ShellClient.java:63)
Error occured: could not connect to the command server

I have tried power cycling the CSPC.

Anyone have a suggestion to remedy this?

[Lynden Price]

Hello, my name is Lynden Price from the Collector Support team. I'd like a little more information to help troubleshoot.

Did you generate a collectorlogin or root login when you first set up your collector. You could try logging in with one of those to help troubleshoot further.

Have you ever been able to log in or is this a recent development?

Thanks,

Lynden

[krinaldo-wab]

Hi Lynden,

I followed the setup guide for the CSPC.  It never prompted me to create a root or collector login, only to use the "admin" built in account and change the default password, which I did.

I configured the collector (through the GUI) with a TACACS-authenticated account for myself, which I use to login to the GUI, however, that account cannot access the SSH / console -- I get an authentication failed message.

When I log into SSH / console using the built-in "admin" account, it accepts the authentication / password, however the error I posted above is then displayed and the session disconnects.

I was able to log into the device using SSH / console and the built-in "admin" account for the first few days I had the CSPC online, only in the past couple days has this problem arisen.  I have power cycled the CSPC VM and it did not clear up the problem.

As it stands, I am able to access the GUI of the collector using my TACACS account, however I am not able to SSH / console to it.

Thanks.

[Lynden Price]

The error message you are getting means that there is an issue with the connection of your client (your PC) to the server (the CSPC). For SSH this could be caused by a variety of settings issues. In your case, since you are getting the same error on the console, it could indicate a more serious problem with the server.

The only thing I can suggest is to check that the resources allocated to the VM meet the minimum requirements:

• 2x 64bit Processor Core
• 4GB RAM available for the virtual server
• 200GB free on HDD
• 1 NIC
• VMware ESXi 4.x or 5.x

If you haven't done much work yet, I would actually recommend redeploying the OVA. It isn't possible to troubleshoot your issue without root access, and you can't enable root if you can't log in. 

From the GUI, you can export your managed device list by going to Reports > Managed Devices and selecting the export option (in csv). 

When you redeploy run the following commands:

pwdreset collectorlogin 180

pwdreset root 180

Then record those passwords.

Once you bring up the new OVA, you can use the same entitlement, and then just reconfigure your SNMP credentials and rediscover your devices by using the IP addresses from the csv.

-Lynden

[krinaldo-wab]

Thanks, Lynden.

A couple points of note -- this appliance is distributed as an OVA/OVF, which allocates the resources for the VM as part of the template.  These aren't things that are configured or chosen when the appliance is deployed, if there's anything wrong with the resources allocated to the appliance, it is part of the template and should be addressed by those building said template.

Also, the error I'm seeing is not related to connectivity, my connectivity to the appliance is fine -- it's a Java application issue because the software layered on top of the Linux shell is Cisco's proprietary shell, written in Java, of all things.  I won't get into the sanity of that design architecture.

The CSPC_Quick_Start_Guide_for_SNTC.pdf does not mention the collectorlogin nor root accounts, the only account referenced is "admin."  Why does there need to be so many local accounts on this appliance?

If this product is intended to be self-supported / community-supported, then the documentation to that end really should be updated to be more complete and accurate.

Having to redeploy the appliance and start over because of a Java application problem that developed magically on its own is frustrating, to say the least.  Not only does it waste time, but it doesn't address the root cause, and I have little confidence that the same thing won't just happen again.

[Lynden Price]

I understand your frustration, but there is simply no way to troubleshoot what the underlying issue could be if we aren't able to log in. There is nothing we can do from the GUI, restarting doesn't work, the only remaining option is to re-deploy.

I agree that the documentation should mention to create the Linux login credentials since, as we can see in this case, creating them is essential to being able to maintain the server. That is a major oversight and I'll make certain it is corrected.

If the issue does reoccur with the new deployment, then we can log into root and determine the cause.

[kgrego]

Thanks for the investigation and explanation Lyndon. 

Just to verify, the issue is that we changed the default admin account after deploying the .ova file, and after a certain time we see this issue? However, we can change the admin account but we must also update the root password. Then if the admin account gets locked out with this error we can log in as root and have the same level of permissions as the admin account?

Regards,

Keith Gregory

[krinaldo-wab]

Hi Lynden, I want to be sure I understand as I just redeployed my CSPC and need to set it up...

There is the "admin" account that is used at the CLI (console/SSH), and the "admin" account that is used through the GUI -- of course, the two are named the same but aren't the same account.

Can I change the CLI "admin" password, but NOT the GUI "admin" password?  Or neither?

[Lynden Price]

Yes, you are prompted to change the admin for the CLI. So long as the one in the GUI isn't changed, there shouldn't be an issue. Also, remember to generate the collectorlogin and root passwords and record those as well.

-Lynden

[D Kendz]

I have just been able to duplicate this issue in version 2.5.1.2 which is reportedly running the aforementioned patch.  See attached screenshot CSPC_Ver.  The funny thing is when you run the "check update" command, you see this (CSPC_patch) in the list of patches.  It exactly matches the version, and even references that this patch will bring your CSPC from ver 2.4 to 2.5 and enable support for SNTC 3.x.  If you take a leap of faith and try to download, you'll get a EULA prompt, but then the CSPC will almost come to a complete stop, for hours, only to tell you your download failed because your CCO credentials are wrong.   Why would I patch a server that is already running the patch?  Why would that fix anything?  So with no way to patch why wouldn't any enterprise serious about security just scrap this thing?  SNTC portal will accept manual uploads via .csv.  Cisco is likely wrapping this functionality into PRIME and will not continue to support or make fixes for this CSPC.  So to all those banging your heads, get the message  BUY PRIME and get ready.   

If for some reason I am reading this wrong, and the patch is either a) completely different than what I'm running and for some unknown reason given the same name (???) , b) needs to be re-installed on 2.5.2.1 to fix this issue, then I will need a manual download since the CSPC is incapable of authenticating correctly on Cisco's servers. 

[Lynden Price · Accepted solution]

Hi again,

On further investigation, there is actually a bug that seems to cause this issue. If you change the default password for "admin" in the GUI, it causes a problem with the connection to the mysql database.

If you changed the admin password last time, then as long as you don't change the admin password on your re-deployment, then you shouldn't run into the issue again.

-Lynden

[krinaldo-wab]

Thanks for the investigation.

I have to ask, though, and I'm not being sarcastic, but is the recommendation seriously to deploy this thing on production networks without changing the default admin password?

The infosec people are going to hit the ceiling.

If I change the GUI "admin" password back to the default, will the problem resolve without me having to redeploy?

[Lynden Price]

Unfortunately, changing it back won't correct the issue. There is a bug for this, so once the patch is available, you should be able to change the admin password.

[josp]

has this "patch" been release yet?

This problem occurs even after a redeploy... ask me how i know this 

[Lynden Price]

The patch hasn't been released yet. The only way to prevent the issue is to not change the admin password for the GUI. If you generated the collectorlogin and root logins, then you should be able to SSH into the Linux interface and do whatever you need to until it is released.