Cisco Prime 3.5 ncs application won't start

[network-pharma]

Hello,

We have Cisco Prime 3.5 installed and it was working just fine. After shutting down and powering on the VM for maintenance, the web console won't start. NCS application won't start as well.

This is the output of ncs status:

cisco-prime/admin# ncs status
Health Monitor is stopped.
Database server is stopped
FTP Service is Stopped
TFTP Service is Stopped
Matlab Server is Stopped
Matlab Server Instance 1 is Stopped
Matlab Server Instance 2 is Stopped
Matlab Server Instance 3 is Stopped
NMS Server is stopped.
Coral Service is stopped..
WSA Service is stopped..
SAM Daemon is stopped.
DA Daemon is stopped.
Compliance engine is not running

We tried starting NCS, it shows absolutely no output. We looked for the launchout.log file but can't find it.

We also tried this link: http://transmitfailure.blogspot.com/2013/04/cisco-prime-infrastructure-bug-id.html but with no result.

Unfortunately our support has ended so we can't open a TAC case. Will you please provide assistance to recover PI, hopefully without losing the data?

[Mark Malone]

Hi
did you try an NCS reload , boot the whole lot
Database maybe corrupt , you could try reinstall it but make a backup copy first though

[network-pharma]

There isn't a reload command. I tried stopping then starting the application, nothing changed.

[marce1000]

 - Also check the VM's primary-console when powering-up. Are there any error messages ?

   M.

[network-pharma]

It starts OS and then just a black screen. No errors, no messages.

[marce1000]

 - You say it starts OS, does this include a trail of Linux services being started with green status on the right ? Also is there a login prompt on the console after the boot ? It should be there.

 M.

[network-pharma]

After multiple VM restarts, it showed the message that Cisco Prime is starting but after it finished, I ran show application status NCS command and it showed that all services are still stopped.
I ran the ncs start command, it showed some error messages. A screenshot is attached.

[network-pharma · Accepted solution]

The issue is finally resolved. I did the following 2 steps:

1. I ran the following commands I came across in another reference which eliminated the error of no space left on device. I found the var filesystem to be completely full after running "show disk" command.

#enter shell:
shell
#get root
sudo su -
#write an empty new wtmp file (this can take a minute)
cat /dev/null > /var/log/wtm

 2. I then came across a link to fix the rpm issue which rebuilds the database and that's what got it up and running.

Log in to shell and switch to root user then execute the following command: rpm --rebuilddb

Thank you for your assistance.

[zuerchermi]

We experienced the exact same problem and your solution worked perfectly, thank you very much!
How did you handle the space problem to prevent future occurrences if I may ask?

Kind regards,

Michael