Cisco Prime 3.0 crashed after installing pi_technology_package-3.0.2-1.0.56

[David Serarols]

Hello,

Our Cisco Prime 3.0 has crashed after installing pi_technology_package-3.0.2-1.0.56. SSH is available but not web.

Here is the story:

1-. Installed OVA

2-. Installed these following patches:

Device-Pack-3-PI3.0-25.ubf

Device-Pack-4-PI3.0-15.ubf

pi302-16.ubf

pi303-10.ubf

Server restarted and all seemed to work fine.

3-. Installed pi_technology_package-3.0.2-1.0.56, server restarted and web does not work any more.

We have tried to restore a previous backup, perform a ncs db reinit and restart the ESXi machine,  but without any success.

genpro01/admin# ncs status
Health Monitor is running, with an error. ( [Role] Primary [State] HA not Configured )
initHealthMonitor(): can not start DB
Database server is stopped
Ftp Server is running
Tftp Server is running
Matlab Server is running
Matlab Server Instance 1 is running
Matlab Server Instance 2 is running
NMS Server is stopped.
CNS Gateway with port 11011 is down
CNS Gateway SSL with port 11012 is down
CNS Gateway with port 11013 is down
CNS Gateway SSL with port 11014 is down
Plug and Play Gateway Broker with port 61617 is down
Plug and Play Gateway config, image and resource are down on https
Plug and Play Gateway is stopped.
SAM Daemon is stopped.
DA Daemon is stopped.
genpro01/admin#

Any suggestions?

Regards,

David

[Marvin Rhoads]

Did you stop/start the ncs application after applying each patch?

The tech pack would not normally let you install over 3.0.3. It should do a check looking for version 3.0.2 to be the current  installed patch level and fail if that's not the case.

At this point you will probably need to open a TAC case and have them repair your system.

[David Serarols]

Hi Marvin. It's nice to see you here again.

I applied patches from web interface, so I didn't stop/start ncs service. I read this patch is just for 3.0.2 but a check was performed by Prime. So since I was not warned by any message, I carried on with the upgrade. Now I realise it was a mistake.

Fortunately this Prime was a new installation with few devices added. Do you think we could perform a system restore? I tried but I obtained this error:

(omitted)

Stage 4 of 9: Decompressing backup ...
-- completed at 04/18/16 17:08:40
Stage 5 of 9: Restoring Support Files ...
-- completed at 04/18/16 17:08:41
Stage 6 of 9: Restoring Database Files ...

-- completed at 04/18/16 17:08:49
Stage 7 of 9: Recovering Database ...
Error while restoring and recovering database, please check the logs.
% Application restore failed
genpro01/admin#

Regards,

David

[Marvin Rhoads · Accepted solution]

David,

Unfortunately things can get really hosed up (technical term) with the database if the application server (ncs) is not restarted after each patch or device pack installation. The later patch versions have begun highlighting this right up front.

I'd say you have two options -

1. Rebuild from scratch and just re-import the devices. That's a self-service option.

2. Open a TAC case and see if they can recover from the current state. The reports I've had from colleagues and read elsewhere here is that recovery from a failed installation of that tech pack has been difficult at best.

Sorry to be the bearer of bad news.

[David Serarols]

Thank you very much for your reply.

We are going to perform a full reinstall. Since there were few devices we will not have much extra work.

Regards,

David

[Claudio Gonzalez Sequeira]

Hi David, 

As Marvin Mentioned, Tech pack should not be install on 3.0 or 3.0.3, you need to install 3.0.2 first. 

since, you did it on 3.0.3, I might think that the database is corrupted, unfortunately there is not way to uninstall the patches.

What I recommend you is to deploy Prime 3.0 from scratch, restore a "safe" backup before applying the patches then proceed to install the patches in the following order : 3.0 > 3.0.2 > Tech pack > 3.0.3

From your previous error, I guess you are restoring a backup with all the patches installed and for that reason there is a database issue. 

Try it with a different backup and if you need help getting 3.0.2 let me know and I can publish that file for you.

Best regards, 

Claudio Gonzalez