NCS 1.1.1.24 migration to PI 1.2.1.012

greg.fuller · ‎02-01-2013

I have an open TAC case on this also....but it's taking longer than usual for them to get pertinent info back to me so figured I'd try here.

I'm in the process of migrating from WCS to Prime Infrastructure 1.2 I've already sucessfully completed the WCS -> NCS 1.0.x upgrade, and then the NCS 1.x to 1.1.1.24 upgrade with no issues. I've followed the suggested migration path to PI 1.2 of deploying a new OVF and them migrating the NCS data from the NCS virtual machine to the new PI virtual machine.

I have PI 1.2.1.012 installed in a new VM and up and running with correct licensing. I installed the "ncs_patch-1.1.1.24-upgrade-12.tar.gz" patch on my NCS install before exporting my data from NCS (and rebooted). NCS data was exported using:

backup ncs-backup repository remoteRepo application NCS

Now when I go to import that data into the PI 1.2.1.012 appliance I get the following:

pi-01/admin# restore ncs-backup-130131-1333.tar.gpg repository remoteRepo application NCS

Restore may require a restart of application services. Continue? (yes/no) [yes] ? yes

Initiating restore. Please wait...

Stage 1 of 9: Transferring backup file ...

-- complete.

Stage 2 of 9: Decrypting backup file ...

-- complete.

Stage 3 of 9: Unpacking backup file ...

--complete.

Stage 4 of 9: Decompressing backup ...

-- complete.

Stage 5 of 9: Restoring Support Files ...

-- complete.

Stage 6 of 9: Restoring Database Files ...

-- complete.

Stage 7 of 9: Recovering Database ...

Failed to restore the DB. Exception: java.lang.Exception: RMAN-00571: ===========================================================

pi-01/admin#

I found bugID CSCub35534 which shows the same "RMAN-00571" java exception error so I might be running into this bug. I followed step 1 in the bugID of stopping my NCS 1.1.1.24 services (ncs stop) and then exporting my data from NCS again, and re-import into PI. Still have the same problem.

Dumping the logs from PI to a remote repository and looking at them doesn't give me much more info. I can see the "Failed to restore the DB. Exception: java.lang.Exception: RMAN-00571:" error in the log file but not much more info that I can figure out around it to determine what's going on.

I've used the PI 1.2 upgrade docs as well as this document through this whole process:

https://supportforums.cisco.com/docs/DOC-26972

Anyone else run into this exact same problem? How were you able to work around this and get you data imported in PI 1.2?

--greg

Rob Johnson · ‎02-02-2013

Go back to the NCS 1.1.1.24 system that has patch 12 applied-

-do an ncs stop

-perform and ncs clean

-run an ncs start

-perform the backup again being sure to include application NCS as arguments to the backup script from the NCS 1.1 server

-take that .gpg file and try restoring it again to the new installed PI 1.2.1.012 VM

greg.fuller · ‎02-04-2013

Nope. No go. Still get the same "Failed to restore the DB. Exception: java.lang.Exception: RMAN-00571" error when importing into PI a backup file made after running the "ncs cleanup" option on NCS.

--greg

Rob Johnson · ‎02-04-2013

Try an inline upgrade----

Performing an Inline Upgrade to Prime Infrastructure 1.2.1.12

Follow these steps to perform an inline upgrade:

Step 1 Ensure that you have installed the appropriate point patch for your existing system, as explained in Patching Your Existing System.

Step 2 Open a console session and log in to the existing server as admin. Enter the password when prompted.

Step 3 Copy the upgrade file downloaded from cisco.com (see Downloading the Upgrade Image and Point Patch) to the default repository:

admin# copy source disk:/defaultRepo

Where:

•source is the application upgrade file's URL, path and filename (for example: FTP:///PI-upgrade-bundle-1.2.1.12.tar.gz).

•disk is the disk and path to the local defaultRepo.

Step 4 Run the application upgrade:

admin# application upgrade PI-upgrade-bundle-1.2.1.12.tar.gz defaultRepo

This step can take 30 minutes or more to complete, depending on the size of the application database.

Rob Johnson · ‎02-04-2013

Are you sure patch 12 is applied to your NCS 1.1.1.24 system? Run show applicaiton to make sure. The reason I say that is because your migration is looking a lot like what's shown in the link below----

"Failed to restore the DB. Exception: java.lang.Exception: RMAN-00571"

http://www.cisco.com/en/US/ts/fn/635/fn63595.html

Rob Johnson · ‎02-05-2013

Also. Are you backing up and restoring to like VM/OVAs?

greg.fuller · ‎02-10-2013

Still no go with this. I've done "ncs db repairdb" on the PI1.2 VM and tried restoring the NCS backup, no go.

Ended up doing an in-line upgrade on my original NCS appliance, that worked. Tried backing up the data from the VM I did the in-line upgade on and importing it into the other PI 1.2 applaince. No go, still get the RMAN-00571 error.

I originally deployed the NCS 1.0 VM appliance with what I think was the "large" appliance as the provisioned disk size is 390.625gigs and 16gigs of RAM.

The PI1.2 VM appliance I believe I may have used the "medium" appliance due to storage space issues and vmware ESXi 4.1 Standard not supporting that many vCPU's on the host I had to deploy this on.

Should I re-deploy with the same size appliance as I used for NCS?

TAC has been zero help in any of this. They just referenced me to the same bug ID I already found (CSCub35534) and really have not been helpful at all (for a change -- usually they are great!).

--greg

Rob Johnson · ‎02-10-2013

Have you tried stopping your NCS 1.1.1.24 application that has patch 12 applied then run a backup from the CLI being sure to include application NCS with the backup command?

greg.fuller · ‎02-11-2013

Finally got this working last evening! Not 100% sure exactly what I did but the restore was sucessful. I think it might have had something to do with making sure that NCS was not running when trying to restore the data. I didn't see any references in any of the documentation to make sure NCS was complete shut down (ncs stop, and ncs status to verify) before doing the restore. But after several db reinit's it appears to have restored everything properly now. For future reference here's what I ended up doing starting with WCS (Linux version):

Make sure WCS was running the latest version v7.0.230.0
Export maps from WCS via the GUI (as a precaution -- we have way too many maps to redo all them again!):
1. In the GUI go to Monitor -> Maps
2. Select which maps you want to export (you can select them all)
3. Select Export Maps and tell it where to store the export
Stop WCS from the cli: /opt/WCS7.0.230.0/StopWCS
Export your user configuration from WCS:
1. : /opt/WCS7.0.230.0/export.sh all /root/wcs-export-TODAY.zip
Copy the backup file off to another SCP/FTP server
Deploy the NCS 1.1.1.24 OVM (NCS-VA-1.1.1.24-xxxx.ova) template and get connectivity working to the VM
STOP the NCS services!! Then verify all services have stopped:
1. ncs stop
2. ncs status
Create a repository where you stored your WCS backup:
1. configure terminal
2. repository wcs-backup-repo
3. url sftp://xxx.xxx.xxx.xxx/home/user/wcs-backups/
4. user myusername password plain mypassword
5. exit
Verify you can see the files in the repository:
1. show repository wcs-backup-repo
Migrate the WCS data into NCS (this will take a LONG time to run!):
1. ncs migrate wcs-data wcs-export-TODAY.zip repository wcs-backup-repo
Start NCS if import was sucessful:
1. ncs start
Verify everything looks good (no errors, all devices show up, maps are there etc)
Download the NCS 1.1.1.24 to Prime Infrastructure 1.2 patch and save it in the same repository as your WCS backup files
Stop NCS and verify all services have stopped:
1. ncs stop
2. ncs status
Install the NCS 1.1.1.24 to Prime Infrastructure 1.2 patch:
1. application upgrade ncs_patch-1.1.1.24-upgrade-12.tar.gz wcs-backup-repo
Reboot the appliance:
1. reboot
Verify that patch was installed (it should show the upgrade-12 patch is installed):
1. show application version NCS
Stop NCS and verify all services have stopped:
1. ncs stop
2. ncs status
Export the NCS application data:
1. backup ncs-export-TODAY repository wcs-backup-repo application NCS
If you are deploying a brand new Prime Infrastructure v1.2 appliance (as suggested by the release notes), deploy the new PI v1.2.1.012 OVM and get connectivity working to it.
Create the repository what contains your backup files:
1. configure terminal
2. repository wcs-backup-repo
3. url sftp://xxx.xxx.xxx.xxx/home/user/wcs-backups/
4. user myusername password plain mypassword
5. exit
Make a local directory on the PI appliance for the local repository:
1. mkdir disk:/defaultRepo
Create a local repository on the PI appliance:
1. configure terminal
2. repository defaultRepo
3. url disk:/defaultRepo
Verify the repositories (defaultRepo should be empty):
1. show repository defaultRepo
2. show repository wcs-backup-repo
Stop NCS and verify all services have stopped:
1. ncs stop
2. ncs status
Import the data from NCS into PI (this will take a LONG time to run!):
1. restore ncs-export-TODAY.tar.gpg repository wcs-backup-repo application NCS
Restart NCS services and verify they are running:
1. ncs start
2. ncs status
Verify you can login and all data has been imported. Hopefully you are all good!

If things go wrong these are some additional commands that might help:

If the database on the appliance you are trying to import data to appears to have something wrong you can try doing this (NOTE all local disk: repositories will lose the data/files!):
- ncs stop
- ncs status
- ncs cleanup
- ncs start
- ncs status
If there is still something wrong with the database you can try re-creating the NCS database from scratch on the appliance (NOTE YOU WILL LOSE EVERYTHING IN THE DATABASE WHEN DOING THIS!):
- ncs stop
- ncs status
- ncs db reinitdb
- ncs start
- ncs status
If you know your linux commands you can also gain true bash shell access to the underlaying system by enabling the root account and setting a password for it:
- root_enable
- Enter the password you would like to give the root account
- root
NCS log files and other goodies are located in the /opt/CSCOlumos/ directory. I found lots of troubleshooting data in there, as long as your good with navigating around the directory structure and reading the logs you should be able to figure out ecactly what might be going on.

Good luck to everyone in their quest to get to the latest version! Cisco definately did not make this process an easy one.

--greg

Michael Grabowski · ‎08-07-2013

I am having the same issues.

I am going to attempt all of these steps (with the exception of step 16: Reboot the applicance)

I will post my results.. Thanks

Michael Grabowski · ‎08-08-2013

I can confirm, doing the steps as stated above have worked with a fresh reload of the .OVA

Process took around 5 hours, local repository, 19GB db

Restore may require a restart of application services. Continue? (yes/no) [yes]

Initiating restore. Please wait...

Stage 1 of 9: Transferring backup file ...

-- complete.

Stage 2 of 9: Decrypting backup file ...

-- complete.

Stage 3 of 9: Unpacking backup file ...

--complete.

Stage 4 of 9: Decompressing backup ...

-- complete.

Stage 5 of 9: Restoring Support Files ...

-- complete.

Stage 6 of 9: Restoring Database Files ...

-- complete.

Stage 7 of 9: Recovering Database ...

-- complete.

Stage 8 of 9: Updating Database Schema ...

: This could take long time based on the existing data size.

Stage 1 of 5: Pre Migration Schema Upgrade ...

-- complete. Time Taken : 0 hr, 2 min, 50 sec

Stage 2 of 5: Schema Upgrade ...

: This could take long time based on the existing data size.

-- complete. Time Taken : 0 hr, 15 min, 15 sec

Stage 3 of 5: Post Migration Schema Upgrade ...

-- complete. Time Taken : 0 hr, 5 min, 39 sec

Stage 4 of 5: Enabling DB Constraints ...

-- complete. Time Taken : 0 hr, 1 min, 42 sec

Stage 5 of 5: Finishing Up ...

-- complete. Time Taken : 0 hr, 0 min, 24 sec

-- complete.

Stage 9 of 9: Re-enabling Database Settings ...

-- complete.