CUCM 9.1.2 --> 10.5(2) failure

[hanlykent]

Hi All,

I am trying to upgrade our CUCM from 9.1.2(SU2) --> 10.5(2) (UCSInstall_UCOS_10.5.2.10000-5.sgn.iso)

I believe I have performed all the upgrade requirements:

• Our hardware is Cisco UCS with old "Medium" UCS Tested Reference Configuration for 2500 users. Each node has 2 x vCPUs. 

• 9.1.2(SU2) tp 10.5(2) is documented as a "Direct Upgrade"

• I have/had at least 30GB free in the common partition on each node

• ciscocm.version3-keys.cop has been installed on all nodes

• I did not install the refresh_upgrade COP file, as I am upgrading from 9.1.2 (i.e. not pre-version 8.5)

• The virtual machine specification meets all the requirements for 10.5(2)

The installation goes for approx 1.5 - 2hrs before it does "Post-installation" tasks where is fails at item 6 or 6 of "Installing database component"

Th install log shows:

01/24/2015 17:24:02 component_install|File:/opt/cisco/install/bin/component_install:807, Function: exec_progmeter(), /opt/cisco/install/bin/progmeter failed (1)|<LVL::Error>

01/24/2015 17:24:02 appmanager.sh|Internal Error, File:/usr/local/bin/base_scripts/appmanager.sh:273, Function: refresh_upgrade(), failed to refresh_upgrade infrastructure_post components|<LVL::Critical>

01/24/2015 17:24:02 post_install|File:/opt/cisco/install/bin/post_install:961, Function: install_applications(), /usr/local/bin/base_scripts/appmanager.sh -refresh-upgrade failed (1)|<LVL::Error>

01/24/2015 17:24:02 post_install|Exiting with result 1|<LVL::Info>

01/24/2015 17:24:02 post_install|INSTALL_TYPE="Refresh Upgrade"|<LVL::Debug>

It seem very similar to the following Cisco support forum issue : https://supportforums.cisco.com/printpdf/12334191; however, I have been unable to remove the AUNP dial plan; I suspect because I have quite a number of route patterns using it and do not wish to remove them.

I upgraded the dial plan as an attempt to rectify the issue, however the upgrade still fails.

Any ideas would be much appreciated.

Thanks

Kent

[Richard Simmons · Accepted solution]

I've run into this issue on several upgrades and always dread seeing a xxNP in use, the way i get around this is by exporting the route patterns, Bulk Administration > Export and select it from there.

TAC will then need to get Root access to remove the xxNP's (all of them) and then the upgrade will succeed.

Once the upgrade is completed and you have installed the xxNP you then need to export the route patterns again from the v10.x server as you will need to edit the header.txt stored within the .tar file. Copy 'CCM : master-10.5.1.11901-1.i386' section from the 10.x export and replace the one in the 9.x export.

You can then import the CUCM 9.x route patterns in CUCM 10.x

Hope this helps,

Richard

[hanlykent]

Hi Richard,

Thanks for your reply,

After attempting this upgrade 2 times and getting assistance from Cisco TAC, the TAC have confirmed that my issue is related to having the AUNP dialplan installed.

The install logs are as above, however the exact issue shows up as:

++ Error on InstallDB logs  ++

16:53:35.301 |   DBUtil::BlockCopyTable ### *ERROR* ###:  (diagnosis):  Bulk Data Migration for table digitdiscardinstructionmember failed due to data constraint issue, (-971).

I am in the midst of attempting your solution; however I was unable to remove the AUNP from the GUI. I escalated to Cisco TAC to have it removed via "Root Access", however the TAC engineer could not find a reference on how to do this.

I have since escalated the TAC case to another Cisco TAC engineer who will hopefully have more information.

With your experience in this issue, I was wondering whether you know the required commands/procedure of the Cisco TAC engineer?

Thanks

Kent

[hanlykent]

Hi Richard,

No need to reply to my previous post. I thought I share my finding with you and others on the forum.

With the assistance from Cisco TAC we were able to uninstall the dialplan from the GUI.

The Cisco documentation states that before you can remove the dial plan you need to remove any of the following that utilise the dial plan:

• Route Patterns
• Translation Patterns
• Route Filters
• Route Lists

(Its probably worth noting here, that you should export your existing Patterns/Filters/Lists before removing them, as you will need to import them back in after the upgrade)

In our case, most configuration were using the "AUNP:Pre-Dot" configuration.

I missed removing some Route Lists, which caused not being able to remove the AUNP.
Interestingly, the GUI did not complain with the Route Lists still being there.Instead, when you pressed "Uninstall" , the status changed to "Dial plan Installation Successful"!!? (Weird)

After removing the appropriate Route Lists, I was able to remove the AUNP from the Publisher and Subscribers using the GUI and then restarted the CM services as instructed in the Admin guide.

As I was removing all the "0.@" route patterns, I created some specific PSTN route patterns (i.e patterns not using the AUNP Dial Plan) to allow our 24/7 NOC maintain outbound dialing. I did some digit manipulation on our gateway to remove the leading "0", as I couldn't use "AUNP: Pre-Dot" on the Route List.

I was then uable to upgrade from 9.1.2 to 10.5.2.11900-3 without any issues.

Your import/export procedure as mentioned above worked a treat too!!

Thanks for sharing!

Kent

[Richard Simmons]

Hi Kent,

Glad yo hear you've got everything working, in the one's I've had issue on I've uninstalled the xxNP from the GUI but it still remained, which then meant the TAC engineer used the root access to delete the file using standard Linux commands but I didn't note this procedure down as Root access should only be done by Cisco.

Regards,

Richard

[pperry]

hanlykent,

Did Richard's suggestion fix your upgrade issue?  I believe we are running into the same issue and like you I am not wanting to mess with the plans...

Thanks in advance for your reply..

[Manish Gogna]

As the suggested action plan requires a root access you should open a TAC case as only then can create an account with root access and also confirm if this is a supported method or not to fix the issue. As per my experience they will not change anything in the root unless the issue and its fix is clearly documented.

HTH

Manish

[Slavik Bialik]

OK, I can totally confirm that this is the solution. I had this now, too. Tried to upgrade from 9.1.2 to 10.5.2(SU3), and had the same error. After removing all records that contained the xxNP (had only NANP) in the discard digits field, and tried to upgrade again, the process were successfully done!

Thanks!

[wsteahr01]

Can you specify what records you removed that contained NANP in your system? Thanks!

[Slavik Bialik]

In my case, in the Route List Details, I was using 'NANP: PreDot and Trailing #'. So every record I had that contained 'NANP: PreDot and Trailing #' I simply changed it to 'None'.

After the upgrade, I changed it back.

Hope you'll have a successful upgrade ;) 

[wsteahr01]

Thank you!

My problem ended up being completely unrelated to NANP dial plans. I dug through the installdb_ru.log.err file and was able to immediately identify the problem. This was the first line in the error log. 

10:34:23.118 |   DBUtil::DoSQLCommand ### *ERROR* ###:  SQL Exception detected while Adding Saved Initialization Records back to table [Error executing "INSERT INTO sipprofile SELECT * FROM TmpSaveTargetInitial": [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UNIQUE INDEX column (Unique Index:).]

I deleted the only non-default SIP Profile from the system and tried the upgrade again and it worked the first try. That profile was a copy of the standard SIP Profile on the system and I think CUCM might not have made the correct database entry when it was created.

For anyone having problems with the database portion of a CUCM upgrade PLEASE look at the installdb_ru.log.err file located in activelog cm/trace/dbl/sdi/. If I would have known to look there right away I would have saved so much time!

Thanks for everyone in this thread, you gave me hope!

[sebastien_michelet]

Thank you for the tip on installdb_ru.log.err, I found a similar error also related to a duplicate SIP profile. My TAC case was going nowhere...