cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2983
Views
20
Helpful
17
Replies
Highlighted
Beginner

6.1.5 to 9.1.2 Jump Upgrade Fails

6.1.5 to 9.1.2 Jump Upgrade Fails. Receive the refresh upgrade fails screen and then a file system error after it attempts to roll back to previous version.

 

upgrade_fail1.jpg

upgrade_fail2.jpg

 

 

A summary of the environment and steps thus far.

 

6.1.5 SU1 - Pub, 2 Subs

 

- Installed three 6.1.5 SU1 servers using v8 ova on UCS C240 M3, VMware ESXi 5.0.

- VM OS changed to RHEL 3 32-bit

- Hostnames, IPs, DNS, Domain, Cert settings during install all matching current cluster. NTP server is different IP but accessible in lab. DNS servers are not.

- Restored entire production cluster DRS to new VMs. NTP server on VM Pub remained with IP from Lab on when VMs were installed with bootable media in first step. Didn't revert to IP from prod environment. Assume NTP server data isn't part of DRS since it didn't revert?

- Installed latest v3 of refresh cop.

- Attempted refresh upgrade to 9.1.2 on Pub.

 

Any ideas? Is DNS not being reachable during refresh an issue? Does the old NTP server IP still exist in the database somewhere? If I look at NTP Server in OS Admin it shows my Lab NTP server IP and is accessible.

 

Thanks in advance.

17 REPLIES 17
Highlighted
Advisor

I haven't seen this issue when doing the jump upgrade.

I think having a different NTP server IP address is fine. I would recommend having a DNS server in the staging environment. When I have done this, I basically segregated the environment and used the DNS and NTP IP addresses from production.

HTH

-Bill
(b) http://ucguerrilla.com
(t) @ucguerrilla

Please remember to rate helpful responses and identify helpful or correct answers.

HTH -Bill (b) http://ucguerrilla.com (t) @ucguerrilla

Please remember to rate helpful responses and identify

Highlighted

Thanks, I have changed the NTP IP to match production and also have DNS in lab environment. Refresh upgrade still fails.

Sent from Cisco Technical Support iPhone App

Highlighted

Are you referring to NTP server on Publisher on in OS Admin or Phone NTP Reference in CUCM Admin?

I have a NTP server configured on the Publisher in OS Admin.

Highlighted
Hall of Fame Community Legend

Hi Aaron,

In addition to the great tips from my friend Bill here (+5 ucguerrilla)

I think if you look closely at this bug we'll see that NTP is the likely

culprit here;

CUCM jump upgrade from 6.1.5 to 9.1.2 fails due to NTP reference missing

CSCui85967

And this thread with nice notes from Aman, Armin and Joe from Cisco;

https://supportforums.cisco.com/message/4026545

Cheers!

Rob

"Seek it out and ye shall find  " 

- OneRepublic

Highlighted

Are you referring to NTP server on Publisher on in OS Admin or Phone NTP Reference in CUCM Admin?

I have a NTP server configured on the Publisher in OS Admin but Phone NTP Reference is missing...

Highlighted
Hall of Fame Community Legend

Hi Aaron,

I see what you are saying here My reference was for the

Pubs NTP not the phone NTP but it seems not to be the case

seeing the adjustments you made between upgrade attempts.

I would open a case with the Drive to 9 TAC so they can turn on the flag

that Joe and Armin reference in the thread I attached so we can see a

better diagnostic message when the upgrade fails. It should point to

the specific cause rather than the current generic message you receive.

Cheers!

Rob

"Seek it out and ye shall find  " 

- OneRepublic

Highlighted

As always, great advice from Mr. Huffman (+5 Huff)

HTH

-Bill
(b) http://ucguerrilla.com
(t) @ucguerrilla

Please remember to rate helpful responses and identify helpful or correct answers.

HTH -Bill (b) http://ucguerrilla.com (t) @ucguerrilla

Please remember to rate helpful responses and identify

Highlighted

Thanks guys. I had come across that thread before and my concern is both of the things mentioned for the fix don't apply in my scenario.

Hardware isn't an issue. These are new UCS servers with TRC Large spec. NTP and DNS servers are also reachable and working in the test environment with same IPs as production environment.

I've done quite a few UC upgrades but never from 6 directly to 9. My gut feeling is I will need to upgrade to 8.X on a physical server and then migrate to virtual. Do you guys know if you can upgrade from 6.1.5 to an 8 release in a VM? I know in the past the hardware validity checks would fail.

Highlighted

So I attempted to upgrade to 7.1.5 in between and same results except the system reverts back to 7.1.5 afterwards. I pulled the following out of the install logs. TAC is no help yet...

01/20/2014 18:23:19 installdb|--- --- Begin Performing Post-Load Target DB Updates (Step 4 of 6)|<:INFO>

01/20/2014 18:23:19 installdb|--- --- Begin Load Rules Enforcement Objects Into Target DB (Step 5 of 6)|<:INFO>

01/20/2014 18:24:37 installdb|--- --- Begin Performing Final Target DB Updates (Step 6 of 6)|<:INFO>

01/20/2014 18:24:37 installdb|--- --- End -- Performing DB Load and Transforming Copy

01/20/2014 18:24:58 IPM|Internal Error, File:ipm.c:2006, Function: ipmReadNormalizedInputLine(), "/usr/local/cm/script/cm-dbl-install RU PostInstall 9.1.2.11900-12 7.1.5.35901-1 /usr/local/cm/ /common/component/database /common/log/install/capture.txt " failed (1)|<:CRITICAL>

01/20/2014 18:25:00 IPM|  end-of-session "Installing database component": 1065.526 secs.|<:INFO>

01/20/2014 18:25:00 IPM|Close progress meter "Component Install"|<:INFO>

01/20/2014 18:25:00 component_install|File:/opt/cisco/install/bin/component_install:756, Function: exec_progmeter(), /opt/cisco/install/bin/progmeter failed (1)|<:ERROR>

01/20/2014 18:25:00 appmanager.sh|Internal Error, File:/usr/local/bin/base_scripts/appmanager.sh:269, Function: refresh_upgrade(), failed to refresh_upgrade infrastructure_post components|<:CRITICAL>

01/20/2014 18:25:00 post_install|File:/opt/cisco/install/bin/post_install:911, Function: install_applications(), /usr/local/bin/base_scripts/appmanager.sh -refresh-upgrade failed (1)|<:ERROR>

01/20/2014 18:25:00 post_install|Exiting with result 1|<:INFO>

01/20/2014 18:25:00 post_install|INSTALL_TYPE="Refresh Upgrade"|<:DEBUG>

01/20/2014 18:25:00 post_install|Process refresh upgrade failure|<:INFO>

01/20/2014 18:25:01 post_install|(CAPTURE) Mail notification cancelled - smtp server address for email not found! [/usr/local/platform/conf/platformConfig.xml]|<:DEBUG>

01/20/2014 18:25:01 post_install|Retore grub.conf from grub.conf.recovery|<:INFO>

01/20/2014 18:25:01 post_install|Copy recovery file, /grub/boot/grub/grub.conf.recovery, to /grub/boot/grub/grub.conf|<:DEBUG>

01/20/2014 18:25:01 post_install|Retore of grub.conf complete|<:INFO>

01/20/2014 18:25:01 post_install|Invalidate upgrade partition|<:INFO>

01/20/2014 18:25:01 post_install|Removing master RPM|<:DEBUG>

01/20/2014 18:25:01 post_install|Removing /etc/opt/cisco/install.conf|<:DEBUG>

01/20/2014 18:25:01 post_install|Invalidate any product configuration file|<:DEBUG>

01/20/2014 18:25:01 post_install|Setting product and deployment version attributes in /etc/opt/cisco/install/conf/callmanager_product.conf to 0.0.0.0000-0000|<:DEBUG>

01/20/2014 18:25:01 post_install|Cleanup refresh upgrade staging areas|<:DEBUG>

01/20/2014 18:25:14 post_install|Parse argument status=upgrade.stage.error|<:DEBUG>

01/20/2014 18:25:14 post_install|_set_upgrade_status_attribute: status set to upgrade.stage.error|<:DEBUG>

01/20/2014 18:25:14 post_install|File:/opt/cisco/install/bin/post_install:580, Function: handle_refresh_upgrade_failure(), Refresh upgrade failed. Trying to reboot to currently active version.|<:ERROR>

01/20/2014 18:26:38 hardware_check.sh|Initializing hardware library on boot|<:INFO>

01/20/2014 18:26:38 ServerApiManager|In main with args -|initialize|- called file |/usr/local/bin/base_scripts/server_api_manager.sh| - and current mode=HSSI_Mode |<:DEBUG>

01/20/2014 18:26:38 ServerApiManager||Method: initialize args: |- current mode=HSSI_Mode completed with code [0]|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|In main with args -|initHardwareLibrary|- called file |/usr/local/bin/base_scripts/server_api_manager.sh| - and current mode=HSSI_Mode |<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|Sam_reset being called|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|No hssi_api.sh found at / for call hssiReset.|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager||Method: initHardwareLibrary args: |- current mode=HSSI_Mode completed with code [0]|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|INITIALIZE: sam_legacy_mode_init=1|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|No hssi_api.sh found at / for call hssiInitialize.|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|INITIALIZE: sam_hssi_mode_init=0|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|In main with args -|initialize|- called file |/usr/local/bin/base_scripts/server_api_manager.sh| - and current mode=HSSI_Mode |<:DEBUG>

01/20/2014 18:26:38 ServerApiManager||Method: initialize args: |- current mode=HSSI_Mode completed with code [0]|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|In main with args -|getHardwareAttribute HWModel|- called file |/usr/local/bin/base_scripts/server_api_manager.sh| - and current mode=HSSI_Mode |<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|No hssi_api.sh found at / for call getHardwareAttribute HWModel.|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager||Method: getHardwareAttribute args: HWModel|- current mode=HSSI_Mode completed with code [0]|<:DEBUG>

01/20/2014 18:26:38 hardware_check.sh|HWModel: "". This hardware is unsupported.|<:WARN>

01/20/2014 18:26:38 ServerApiManager|In main with args -|initialize|- called file |/usr/local/bin/base_scripts/server_api_manager.sh| - and current mode=HSSI_Mode |<:DEBUG>

01/20/2014 18:26:38 ServerApiManager||Method: initialize args: |- current mode=HSSI_Mode completed with code [0]|<:DEBUG>

01/20/2014 18:26:38 ServerApiManager|In main with args -|getValidateErrorFile|- called file |/usr/local/bin/base_scripts/server_api_manager.sh| - and current mode=HSSI_Mode |<:DEBUG>

01/20/2014 18:26:38 ServerApiManager||Method: getValidateErrorFile args: |- current mode=HSSI_Mode completed with code [0]|<:DEBUG>

01/20/2014 18:26:38 hardware_check.sh|The hardware you are using is not supported for this product.  Upgrade the hardware and try again|<:ERROR>

01/20/2014 18:26:38 display_screen|Arguments: "Hardware Configuration Error" "WARNING.   This system has been identified as an unsupported hardware platform. This platform may have performance issues and must not be used in a production system of any kind.   By continuing with this platform, you hereby acknowledge that all applicable warranties and indemnities, and any applicable Cisco services contracts that may be associated with this system, are null and void." "Continue"|<:DEBUG>

01/20/2014 18:26:38 display_screen|Button label size information 66, 1, 66|<:DEBUG>

01/20/2014 18:26:44 display_screen|Arguments: "Hardware Configuration Error" "Additionally, by continuing to use this unsupported platform, you agree that your system will not be supported by Cisco TAC, and will not be used in a production system environment.   You further agree that you shall be solely responsible for product support and any and all liabilities associated with  your use of Cisco products on an unsupported configuration." "Disagree" "Agree"|<:DEBUG>

01/20/2014 18:26:44 display_screen|Button label size information 66, 2, 33|<:DEBUG>

01/20/2014 18:26:46 hardware_check.sh|Continue with incorrect hardware configuration|<:INFO>

01/20/2014 18:26:46 hardware_check.sh|The hardware you are using is not supported for this product. Upgrade the hardware and try again|<:INFO>

01/20/2014 18:27:23 /usr/local/bin/base_scripts/rclocalmgr.sh|action type is commit_del|<:DEBUG>

01/20/2014 18:27:23 /usr/local/bin/base_scripts/rclocalmgr.sh|running commit_del|<:INFO>

01/20/2014 18:27:23 /usr/local/bin/base_scripts/rclocalmgr.sh|No scheduled deletions|<:INFO>

Highlighted

Have you checked into this defect https://tools.cisco.com/bugsearch/bug/CSCtt17619?  The subnet mask can be set to 255.255.255.000 which causes the database process to fail to bind to the network interface.

Highlighted

Update to the thread.

file view inactivelog cm/trace/dbl/sdi/installdb_ru.log.err (some output changed to hide data)


12:23:09.085 |*ERROR* Error executing "insert into EndUser (allowcticontrolflag,assocpc,building,deletedtimes tamp,department,enablemobilevoice,enablemobility,f acsimiletelephonenumber,firstname,fkdirectoryplugi nconfig,fkmatrix_presence,

homephone,mailid,manager,maxdeskpickupwaittime,mid dlename,mobile,nickname,ocsprimaryuseraddress,page r,passwordreverse,pkid,remotedestinationlimit,site ,status,telephonenumber,title,tkuserprofile,unique identifier,userid) values

('T','','',123456,'','F','F','','******','32b66ffb-48d6-c3d5-33c1-1653b28396fb','ad243d17-98b4-4118-8feb-5ff2e1b781ac','XXX XXX-XXXX','*****.Null@*******.com','G***A',10000,'','X XX XXX-XXXX','',

'sip:*****.Null@************.com','','69c4f936f9cd f45f6bbca2570c31215629bb5d6fb97493478b8ff3db6fffbc 55','a677593d-d61e-4327-91e1-031bf4bc90ad',4,'',1,'2848','District',1,'f195e958 a1652943a1a2c7ab03885743',

'null'): [Informix][Informix ODBC Driver][Informix]Cannot insert a null into column (enduser.lastname).

12:23:16.504 |*ERROR* Error executing "insert into credential"


This particular end user has a last name of NULL, which is a valid SQL command against the Informix database. So when it tried to insert his last name in that field it ran the null command instead of just entering it as a value. This kills the whole database upgrade. Changed his last name in AD to aNULL, force synced with CUCM, new DRS and started the process above again and it worked.


If you ever run into problems upgrading CUCM, Unity, CUPS, etc. Make sure you don't have an end user with the name NULL anywhere!!!!!!!!!

Highlighted
Hall of Fame Community Legend

Hi Aaron,

Thanks for this update! That is one crazy reason to kill an entire upgrade

+5 for sharing your results with the community.

Cheers!

Rob

"Seek it out and ye shall find  " 

- OneRepublic

Highlighted

I have the same symptoms with a 6.1.4 to 9.1.2 jump upgrade, in my case the output of

file view inactivelog cm/trace/dbl/sdi/installdb_ru.log.err

is: -

15:59:29.436 |   CSV::DeleteFile *ERROR* Unable to read file delete_Product_30054.csv

15:59:47.217 |*ERROR* Error executing "Update ProductCapabilities set

tkProduct='30029', tkProductConfig='32', enumValue='70', moniker='NL_CROATIA' where enum = '36594'": [Informix][Informix ODBC Driver][Informix]Missing key

in referenced table for referential constraint (informix.tk_productcapabilities_tkproduct).

16:00:05.282 |   installFull *ERROR* Prior Cancel or Error Processing

installCsv (/usr/local/cm/db/csv/products)

Can anyone point me in the right direction before I open a TAC case?

Regards,

Andy

Highlighted

That error indicates you have some Tandberg device, which had to have been added into the database at one point with a COP file.  The production system will likely show what COP file was installed with "show version active" and compare that to your VM 6.1.4 server that you are using.  Re-install the missing COP file and your upgrade should succeed after that.

Content for Community-Ad