UCCX Upgrade from 10.6 to 11 (Log details)

juroda · ‎08-22-2016

Introduction

This document will help you to understand logs during Upgrade of UCCX from 10.6 to 11

Problem

Upgrade stuck. Status: Running after hours.

Solution

Collect:

1. file view install uccx-install.log

2. show uccx version

3. file view install system-history.log

Log analysis:

Here you can find the logs of a successful installation:

Run on CLI: “file view install uccx-install.log”, you should see the following important logs if the upgrade is successful:

file_list.sh|iso image detected|<LVL::Debug>

filter|INorOUT=IN UCSInstall_UCCX_11_0_1_UCOS_11.0.1.10000-75.sgn.iso: Filter passed.|<LVL::Debug>5.sgn.iso|<LVL::Debug>

upgrade_validate_file.sh|There is enough space on device to proceed.|<LVL::Info>

upgrade_validate_file.sh|upgrade_validate_file.sh complete (rc=0)|<LVL::Info>

upgrade_get_file.sh|Create md5 complete|<LVL::Info>

upgrade_get_file.sh|Authenticate file "/common/download/11.0.1.10000-75/checksum_file.sgn"|<LVL::Info>

upgrade_get_file.sh|File authentication complete|<LVL::Debug>

upgrade_install.sh|Starting Upgrade -- upgrade_install.sh|<LVL::Info>

upgrade_install.sh|Upgrade the system|<LVL::Info>

upgrade_manager.sh|new upgrade version=11.0.1.10000-75|<LVL::Info>

PlatformXmlUpg|Migrating to version .|<LVL::Info>

upgrade_manager.sh|Upgrade (L2) Starting|<LVL::Info>

upgrade_manager.sh|Upgrade from 10.6.1.10000-39 allowed|<LVL::Info>

upgrade_manager.sh|Free space validation complete|<LVL::Info>

upgrade_manager.sh|Install platform on inactive partition|<LVL::Info>

upgrade_manager.sh|Install the OS on inactive partition|<LVL::Info>

install_os|Upgrade OS successful|<LVL::Info>

upgrade_manager.sh|OS install on inactive partition complete|<LVL::Info>

upgrade_manager.sh|Platform install on inactive partition complete|<LVL::Info>

upgrade_manager.sh|Upgrade complete|<LVL::Info>

upgrade_install.sh|System upgrade complete|<LVL::Info>

upgrade_install.sh|No switch to new version|<LVL::Info>

upgrade_install.sh|Upgrade Complete -- (0)|<LVL::Info>

It confirms that Upgrade was completed.

a) If the upgrade is stuck, and after the following line: “upgrade_install.sh|Upgrade Complete -- (0)|<LVL::Info>” at the end you see:

upgrade_install.sh|is_upgrade_lock_available: Upgrade lock is not available.|<LVL::Debug>

upgrade_install.sh|is_upgrade_in_progress: Already locked by this process (pid: 29492).|<LVL::Debug>

upgrade_install.sh|release_upgrade_lock: Releasing lock (pid: 29492)|<LVL::Debug>

I will suggest you the following:

a-1) Check “show uccx version”

If Active partition shows: current UCCX version running and Inactive partition shows: 11.0.1.10000-75 and

a-2) Also running “file view install system-history.log” and you can see:

root: Upgrade 11.0.1.10000-75 Start

root: Upgrade 11.0.1.10000-75 Success

If you confirm that both (a&b) conditions are on the outputs, let’s go ahead and start switch version via CLI “utils system switch-version”.

If you prefer, you can reboot the server first, then do the switch.

More know issues related to the upgrade from 10.6 to 11:

b) If for some reason the installation fails because the COP file is missing, you will see the following on the logs:

upgrade_install.sh|IOWAIT monitor stopped|<LVL::Info>

upgrade_install.sh|File:/usr/local/bin/base_scripts/upgrade_install.sh:604, Function: main(), Upgrade Failed -- (1)|<LVL::Error>

upgrade_install.sh|Parse argument status=upgrade.stage.error|<LVL::Debug>

upgrade_install.sh|_set_upgrade_status_attribute: status set to upgrade.stage.error|<LVL::Debug>

As per documentation:

For Unified CCX 10.6(1) and the related SUs to be compatible with Unified CM 11.x, install the COP version ciscouccx.106x.cucm11x-compatible.14.cop.sgn.

http://docwiki.cisco.com/wiki/Unified_CCX_Software_Compatibility_Matrix_for_10.6(1)SU1

c) Important: COP ciscouccx.106x.cucm11x-compatible.14.cop.sgn. is only needed for 10.6 Base

If you are running 10.6.1SU1, COP file is not needed, the documentation is wrong. You will get the following error from the logs:

sd_cancelInstall.sh|Pre-upgrade cancel/clean-up initiated
sd_cancelInstall.sh|Canceling active side processes
sd_cancelInstall.sh|Canceling active side processes done
sd_cancelInstall.sh|Canceling inactive side processes
sd_cancelInstall.sh|Skipping IDS shared memory clean up since this is a COP
sd_cancelInstall.sh|Canceling inactive side processes done

locale_install.sh: Successful untarring of /common/download//ciscouccx.106x.cucm11x-compatible.14.cop
Validate the ES compatibility
Install Failed...The ES is not compatible with the Service Update Installed in the system
locale_install.sh: ERROR: Copstart script of /common/download//ciscouccx.106x.cucm11x-compatible.14.cop failed

Also, you can confirm it on the download page:

10.6.1 (COP listed):

https://software.cisco.com/download/release.html?mdfid=286279660&flowid=72862&softwareid=280840578&r...

10.6.1SU1 (COP not listed):

https://software.cisco.com/download/release.html?mdfid=286279660&flowid=72862&softwareid=280840578&r...

PierMatteo · ‎02-03-2017

The stucked upgrade is also listed as a bug: CSCuu34496

Facing the same behaviour when upgrading to version 11.5(1).

etmarcof · ‎03-02-2017

Hi,

Did you manage to solve your problem?

I' want upgrade from UCCX 10.6.1 SU1 to 11.5.1.

I could upgrade First node but i couldn't upgrade second node it fails. Any suggestions?

I'm using same iso.

Regards,

PierMatteo · ‎03-02-2017

Hello etmarcof,

if you've hit this BUG (CSCuu34496) the upgrade was successful in the background - only the GUI stucks.

Verify this by issuing "show uccx version" in UCCX's CLI.

What exactly is your error message? At what point of upgrade does the error occure?

Hope this helps.

Regards,

Pier

etmarcof · ‎03-02-2017

Hi Pier,

Only second onde failed upgrade on logs i saw

upgrade_install.sh|IOWAIT monitor stopped|<LVL::Info>

upgrade_install.sh|File:/usr/local/bin/base_scripts/upgrade_install.sh:604, Function: main(), Upgrade Failed -- (1)|<LVL::Error>

upgrade_install.sh|Parse argument status=upgrade.stage.error|<LVL::Debug>

upgrade_install.sh|_set_upgrade_status_attribute: status set to upgrade.stage.error|<LVL::Debug>

But my uccx is 10.6 Su1 so i dont need that cop.

show uccx version dont show Nothing in inactive version.

regards

dstetter · ‎03-04-2017

Hello etmarcof, I am having exactly the same issue as you upgrading from 10.6.1 to 11.5.1

Were you able to find a solution for this?

I am working with TAC now to try to find a resolution, my publisher is fine, but can't upgrade the subscriber and the inactive version is showing as NA for me.

Thanks

Dennis

dstetter · ‎03-05-2017

We ended up hitting this bug (CSCuw65303):

http://www.uccollabing.com/2016/04/27/installing-or-upgrading-to-uccx-10-6-1-su1/

Even though I was upgrading from 10.6.1 to 11.5.1, once TAC updated the publisher, install went by with no issues.

etmarcof · ‎03-06-2017

Hi Dennis,

I solved the issue removing second node from first node.

Then i upgrade first node to 11.5.1 and switch-version.

I Added second node on first node and i did a fresh install of second node with 11.5.1

I 'm lucky because i have new servers so i don't need to do this in production.

When you have a solution from TAC please share.

Regards

dstetter · ‎03-06-2017

You need Root access, so you will have to work with TAC, but basically these are the steps.

1. Access root from Publsiher node

2. navigate to /opt/cisco/uccx/Post.d

cd /opt/cisco/uccx/Post.d

3. run script ./authorize_remote_node.sh