Re: CUCM 10.5(2)SU3 upgrade got stuck

HRS · ‎04-01-2016

Upgrade from CUCM 10.5.2 to 10.5.2 SU3 getting stuck on

On Install Logs.. It got stuck on Releasing Lock.

03/27/2016 20:47:22 upgrade_install.sh|Upgrade Complete -- (0)|<LVL::Info>
03/27/2016 20:47:22 upgrade_install.sh|is_upgrade_lock_available: Upgrade lock is not available.|<LVL::Debug>
03/27/2016 20:47:22 upgrade_install.sh|is_upgrade_in_progress: Already locked by this process (pid: 26363).|<LVL::Debug>
03/27/2016 20:47:22 upgrade_install.sh|release_upgrade_lock: Releasing lock (pid: 26363)|<LVL::Debug>

System History shows >>>> 03/27/2016 20:47:06 | root: Upgrade 10.5.2.13900-12 Success

When Checked on RTMT, Can see Low Virtual Memory is more than 95% sometimes even reaches 100%, also LowSwapPartitionAvailableDiskSpace with available is only 3%. When upgrading the CUCM the Available Hard Disk space is 9 GB.

Will these COP Files Help our upgrade?

The Cisco Free Common Space COP file can be used to free up space when the upgrade runs out of space on an OVA with a disk that is 80 GB or less.

ciscocm.free_common_space_v1.1.cop.sgn

10-MAR-2015

0.01 MB

The Cisco Free Common Space COP file can be used to free up space when the upgrade runs out of space on an OVA with a disk that is 80 GB or less.

ciscocm.free_common_space_v1.2.k3.cop.sgn

10-MAR-2015

0.01 MB

The Cisco Free Common Space COP file can be used to free up space when the upgrade runs out of space on an OVA with a disk that is 80 GB or less.

ciscocm.free_common_space_v1.3.k3.cop.sgn

Thanks,

Hrs

Deepak Rawat · ‎04-02-2016

Yes, you should absolutely install one of the cop files preferably ciscocm.free_common_space_v1.3.k3.cop.sgn to free the common partition. You can also check the common partition by issuing below command:

show status

http://docwiki.cisco.com/wiki/CUCM_Upgrades_and_Migrations_FAQ#What_COP_files_do_I_need.3F.3F

Regards

Deepak

Suresh Hudda · ‎04-02-2016

Yes, cop file will help to free up space but then u won't be able to switch back version. Hope you're upgrading from unrestricted to unrestricted version.

Suresh

Jaime Valencia · ‎04-02-2016

As Suresh mentions, installing that COP is a one way street only, it's far easier to try to simply download logs/traces from the server and choose the option to delete them than using the COP file.

If you were already able to do the install, that actually means there should be enough space in the common partition, otherwise, you would not have gotten far as an error would have prevented the upgrade until enough space was available.

High usage in swap should not affect the upgrade

And if those are the last lines of the upgrade logs, that's WAD, you analysis is wrong, this is from logs from my lab:

02/26/2016 17:28:25 upgrade_install.sh|Initiated cluster manager to send updated data to nodes|<LVL::Info>
02/26/2016 17:28:25 upgrade_install.sh|Upgrade Complete -- (0)|<LVL::Info>
02/26/2016 17:28:25 upgrade_install.sh|is_upgrade_lock_available: Upgrade lock is not available.|<LVL::Debug>
02/26/2016 17:28:25 upgrade_install.sh|is_upgrade_in_progress: Already locked by this process (pid: 6349).|<LVL::Debug>
02/26/2016 17:28:25 upgrade_install.sh|release_upgrade_lock: Releasing lock (pid: 6349)|<LVL::Debug>

That's a successful upgrade

Exactly how did you determine the upgrade failed???

HTH

java

if this helps, please rate

HRS · ‎04-02-2016

Thank you for All of your Reply...

@Jamie, Yes system history shows that the upgrade is successful but When we do "utils system upgrade status" the output shows as "Installing" we have waited for more than 24 hours.

and on GUI it was showing as "running"

Thanks,

Hari

Jaime Valencia · ‎04-02-2016

Have you tried switching the versions from CLI???

HTH

java

if this helps, please rate

d.haeni · ‎10-12-2016

When doing a L2 upgrade from 10.5(2) to 10.5(2)SU3a CUCM and Connection publisher servers were both showing the same behaviour (GUI kept showing as "running", utils system upgrade status" kept showing "Installing")

After waiting for one hour without any result, I started upgrading the Subscribers of both CUCM and Connection. Both completed just fine.

After switching versions CUCM/Connection Pub and the Sub, they all nicely booted into the upgraded partition.

Thanks for suggesting to keep calm and continue :)

/David

bradley.mcrae · ‎10-06-2017

I had the same issue in my lab going from 10.5.2SU3 to 10.5.2SU6. The Publisher hung at the end, but the Subscriber finished the upgrade normally.

I let the Publisher sit for at least 16hrs just to see if it would finish, or timeout, or other. It was happy to sit in upgrading mode. Nothing in the logs indicate why it hung at that point.

I rebooted the publisher, then switched versions, and followed it with a switch versions on the Subscriber. Everything seems to have come back just fine (it's a lab though, so I haven't verified most of the features).

I seem to recall having similar issues when I upgraded to 10.5.2SU3. I was hoping to find a root cause, but the workaround seems to have done the trick.

HRS · ‎10-27-2017

Here is what I have analyzed on the Lab, if think your upgrade got stuck. View the Install log and not just the last lines of the logs. If you upgrade got stuck on "release_upgrade_lock: Releasing lock" after couple of lines of "Upgrade Complete" this is a successful upgrade.

In order to confirm, Download or view the install log and look for keyword / lines like below,

(Omitted lines in between with "|")

""post_upgrade|========================= Upgrade complete. Awaiting switch to version. =========================

|

upgrade_manager.sh|_set_upgrade_status_attribute: status set to upgrade.stage.completed|<LVL::Debug>

upgrade_manager.sh|Upgrade complete|<LVL::Info>

|

upgrade_install.sh|System upgrade complete

|

upgrade_install.sh|release_upgrade_lock: Releasing lock (pid: 27224)|<LVL::Debug>

""

If you see most of these line then it safe to try switch version. We will come into this situation when there is an interruption on Webpage or CLI.

Also run show version inactive - you should see the new version.

Hope this helps!