CUCM upgrade from 9.1.2 to 11.5 failed. Cannot cancel upgrade process

[Hoa Dao]

I was upgrading one of the subscribers to version 11.5.  During the uprade, the VM host ran out of storage; the subscriber hang.  I had no choice but to reboot it to regain access.

Since then the server upgrade status is always "installing".  I cannot reset the upgrade status. 

I try to cancel it with CLI commands:

admin:utils system upgrade cancel

Canceling the upgrade.  Please wait...

Upgrade status: installing

From the GUI, I select "Resume control".  The status is "running".  I then click "Cancel", the system returns "Installation or upgrade was canceled succesfully".  However, when I try the upgrade again, it shows "Warning: Another session is installing the software..."

Reboot the server did not help either

Do you know a way to reset the upgrade status?

Please help!!!

[Manish Gogna]

That's a bit strange. How many nodes are there in this cucm cluster? How many of them are already upgraded to the desired version? What are the exact existing and target cucm versions? 

Manish

[Hoa Dao]

Manish,

It is 7-node cluster, and we are upgrading from 9.1.2.11900-12 to 11.5.1.11900-26.  6 nodes were upgraded successfully.  During the upgrade of the last node, for some reason, the VM back-end storage infrastructure, NOT the disk space on the CUCM node, ran out of LUN storage space.  (This is an issue on the VM infrastructure)  The node was frozen, I had to recycle the power to regain access to the node.

Since then, the node always stucks in upgrading status.  I cannot cancel the upgrade process.  Is there another way to cancel the upgrade process besides the CLI command "utils system upgrade cancel" or from the CUCM OS admin web page?

[Jaime Valencia]

That sounds to me like this is breaking the storage requirements for Cisco UC apps (and that means your deployment would be not supported)

Storage

The following apply to supported DAS, SAN and NAS storage

• Must map 1 GB of VM vDisk to 1 GB of physical storage.
  • Storage thin provisioning is not recommended (whether at VM layer or storage array layer).
  • Any other form of storage oversubscription is not supported.
  • The sum of virtual machines' vDisks may not exceed the physical disk space of the physical server's logical volume capacity (i.e. capacity net of overhead for the VM itself, VMFS in vSphere and physical RAID configuration).
  • Cisco recommends 10% buffer on top of vDisk values to handle overhead within the VM (such as swap files which are the size of the VM's vRAM). See Shared Storage Considerations for more details.
• The DAS, NAS or SAN storage solution must also supply enough performance to handle the total load of the VMs.
  • Must provide enough IOPS to handle sum of the VMs.
  • Kernel command latency must not exceed 2-3 milliseconds.
  • Physical device command latency must not exceed 15-20 milliseconds.
  • See Shared Storage Considerations and Storage Performance Requirements for more details.
• If the above capacity or performance requirements are not met, the storage system is overloaded and must be "fixed" by either moving virtual machines to alternate storage, or improving storage hardware.

http://www.cisco.com/c/dam/en/us/td/docs/voice_ip_comm/uc_system/virtualization/collaboration-virtualization-sizing.html#generalrulestorage

You're facing a scenario that should not be faced if you abide by the above rules, I'd suggest you pass the above guidelines to whoever is responsible for your storage network, and I'd probably just reinstall that server, there really is no way to tell what happens if you run out of space.

[Hoa Dao]

Yes, our storage is now fixed. 

Is there another way besides reinstalling version 9.1.2.11900-12; ie. is there a way to reset the current "installing" upgrade status  and re-initiate the11.5.1.11900-26 upgrade???

[Jaime Valencia]

Not that I'm aware of, it's either via CLI, GUI or a reboot. If none of those have done the trick, right now you're on uncharted territory due to the problem with your storage.

You can try TAC, but you'll have to explain them what happened, and see if they have any idea of what to do, but chances are, they will also tell you to rebuild that server as the upgrade procedure was halted in a way that's never supposed to happen.

[Hoa Dao]

FYI
I had to reinstall the subscriber.

[Sushant Sharma]

You have already done with publisher so better to rebuild the server again with new version .