CUCM CLUSTER RU UPGRADE FAILED FROM 11.5.1 TO 12.5.1

[BrkyRivera]

Hi people,

I am having an issue upgrading CUCM cluster for a customer from version 11.5.1.11900-26 to version 12.5.1.10000-22. Also tried 12.5.1SU3 failed that one too. Cluster is made of 1 Publisher and 1 subscriber, following IM/P and UnityC.

Here is the scenario for both hardware and software:

Hardware platform: BE6000 M4 appliance

Host: ESXi 6.5U3

CUCM OVA version 1.0

Steps taken prior to upgrade:

*Preupgrade task ran with no fails, just warning for smart licensing, so everyting passed fine.

*Ran free common space cop file, after resize of vdisk from 80GB to 110GB as per requirement to get enough free space.

Performed RU upgrade through the OS Administration GUI, restarts and does about 1.5-2 hrs of installing components and seems like last reboot when it crashes and will not boot with this error message (see attached): 

error: file `/refresh_upgrade/isolinux/initrd.img` not found.

Press any key to continue...

Remark: I have tested it prior to this upgrade on a single CUCM server, same scenario completely, upgrades easy and fine.

Thanks in advance people. Any help is appreciated.

[Roger Kallberg]

Yes I did mean @BrkyRivera, I accidental wrote @mmoulson2 in my reply, but I edited it later to correct it. If not even a freshly built and restored node works I would think that you'd best bet is to reach out to TAC to get their help on this.

[goranpilat]

Hi,

we had similar issue few days ago. What seemed to resolve the problem (and it wasn't diagnosed as problem on pre-upgrade-check) was to configure DNS servers and domain on cucm. Both DNS forward and reverse lookup have to be available on DNS servers. Also make sure NTP servers are of stratum 4 or less.

Hope it helps

Goran

[mmoulson2]

We tried to upgrade the publisher twice with both attempts resulting in a VM that would not boot. The issue only appeared to happen to the publisher node. We were fortunate that we had a VM level backup to be able to quickly restore the server and get services running again.

Unfortunately the process to upgrade for me was to rebuild the CUCM publisher as vanilla 11.5.1.11900-26. Then perform a DRS restore to the complete cluster. After doing that I attempted to upgrade to 12.5.1.10000-22 again and the upgrade was successful. This is not ideal at all and requires an extended maintenance window.

There was clearly some issue with the old 11.5 VM, however with it not booting up there was not much I could give to TAC in terms of logs. I'm sure there are people out there who could use a Linux rescue image to get file system access. However I didn't have the time to look into it as we were under pressure from the customer to get the work completed.

[Sirtalal]

Thanks for your reply, I really appreciate it

[BrkyRivera]

Hi Sirtalal,

It passed some time since I've had that issue, and it really took me a lot of time trying to avoid recovery and do sort of a fixup of VM. That wont work believe me, no fixup even if your good with linux. If you have a subscriber up and running perform a backup. Then use that link in my post for recovery procedure for publisher from subscriber. Once its back up and before you try an upgrade again, if I remember well you have to install that one .cop file which cleans and extends partitions. Also, preupgrade check cop file is mandatory as it runs checks on a lot of important thing such as free space, DNS, NTP accessibility etc...be clear that all tests pass before you proceed.   

[BrkyRivera]

Well there is a problem when doing VM recovery with your backup storage and not using reinstall with cucm disaster recovery/restore. When you do VM restore whit whatever backup solution for the clustered environment problem happens because of inconsistent database replication. Avoid that if all possible. You would have to do database replication fixup for all the servers in cluster.

OK, what you could do is clean install and restore publisher from subscriber if you dont have good backup of it. Make sure you make that disaster recovery backups before you start the process.

You can follow this guide: https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/116946-technote-product-00.html

[Laith Ibrahim]

Dears,

I faced this error while upgrading from 12.0.1(SU1) to 14.0(SU2),

All my pre upgrades tests were successful, and all needed cop files were installed,

Direct upgrade will never work in this scenario, and the TAC confirmed as well,

It worked with me with PCD 14.0, the PCD task need to be a Migration task only, direct upgrade through PCD will not work as well,

ofcourse here you need to install the migration cop file as per the cucm version (Very important),

I hope this helps,

[nadeem1nadeem]

Hello,

I am facing same error message. I was performing upgrade from 11.5 to 14.0 Su2.

I have restored the VM twice and tried upgrade but same error.

Let me know if anyone has fix for this upgrade error.

Nadeem

[Sirtalal]

Hi Nadeem,

 When I faced the issue I had to rebuild the node, Tac engineer said this issue happened because I changed the size of hard disk.

 I was following cisco guide when I change the size of hard disk but there was a bug in the guide.

[Laith Ibrahim]

Hello Nadeem,

Use Prime Collaboration Deployment (PCD v14.x) to do the migration for your CUCM, it is the best way to avoid all issues,

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/pcdadmin/14/cucm_b_pcd-admin-guide-1401/cucm_m_cisco-prime-collaboration-deployment-features.html#CUCM_TP_M7AF10F2_00

Thanks,

[mmoulson2]

Hi Nadeem, I am interested to know what method you used to "restore the VM"?

[nadeem1nadeem]

Hi,

Initially i have reverted VM snapshot from backup which didn't helped me.

Then TAC engineer reviewed the  logs and found in the past VM restarted ungracefully and advised to rebuild the CUCM node and then go for the upgrade.

After rebuilding CUCM node from the scratch my upgrade went well from 11.5 to 14.0.

Thank you all.