power failure: cuc sub, cucm sub, imp pub and imp sub not booting, black screen

[jaheshkhan]

implemented as follows in two BE6000M M4 servers

1) BE6000M M4 - Primary

==============

CUCM publisher
CUC publisher

IMP publisher

2) BE6000M M4 - secondary

==============

CUCM subscriber
CUC subscriber

IMP subscriber

all are in version 12.0

All were working without issue.

Today we found there was an power failure occured and both servers esxi were not up. so we powered on both esxi. and then we power on each servers one by one as first power on publisher and then subscriber.

then we noticed that cucm publisher and cuc publisher only powered on and up.

All other server did not show any sign of power on and endup in black screen with curser blinking.

What could be the reason? Are the VMs corrupted? or ESxi corrupted?  why only subscriber servers(even IMP publisher) showing this behavior ? Please help in this regard

[jaheshkhan]

can anybody help me in this regard? why its behaving like this? is it crashed? there is no backup. because its new installation so we didnt setup yet.

[Flo.Matalis]

Have you tried deploying VMs again on the secondary server? Is it still working?

If it is still working, Subscriber VMs could only be corrupted because they were not gracefully shutdown. Good thing your CUCM and CUC pubs are still UP. Since your CUCM and CUC Publishers are still UP, you may create subscriber server again and since you were not able to do backups, you need to create IMP servers again. Best if after implementation, you need to add get backup just in case an issue happens.

[jaheshkhan · Accepted solution]

Hmm... Raised the issue with TAC and issue has solved. It is related to one bug (CSCvh55176).

BUG 1 : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvh55176/

Solution

1. Power off/shudown the server if it is ON.
2. Mount respective ISO image of CUCM/IMP server recovery CD to corresponding Server VM.
3. Boot the server to recovery CD.
4. After fully loaded and while at command prompt, Press ALT+F2. This will shift to Root commandline.
5. Enter the following commands:
[anaconda root@localhost /]# find / -name initramfs* ---> This will find which folder the img file reside.
[anaconda root@localhost /]# ls -l /mnt/part1/boot
[anaconda root@localhost /]# chroot /mnt/part1
[anaconda root@localhost /]# /usr/bin/vmware-config-tools.pl -d

After entering the above commands, quit from command line and power off the VM and then power on VM again. After this, it will boot properly.

[joshua.gertig]

Just wanted to chime in that we ran across this bug as well, on Unity Connection 11.5 SU5.
SU5 is listed as fixed, but for CCM not Unity. 
We followed the commands that were listed in the linked bug search (https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvh55176/). As the instructions said,  we did see errors after running this command:
/usr/bin/vmware-config-tools.pl -d

But it ended up booting ok. Thanks!!!

[lior look]

Yesterday, I upgraded some CUCM servers to 11.5(1)SU6 and it happened on this version also, it fixed by the workaround of CSCvh55176

---

when you mount the recovery iso on the vm, and you do not succeed to boot from this after reboot/recycling, make sure the vm configured to boot first from CD drive.

set the vm to load the BIOS blue screen after next reboot, go to boot tab and make sure the CD drive in the first line.

after you set the workaround steps, go back to BIOS screen and turn back the boot list as was before, move the CD drive to the last line.

good luck!