CUC 14 critical error:CUC subscriber installation failed at the end

[jaheshkhan]

I was trying to Fresh install CUC cluster (version 14) over WAN using PCD 14.

CUC publisher successfully installed but CUC subscriber in other site Host machine failed as below:

CUC version : 14.0

ESXi verion : 7.0

hardware: BE7000M

please help me 

Attaching error pic

Im using 15 character long security password with underscore("_"). Is this making an issue?

Im now worried that CUCM cluster will also failed if so 

very urgent situation. This was my test installation. but it failed at the end. every thing was going so smoothly.

[Nithin Eluvathingal]

 As PCD installation failed for sub,  install the sub  node using normal process. 

I assume, that the installation may be halt for more than an hour and that could be the reason for failed installation at branch. Since the site is connected through wan, there could be chances of network issues. 

Make sure you met all requirement for such designs(clustering through Wan)

[jaheshkhan]

kindly note the requiremets are all met thats the reason we went with clustering over WAN. they have 1GB speed.

installaiton didnt halt for an hour. im infront of it. it ended up all of sudden. some kind of sftp action then it stop 

08/09/2021 10:51:19
ERROR
cucsub
ks_post.sh|File:/etc/opt/cisco/install/setup/setup_root_access.sh:29, Function: setup_root_access(), Failed to disable root access|
08/09/2021 17:56:01
ERROR
cucsub
anaconda.install|File:/sbin/anaconda.install:2595, Function: main(), check_for_critical_error, real anaconda returned with - 0 code, continuing|
08/10/2021 04:58:07
ERROR
cucsub
ntp_one_shot.sh|File:/usr/local/bin/base_scripts/ntp_one_shot.sh:286, Function: ntpdateError(), NTP server cucpub did not reply as expected. 'ntpdate -q' failed (ntpdate rc=1).|
08/10/2021 04:58:07
ERROR
cucsub
ntp_one_shot.sh|File:/usr/local/bin/base_scripts/ntp_one_shot.sh:611, Function: setTime(), The primary node NTP server, cucpub, did not respond (rc=1).|
8/9/2021 22:24:22
ERROR
cucsub
Install Failed

[Roger Kallberg]

Looks like the NTP service on the CUC pub can not be reached by your error message.

[Nithin Eluvathingal]

Whats the NTP status on CUC primary node, is it synchronised ? 

Could you please reach check the rechability  between the Hq and Branch. 

reinstall the node. 

[jaheshkhan]

reachability is there.. 

esxi host IP and UC application IPs are in same subnet.

[Nithin Eluvathingal]

When I say reachability, its just IP ping between devices    . You need to make sure that asper the guides all required ports are opened . 

clustering over wan had  some requirement, refer SRND and also verify the security guide. It has mentioned all required port for clustering and it should be open.

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/connection/12x/security/b_12xcucsecx/b_12xcucsecx_chapter_00.html

In your case, The primary node NTP server, cucpub, did not respond... check if the NTP is synced on primary ? 

[jaheshkhan]

As you mentioned NTP servers are not accessible from CUC publisher. Attached the image.

But  CUC publisher can ping NTP servers.

[Nithin Eluvathingal]

Your CUC primary should be synced with a NTP server. Use   a working/reachable  NTP  server for CUC primary. Then restart the subscriber installation. 

[jaheshkhan]

what to do. I already mentioned client there should not be any kind of firewall blockage between NTP servers, DNS servers etc...

HQ and Branch there is no block between each other..

NTP server and CUC publisher are in HQ but on different subnets 

[jaheshkhan]

I dont know whats happening. now its showing  both NTP servers are accessible.

no idea whats happening

[jaheshkhan]

now error is as below:

is this because of bug? im now trying CUCM cluster. 

08/10/2021 00:33:10
ERROR
cucsub
IPM|Internal Error, File:ipm.c:2011, Function: ipmReadNormalizedInputLine(), "rpm --percent -iv --replacefiles /common/rpm-archive/14.0.1.10000-19/RPMS/thirdparty/cm-lib-1.0.0.0-1.i386.rpm " failed (1).|
08/10/2021 05:22:40
ERROR
cucsub
appmanager.sh|Internal Error, File:/usr/local/bin/base_scripts/appmanager.sh:164, Function: install(), failed to install application components|
08/10/2021 06:52:52
ERROR
cucsub
IPM|Internal Error, File:ipm.c:2011, Function: ipmReadNormalizedInputLine(), "rpm --percent -iv --replacefiles /common/rpm-archive/14.0.1.10000-19/RPMS/thirdparty/cm-lib-1.0.0.0-1.i386.rpm " failed (1).|
08/10/2021 06:53:29
ERROR
cucsub
appmanager.sh|Internal Error, File:/usr/local/bin/base_scripts/appmanager.sh:156, Function: install(), failed to install infrastructure_post components|
08/10/2021 07:01:03
ERROR
cucsub
IPM|Internal Error, File:ipm.c:2011, Function: ipmReadNormalizedInputLine(), "rpm --percent -iv --replacefiles /common/rpm-archive/14.0.1.10000-19/RPMS/thirdparty/cm-lib-1.0.0.0-1.i386.rpm " failed (1).|
08/10/2021 07:01:41
ERROR
cucsub
appmanager.sh|Internal Error, File:/usr/local/bin/base_scripts/appmanager.sh:156, Function: install(), failed to install infrastructure_post components|
8/10/2021 1:2:30
ERROR
cucsub
Install Failed

[Nithin Eluvathingal]

Did you resolved your NTP issue for CUC ? did your CUC synced with NTP ?

Installing Nodes over Wan, I always faced network challenges. Make sure your firewall has opened required port for CUC and CUCM. 

Here are the port users for CUCM. refer clustering port section and ask your firewall team to check it.

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/im_presence/configAdminGuide/14_0_1/cup0_b_config-and-admin-guide-1401/cup0_b_config-and-admin-guide-1401_chapter_0100001.html

Last week when I did a similar installation, it didnt went through. After the network team did some major changes installation went through. 

Its not just NTP and DNS. when clustering through wan your need to refer the port usage guides for CUC and CUCM, IM etc... If not clustering won't happen.

Last option, bring the server to HQ. install the nodes and then  move to branch. 

[jaheshkhan]

ports between two sites(HQ and branch) are completely open on both side for incoming and outgoing at the moment . no restriction

PCD and publishers are in the same subnet. EsxiHQ also in the same subnet

esxiBRANCH is in the same subnet of UC subscribers.

PCD DB taking time to reflect VM in the cluster once we delete and recreate the cluster again..  This is really bad. how long we need to wait?

[jaheshkhan]

In PCD I cannot select the VM . its greyed out. I deleted the VM and recreated the VM again with same name. I want the same. But the PCD still considering its already taken. i removed all clusters and task. sitll this particular VM is not able to selectable. 

also it take some times to get reselect in PCD..  

attaching the image

this details is not removing from PCD database. i restarted PCD . still the same. i cannot use the same name any more for PCD.