CUCM 10.5.2.13901-2: issue during Suscriber installation

[raziel78kain]

Hello,

we have a CUCM 10.5.2.13901-2 Publisher, and we are trying to install a Subscriber, but the installation fails during the Connectivity Validation step to the Publisher.

Keep in mind that everything should be OK at network/configuration side:

• DNS OK (A and PTR records configured for every CUCM node)
• NTP OK (NTP Servers configured on Pub at Stratum 1)
• Reachability OK (Pub and Sub are in the same VLAN, so no firewall in the middle)
• Sub's server node name configured on Pub

Here's the snippet of the install logs related to the Connectivity Validation step:

----------

09/16/2016 01:09:46 InstallWizard|ipmScriptListRun: desc="Running Connectivity Validation", title="Connectivity Validation", items=3|<LVL::Debug>

09/16/2016 01:09:46 IPM|Open progress meter "Connectivity Validation"|<LVL::Info>

09/16/2016 01:09:46 IPM|  begin-of-session "Running Connectivity Validation", 3 items|<LVL::Info>

09/16/2016 01:09:46 IPM|    ipmDoTimedCommand: cmd="/usr/local/bin/base_scripts/primNodeVerify.sh", est(sec)=60, max(sec)=300|<LVL::Debug>

09/16/2016 01:09:46 IPM|    begin-of-work: [cmd="/usr/local/bin/base_scripts/primNodeVerify.sh"]|<LVL::Info>

09/16/2016 01:09:46 IPM-Child|execlp(/tmp/.ipmSnHa0a) for cmd "/usr/local/bin/base_scripts/primNodeVerify.sh"|<LVL::Debug>

09/16/2016 1 0:09:49 InstallWizard|Platform Install: installWizMain.c@@/main/cm_su3_10_5_2/1|<LVL::Info>

09/16/2016 01:16:11 IPM|(CAPTURE) head: cannot open `/usr/local/platform/.security/CCMEncryption/keys/encrypted_Dkey.txt' for reading: No such file or directory|<LVL::Debug>

09/16/2016 01:16:11 IPM|(CAPTURE) isftp using password failed (11). Trying sftp using keys.|<LVL::Debug>

09/16/2016 01:16:11 IPM|(CAPTURE) isftp using password failed (12). Trying sftp using keys.|<LVL::Debug>

09/16/2016 01:16:11 IPM|(CAPTURE) File /usr/local/platform/conf/cluster.conf not found on the first node. rc=12|<LVL::Debug>

09/16/2016 01:16:11 IPM|Saw potential hang while executing "/usr/local/bin/base_scripts/primNodeVerify.sh".|<LVL::Warn>

09/16/2016 01:16:11 IPM|    end-of-work: [cmd="/usr/local/bin/base_scripts/primNodeVerify.sh"]|<LVL::Info>

09/16/2016 01:16:11 IPM|Child's return-status = 0x00000009|<LVL::Debug>

09/16/2016 01:16:11 IPM|Internal Error, File:ipm.c:1919, Function: ipmDoTimedCommand(), Installation appeared to hang while executing "/usr/local/bin/base_scripts/primNodeVerify.sh" (estimated time  0:01:00, lapsed t ime  0:05:00)|<LVL::Critical>

09/16/2016 01:16:13 IPM|  end-of-session "Running Connectivity Validation": 387.156 secs.|<LVL::Info>

09/16/2016 01:16:13 IPM|Close progress meter "Connectivity Validation"|<LVL::Info>

09/16/2016 01:16:13 post_install|Exiting with result 1|<LVL::Info>

09/16/2016 01:16:13 post_install|INSTALL_TYPE="Basic Install"|<LVL::Debug>

09/16/2016 01:16:13 post_install|File:/opt/cisco/install/bin/post_install:720, Function: check_for_critical_error(), check_for_critical_error, found /common/log/install/critical.log, exiting|<LVL::Error>

09/16/2016 01:16:13 post_install|(CAPTURE) Mail notification cancelled - smtp server address for email not found! [/tmp/platformConfig.xml]|<LVL::Debug>

09/16/2016 01:16:13 display_screen|Arguments: "Critical Error" "The installation has encountered a unrecoverable internal error. For further assistance report the following information to your support provider.

Installation appeared to hang while executing "/usr/local/bin/base_scripts/primNodeVerify.sh" (estimated time  0:01:00, lapsed time  0:05:00)

----------

What's wrong?

TIA and regards.

[Aman Soi]

Hi,

Check  the discussion if it relates  your issue

https://supportforums.cisco.com/discussion/12451881/cucm-105-fresh-install-suscribers-fails

regds,

aman

[raziel78kain]

Hello Aman,

thanks for your answer.

I don't think that the bug CSCus35964 (mentioned in that discussion) is our issue, because:

• we have a CUCM 10.5.2.13901-2, while the known affected release is 10.5(2.10000.1) and one of the known fixed releases is 10.5(2.11900.2), and both of them are older than our release, so this bug should be not present in our release;
• we have made a fresh installation of a brand new Publisher, followed by a fresh installation of a brand new Subscriber, so no backup/restore activities have been made in our scenario.

Regards.

[Evgeny Izetov]

Please check that you haven't mistyped the security password.

[raziel78kain]

Hello Evgeny,

we have tried to modify the security password on the Publisher with the same that we have; here's the output of the command "set password user security":

----------

admin:set password user security
Please enter the old password: *********
Please enter the new password: *********
Old and New password are the same.Please use a password different from the existing one

Please enter the new password:
Control-C pressed

----------

So, the security password should be correct...

Regards.

[Aman Soi]

Suggest opening TAC case.

change of security password would require reboot of server.

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/cli_ref/9_1_1/CUCM_BK_C6AE17AA_00_cucm-cli-reference-guide-91/CUCM_BK_C6AE17AA_00_cucm-cli-reference-guide-91_chapter_0110.html

regds,

aman

[raziel78kain]

Hello Aman,

at last, we have found the issue.

It wasn't a password issue, but probably a VLAN/MTU issue.

In fact, with everything configured with an MTU = 1500 (on the VMs, on the VMware ESXi nodes, on the switches), it wasn't possible to make a ping from a VM to another one (on the same node or on different nodes) with a payload from 1469 to 1472 bytes (i.e., an MTU from 1497 to 1500 bytes) with the Don't Fragment (DF) bit set; instead, a ping with a payload of 1468 bytes (i.e., an MTU of 1496 bytes) with the DF set worked correctly...

And we know that 4 bytes (1500 - 1496) is the size of the VLAN ID in the 802.1Q... But this overhead should be automatically kept into account, with no effects to the VM's MTU, isn't it?

Furthermore, the weird thing is that this issue was present only on a couple of UCS-switch connections; in fact, other connections worked correctly.

Finsally we have simply solved by issuing a "shutdown/no shutdown" on the switch interface.

Why, in your opinion, has this operation worked?

Thanks and regards.

[Manish Gogna]

Hi,

A few imp points regarding MTU size in the CUCM install guides are the following:

The maximum transmission unit (MTU) represents the largest packet, in bytes, that this host will transmit on the network.

The MTU size that you configure must not exceed the lowest MTU size that is configured on any link in your network.

Default: 1500 bytes

The MTU setting must be the same on all nodes in a cluster.

Manish

[raziel78kain]

Hello Manish,

we already knew that, but why the "actual" MTU was 1496 before the "shut/no shut" on the switch, and it's 1500 after the "shut/no shut", even if we had configured everything with 1500? Do you have any ideas about that?

Regards.

[Manish Gogna]

Hi,

I am not sure if a shut / no shut of port will affect the MTU value, the issue could have been something else. However, please remember that when a VLAN has switch ports with different MTU size, packets received from a port with a larger MTU might be dropped when they are forwarded to a port with a smaller MTU. This is mentioned in the Switch IOS config guides.

Manish

[gary.w.johnson]

The shutdown/no shutdown worked for me as well.  I'm deploying a new install of CUCM 11.5(1)SU3 on BE6K-M's (C220-M4 platform).  I ran into the same error(s).

When I saw your post, I took a few seconds and "bumped" all of the publisher and subscriber ports in the VoIP vlan on my lab switch 3750E-24TD-E running IOS 12.2 (58)SE2.

I can't explain why this worked, but I'm thankful that it did.

Thanks for posting.

[Rana21]

The shutdown/no shutdown worked for me as well.

[HARIS_HUSSAIN]

I have faced similar issue during installation.

If it is fresh installation of publisher try reinstalling the publisher and add the subscriber.

If not raise case with Cisco TAC.

Thanks

Haris

[raziel78kain]

Hello Haris,

have you had the issue with the same identical CUCM version (10.5.2.13901-2)?

Regards.