Re: CUCM 12 Unable to Add Additional Subsriber

mumbles202 · ‎11-05-2019

Trying to add an additional subscriber to an already deployed cluster. The Cluster is currently running 12.0.1.22900-11 so I used the bootable base 12 image and used UCSInstall_UCOS_12.0.1.22900-11.sgn.iso to update the install media to the correct version before adding the cluster information. The server is added into the Publisher under servers but the installs keeps failing and displaying the following:

the installation has encountered a unrecoverable internal error. for further assistance report the following information to your support provider.

"/usr/local/cm/script/cm-dbl-install install PostInstall 12.0.1.22900-11 12.0.1.22900-11 /usr/local/cm/usr/local/cm / /common/log/install/capture.txt" failed (1)

The hardware is different but should be supported as far as i know, though the VMware environment is different for the new subscriber if that makes a difference.

Jaime Valencia · ‎11-05-2019

You'd need to dump and review the install logs for more information as to why it's failing

HTH

java

if this helps, please rate

mumbles202 · ‎11-06-2019

Yes that would be how I would normally proceed. Problem is server is on VCloud rather than traditional ESXi/VI so the option to add a serial port isn't available.

Vaijanath Sonvane · ‎11-07-2019

Hi,

What version of ESXi is running on vCloud? Please note CUCM 12.0 is not supported on ESXi 6.7.

Are you using the correct security password when you are installing new subscriber?

Please rate helpful posts and if applicable mark "Accept as a Solution".
Thanks, Vaijanath S.

mumbles202 · ‎11-14-2019

Security password seems to be correct as when I use a password I know is incorrect the validation fails when reaching out to the publisher.

For a failed installation is the correct procedure to delete the node definition
from within CUCM, reboot the cluster and then readd the the server name and try the install?

mumbles202 · ‎11-15-2019

I'm seeing this error in the logs:

2019-11-15 23:26:52,081 ERROR [main] - trust directory parameter is null
2019-11-15 23:26:52,081 ERROR [main] - Directory '/usr/local/platform/.security/ipsec/trust-certs' does not exist
com.cisco.cpi.certMgmt.CertMgrException: Directory '/usr/local/platform/.security/ipsec/trust-certs' does not exist

This is CUCM 12.0.1.22900-11.

Vaijanath Sonvane · ‎11-15-2019

Hi,

There is a software bug CSCvc83768 which relates to same issue but not with CUCM 12.0. Please open Cisco TAC case for further help.

Please rate helpful posts and if applicable mark "Accept as a Solution".
Thanks, Vaijanath S.

Juraj Papic · ‎11-15-2019

Hello,

I have the same issue, but when I add my first sub to my pub, this is a brand new install.

Im working with esxi and when I want to send the install log via serial port its allways blank.

thanks.

mumbles202 · ‎11-16-2019

Here some more information from the install file if it helps to determine the error:

2019-11-15 17:08:29,169 ERROR [main] - The keystore does not exisit, loading empty.
2019-11-15 17:08:29,169 INFO [main] - OUT -- RSACryptoEngine.java - loadKeyStore - 
2019-11-15 17:08:29,180 ERROR [main] - Failed to merge tomcat-ECDSA keystore into tomcat keystore

2019-11-15 17:08:29,183 ERROR [main] - Merge of tomcat and tomcat-ECDSA keystore failed.
com.cisco.cpi.certMgmt.CertMgrException: Merge of tomcat and tomcat-ECDSA keystore failed.

2019-11-15 23:22:30,098 INFO [main] - IN -- ITLRecoveryCertMgr - decryptCertificateAndKey()
2019-11-15 23:22:30,105 INFO [main] - Exiting encryptPassword.fail. failed to read status file
2019-11-15 23:22:30,106 FATAL [main] - java.io.FileNotFoundException: /usr/local/platform/.security/dkey_status.txt (No such file or directory)
2019-11-15 23:22:30,107 INFO [main] - Entering decryptPassword
2019-11-15 23:22:30,113 INFO [main] - Use Dkey to decrypt data


2019-11-15 23:22:30,798 DEBUG [main] - CryptoFeatureEntry - Check of envvar CISCOJ_PERMIT_UNIMPLEMENTED_CIPHER_DESEDEWRAP for key/value Cipher.DESedeWrap/com.cisco.ciscossl.provider.ciscojce.CiscoJCEFirewall$CiscoJCEUnimplementedFIPS yields permission false
2019-11-15 23:22:30,798 DEBUG [main] - CryptoFeatureEntry - Check of envvar CISCOJ_PERMIT_UNIMPLEMENTED_CIPHER_AESWRAP for key/value Cipher.AESWrap/com.cisco.ciscossl.provider.ciscojce.CiscoJCEFirewall$CiscoJCEUnimplementedFIPS yields permission false
2019-11-15 23:22:30,798 WARN [main] - configureCiscoJCEFirewall - No non-FIPS crypto features being intercepted by CiscoJCE
2019-11-15 23:22:30,798 DEBUG [main] - configureCiscoJCEFirewall - Installing handler Cipher.DESedeWrap/com.cisco.ciscossl.provider.ciscojce.CiscoJCEFirewall$CiscoJCEUnimplementedFIPS to disallow Cipher.DESedeWrap 
2019-11-15 23:22:30,798 DEBUG [main] - configureCiscoJCEFirewall - Installing handler Cipher.AESWrap/com.cisco.ciscossl.provider.ciscojce.CiscoJCEFirewall$CiscoJCEUnimplementedFIPS to disallow Cipher.AESWrap 
2019-11-15 23:22:30,808 DEBUG [main] - engineSetPadding - Setting padding for aes-0-cbc to PKCS5Padding
2019-11-15 23:22:30,837 DEBUG [main] - engineSetPadding - Setting padding for aes-0-cbc to PKCS5Padding
2019-11-15 23:22:30,846 INFO [main] - Exiting decryptPassword.fail
2019-11-15 23:22:30,846 FATAL [main] - Decryption with static key failed as well. Fatal error javax.crypto.BadPaddingException: ../Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
2019-11-15 23:22:30,846 INFO [main] - CCMENC::ERROR : Dkey decryption failed. Use recovery mechanism to decrypt data.
2019-11-15 23:22:30,846 INFO [main] - Using static key to decrypt data
2019-11-15 23:22:30,846 DEBUG [main] - engineSetPadding - Setting padding for aes-0-cbc to PKCS5Padding
2019-11-15 23:22:30,846 DEBUG [main] - engineSetPadding - Setting padding for aes-0-cbc to PKCS5Padding
2019-11-15 23:22:30,846 INFO [main] - Exiting decryptPassword.fail
2019-11-15 23:22:30,846 FATAL [main] - Decryption with static key failed as well. Fatal error javax.crypto.BadPaddingException: ../Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
2019-11-15 23:22:30,847 INFO [main] - CCMENC::ERROR : static key decryption failed. Use old keys to decrypt data
2019-11-15 23:22:30,847 INFO [main] - Exiting DecryptPassword.fail. failed to read oldkey file
2019-11-15 23:22:30,847 FATAL [main] - java.io.FileNotFoundException: /usr/local/platform/.security/CCMEncryption/keys/oldkeys.txt (No such file or directory)
2019-11-15 23:22:30,847 INFO [main] - CCMENC::ERROR : decryptPassword - recovery mechanism failed



23:22:08.590 |Do not write /etc/hosts until Platform Config is done.
23:22:08.590 |adding new entry in etc/hosts: myorg-subscriber2(10.0.17.8)
23:22:08.590 |succeeded to create sm for: myorg-subscriber2
23:22:08.590 |exec'ing: sudo /root/.security/ipsec/disable_ipsec.sh --desthostName=myorg-subscriber2 --op=delete
23:22:09.177 |Timeout or error() 115 - Operation now in progress, port 8500


2019-11-15 23:22:30,847 INFO [main] - CCMENC::ERROR : decryptPassword - recovery mechanism failed
2019-11-15 23:22:30,847 ERROR [main] - Exception while Decrypting the string
2019-11-15 23:22:30,849 ERROR [main] - Error. Unable to decrypt Certificate-key pair. 
2019-11-15 23:22:30,849 ERROR [main] - Error. Unable to decrypt Certificate-key pair. 

11/15/2019 16:39:03 IPM|      command #284: 60628 bytes, 0 seconds: cp /mnt/source/RedHat/RPMS/libgpg-error-1.7-4.el6.x86_64.rpm /mnt/sysimage/common/rpm-archive/12.0.1.10000-10/RPMS/libgpg-error-1.7-4.el6.x86_64.rpm|<LVL::Info>
11/15/2019 16:39:07 IPM|Execute copy cmd "cp /mnt/source/RedHat/RPMS/libgpg-error-1.7-4.el6.i686.rpm /mnt/sysimage/common/rpm-archive/12.0.1.10000-10/RPMS/libgpg-error-1.7-4.el6.i686.rpm:60532"|<LVL::Debug>
11/15/2019 16:39:07 IPM|Execute copy cmd "cp /mnt/source/RedHat/RPMS/libgpg-error-1.7-4.el6.i686.rpm /mnt/sysimage/common/rpm-archive/12.0.1.10000-10/RPMS/libgpg-error-1.7-4.el6.i686.rpm:60532"|<LVL::Debug>
11/15/2019 16:39:07 IPM|Execute copy cmd "cp /mnt/source/RedHat/RPMS/libgpg-error-1.7-4.el6.x86_64.rpm /mnt/sysimage/common/rpm-archive/12.0.1.10000-10/RPMS/libgpg-error-1.7-4.el6.x86_64.rpm:60628"|<LVL::Debug>
11/15/2019 16:39:07 IPM|Execute copy cmd "cp /mnt/source/RedHat/RPMS/libgpg-error-1.7-4.el6.x86_64.rpm /mnt/sysimage/common/rpm-archive/12.0.1.10000-10/RPMS/libgpg-error-1.7-4.el6.x86_64.rpm:60628"|<LVL::Debug>
11/15/2019 16:46:14 security_postinstall|Successful in migrating tomcat manager error page to not disclose information.|<LVL::Info>
11/15/2019 16:46:55 ks_post.sh|kerneldump - Successfully modified /sbin/mkdumprd to suppress harmless errors while loading block device module|<LVL::Info>
11/15/2019 16:47:35 ks_post.sh|(CAPTURE) error reading information on service smartd: No such file or directory|<LVL::Debug>
11/15/2019 16:58:45 upgrade_manager.sh|(CAPTURE) shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory|<LVL::Debug>
11/15/2019 16:58:54 upgrade_manager.sh|(CAPTURE) shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory|<LVL::Debug>
11/15/2019 16:58:54 upgrade_manager.sh|(CAPTURE) shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory|<LVL::Debug>
11/15/2019 16:58:56 populate_rpm_archive|Copy libgpg-error-1.7-4.el6.i686.rpm to /common/rpm-archive/12.0.1.22900-11/RPMS|<LVL::Info>
11/15/2019 16:58:56 populate_rpm_archive|Located libgpg-error-1.7-4.el6.i686.rpm in /mnt/source/RedHat/RPMS|<LVL::Debug>
11/15/2019 16:58:56 populate_rpm_archive|Add "cp /mnt/source/RedHat/RPMS/libgpg-error-1.7-4.el6.i686.rpm /common/rpm-archive/12.0.1.22900-11/RPMS/libgpg-error-1.7-4.el6.i686.rpm" to command list|<LVL::Debug>


FATAL [main] - java.io.FileNotFoundException: /usr/local/platform/.security/dkey_status.txt (No such file or directory)

FATAL [main] - Decryption with static key failed as well. Fatal error javax.crypto.BadPaddingException: ../Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
2019-11-15 23:22:31,805 INFO [main] - CCMENC::ERROR : Dkey decryption failed. Use recovery mechanism to decrypt data.

BTW, not sure if it is relevant but I booted off the bootable 12.0.1.10000-10 media and then used the iso 12.0.1.22900-11 to upgrade.