cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1122
Views
15
Helpful
6
Replies

CUCM upgrade from 11.5 SU3 to 11.5 SU11 is failing

Santosh Mohite
Level 1
Level 1

Hi,

I am trying to upgrade CUCM from 11.5 SU3 to 11.5 SU11 but it's failing with an unknown error. DBreplication is OK.

 

File UCSInstall_UCOS_UNRST_11.5.1.23900-30.sha512.iso
Start Time Thu Sep 15 21:56:49 SGT 2022
Status Error encountered: An unknown error occurred while accessing the upgrade file. Warning: A system reboot is required when the upgrade process completes or is canceled. This will ensure services affected by the upgrade process are functioning properly.
Action Download the file from the link provided and reboot the system.
Log File Install Log
Advanced Error Messages Hide

/usr/local/cm/script/cm-dbl-install L2 PostInstall 11.5.1.23900-30 11.5.1.13900-52 /usr/local/cm/ /partB/usr/local/cm/ /common/log/install/capture.txt failed (1)

failed to upgrade infrastructure_post components

/usr/local/bin/base_scripts/appmanager.sh -upgrade -basepath /common/rpm-archive/11.5.1.23900-30 -logfile /common/log/install/install_log_2022-09-15.21.11.50.log -resultfile /common/log/install/upgrade-results.xml failed (1)

 

============================================
admin:utils dbreplication runtimestate

Server Time: Sat Sep 17 13:02:35 +08 2022

Cluster Replication State: Replication status command started at: 2022-09-17-13-00
Replication status command ENDED. Checked 706 tables out of 706
Last Completed Table: devicenumplanmapremdestmap
No Errors or Mismatches found.

Use 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2022_09_17_13_00_29.out' to see the details


DB Version: ccm11_5_1_13900_52

Repltimeout set to: 300s
PROCESS option set to: 1


Cluster Detailed View from SGPCUCMPUB (2 Servers):

PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
ABC X.244.225.241 0.012 Y/Y/Y 0 (g_2) (2) Setup Completed
XYZ X 0.070 Y/Y/Y 0 (g_3) (2) Setup Completed

 

I already attempted the upgrade couple of times with cluster reboot but no luck. Any help would be appreciated.

 

Thanks,

Santosh

1 Accepted Solution

Accepted Solutions

     -" Standard CTI Secure Connection " present in "dirgroup" table. This must be present in db dirgroup.

can you check the decency record for this role and if possible try to remove it from access group and rerun the preupgrade check

 



Response Signature


View solution in original post

6 Replies 6

Before the upgrade, download and install/run the latest version of preUpgradeCheck COP file. Look at PASS / WARNING / FAIL output.  Resolve all FAIL and maybe the WARNINGs. Repeat until happy.

And its mandatory you read the below

- Release note of upgrade file

- Install upgrade Guide

 



Response Signature


Santosh Mohite
Level 1
Level 1

Hi Nithin,

Can you please provide the link to preUpgradeCheck cop file? I couldn't find the file.

Thanks

https://software.cisco.com/download/home/286322286/type/286319173/release/COP-Files

 

As the above file is sha 512 signed, you need to install the below cop first. you can download both files from above link.

ciscocm.enable-sha512sum-2021-signing-key-v1.0.cop.sgn



Response Signature


Hi Nithin,

There are 3 failures. I am not sure how to fix dataBase sanity.

1) NTP: I added another free NTP server 162.159.200.1 (in-country) as our internal NTP seems to be taking a longer time to sync

admin:utils ntp status
ntpd (pid 19380) is running...

remote              refid                      st  t when poll reach delay offset jitter
==============================================================================
+X.X.X.X            162.159.200.1     4  u  7   64 377 232.796 -0.997 6.054
*162.159.200.1 10.23.8.4             3  u  67 64 377 34.207 2.694 1.132


synchronised to NTP server (162.159.200.1) at stratum 4
time correct to within 104 ms
polling server every 64 s

Current time in UTC is : Sat Sep 17 17:12:39 UTC 2022
Current time in Asia/Singapore is : Sun Sep 18 01:12:39 +08 2022

2) Test dataBase Sanity: 
-" Standard CTI Secure Connection " present in "dirgroup" table.
It is recommended to fix the entries before upgrade

3)File system checks: It seems to be ok for 11.5 versions upgrade

 

admin:file view install PreUpgradeReport.txt
 
Use "file view install PreUpgradeReport.txt" to view the report
================================================================================
                          Pre Upgrade Test         Date: 09/17/2022 16:39:23 UTC
================================================================================
     Active Version:   11.5.1.13900-52
     Server:   CUCMPUB , CUCM Publisher
================================================================================
 
     Result  Test
     ------  -------------------------------------------------------------------
1.1  PASS    Common Security Password Length
     System not in FIPS mode, Common Security Password's Minimum length
     requirement not enforced
 
1.2  FAIL    Network status (NTP, DNS & Cluster node connectivity)
     ntp_stratum         : Failed
     The reference NTP server is a stratum 5 clock.
     NTP servers with stratum 5 or worse clocks are deemed unreliable.
     Please consider using an NTP server with better stratum level.
 
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 183) :
     Please use OS Admin GUI to add/delete NTP servers.
 
1.3  PASS    Deprecated Phone Models
1.4  PASS    Cluster Database Status
1.5  PASS    PLM License Status
1.6  FAIL    Test dataBase Sanity
 
     -" Standard CTI Secure Connection " present in "dirgroup" table.
     It is recommended to fix the entries before upgrade
 
1.7  WARN    DRS backup status
     Last Successful backup taken on: Wed Sep 14 01:29:36 SGT 2022
 
1.8  FAIL    Filesystem Checks
 
     *) Upgrade of the server to some or all of the following version(s) will
     fail -
11.5
     PASS: You have 27558012093 bytes of required 25417837269 bytes common
     partition space required for L2 upgrade to 11.5.
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 183) :
 
12
     PASS: You have 27557999805 bytes of required 25168436072 bytes common
     partition space required for L2 upgrade to 12.0.
 
12.5
     Failed: You only have 27557967037 bytes of required 29050540694 bytes
     common partition space required for RU upgrade to 12.5. You do not need to
     cleanup inactivelog. /common/log/inactive space will be automatically
     reclaimed. Cleanup 1492573657 bytes of space from /common/log/active,
     /common to free up the required space.
 
14
     Failed: You only have 27557700797 bytes of required 29058810518 bytes
     common partition space required for RU upgrade to 14. You do not need to
     cleanup inactivelog. /common/log/inactive space will be automatically
     reclaimed. Cleanup 1501109721 bytes of space from /common/log/active,
     /common to free up the required space.
 
     Before starting the upgrade, clear space via the following methods and
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 60 of 183) :
     reinstall this COP:
     - Install latest version of ciscocm.free_common_space COP file from
     cisco.com (this will delete trace/log files, current inactive version, and
     backed up COP files).
     - For source 11.0+, use OS Admin GUI's Device Load Management to Remove
     Unused Firmware from the System.
     - Delete any unneeded core files with "file delete" CLI.
     - For source 10.x and higher, increase size of your vdisk via power down
     VM, edit VM specs, powerup.
     Details of directories occupying most space in /common partition are as
     below...
     ********************************************
     Used /var/log/active(/common/log/taos-log-a/b) directory space: 20G
     Detailed view:
     1.2G    /var/log/active/audit
     9.1G    /var/log/active/car_db
     9.8M    /var/log/active/ccm_db
     6.1G    /var/log/active/cm
     99M     /var/log/active/core
     538M    /var/log/active/dp_db
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 61 - 80 of 183) :
     45M     /var/log/active/elm
     60M     /var/log/active/elm_db
     20K     /var/log/active/installed_options
     4.0K    /var/log/active/mgetty
     18M     /var/log/active/mohprep
     4.0K    /var/log/active/patches
     552M    /var/log/active/platform
     0       /var/log/active/sa
     8.0K    /var/log/active/sso
     95M     /var/log/active/syslog
     1.5G    /var/log/active/tomcat
     112K    /var/log/active/uccx
     ********************************************
     Used /var/log/active/cm directory space: 6.1G
     Detailed view:
     895M    /var/log/active/cm/bin
     4.0K    /var/log/active/cm/CCMEncryptionMigration.log
     0       /var/log/active/cm/cdr
     0       /var/log/active/cm/cdr_repository
     180M    /var/log/active/cm/log
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 81 - 100 of 183) :
     139M    /var/log/active/cm/phoneguide_locales
     9.8M    /var/log/active/cm/report
     1.6G    /var/log/active/cm/tftpdata
     3.3G    /var/log/active/cm/trace
     ********************************************
     Used /var/log/inactive((/common/log/taos-log-a/b) directory space: 895M
     ********************************************
     Used /common directory space: 21G
     Detailed view:
     36K     /common/adminsftp
     4.0K    /common/backup-sysctl.conf.append
     3.6M    /common/bps
     4.0K    /common/cancel_upgrade
     4.0K    /common/capture.txt
     4.0K    /common/certs
     2.2M    /common/download
     28K     /common/drf
     288K    /common/elm
     21G     /common/log
     16K     /common/lost+found
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 101 - 120 of 183) :
     4.0K    /common/mnt
     2.6M    /common/moh
     12K     /common/pns
     8.0K    /common/preflight
     4.0K    /common/rpm-archive
     32K     /common/sdlhelper
     104K    /common/spy.py
     20K     /common/stlphelper
     4.0K    /common/sysctl.conf.append
     4.0K    /common/upload
     8.0K    /common/var
     4.0K    /common/whousesme
     ********************************************
     ********************************************
 
     Consider cleaning up space from(but not limited to)
     /var/log/active/cm/tftpdata, /var/log/active/cm/trace,
     /var/log/active/cm/log and /var/log/active/audit folders and retry the
     upgrade.
 
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 121 - 140 of 183) :
1.9  PASS    Network Adapter Type
================================================================================
 
     System Status List
     ---------------------------------------------------------------------------
2.1  Cops Installed (PASS)
     ciscocm.enable-sha512sum-2021-signing-ke
     cmterm-7832-sip.12-8-1-0101-482.k3.cop
     cmterm-8832-sip.12-8-1-0101-482.k3.cop
     cmterm-devicepack11.5.1.23164-1.cop
 
2.2  Version             VMTools Type (WARN)
     9.10.5.49873        native vm-tools
     WARNING: Update the native vmtools to latest for optimal system
     performance.
 
2.3  Status              Service Name (PASS)
     No Issues Found
 
2.4  Upgrade Checks (WARN)
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 141 - 160 of 183) :
 
      *) Smart Licensing requires you to have a Smart Account created and
     configured before you upgrade or migrate the Cisco Unified Communications
     Manager server to 12.0 or higher releases.
 
2.5  Count               Phone Status (PASS)
     0                   Registered
     0                   Unregistered
 
================================================================================
 
Summary:
  Total Test Run : 14
  Total Passed   : 8
  Total Warnings : 3
  Total Failed   : 3
 
Note: Please refer to the readme of Pre Upgrade cop for test details and
      pass/fail/warn/criteria
 
 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 161 - 180 of 183) :
Duration for running tests: 0:02:54
================================================================================
Use "file view install PreUpgradeReport.txt" to view the report
 
end of the file reached
options: q=quit, n=next, p=prev, b=begin, e=end (lines 181 - 183 of 183) :

     -" Standard CTI Secure Connection " present in "dirgroup" table. This must be present in db dirgroup.

can you check the decency record for this role and if possible try to remove it from access group and rerun the preupgrade check

 



Response Signature


Hi Nithin,

First of all running preupgrade cop really helped me to narrow down the issue. Thank you for suggesting that.

I already deleted the Standard CTI Secure Connection from the access control group on CUCM which had no role allocated to it.

SantoshMohite_0-1663482070304.png

This can also be checked from CLI. admin:run sql select * from dirgroup

Everything looked good and I successfully upgraded the CUCM. Thank you for your great help!

 

Regards,

Santosh