09-16-2022 10:26 PM - edited 09-16-2022 10:31 PM
Hi,
I am trying to upgrade CUCM from 11.5 SU3 to 11.5 SU11 but it's failing with an unknown error. DBreplication is OK.
File UCSInstall_UCOS_UNRST_11.5.1.23900-30.sha512.iso
Start Time Thu Sep 15 21:56:49 SGT 2022
Status Error encountered: An unknown error occurred while accessing the upgrade file. Warning: A system reboot is required when the upgrade process completes or is canceled. This will ensure services affected by the upgrade process are functioning properly.
Action Download the file from the link provided and reboot the system.
Log File Install Log
Advanced Error Messages Hide
/usr/local/cm/script/cm-dbl-install L2 PostInstall 11.5.1.23900-30 11.5.1.13900-52 /usr/local/cm/ /partB/usr/local/cm/ /common/log/install/capture.txt failed (1)
failed to upgrade infrastructure_post components
/usr/local/bin/base_scripts/appmanager.sh -upgrade -basepath /common/rpm-archive/11.5.1.23900-30 -logfile /common/log/install/install_log_2022-09-15.21.11.50.log -resultfile /common/log/install/upgrade-results.xml failed (1)
============================================
admin:utils dbreplication runtimestate
Server Time: Sat Sep 17 13:02:35 +08 2022
Cluster Replication State: Replication status command started at: 2022-09-17-13-00
Replication status command ENDED. Checked 706 tables out of 706
Last Completed Table: devicenumplanmapremdestmap
No Errors or Mismatches found.
Use 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2022_09_17_13_00_29.out' to see the details
DB Version: ccm11_5_1_13900_52
Repltimeout set to: 300s
PROCESS option set to: 1
Cluster Detailed View from SGPCUCMPUB (2 Servers):
PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
ABC X.244.225.241 0.012 Y/Y/Y 0 (g_2) (2) Setup Completed
XYZ X 0.070 Y/Y/Y 0 (g_3) (2) Setup Completed
I already attempted the upgrade couple of times with cluster reboot but no luck. Any help would be appreciated.
Thanks,
Santosh
Solved! Go to Solution.
09-17-2022 10:49 PM - edited 09-17-2022 10:49 PM
-" Standard CTI Secure Connection " present in "dirgroup" table. This must be present in db dirgroup.
can you check the decency record for this role and if possible try to remove it from access group and rerun the preupgrade check
09-17-2022 06:49 AM
Before the upgrade, download and install/run the latest version of preUpgradeCheck COP file. Look at PASS / WARNING / FAIL output. Resolve all FAIL and maybe the WARNINGs. Repeat until happy.
And its mandatory you read the below
- Release note of upgrade file
- Install upgrade Guide
09-17-2022 08:08 AM
Hi Nithin,
Can you please provide the link to preUpgradeCheck cop file? I couldn't find the file.
Thanks
09-17-2022 08:47 AM
https://software.cisco.com/download/home/286322286/type/286319173/release/COP-Files
As the above file is sha 512 signed, you need to install the below cop first. you can download both files from above link.
ciscocm.enable-sha512sum-2021-signing-key-v1.0.cop.sgn
09-17-2022 10:14 AM
Hi Nithin,
There are 3 failures. I am not sure how to fix dataBase sanity.
1) NTP: I added another free NTP server 162.159.200.1 (in-country) as our internal NTP seems to be taking a longer time to sync
admin:utils ntp status
ntpd (pid 19380) is running...
remote refid st t when poll reach delay offset jitter
==============================================================================
+X.X.X.X 162.159.200.1 4 u 7 64 377 232.796 -0.997 6.054
*162.159.200.1 10.23.8.4 3 u 67 64 377 34.207 2.694 1.132
synchronised to NTP server (162.159.200.1) at stratum 4
time correct to within 104 ms
polling server every 64 s
Current time in UTC is : Sat Sep 17 17:12:39 UTC 2022
Current time in Asia/Singapore is : Sun Sep 18 01:12:39 +08 2022
2) Test dataBase Sanity:
-" Standard CTI Secure Connection " present in "dirgroup" table.
It is recommended to fix the entries before upgrade
3)File system checks: It seems to be ok for 11.5 versions upgrade
admin:file view install PreUpgradeReport.txt |
Use "file view install PreUpgradeReport.txt" to view the report |
================================================================================ |
Pre Upgrade Test Date: 09/17/2022 16:39:23 UTC |
================================================================================ |
Active Version: 11.5.1.13900-52 |
Server: CUCMPUB , CUCM Publisher |
================================================================================ |
Result Test |
------ ------------------------------------------------------------------- |
1.1 PASS Common Security Password Length |
System not in FIPS mode, Common Security Password's Minimum length |
requirement not enforced |
1.2 FAIL Network status (NTP, DNS & Cluster node connectivity) |
ntp_stratum : Failed |
The reference NTP server is a stratum 5 clock. |
NTP servers with stratum 5 or worse clocks are deemed unreliable. |
Please consider using an NTP server with better stratum level. |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 183) : |
Please use OS Admin GUI to add/delete NTP servers. |
1.3 PASS Deprecated Phone Models |
1.4 PASS Cluster Database Status |
1.5 PASS PLM License Status |
1.6 FAIL Test dataBase Sanity |
-" Standard CTI Secure Connection " present in "dirgroup" table. |
It is recommended to fix the entries before upgrade |
1.7 WARN DRS backup status |
Last Successful backup taken on: Wed Sep 14 01:29:36 SGT 2022 |
1.8 FAIL Filesystem Checks |
*) Upgrade of the server to some or all of the following version(s) will |
fail - |
11.5 |
PASS: You have 27558012093 bytes of required 25417837269 bytes common |
partition space required for L2 upgrade to 11.5. |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 183) : |
12 |
PASS: You have 27557999805 bytes of required 25168436072 bytes common |
partition space required for L2 upgrade to 12.0. |
12.5 |
Failed: You only have 27557967037 bytes of required 29050540694 bytes |
common partition space required for RU upgrade to 12.5. You do not need to |
cleanup inactivelog. /common/log/inactive space will be automatically |
reclaimed. Cleanup 1492573657 bytes of space from /common/log/active, |
/common to free up the required space. |
14 |
Failed: You only have 27557700797 bytes of required 29058810518 bytes |
common partition space required for RU upgrade to 14. You do not need to |
cleanup inactivelog. /common/log/inactive space will be automatically |
reclaimed. Cleanup 1501109721 bytes of space from /common/log/active, |
/common to free up the required space. |
Before starting the upgrade, clear space via the following methods and |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 60 of 183) : |
reinstall this COP: |
- Install latest version of ciscocm.free_common_space COP file from |
cisco.com (this will delete trace/log files, current inactive version, and |
backed up COP files). |
- For source 11.0+, use OS Admin GUI's Device Load Management to Remove |
Unused Firmware from the System. |
- Delete any unneeded core files with "file delete" CLI. |
- For source 10.x and higher, increase size of your vdisk via power down |
VM, edit VM specs, powerup. |
Details of directories occupying most space in /common partition are as |
below... |
******************************************** |
Used /var/log/active(/common/log/taos-log-a/b) directory space: 20G |
Detailed view: |
1.2G /var/log/active/audit |
9.1G /var/log/active/car_db |
9.8M /var/log/active/ccm_db |
6.1G /var/log/active/cm |
99M /var/log/active/core |
538M /var/log/active/dp_db |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 61 - 80 of 183) : |
45M /var/log/active/elm |
60M /var/log/active/elm_db |
20K /var/log/active/installed_options |
4.0K /var/log/active/mgetty |
18M /var/log/active/mohprep |
4.0K /var/log/active/patches |
552M /var/log/active/platform |
0 /var/log/active/sa |
8.0K /var/log/active/sso |
95M /var/log/active/syslog |
1.5G /var/log/active/tomcat |
112K /var/log/active/uccx |
******************************************** |
Used /var/log/active/cm directory space: 6.1G |
Detailed view: |
895M /var/log/active/cm/bin |
4.0K /var/log/active/cm/CCMEncryptionMigration.log |
0 /var/log/active/cm/cdr |
0 /var/log/active/cm/cdr_repository |
180M /var/log/active/cm/log |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 81 - 100 of 183) : |
139M /var/log/active/cm/phoneguide_locales |
9.8M /var/log/active/cm/report |
1.6G /var/log/active/cm/tftpdata |
3.3G /var/log/active/cm/trace |
******************************************** |
Used /var/log/inactive((/common/log/taos-log-a/b) directory space: 895M |
******************************************** |
Used /common directory space: 21G |
Detailed view: |
36K /common/adminsftp |
4.0K /common/backup-sysctl.conf.append |
3.6M /common/bps |
4.0K /common/cancel_upgrade |
4.0K /common/capture.txt |
4.0K /common/certs |
2.2M /common/download |
28K /common/drf |
288K /common/elm |
21G /common/log |
16K /common/lost+found |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 101 - 120 of 183) : |
4.0K /common/mnt |
2.6M /common/moh |
12K /common/pns |
8.0K /common/preflight |
4.0K /common/rpm-archive |
32K /common/sdlhelper |
104K /common/spy.py |
20K /common/stlphelper |
4.0K /common/sysctl.conf.append |
4.0K /common/upload |
8.0K /common/var |
4.0K /common/whousesme |
******************************************** |
******************************************** |
Consider cleaning up space from(but not limited to) |
/var/log/active/cm/tftpdata, /var/log/active/cm/trace, |
/var/log/active/cm/log and /var/log/active/audit folders and retry the |
upgrade. |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 121 - 140 of 183) : |
1.9 PASS Network Adapter Type |
================================================================================ |
System Status List |
--------------------------------------------------------------------------- |
2.1 Cops Installed (PASS) |
ciscocm.enable-sha512sum-2021-signing-ke |
cmterm-7832-sip.12-8-1-0101-482.k3.cop |
cmterm-8832-sip.12-8-1-0101-482.k3.cop |
cmterm-devicepack11.5.1.23164-1.cop |
2.2 Version VMTools Type (WARN) |
9.10.5.49873 native vm-tools |
WARNING: Update the native vmtools to latest for optimal system |
performance. |
2.3 Status Service Name (PASS) |
No Issues Found |
2.4 Upgrade Checks (WARN) |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 141 - 160 of 183) : |
*) Smart Licensing requires you to have a Smart Account created and |
configured before you upgrade or migrate the Cisco Unified Communications |
Manager server to 12.0 or higher releases. |
2.5 Count Phone Status (PASS) |
0 Registered |
0 Unregistered |
================================================================================ |
Summary: |
Total Test Run : 14 |
Total Passed : 8 |
Total Warnings : 3 |
Total Failed : 3 |
Note: Please refer to the readme of Pre Upgrade cop for test details and |
pass/fail/warn/criteria |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 161 - 180 of 183) : |
Duration for running tests: 0:02:54 |
================================================================================ |
Use "file view install PreUpgradeReport.txt" to view the report |
end of the file reached |
options: q=quit, n=next, p=prev, b=begin, e=end (lines 181 - 183 of 183) : |
09-17-2022 10:49 PM - edited 09-17-2022 10:49 PM
-" Standard CTI Secure Connection " present in "dirgroup" table. This must be present in db dirgroup.
can you check the decency record for this role and if possible try to remove it from access group and rerun the preupgrade check
09-17-2022 11:24 PM
Hi Nithin,
First of all running preupgrade cop really helped me to narrow down the issue. Thank you for suggesting that.
I already deleted the Standard CTI Secure Connection from the access control group on CUCM which had no role allocated to it.
This can also be checked from CLI. admin:run sql select * from dirgroup
Everything looked good and I successfully upgraded the CUCM. Thank you for your great help!
Regards,
Santosh
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide