cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4471
Views
30
Helpful
16
Replies

My Unity Connection upgrade attempts keep failing---8.6 to 11 failed, 8.6 to 10.5 failed

kwajmahall
Level 1
Level 1
Would you guys have any ideas or hints to maybe help me?
We have been trying since Friday morning to upgrade our Cisco Unity Connection Servers.


We use virtual servers and our current version of Cisco Unity Connection is 8.6.2ES44.22900-44

We first installed this COP file: ciscocm.version3-keys.cop.sgn

Then attempted to upgrade using this file:

UCSInstall_UCOS_11.5.1.10000-6.sgn.iso

The
result was that it went through a very long process like it was upgrading but it did not.

We then tried to uprade to an earlier version using this file:

UCSInstall_UCOS_10.5.2.13901-2.sgn.iso

We got the same result, failure to upgrade.

I will attach the most recent install log file. Notice the "No switch to new version" remark.


VMware Installation:
2 vCPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Disk 1: 200GB
6144 Mbytes RAM

admin:show version active
Active Master Version: 8.6.2.22900-9
Active Version Installed Software Options:
ciscocm.version3-keys.cop
admin:show version inactive

No Inactive side available

Executed command unsuccessfully

admin:
1 Accepted Solution

Accepted Solutions

I've done some deeper checks of those logs already provided, and looks like one table is corrupted:

2016-06-15 12:46:26 DEBUG    - migrating server-specific configuration item (System.Messaging.VMWS.ReverseTrapPilotNumber, 284f4aab-254d-478b-8b83-0f301ee67495) (tbl_configuration:114)

Unfortunately the only way to check which exactly and to fix it would be from root access, and root access is only available for TAC. So you'd need to open Cisco and once they enter the errors I've highlighted in my previous post that should be a quick fix on your system from root that should to the trick.

Leszek

View solution in original post

16 Replies 16

kwajmahall
Level 1
Level 1

Is there any type of upgrade patch that I need to install first?  I am fresh out of ideas.

Can you get switch version logs as well:

file view install cuc-version-switch.log

Can you check egenral health of the cluster using:

show cuc cluster status

show network cluster

Leszek

Hi Leszek,

Here are the results:

admin:file view install cuc-version-switch.log
cuc-version-switch.log: File not found.
admin:show cuc cluster status
ACE_File_Lock::ACE_File_Lock: Permission denied /dev/shm/CCM_GENstatusLock_0

Server Name Member ID Server State Internal State Reason
----------- --------- ------------ -------------- ------
usxhqvuc 0 Primary Pri Active Normal
usxhqvuc2 1 Secondary Sec Active Normal

SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_pub 100 Active Local 0
g_ciscounity_sub1 101 Active Connected 1793 Jun 14 19:47:11

SERVERS
Server Peer ID State Status Queue Connection Changed
---------------------------------------------------------------------------
g_ciscounity_pub g_ciscounity_pub 100 Active Local 0
g_ciscounity_sub1 101 Active Connected 1793 Jun 14 19:47:11
g_ciscounity_sub1 g_ciscounity_pub 100 Active Connected 0 Jun 14 19:47:11
g_ciscounity_sub1 101 Active Local 0

STATE
Source ER Capture Network Apply
State State State State
---------------------------------------------------------------------------
g_ciscounity_pub Active Running Running Running
g_ciscounity_sub1 Active Running Running Running

admin:show network cluster
10.10.9.11 usxhqvuc.usxpress.com usxhqvuc Publisher authenticated
10.10.10.27 usxhqvuc2.usxpress.com usxhqvuc2 Subscriber authenticated using TCP since Tue Jun 14 19:46:37 2016
admin:

Can you please get couple more outputs:

utils dbreplication runtimestate
file dump install system-history.log
utils ntp status
show status

Additionally, have you tried to perform cluster reboot before the upgrade?

Leszek

Yes sir, I did reboot prior to install.

Here is the output of the commands.  I have truncated the install log to the interesting part----

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2014-10-22-17-50
Last Sync Result: SYNC COMPLETED 541 tables sync'ed out of 541
Sync Errors: NO ERRORS

DB Version: ccm8_6_2_22900_9
Number of replicated tables: 541

Cluster Detailed View from PUB (2 Servers):

PING REPLICATION REPL. DBver& REPL. REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) RPC? STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- ----------- ----- ------- ----- -----------------
usxhqvuc 10.10.9.11 0.051 Yes Connected 0 match Yes (2) PUB Setup Completed
usxhqvuc2 10.10.10.27 2.34 Yes Connected 0 match Yes (2) Setup Completed

admin:utils ntp status
ntpd (pid 22840) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
*10.10.201.254 10.10.90.9 4 u 551 1024 377 4.947 -11.446 2.342


synchronised to NTP server (10.10.201.254) at stratum 5
time correct to within 114 ms
polling server every 1024 s

Current time in UTC is : Thu Jun 16 20:33:00 UTC 2016
Current time in America/New_York is : Thu Jun 16 16:33:00 EDT 2016
admin:show status

Host Name : usxhqvuc
Date : Thu Jun 16, 2016 16:33:15
Time Zone : Eastern Daylight Time (America/New_York)
Locale : en_US.UTF-8
Product Ver : 8.6.2.22900-9
OS Ver : 5.0.0.0-2
License MAC : 6397293e2882

Uptime:
16:33:16 up 1 day, 3:46, 1 user, load average: 0.25, 0.22, 0.18

CPU Idle: 99.02% System: 00.98% User: 00.00%
IOWAIT: 00.00% IRQ: 00.00% Soft: 00.00% Intr/sec: 609.80

Memory Total: 6097688K
Free: 231492K
Used: 5866196K
Cached: 2430788K
Shared: 0K
Buffers: 184232K

Total Free Used
Disk/active 36660728K 27946548K 8335712K (23%)
Disk/inactive 37252664K 27972880K 8901316K (25%)
Disk/logging 127599608K 63597892K 57415484K (48%)

06/10/2016 09:39:14 | root: Cisco Option Install ciscocm.version3-keys.cop Start
06/10/2016 09:39:51 | root: Cisco Option Install ciscocm.version3-keys.cop Success
06/10/2016 13:34:04 | root: Upgrade 11.5.1.10000-6 Start
06/10/2016 13:34:06 | root: Upgrade (refresh) 11.5.1.10000-6 Start
06/10/2016 13:43:05 | root: Restart 8.6.2.22900-9 Start
06/10/2016 13:45:22 | root: Upgrade (refresh) 11.5.1.10000-6 Start-Install
06/10/2016 10:57:07 | root: Boot 11.5.1.10000-6 Start
06/10/2016 14:34:26 | root: Upgrade (refresh) 11.5.1.10000-6 Failure
06/10/2016 14:34:49 | root: Restart package master
06/10/2016 18:35:42 | root: Boot 8.6.2.22900-9 Start
06/10/2016 17:51:18 | root: Upgrade 11.5.1.10000-6 Start
06/10/2016 17:51:24 | root: Upgrade (refresh) 11.5.1.10000-6 Start
06/10/2016 18:04:00 | root: Restart 8.6.2.22900-9 Start
06/10/2016 18:06:17 | root: Upgrade (refresh) 11.5.1.10000-6 Start-Install
06/10/2016 15:18:47 | root: Boot 11.5.1.10000-6 Start
06/10/2016 18:56:54 | root: Upgrade (refresh) 11.5.1.10000-6 Failure
06/10/2016 18:57:17 | root: Restart package master
06/10/2016 22:58:08 | root: Boot 8.6.2.22900-9 Start
06/11/2016 01:00:06 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
6/11/2016 01:01:26 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/12/2016 01:00:21 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/12/2016 01:01:36 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/13/2016 01:00:21 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/13/2016 01:01:39 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/13/2016 17:15:58 | root: Upgrade 10.5.2.13901-2 Start
06/13/2016 17:16:00 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/13/2016 17:28:30 | root: Restart 8.6.2.22900-9 Start
06/13/2016 17:30:41 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
06/13/2016 14:41:52 | root: Boot 10.5.2.13901-2 Start
06/13/2016 18:19:25 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
06/13/2016 18:19:49 | root: Restart package master
06/13/2016 22:20:42 | root: Boot 8.6.2.22900-9 Start
06/14/2016 01:00:07 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/14/2016 01:01:20 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/14/2016 17:39:27 | root: Restart 8.6.2.22900-9 Start
06/14/2016 17:40:27 | root: Boot 8.6.2.22900-9 Start
06/14/2016 17:56:29 | root: Cisco Option Install ciscocm.version3-keys.cop Start
06/14/2016 17:57:04 | root: Cisco Option Install ciscocm.version3-keys.cop Success
06/14/2016 18:40:55 | root: Upgrade 10.5.2.13901-2 Start
06/14/2016 18:41:04 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/14/2016 18:53:57 | root: Restart 8.6.2.22900-9 Start
06/14/2016 18:56:08 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
06/14/2016 16:07:29 | root: Boot 10.5.2.13901-2 Start
06/14/2016 19:44:37 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
06/14/2016 19:45:00 | root: Restart package master
06/14/2016 23:45:53 | root: Boot 8.6.2.22900-9 Start
06/15/2016 01:00:05 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/15/2016 01:01:26 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/15/2016 11:45:22 | root: Upgrade 10.5.2.13901-2 Start
06/15/2016 11:45:29 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/15/2016 11:56:04 | root: Restart 8.6.2.22900-9 Start
06/15/2016 11:58:16 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
06/15/2016 09:09:19 | root: Boot 10.5.2.13901-2 Start
06/15/2016 12:46:32 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
06/15/2016 12:46:56 | root: Restart package master
06/15/2016 16:47:46 | root: Boot 8.6.2.22900-9 Start
06/16/2016 01:00:12 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/16/2016 01:01:35 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
admin:

Looks like we don;t have appropriate logs for the time of the failure:


06/13/2016 17:15:58 | root: Upgrade 10.5.2.13901-2 Start
06/13/2016 17:16:00 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/13/2016 17:28:30 | root: Restart 8.6.2.22900-9 Start
06/13/2016 17:30:41 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
<< This is where provided Install logs finish

06/13/2016 14:41:52 | root: Boot 10.5.2.13901-2 Start
06/13/2016 18:19:25 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
<< We'd need some logs from the time it failed (since this was RU from what i recall they should be somweher on inactive partition)

06/13/2016 18:19:49 | root: Restart package master
06/13/2016 22:20:42 | root: Boot 8.6.2.22900-9 Start

Can you try to get install logs using RTMT frominactive partition:

Leszek

Yes sir, here are both pub and sub, inactive and active

So here is what we know now, install is failing because some errors in cuc-import as per:

#Install logs

06/15/2016 12:44:30 IPM|Execute shell command "/opt/cisco/connection/lib/install/cuc-import RU Import 10.5.2.13901-2 8.6.2.22900-9 /usr/local/cm/ /common/component/connection /common/log/install/capture.txt "|<LVL::Debug>
06/15/2016 12:46:30 IPM|Internal Error, File:ipm.c:2011, Function: ipmReadNormalizedInputLine(), "/opt/cisco/connection/lib/install/cuc-import RU Import 10.5.2.13901-2 8.6.2.22900-9 /usr/local/cm/ /common/component/connection /common/log/install/capture.txt " failed (1)|<LVL::Critical>

# So then I checked import logs, and it looks like it's failing because of some issues on the database most likely soem duplicate entries in DB from some reason

2016-06-15 12:46:26 ERROR csp_getlasterror() returned code = 1010, message = Duplicate configuration node. Column = [pParentId] (sql:289)
2016-06-15 12:46:26 ERROR csp_getlasterror() returned code = 1010, message = Duplicate configuration node. Column = [pParentId] (sql:289)
2016-06-15 12:46:26 check_last_error(**kwargs)
2016-06-15 12:46:26 File "/opt/cisco/connection/lib/python2.6/site-packages/connection/sql.py", line 290, in check_last_error
2016-06-15 12:46:26 raise informixdb.DatabaseError, informixdb.DatabaseError(str(message), int(code), [])
2016-06-15 12:46:26 raise informixdb.DatabaseError, informixdb.DatabaseError(str(message), int(code), [])
2016-06-15 12:46:26 _informixdb.DatabaseError: SQLCODE 1010 in Duplicate configuration node. Column = [pParentId]:

2016-06-15 12:46:26 ++ echo 'cuc-import failed.'
2016-06-15 12:46:26 cuc-import failed.

Any chance you can get additionally DB installation logs as well, maybe they will have some what table exacly has this error.

Leszek

You have been an amazing help and I am thankful to you!

If you will tell me which logs to pull and where to find them, I will be glad to look and upload any that I find.

I've done some deeper checks of those logs already provided, and looks like one table is corrupted:

2016-06-15 12:46:26 DEBUG    - migrating server-specific configuration item (System.Messaging.VMWS.ReverseTrapPilotNumber, 284f4aab-254d-478b-8b83-0f301ee67495) (tbl_configuration:114)

Unfortunately the only way to check which exactly and to fix it would be from root access, and root access is only available for TAC. So you'd need to open Cisco and once they enter the errors I've highlighted in my previous post that should be a quick fix on your system from root that should to the trick.

Leszek

Thank you sir!  I will provide this info to our TAC engineer.

I'm glad I could give some directions here.

If my input was useful please rate the responses.

Leszek

Shashank Mahajan
Cisco Employee
Cisco Employee

I could see the following in attached install logs:

06/13/2016 17:28:17 upgrade_install.sh|Selinux mode not in enforcing after upgrade because of prior mode|<LVL::Info>
06/13/2016 17:28:17 upgrade_install.sh|No switch to new version|<LVL::Info>

Since this is a refresh upgrade to CUC version 10.5.2.13901-2, change the OS secure mode on both CUC 8.6.2 servers to permissive mode - 'utils os secure permissive'.

Perform the upgrade on Publisher and Subscriber and once the upgrade is complete you may revert it back to enforce mode - 'utils os secure enforce'

Thank you for the suggestion Sir, however is was already in permissive mode per the CLI

admin:utils os secure permissive
OS security is already permissive, command cannot be executed