06-14-2016 11:47 AM - edited 03-17-2019 07:14 AM
Solved! Go to Solution.
06-17-2016 04:40 AM
I've done some deeper checks of those logs already provided, and looks like one table is corrupted:
2016-06-15 12:46:26 DEBUG - migrating server-specific configuration item (System.Messaging.VMWS.ReverseTrapPilotNumber, 284f4aab-254d-478b-8b83-0f301ee67495) (tbl_configuration:114)
Unfortunately the only way to check which exactly and to fix it would be from root access, and root access is only available for TAC. So you'd need to open Cisco and once they enter the errors I've highlighted in my previous post that should be a quick fix on your system from root that should to the trick.
Leszek
06-14-2016 11:49 AM
Is there any type of upgrade patch that I need to install first? I am fresh out of ideas.
06-15-2016 07:13 AM
Can you get switch version logs as well:
file view install cuc-version-switch.log
Can you check egenral health of the cluster using:
show cuc cluster status
show network cluster
Leszek
06-15-2016 07:23 AM
Hi Leszek,
Here are the results:
admin:file view install cuc-version-switch.log
cuc-version-switch.log: File not found.
admin:show cuc cluster status
ACE_File_Lock::ACE_File_Lock: Permission denied /dev/shm/CCM_GENstatusLock_0
Server Name Member ID Server State Internal State Reason
----------- --------- ------------ -------------- ------
usxhqvuc 0 Primary Pri Active Normal
usxhqvuc2 1 Secondary Sec Active Normal
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_pub 100 Active Local 0
g_ciscounity_sub1 101 Active Connected 1793 Jun 14 19:47:11
SERVERS
Server Peer ID State Status Queue Connection Changed
---------------------------------------------------------------------------
g_ciscounity_pub g_ciscounity_pub 100 Active Local 0
g_ciscounity_sub1 101 Active Connected 1793 Jun 14 19:47:11
g_ciscounity_sub1 g_ciscounity_pub 100 Active Connected 0 Jun 14 19:47:11
g_ciscounity_sub1 101 Active Local 0
STATE
Source ER Capture Network Apply
State State State State
---------------------------------------------------------------------------
g_ciscounity_pub Active Running Running Running
g_ciscounity_sub1 Active Running Running Running
admin:show network cluster
10.10.9.11 usxhqvuc.usxpress.com usxhqvuc Publisher authenticated
10.10.10.27 usxhqvuc2.usxpress.com usxhqvuc2 Subscriber authenticated using TCP since Tue Jun 14 19:46:37 2016
admin:
06-16-2016 01:32 AM
Can you please get couple more outputs:
utils dbreplication runtimestate
file dump install system-history.log
utils ntp status
show status
Additionally, have you tried to perform cluster reboot before the upgrade?
Leszek
06-16-2016 01:35 PM
Yes sir, I did reboot prior to install.
Here is the output of the commands. I have truncated the install log to the interesting part----
admin:utils dbreplication runtimestate
DB and Replication Services: ALL RUNNING
Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2014-10-22-17-50
Last Sync Result: SYNC COMPLETED 541 tables sync'ed out of 541
Sync Errors: NO ERRORS
DB Version: ccm8_6_2_22900_9
Number of replicated tables: 541
Cluster Detailed View from PUB (2 Servers):
PING REPLICATION REPL. DBver& REPL. REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) RPC? STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- ----------- ----- ------- ----- -----------------
usxhqvuc 10.10.9.11 0.051 Yes Connected 0 match Yes (2) PUB Setup Completed
usxhqvuc2 10.10.10.27 2.34 Yes Connected 0 match Yes (2) Setup Completed
admin:utils ntp status
ntpd (pid 22840) is running...
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.10.201.254 10.10.90.9 4 u 551 1024 377 4.947 -11.446 2.342
synchronised to NTP server (10.10.201.254) at stratum 5
time correct to within 114 ms
polling server every 1024 s
Current time in UTC is : Thu Jun 16 20:33:00 UTC 2016
Current time in America/New_York is : Thu Jun 16 16:33:00 EDT 2016
admin:show status
Host Name : usxhqvuc
Date : Thu Jun 16, 2016 16:33:15
Time Zone : Eastern Daylight Time (America/New_York)
Locale : en_US.UTF-8
Product Ver : 8.6.2.22900-9
OS Ver : 5.0.0.0-2
License MAC : 6397293e2882
Uptime:
16:33:16 up 1 day, 3:46, 1 user, load average: 0.25, 0.22, 0.18
CPU Idle: 99.02% System: 00.98% User: 00.00%
IOWAIT: 00.00% IRQ: 00.00% Soft: 00.00% Intr/sec: 609.80
Memory Total: 6097688K
Free: 231492K
Used: 5866196K
Cached: 2430788K
Shared: 0K
Buffers: 184232K
Total Free Used
Disk/active 36660728K 27946548K 8335712K (23%)
Disk/inactive 37252664K 27972880K 8901316K (25%)
Disk/logging 127599608K 63597892K 57415484K (48%)
06/10/2016 09:39:14 | root: Cisco Option Install ciscocm.version3-keys.cop Start
06/10/2016 09:39:51 | root: Cisco Option Install ciscocm.version3-keys.cop Success
06/10/2016 13:34:04 | root: Upgrade 11.5.1.10000-6 Start
06/10/2016 13:34:06 | root: Upgrade (refresh) 11.5.1.10000-6 Start
06/10/2016 13:43:05 | root: Restart 8.6.2.22900-9 Start
06/10/2016 13:45:22 | root: Upgrade (refresh) 11.5.1.10000-6 Start-Install
06/10/2016 10:57:07 | root: Boot 11.5.1.10000-6 Start
06/10/2016 14:34:26 | root: Upgrade (refresh) 11.5.1.10000-6 Failure
06/10/2016 14:34:49 | root: Restart package master
06/10/2016 18:35:42 | root: Boot 8.6.2.22900-9 Start
06/10/2016 17:51:18 | root: Upgrade 11.5.1.10000-6 Start
06/10/2016 17:51:24 | root: Upgrade (refresh) 11.5.1.10000-6 Start
06/10/2016 18:04:00 | root: Restart 8.6.2.22900-9 Start
06/10/2016 18:06:17 | root: Upgrade (refresh) 11.5.1.10000-6 Start-Install
06/10/2016 15:18:47 | root: Boot 11.5.1.10000-6 Start
06/10/2016 18:56:54 | root: Upgrade (refresh) 11.5.1.10000-6 Failure
06/10/2016 18:57:17 | root: Restart package master
06/10/2016 22:58:08 | root: Boot 8.6.2.22900-9 Start
06/11/2016 01:00:06 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
6/11/2016 01:01:26 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/12/2016 01:00:21 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/12/2016 01:01:36 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/13/2016 01:00:21 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/13/2016 01:01:39 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/13/2016 17:15:58 | root: Upgrade 10.5.2.13901-2 Start
06/13/2016 17:16:00 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/13/2016 17:28:30 | root: Restart 8.6.2.22900-9 Start
06/13/2016 17:30:41 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
06/13/2016 14:41:52 | root: Boot 10.5.2.13901-2 Start
06/13/2016 18:19:25 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
06/13/2016 18:19:49 | root: Restart package master
06/13/2016 22:20:42 | root: Boot 8.6.2.22900-9 Start
06/14/2016 01:00:07 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/14/2016 01:01:20 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/14/2016 17:39:27 | root: Restart 8.6.2.22900-9 Start
06/14/2016 17:40:27 | root: Boot 8.6.2.22900-9 Start
06/14/2016 17:56:29 | root: Cisco Option Install ciscocm.version3-keys.cop Start
06/14/2016 17:57:04 | root: Cisco Option Install ciscocm.version3-keys.cop Success
06/14/2016 18:40:55 | root: Upgrade 10.5.2.13901-2 Start
06/14/2016 18:41:04 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/14/2016 18:53:57 | root: Restart 8.6.2.22900-9 Start
06/14/2016 18:56:08 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
06/14/2016 16:07:29 | root: Boot 10.5.2.13901-2 Start
06/14/2016 19:44:37 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
06/14/2016 19:45:00 | root: Restart package master
06/14/2016 23:45:53 | root: Boot 8.6.2.22900-9 Start
06/15/2016 01:00:05 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/15/2016 01:01:26 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
06/15/2016 11:45:22 | root: Upgrade 10.5.2.13901-2 Start
06/15/2016 11:45:29 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/15/2016 11:56:04 | root: Restart 8.6.2.22900-9 Start
06/15/2016 11:58:16 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
06/15/2016 09:09:19 | root: Boot 10.5.2.13901-2 Start
06/15/2016 12:46:32 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
06/15/2016 12:46:56 | root: Restart package master
06/15/2016 16:47:46 | root: Boot 8.6.2.22900-9 Start
06/16/2016 01:00:12 | root: DRS Backup UCMVersion:8.6.2.22900-9 Start
06/16/2016 01:01:35 | root: DRS Backup UCMVersion:8.6.2.22900-9 Success
admin:
06-17-2016 02:00 AM
Looks like we don;t have appropriate logs for the time of the failure:
06/13/2016 17:15:58 | root: Upgrade 10.5.2.13901-2 Start
06/13/2016 17:16:00 | root: Upgrade (refresh) 10.5.2.13901-2 Start
06/13/2016 17:28:30 | root: Restart 8.6.2.22900-9 Start
06/13/2016 17:30:41 | root: Upgrade (refresh) 10.5.2.13901-2 Start-Install
<< This is where provided Install logs finish
06/13/2016 14:41:52 | root: Boot 10.5.2.13901-2 Start
06/13/2016 18:19:25 | root: Upgrade (refresh) 10.5.2.13901-2 Failure
<< We'd need some logs from the time it failed (since this was RU from what i recall they should be somweher on inactive partition)
06/13/2016 18:19:49 | root: Restart package master
06/13/2016 22:20:42 | root: Boot 8.6.2.22900-9 Start
Can you try to get install logs using RTMT frominactive partition:
Leszek
06-17-2016 03:59 AM
06-17-2016 04:22 AM
So here is what we know now, install is failing because some errors in cuc-import as per:
#Install logs
06/15/2016 12:44:30 IPM|Execute shell command "/opt/cisco/connection/lib/install/cuc-import RU Import 10.5.2.13901-2 8.6.2.22900-9 /usr/local/cm/ /common/component/connection /common/log/install/capture.txt "|<LVL::Debug>
06/15/2016 12:46:30 IPM|Internal Error, File:ipm.c:2011, Function: ipmReadNormalizedInputLine(), "/opt/cisco/connection/lib/install/cuc-import RU Import 10.5.2.13901-2 8.6.2.22900-9 /usr/local/cm/ /common/component/connection /common/log/install/capture.txt " failed (1)|<LVL::Critical>
# So then I checked import logs, and it looks like it's failing because of some issues on the database most likely soem duplicate entries in DB from some reason
2016-06-15 12:46:26 ERROR csp_getlasterror() returned code = 1010, message = Duplicate configuration node. Column = [pParentId] (sql:289)
2016-06-15 12:46:26 ERROR csp_getlasterror() returned code = 1010, message = Duplicate configuration node. Column = [pParentId] (sql:289)
2016-06-15 12:46:26 check_last_error(**kwargs)
2016-06-15 12:46:26 File "/opt/cisco/connection/lib/python2.6/site-packages/connection/sql.py", line 290, in check_last_error
2016-06-15 12:46:26 raise informixdb.DatabaseError, informixdb.DatabaseError(str(message), int(code), [])
2016-06-15 12:46:26 raise informixdb.DatabaseError, informixdb.DatabaseError(str(message), int(code), [])
2016-06-15 12:46:26 _informixdb.DatabaseError: SQLCODE 1010 in Duplicate configuration node. Column = [pParentId]:
2016-06-15 12:46:26 ++ echo 'cuc-import failed.'
2016-06-15 12:46:26 cuc-import failed.
Any chance you can get additionally DB installation logs as well, maybe they will have some what table exacly has this error.
Leszek
06-17-2016 04:26 AM
You have been an amazing help and I am thankful to you!
If you will tell me which logs to pull and where to find them, I will be glad to look and upload any that I find.
06-17-2016 04:40 AM
I've done some deeper checks of those logs already provided, and looks like one table is corrupted:
2016-06-15 12:46:26 DEBUG - migrating server-specific configuration item (System.Messaging.VMWS.ReverseTrapPilotNumber, 284f4aab-254d-478b-8b83-0f301ee67495) (tbl_configuration:114)
Unfortunately the only way to check which exactly and to fix it would be from root access, and root access is only available for TAC. So you'd need to open Cisco and once they enter the errors I've highlighted in my previous post that should be a quick fix on your system from root that should to the trick.
Leszek
06-17-2016 04:48 AM
Thank you sir! I will provide this info to our TAC engineer.
06-17-2016 06:51 AM
I'm glad I could give some directions here.
If my input was useful please rate the responses.
Leszek
06-14-2016 04:33 PM
I could see the following in attached install logs:
06/13/2016 17:28:17 upgrade_install.sh|Selinux mode not in enforcing after upgrade because of prior mode|<LVL::Info>
06/13/2016 17:28:17 upgrade_install.sh|No switch to new version|<LVL::Info>
Since this is a refresh upgrade to CUC version 10.5.2.13901-2, change the OS secure mode on both CUC 8.6.2 servers to permissive mode - 'utils os secure permissive'.
Perform the upgrade on Publisher and Subscriber and once the upgrade is complete you may revert it back to enforce mode - 'utils os secure enforce'
06-15-2016 06:12 AM
Thank you for the suggestion Sir, however is was already in permissive mode per the CLI
admin:utils os secure permissive
OS security is already permissive, command cannot be executed
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: