05-14-2020 06:05 PM - edited 05-14-2020 06:09 PM
Hello Everyone,
Upgrading UCCX 10.6 to 11.6. The switch version fails with the following error message.
ERROR: Sync after switch version failed
: Intelligence Center Switch Action Switch version has failed due to: Intelligence Center Switch Action failed Please collect the logs via 'file get install /*'. Reboot this UCCX node with the command 'utils system restart'. Issue the command 'utils uccx switch-version db-check'.If database corruption is found issue the command 'utils uccx switch-version db-recover'.Contact Cisco TAC for assistance.
I thought I am hitting the bug CSCvf21099 but the workaroung solution did not help.
I am uploading the logs I collected. Has anyone experienced such issue?
Thanks,
MK
Solved! Go to Solution.
05-14-2020 07:48 PM
You seem to be matching this bug, albeit, the source version doesn't match, however, that's likely not too critical.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf35191
2020-05-12 20:53:37 cuic_l2_upgrade.py|command [su -c 'source /usr/local/cm/db/informix/local/ids.env l2; /usr/local/cm/db/informix/bin/dbaccess - /opt/cisco/cuic/sql/cuic_upgrade_14_to_15.sql' informix] failed with error code [65280] 2020-05-12 20:53:37 cuic_l2_upgrade.py|Exception during execution of [source /usr/local/cm/db/informix/local/ids.env l2; /usr/local/cm/db/informix/bin/dbaccess - /opt/cisco/cuic/sql/cuic_upgrade_14_to_15.sql]
05-14-2020 06:44 PM
05-14-2020 06:47 PM
Thank you Anthony,
Do you see anything wrong in the logs?
MK
05-14-2020 06:58 PM - edited 05-14-2020 07:00 PM
I didn't look, but now that I downloaded them, are you sure these are the install logs? I would have expected a folder called install here, and not jmx or cuic.
EDIT: I mean this looks like something of note:
0000000021: 10.89.0.17: May 13 2020 22:20:18.138 -0400: %CCBU_INFRASTRUCTURE-3-MSG_PROPERTY_SYNTAX_ERROR:
%[exception=Property defined twice in property files. All property names must be unique or some values
will be overwritten: /opt/cisco/cuic/conf/cuic][reason_string=catalinaOptsGc]: There was a syntax error
in one of the properties files. ; Resolution requires Cisco TAC to be contacted.
05-14-2020 07:32 PM
05-14-2020 07:48 PM
You seem to be matching this bug, albeit, the source version doesn't match, however, that's likely not too critical.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf35191
2020-05-12 20:53:37 cuic_l2_upgrade.py|command [su -c 'source /usr/local/cm/db/informix/local/ids.env l2; /usr/local/cm/db/informix/bin/dbaccess - /opt/cisco/cuic/sql/cuic_upgrade_14_to_15.sql' informix] failed with error code [65280] 2020-05-12 20:53:37 cuic_l2_upgrade.py|Exception during execution of [source /usr/local/cm/db/informix/local/ids.env l2; /usr/local/cm/db/informix/bin/dbaccess - /opt/cisco/cuic/sql/cuic_upgrade_14_to_15.sql]
05-14-2020 08:02 PM
Thanks Anthony, will open a TAC case.
Regards,
MK
05-14-2020 08:09 PM
05-20-2020 06:08 PM
Hi Anthony,
The TAC engineer believes the issues is because of the hight percetage of IOWAIT but has not been able to identify the cause yet. I am sharing with you the log file he captured from UCCX Pub.
As you can see below the IOWAIT is 35.37%. The same value in Sub shows 0.0%
admin:show status
Host Name : XXXXXXXXXXXX
Date : Wed May 20, 2020 08:55:02
Time Zone : Eastern Daylight Time (America/Montreal)
Locale : en_US.UTF-8
Product Ver : 10.6.1.11002-15
Unified OS Version : 6.0.0.0-2
License MAC : XXXXXXXXXX
Uptime:
08:55:04 up 6 days, 9:58, 1 user, load average: 4.50, 3.51, 3.28
CPU Idle: 36.76% System: 05.66% User: 20.82%
IOWAIT: 35.73% IRQ: 00.26% Soft: 00.77%
Memory Total: 16336516K
Free: 5983608K
Used: 10352908K
Cached: 3631348K
Shared: 0K
Buffers: 430432K
Total Free Used
Disk/active 37090880K 23686324K 13027724K (36%)
Disk/inactive 37090816K 23215904K 13498084K (37%)
Disk/logging 150689284K 84741520K 58293200K (41%)
admin:show process load
top - 21:04:22 up 6 days, 22:07, 1 user, load average: 2.72, 3.11, 3.14
Tasks: 229 total, 1 running, 228 sleeping, 0 stopped, 0 zombie
Cpu(s): 22.7%us, 4.9%sy, 0.0%ni, 43.4%id, 28.2%wa, 0.3%hi, 0.6%si, 0.0%st
Mem: 16336516k total, 10357104k used, 5979412k free, 440508k buffers
Swap: 2064280k total, 0k used, 2064280k free, 3612976k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
804 informix 10 -10 269m 183m 181m S 51.2 1.1 5893:14 uccxoninit
1559 root 1 -19 269m 121m 120m D 9.8 0.8 987:50.21 uccxoninit
1604 root 1 -19 269m 120m 119m D 7.9 0.8 924:02.26 uccxoninit
1606 root 1 -19 269m 119m 117m D 5.9 0.7 850:06.27 uccxoninit
1614 root 1 -19 269m 118m 117m D 5.9 0.7 806:16.17 uccxoninit
15459 uccxuser 20 0 1032m 748m 30m S 3.9 4.7 157:36.16 UCCX_Engine
9 root 20 0 0 0 0 S 2.0 0.0 4:17.64 ksoftirqd/1
4851 tomcat 20 0 2010m 902m 31m S 2.0 5.7 348:27.81 tomcat
14982 ccmservi 20 0 327m 62m 20m S 2.0 0.4 248:45.07 RisDC
17634 root 20 0 1340m 659m 25m S 2.0 4.1 49:53.57 java
admin:
05-26-2020 11:11 AM
Hi Anthony,
I am receiving a very poor service from Cisco TAC and have decided to rebuild the servers. I am not even sure if the reinstall will help to go through the upgrade but have no other choices at the moment.
This is a 24/7 production environment and here's the strategy I am planing to undertake:
1) Take a DRS backup from HA Cluster
2) Stop the UCCX Engine on the Pub in order to make the Sub Master.
2) Shutdown the Pub and restart with the Installation of the production version.
3) Perform a DRS Restor from the backup taken in the step 1
4) Make sure the DB Replication is 2 within the cluster
Questions:
1) After rebuilding the Pub, do I need to reinstall the licenses or the DRS restore will take care of it?
2) WIll DRS restore affect the call center functionality? As mentioned, this is a 24/7 call center.
Thanks,
MK
09-30-2020 08:47 PM
I finally got a solid TAC engineer who helped to solve this issue. He had to use root access to delete some files and rebooting the servers. The switch version worked right after.
Thanks,
MK
09-30-2020 11:11 PM
Wow! 4 months later! At least it was resolved.
10-01-2020 05:49 AM
LOL!
It was resolved weeks after I opened the TAC case but I had completely forgotten to udpdate the case here.
Thanks,
MK
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide