UCCX Sync after switch version failed

[mightyking]

Hello Everyone,

Upgrading UCCX 10.6 to 11.6. The switch version fails with the following error message.

ERROR: Sync after switch version failed
: Intelligence Center Switch Action Switch version has failed due to: Intelligence Center Switch Action failed Please collect the logs via 'file get install /*'. Reboot this UCCX node with the command 'utils system restart'. Issue the command 'utils uccx switch-version db-check'.If database corruption is found issue the command 'utils uccx switch-version db-recover'.Contact Cisco TAC for assistance.
 

I thought I am hitting the bug CSCvf21099 but the workaroung solution did not help.

I am uploading the logs I collected. Has anyone experienced such issue?

Thanks,

MK

[Anthony Holloway]

I just search through some old closed cases and I haven't hit anything like this. Good luck.

[mightyking]

Thank you Anthony,

Do you see anything wrong in the logs?

MK

[Anthony Holloway]

I didn't look, but now that I downloaded them, are you sure these are the install logs? I would have expected a folder called install here, and not jmx or cuic.

EDIT: I mean this looks like something of note:

0000000021: 10.89.0.17: May 13 2020 22:20:18.138 -0400: %CCBU_INFRASTRUCTURE-3-MSG_PROPERTY_SYNTAX_ERROR:
%[exception=Property defined twice in property files.  All property names must be unique or some values
will be overwritten: /opt/cisco/cuic/conf/cuic][reason_string=catalinaOptsGc]: There was a syntax error
in one of the properties files.  ; Resolution requires Cisco TAC to be contacted. 

[mightyking]

Anthony,

I just uploaded the intall logs. Could you please have a look?

Thanks,

MK

[Anthony Holloway · Accepted solution]

You seem to be matching this bug, albeit, the source version doesn't match, however, that's likely not too critical.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf35191

2020-05-12 20:53:37 cuic_l2_upgrade.py|command [su -c 'source /usr/local/cm/db/informix/local/ids.env l2; /usr/local/cm/db/informix/bin/dbaccess - /opt/cisco/cuic/sql/cuic_upgrade_14_to_15.sql' informix] failed with error code [65280]
2020-05-12 20:53:37 cuic_l2_upgrade.py|Exception during execution of [source /usr/local/cm/db/informix/local/ids.env l2; /usr/local/cm/db/informix/bin/dbaccess - /opt/cisco/cuic/sql/cuic_upgrade_14_to_15.sql]

[mightyking]

Thanks Anthony, will open a TAC case.

Regards,

MK

[Anthony Holloway]

I'd like to know how it goes. good luck.

[mightyking]

Hi Anthony,

The TAC engineer believes the issues is because of the hight percetage of IOWAIT but has not been able to identify the cause yet.  I am sharing with you the log file he captured from UCCX Pub.

As you can see below the IOWAIT is 35.37%. The same value in Sub shows 0.0%

admin:show status

Host Name          : XXXXXXXXXXXX
Date               : Wed May 20, 2020 08:55:02
Time Zone          : Eastern Daylight Time (America/Montreal)
Locale             : en_US.UTF-8
Product Ver        : 10.6.1.11002-15
Unified OS Version : 6.0.0.0-2
License MAC        : XXXXXXXXXX

Uptime:
 08:55:04 up 6 days,  9:58,  1 user,  load average: 4.50, 3.51, 3.28

CPU Idle:   36.76%  System:   05.66%    User:   20.82%
  IOWAIT:   35.73%     IRQ:   00.26%    Soft:   00.77%

Memory Total:       16336516K
        Free:        5983608K
        Used:       10352908K
      Cached:        3631348K
      Shared:              0K
     Buffers:         430432K

                        Total            Free            Used
Disk/active         37090880K       23686324K       13027724K (36%)
Disk/inactive       37090816K       23215904K       13498084K (37%)
Disk/logging       150689284K       84741520K       58293200K (41%)

admin:show process load
top - 21:04:22 up 6 days, 22:07,  1 user,  load average: 2.72, 3.11, 3.14
Tasks: 229 total,   1 running, 228 sleeping,   0 stopped,   0 zombie
Cpu(s): 22.7%us,  4.9%sy,  0.0%ni, 43.4%id, 28.2%wa,  0.3%hi,  0.6%si,  0.0%st
Mem:  16336516k total, 10357104k used,  5979412k free,   440508k buffers
Swap:  2064280k total,        0k used,  2064280k free,  3612976k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
  804 informix  10 -10  269m 183m 181m S 51.2  1.1   5893:14 uccxoninit        
 1559 root       1 -19  269m 121m 120m D  9.8  0.8 987:50.21 uccxoninit        
 1604 root       1 -19  269m 120m 119m D  7.9  0.8 924:02.26 uccxoninit        
 1606 root       1 -19  269m 119m 117m D  5.9  0.7 850:06.27 uccxoninit        
 1614 root       1 -19  269m 118m 117m D  5.9  0.7 806:16.17 uccxoninit        
15459 uccxuser  20   0 1032m 748m  30m S  3.9  4.7 157:36.16 UCCX_Engine       
    9 root      20   0     0    0    0 S  2.0  0.0   4:17.64 ksoftirqd/1       
 4851 tomcat    20   0 2010m 902m  31m S  2.0  5.7 348:27.81 tomcat            
14982 ccmservi  20   0  327m  62m  20m S  2.0  0.4 248:45.07 RisDC             
17634 root      20   0 1340m 659m  25m S  2.0  4.1  49:53.57 java              
admin:

[mightyking]

Hi Anthony,

I am receiving a very poor service from Cisco TAC and have decided to rebuild the servers. I am not even sure if the reinstall will help to go through the upgrade but have no other choices at the moment.

This is a 24/7 production environment and here's the strategy I am planing to undertake:

1) Take a DRS backup from HA Cluster

2) Stop the UCCX Engine on the Pub in order to make the Sub Master.

2) Shutdown the Pub and restart with the Installation of the production version.

3) Perform a DRS Restor from the backup taken in the step 1

4) Make sure the DB Replication is 2 within the cluster

Questions:

1) After rebuilding the Pub, do I need to reinstall the licenses or the DRS restore will take care of it?

2) WIll DRS restore affect the call center functionality?  As mentioned, this is a 24/7 call center.

Thanks,

MK

[mightyking]

I finally got a solid TAC engineer who helped to solve this issue. He had to use root access to delete some files and rebooting the servers. The switch version worked right after.

Thanks,

MK

[Anthony Holloway]

Wow! 4 months later!  At least it was resolved.

[mightyking]

LOL!

It was resolved weeks after I opened the TAC case but I had completely forgotten to udpdate the case here.

Thanks,

MK