UCCX "there was an error while interacting with the database" error

[Agustin Varela]

Hi Everyone, first of all, thanks for any help you can give me.

so, we have 2 UCCX HA Servers WAN deployment version 9.0.2.10000-71. Today we had problems with one HDD on the UCS in one site, and the complete server had to be turned off and on. after that we turned on the virtual machines (uccx pub). the servers started well but now we can not write on the database of any servers, we can not make any change in the configuration and this errors appears: there was an error while interacting with the database.

then I check the services on that server and the Subsystem Manager is in Partial Service. (in the other server its all on IN SERVICE state)

then ill do the dbreplication runtimestate command, and the output is in the picture.

then i ll do som diag commands:

but as you can see there are no major problems with the database.

my action plan is the follow:

1. replace the failure disk in the server and check the array

2. after the array is OK in the UCS server, log into the UCCX pub cli an do "utils dbreplication repair all" command

3. if this does not help, reboot the entire UCCX cluster.

4 if this does not help, ..... i dont know jajajaj

can you please give me som opinions about my action plan, and othes ideas that you can share if this does not work.

thanks in advanced

beste regards

[Anthony Holloway]

Did you look at CCX Serviceability and look at the config datastore? It should say Read Acces=YES, Write Access=YES, Replicate Status=RUNNING for both nodes.

How about looking at replication in CCX Serviceability? It should say State=Connected, Job Status=Active, and at the top the two buttons should say Reset Replication and Disable CDS and HDS.

If it's not, you can try resetting or disabling and enabling replication.

From the help page:

Click Reset Replication to reset the replication if the replication is not functional between the two nodes. The Reset Replication button will be enabled only when the database on both the nodes are enabled. When the subscriber goes down and it is required to make configuration updates from the publisher, you can disable Config Datastore (CDS) and Historical Datastore (HDS) on the subscriber using Disable CDS and HDS icon or button. The database information for the cluster is displayed at the bottom of the window. Once the subscriber is up, you can enable CDS and HDS on the subscriber using the same toggle button. Caution Any configuration in Application Administration and Historical data on the Subscriber node would get over written, when CDS is enabled again.

Alternatively, or in addition to, you can run this command on CLI: utils uccx dbreplication status

[Agustin Varela]

Hi Anthony, thanks for you reply.

i ll check all you said in both servers and all is good.

all four are in the same state:

all that ou said is in the normal state,

thanks

best regards.

[deepakcn]

Hi,

The subsystem Manager is in partial service because the CM telephony is in partial service. This cannot block config changes on UCCX. Also the partial service is displayed for the Slave node. So did the UCCX fail over master ship from node 1 to node 2 or did you get a screenshot from UCCX node 2? Also let us know if the CCX shows partial on both nodes or just one.

 Increase the trace "ADM_CFG" for MADM (CCX Administration) on the CCX serviceability page to xdebuggining 3:

- Now re-create the issue by making the same config change to re-produce the error. (Also let us know what config change are you making on the CCX for eg adding skills,creating CSQ,updating application etc)

- Once done, collect the CCX Administration logs. Open the logs and let me know if you see the below errors or not :

%MADM-ADM_CFG-7-UNK:AppController.writeApp() - Config Exception Occured -  DB_ACCESS_ERROR:Replace; nested exception is: 
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not execute update query

%MADM-ADM_CFG-3-ADM_EXCEPTION:Unknown ADM Exception: Exception=com.cisco.config.ConfigException: DB_ACCESS_ERROR:Replace; nested exception is: 

javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not execute update query


If you see the above events in the logs, then it looks like update query failed due to a DB access error. Let us know if you have any third party servers like wallboard,QM etc querying the CCX DB presently.  In some cases,CCX database service can be restarted to get past this error and in others a complete server reboot is needed.


If you don't see the above events in the logs, then kindly attach the MADM logs here for further analysis. 

Thanks and Regards,

Deepak Nair

[Agustin Varela]

HI Deepakcn, thanks so much for you reply, fortunately the problem was solved.

we change the fail disk and the array rebuilds ok, then i did the stop and reset all the replication  and after 2 hours the tables are in sync and all DB is in 2 - GOOD

thank you very much for your help and replys.

best regards!!