CUCM cannot delete/modify axluser

[GlKlondike]

Hi,

Cisco Unified CM System version: 9.1.2.11900-12

After drop presence sync, i try to modify "axluser" password, and see this error:

Error: [2654] IMS_808

When i try to delete "axluser":

Delete failed. Could not delete a row.

Everything without this, works fine.

This is database error? How repair?

Thanks

[Jorie van Santvoort]

Hi,

This could well be a database problem indeed, to check this you could do the following:

go to CUCM > Cisco Unified Reporting > System Reports > Unified CM Database status > Generate Report.

Check the Unified CM Database Status if the replication state is 2 - Good for all servers.

If not you might have a database problem and you might have to fix the db_replication in CLI. You can follow the steps in this troubleshooting guide. http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200396-Steps-to-Troubleshoot-Database-Replicati.html#anc7

If db status is healthy on all servers, let us know so we can help you further if necessary

Hope this helps.

[GlKlondike]

Thank you for answer

I try to repair and reset dbreplication, but setup not completed

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

DB CLI Status: No other dbreplication CLI is running...

Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2017-07-17-13-59
Last Sync Result: SYNC COMPLETED 603 tables sync'ed out of 603
Sync Errors: 4 Errors or failures were found
Use this command to see the output- file view activelog /cm/trace/dbl/2017_07_17_13_57_17_dbl_repl_cdr_Broadcast.log

DB Version: ccm9_1_2_11900_12
Repltimeout set to: 300s
PROCESS option set to: 1

Cluster Detailed View from callmanager1 (2 Servers):

PING CDR Server REPL. DBver& REPL. REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) RPC? (ID) & STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- -------------- ----- ------- ----- -----------------
callmanager1 10.0.3.11 0.071 Yes (2) Connected 0 match No (2) PUB Setup Completed
callmanager2 10.0.4.11 7.001 Yes (4) Connected 0 match N/A (4) Setup Completed

With current "axluser"  i can connected to cucm Sub, but Pub with same password show login error.

I think this problem in corrupted "axluser" on Pub and maybe something else.

What i can do in this situation?

[Jitender Bhandari]

Hi,

Did you try "run loadxml" and restarted tomcat "utils service restart Cisco Tomcat"?

JB

[GlKlondike]

Thank you for taking time

Yeah, I restart and do reset/repair db replication  full cluster several times.

In dbreplication log, this error:

CUCM have instruments to restore mismatch row in database table?

Table scan for 
ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1123 0 0 0 2
g_callmanager2_ccm9_1_2_11900_12 1123 0 0 2 0

The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733

[Rajan]

It seems like a specific table is corrupted. Can you do the following for repairing this table alone.

utils dbreplication repairtable credentialdynamic all

If you see mismatch for any other table follow the same for those tables.

HTH

Rajan

Pls rate all useful posts

[GlKlondike]

Thank you for answer

After this command, log 

utils dbreplication repairtable tablename [nodename]|all output

To determine if replication is suspect, look for the following:
(1) Number of rows in a table do not match on all nodes.
(2) Non-zero values occur in any of the other output columns for a table

First, a cdr list of the replication status of the servers

SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_callmanager1_ccm9_1_2_11900_12 2 Active Local 0
g_callmanager2_ccm9_1_2_11900_12 4 Active Connected 0 Jun 27 15:23:44

Jul 18 2017 12:53:35 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0

The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 18 2017 12:53:41 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------

command failed -- WARNING: replicate is not in sync (178)

Running a cdr check after repair to insure table is in sync

Jul 18 2017 12:53:42 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------

data mismatch

key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 0
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0

WARNING: replicate is not in sync
Jul 18 2017 12:53:43 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------

command failed -- WARNING: replicate is not in sync (178)

end of the file reached

Full sync not completed, in table i see "callmanager2" have 1 mismatch, how i can fix this? 

Maybe execute "utils dbreplication repairtable credentialdynamic callmanager2"?

[Rajan]

You can try that "utils dbreplication repairtable credentialdynamic callmanager2"

Also can you login to both pub & subscriber web page and see whether you still see the user "axluser" which you tried deleting. If you see this in one server and not in other, then you need to try deleting this stale entry to solve this.

HTH

Rajan

[GlKlondike]

Repairtable did not help

Have idea how CUCM can make fix mismatch on database?

Axluser looks right on PUB and SUB, every changes i see on both.

But if I trying to change password or delete "axluser" i see error.

[Jitender Bhandari]

Hi,

Also looks like some error on DB, try below

(1) Stopped the replication on all the nodes by running the command "utils dbreplication stop" (1st on sub's and then on pub)
(2) Ran the command "utils dbreplication dropadmindb" on all the nodes (1st the pub and then the sub's)
(3) Ran the command "utils dbreplication reset all" on the publisher.

(Rate if it helps)

JB

[Jitender Bhandari]

Hi,

If replication checks out correct try "run loadxml" and restarted tomcat "utils service restart Cisco Tomcat"

(Rate if it helps)

JB