UCCX replacing a crashed node 8.5 issues

[Geevarghese Cheria]

Introduction

This  document discusses the steps to recover when one of the node crashes in an High Availability Environment.

1. After rebuilding the second node in the cluster, and adding it again. Replication is broken.
2. Restoring uccx Publisher

Problem1

You are having a standby node failure in UCCX cluster. When adding the new node it failed due to the previous node existing on the system. You progressed to delete the old node from CCX and add the new node. This seemed to work but failed the datastore replication. At the moment the second server is standby/partial service and the replication is not availalbe for the second node.

Solution

Follow the steps below that needs to be performed after working hours.it will be better to have a valid backup of uccx, 
before proceeding with the below action plan. 

1.From the CLI of publisher, run the command utils uccx syncusers.

2.From the CLI of subscriber, run the command utils uccx syncusers.

3.Deactivate subscriber config and historical datastores from the Serviceabilty page,if it is already activated.

4.Setup replication from the subscriber CLI by running the command "utils uccx dbreplication reset".

This can be run from the Serviceability -> DB Control Center page as well.

5.Activate subscriber config and historical datastores from the Serviceabilty page.

6.Wait for some time and check the status from the Serviceabilty->DB Control Center page.

7.Enable subscriber datastore components from the subscriber appadmin setup wizard if it not activated successfully.

8.Verify the replication status from the Serviceabilty->DB Control Center page.

After performing the above steps If the replication not working properly collect and check the logs. 
Follow the steps below
1)Run the recovery DVD to repair UCCX Sub secondary Only, to repair file system.
2)During out of working hours run this command on the SUB server only
utils uccx database forcedatasyncAfter this restart the cluster (starting with the primary node, one at a time)and
Reset database replication(before resting the replication make sure that service and up
are running on BOTH nodes, you can do the reset either from the GUI or CLI).

Problem2

Issue with UCCX 8.0(2) SU4 in HA deployment.

• The HRC user could not log in anymore.
  
• The node 1 database was "Dropped/Timeout"
  
• The node 2 database was running, but contain no data.
  
• When tried to reboot the node 1 but it could not be restarted.
• At this time there was not any data logging for incomming calls.

Solution

If possible please take the back up of the UCCX Subscriber before you run a repair on the Publisher Server.

Follow the procedure below to restore latest data to the publisher:

Restoring only the Publisher Node in an HA Setup (with Rebuild)

 In a high availability (HA) setup , if there is a hard-drive failure or any other critical hardware or Software failure which needs rebuild of the
Publisher ( first ) node, then follow the below procedure to recover the publisher node to the last backed up state of the publisher. Run the below
procedure if you have a valid backup taken before the failure of the node.

 Procedure:
 Step 1 Perform a fresh installation of the same version of Cisco Unified Contact Center Express (using the same administrator credentials, network
configuration and security password used earlier) on the node prior to restoring it. For more information on installing Cisco Unified Contact
Center Express, see the Installing Cisco Unified Contact Center Express
available here:http://www.cisco.com/en/US/products/sw/custcosw/ps1846/prod_installation_guides_list.html

Step 2 Navigate to Cisco Unified CCX Administration, select Disaster Recovery System from the Navigation drop-down list box in the upper-right
corner of the Cisco Unified CCX Administration window, and click Go. The Disaster Recovery System Logon window displays.

Step 3 Log in to the Disaster Recovery System with the same Platform Administrator username and password that you use to log in to Cisco Unified
Operating System Administration.

Step 4 Configure the backup device. For more information, see Managing Backup Devices. 

Step 5 Navigate to Restore > Restore Wizard. The Restore Wizard Step 1 window displays.

Step 6 In the Select Backup Device area, choose the backup device from which to restore.

Step 7 Click Next. The Restore Wizard Step 2 window displays.

Step 8 Choose the backup file that you want to restore.

Note: The backup filename indicates the date and time that the system created the backup file.

Step 9 Click Next. The Restore Wizard Step 3 window displays.

Step 10 Select the feature UCCX.

Step 11 Click Next. The Restore Wizard Step 4 window displays,

Step 12 When you get prompted to choose the nodes to restore, choose only the first node (the publisher).

Caution: Do not select the second (subscriber) node in this condition as this will result in failure of the restore attempt.

Step 13 To start restoring the data, click Restore.

Note: During the restore process, do not perform any tasks with Cisco Unified Contact Center Express Administration or User Options.
Restoring the first node may take up to several hours based on the size of database that is being restored. Depending on the size of your database that
you choose to restore, the system can require one hour or more to restore.

Note: Based on the requirements, you have the option to either retrieve the existing publisher node data from the DRS backup to be available on all the
nodes in the cluster or retrieve the more recent data (if available) from the subscriber node to be available in the cluster.

Step 14 Run the following CLI command from the Subscriber node after the restore process is successful (restore status indicates 100 per cent) to
inititate restoring the Publisher node only (with rebuild). utils uccx setuppubrestore

Important Step:

Step 15 Run the following CLI command on the target node; that is if you want to retrieve the publisher node's data, then run this command on the
subscriber node, but if you want to retrieve the subscriber node's data (which is more up-to-date), then run this command on the publisher node.
utils uccx database forcedatasync 

 Warning: In any case, you must execute this command on either of the nodes after restoring the publisher node.

Step 16 Restart both the nodes and run the following CLI command on the Publisher node to set up replication. utils uccx dbreplication reset

For more information on restarting, see the Cisco Unified Communications
Operating System Administration Guide available here:
http://www.cisco.com/en/US/products/sw/custcosw/ps1846/prod_maintenance_guides_list.html.
Caution: If you have done some configuration or hardware changes while performing fresh installation in Step 1 that might impact the License MAC,
then rehost your license again using the license rehosting mechanism before
running the CLI command "utils uccx dbreplication reset".

For more information on the licensing rehosting mechanism, see the
Installing Cisco Unified Contact Center Express available here:
http://www.cisco.com/en/US/products/sw/custcosw/ps1846/prod_installation_guides_list.html

 Step 17 Your data gets restored on the publisher node. To view the status of the restore, see the "Viewing the Restore Status" section on page 19 of
http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact/contact_center/crs/express_8_0/configuration/guide/uccx802drs.pdf

Related Links

Disaster Recovery System Administration Guide for Cisco Unified Contact Center Express Release 8.5(1)

https://supportforums.cisco.com/thread/2175372?tstart=0