CUIC primary node crashed Agents UCCE reports are not available

[serotonin01]

We had a major server crash losing primary AW and CUIC 10.0 which is legacy.

CUIC data sources are running off Secondary Node, however the UCCE Realtime and Historical value list show off line.

None of the reports including agent, skills etc are available either.  I checked the SQL DB and it appears none of the t_Termination_Call_Detail table is not getting updated either.

In addition because the CUIC Primary node is down we do not have access to the OAMP for CUIC Administration.

I am kind of at a loss. It appears the primary servers are lost and will have to be rebuilt from scratch.

Is there anything I can check to make the Reports run using Secondary Node?

Also is there a way to install or start OAMP plugin on the secondary node?

Any hints and suggestions are appreciated.

[Ramamoorthy Shanmugam]

Hi ,

CUIC OAMP will work only on the Publisher node.

You have to fix the issue or start rebuilding the same IP and Hostname.

For restoration, you need to use the latest /Old CUIC DRS backup otherwise you will lose all the custom reports Definition, Value list and Datasource etc.

If you don’t have the latest backup download all the Custom reports from the Secondary node later we can import them into the Primary node. Don’t try sync from SUB node to PUB you will lose all the cuic custom configurations J

For UCCE AW-HDS A data can be recovered in many ways.

Ram.S

[serotonin01]

Hi Ram.S,

Thank you for you reply. I don't think we have old DRS.  Is there an easy way to backup all the custom reports, definitions, etc from Secondary CUIC node or I have to do one by one? Would they be in the SQL db anywhere?

I cannot make the Secondary node work to populate data in the reports either.

Thank you, Daniel

[Ramamoorthy Shanmugam]

Hi,

There is no bulk Export option. We need to Export one by one for Reports ,Report defination and value list for Version 10.x and before 11.5.x. After Version 11.6.x Report exports will have report definations.

if you have store Procedure reports will be there in DB otherwise we need to follow the above steps.

Ram.S

[serotonin01]

Hi Ram.S

Thank you again for your help.I will start manual export then.

Do you know what could cause Agent Real time reports not run on from Secondary node? When I run Agent real time report it all shows all blank.  I can run Precision Queue reports but nothing related to Agents status etc.

I suspect it's all related to the Value List showing offline, and I cannot get it to work even though Data Source should Connected to Secondary Node.

Thank you, Daniel

[serotonin01]

I am unable to export the reports individually nor any users, etc, from the Secondary CUIC node, the save is failing, asking to retry.

Is there a place on the server I can check and export the data?

Thank you, Daniel.

[bill.king1]

I may not be following you, but you may be hitting this defect. If you log into the Data Sources tab and do Switch Over to get away from your failed node, does it work?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvb20035

[serotonin01]

Hi Bill King,

Thanks for the response. Yes I did have to switch manually but that did not help.

The Data Sources show connected and Online but when checked the Value List corresponding to Agents and other show offline.

None of the reports are updating data.

I am trying couple things. I want to backup CUIC custom reports from Secondary

[serotonin01]

I am also trying to restore AW server with SQL DB on Primary. 

Do I need to restore the awdb and hds on the Primary from the Secondary backup? Or the Primary node will synch with Secondary and copy it to Primary automatically?

Anyone with such experience?

Thanks, Daniel

[bill.king1]

The AW/HDS is independent of CUIC from that perspective. It is also likely that your A side AW/HDS is independent of your secondary AW/HDS.

Once you bring the A side AW/HDS back online, it will get its config from the Central controller, not from the other AW/HDS. It will also get limited historical data from the central controller.

You will need to manually copy the extended historical data for the A side HDS from the other HDS. There are a number of posts about that on the forum, but here's one of them.

https://community.cisco.com/t5/contact-center/hds-a-and-hds-b-can-not-synchronized/td-p/2303002

[serotonin01]

Hi bill.king1

Thanks, It is strange I would expect the B side pick up the data from the Central Controller too.

I am reading the suggested posts now. You may be right there is probably some synchronization issue between Logger B and HDS B.

I can see the AW data is in synch but not the HDS. I can modify agents and other configuration successfully. It is just the HDS data is not updating.

I am rebuilding AW/HDS on side A now. I am trying to be careful about not to lose any data.

Thanks.

[bill.king1]

So you're saying both the HDS A which crashed and the B side HDS which is up, neither have good historical data?

[serotonin01]

Hi bill.king1

Correct neither side have the good historical data. Side B is up but apparently is not syncing with Logger B. HDS B is still not getting updated. I think I may need to get it truncated or something.

When I rebuild Side A it will start with 0.

I do not see any errors either.

Thanks

[serotonin01]

Here is an update. I was able to restore HDS which is in sync and new calls are getting updated in the Termination_Call_Detail table.

However, the secondary CUIC which is the only CUIC up right now still cannot generate Agent Realtime nor Historical reports.

I can generate Call Type, Precision Queue reports, etc but nothing related to Agents.

My Data Source show online and Agent value list successfully shows all agents.

What does CUIC need to generate Agent Reports?

Am I missing something?

[bill.king1]

If you run an agent report, what happens (i.e. error message, no results, etc.)?

When you run the report, copy the SQL from the report(s) in CUIC you ran and try running it on both AW/HDS to verify that the HDS does in fact have the data you're trying to report on.