Ask the Expert: Troubleshooting Unified Contact Center Enterprise

[ciscomoderator]

            With Goran Selthofer

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about integrating Unified Contact Center Enterprise into your environment and troubleshooting the many features that are available with the Unified Contact Center Enterprise solution.

Cisco Unified Contact Center Enterprise delivers intelligent contact routing, call treatment, network-to-desktop computer telephony integration (CTI), and multichannel contact management over an IP infrastructure. It combines multichannel automatic call distributor (ACD) functionality with IP telephony in a unified solution. This makes it easier for your company to rapidly deploy a distributed contact center infrastructure.

Goran Selthofer is a team lead for the Cisco TAC EMEAR Contact Center team based in Brussels. He has supported UCCE, UCCX, CVP, and UCCE applications for the past seven years within the Cisco TAC. He has more than 13 years of overall experience in the industry, with broad experience in Cisco Unified Communications infrastructure solutions as he has been also working for Cisco Gold Partner prior to joining Cisco TAC. Goran also provides internal training to TAC engineers on Contact Center topics. He graduated with a master's degree at the Technical Military Academy - Belgrade University. He also holds CCIE certification (number 27211) in voice as well as VMware Certified Professional certifications. 

Remember to use the rating system to let Goran know if you have received an adequate response. 

Goran might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation in Collaboration, Voice and Video community,  sub-community, Contact Center discussion forum shortly after the event. This event lasts through February 14, 2014. Visit this forum often to view responses to your questions and the questions of other community members.

[Chitrangad Pathak]

Goran,

Thank you for initiating this discussion. Want some help on advanced troubleshooting like how to read the logs once they are collected. Is there any tool available for the partners to do that?

Also When working with Agent State Trace, what exactly happens at the logging level

Regards

Chitrangad Pathak

[Goran Selthofer]

Hello Chitrangad!

Nice to see you here, thanks for posting the question!

Well, honestly, there is no 'tool' which is used by TAC to read UCCE traces. Far from that . We use text editors with coloring schemes when reading logs and that is a long manual process.

One exception is a basic Call Flow tool which is distributed with CVP software. That one is promissing but currently it is still not widely used as it requires pre-created log templates and currently there are only few. However, it works very well for CVP SIP tracing.

However, back to UCCE logs, we are not using any special tools and that is why TAC would generally ask you to provide as much info as possible (like ANI, time stamps, Agent ID, Extension ...etc) in order to analyze logs as it is not enough just to send logs to TAC. So, unfortunatelly there is no magic buttons or crystal balls (YET! ) which TAC or partners and customers can use when reading logs.

With that being said, foundation for reading logs is to really understand processes and tasks, to know the exact call flow and expected behavior and to gather as much info as possible about BAD but also GOOD examples.

Now, related to the second part of your question, with intention to make it more actual by introducing Finesse in the same story and also as I have promised David from above post, I invite you to read great example written by my good friend and colleague Linda, who has created the following:

Finesse Agent Login Trace with the Use of Logs

http://www.cisco.com/en/US/products/ps11324/products_tech_note09186a0080c14a55.shtml

I think this should give you a pretty good overview of what is happening there...

Otherwise, there is also another example from her which is yet to be published and it is about:

How to Identify CTI Server errors in Finesse Logs

Introduction

This document will show you how to quickly identify CTI Server peripheral errors in the Finesse Logs.

Prerequisites

Requirements

Cisco recommends that you have knowledge of Cisco Finesse,  Voice Operating System (VOS) CLI command prompt and UCCE CTI Server messages.

Components Used

The information in this document is based on Cisco Finesse Version 9.1(1).

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Finesse Error-Desktop-webservices Log File

First locate any CTI server errors log in the Error-Desktop-webservices log.  This log will help you identify if the Finesse Server is receiving error messages from CTI Server.  In this log we see the following message at 14:43:50.838.

0000063104: 10.10.10.38: Jun 19 2013 14:43:50.838 -0400: %CCBU_pool-6-thread-57-3-CMD_FAILED: %[ENTITY_ID=2005][ERROR_DESCRIPTION=errorCode 70][command_name=LOGIN]: Received failed command response

The error message indicates ENTITY_ID=2005 or agentID 2005 encountered an error when trying to login.

Finesse Desktop-webservices Log File

Open the Desktop-webservices log for the same timestamp that the error was found in the Error-Desktop-webservices log.

Search the log file to locate the error message.  Search on the timestamp of the error (14:43:50) or the failure code, i.e.failureCode=70 to locate the error message.   The webservices log will provide the "peripheralErrorCode" in this example the peripheral error code received from the backend cti server is 12005.

0000063103: 10.10.10.38: Jun 19 2013 14:43:50.836 -0400: %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=CTIControlFailureConf [failureCode=70, peripheralErrorCode=12005, text=null]CTIMessageBean [invokeID=19976, msgID=35, timeTracker={"id":"ControlFailureConf","CTI_MSG_NOTIFIED":1371667430836,"CTI_MSG_RECEIVED":1371667430836}, msgName=ControlFailureConf, deploymentType=CCE]][cti_response_time=0]: Decoded Message to Finesse from backend cti server

CTI Server Log

Open the CTI Server log for the same timestamp that the error was found in the Error-Desktop-webservices log and locate the peripheral error code 12005 with approximately the same time stamp.  In this example search on 14:43.

14:43:48:898 cg1A-ctisvr SESSION 5: MsgType:SET_AGENT_STATE_REQ (InvokeID:0x4e08 PeripheralID:5001 AgentState:LOGIN

14:43:48:898 cg1A-ctisvr SESSION 5:         AgentWorkMode:AWM_UNSPECIFIED NumSkillGroups:0 EventReasonCode:50004 ForcedFlag:1

14:43:48:898 cg1A-ctisvr SESSION 5:         AgentServiceReq:0 AgentInstrument:"2005" AgentID:"2005" )

14:43:48:898 cg1A-ctisvr Trace: ProcessSetAgentStateRequest - sessionID 5

14:43:48:898 cg1A-ctisvr Trace: *** AddToAssociateAgentList();           ADDED: SessionID=5 AgentID=2005 PeripheralID=5001

14:43:48:898 cg1A-ctisvr Trace: CSTASetAgentState: InvokeID=0x2f0b1596 Dev=2005 AgentMode=LOG_IN AGID=2005 SG=-1(0xffffffff))

14:43:48:898 cg1A-ctisvr Trace: PrivateData: EventReasonCode=50004 WorkMode=0 NumAdditionalGroups=0 PositionID= SupervisorID= ClientAddress=

14:43:48:900 cg1A-ctisvr Trace:

14:43:48:900 cg1A-ctisvr Trace: CSTAUniversalFailureConfEvent: InvokeID=0x2f0b1596 Error=GENERIC_UNSPECIFIED_REJECTION

14:43:48:900 cg1A-ctisvr Trace: PRIVATE_DATA: PeripheralErrorCode=0x2ee5(12005)

14:43:48:900 cg1A-ctisvr SESSION 5: MsgType:CONTROL_FAILURE_CONF (InvokeID:0x4e08 FailureCode:CF_GENERIC_UNSPECIFIED_REJECTION

14:43:48:900 cg1A-ctisvr SESSION 5:         PeripheralErrorCode:12005 )

14:44:06:483 cg1A-ctisvr Trace:

The CTI server log will show that CTI Server received a SET_AGENT_STATE_REQ for agent 2005 using AgentInstrument 2005. This message was sent to CTI server from Session 5 or the Finesse Server.  CTI server responded to the request with a PeripheralErrorCode:12005.

To determine which session your Finesse server you can use procmon.  Using procmon we can verify that Session 5 is 10.10.10.38 with a client ID of Finesse.  Where 10.10.10.38 is the IP address of our primary Finesse Server.

C:\icm>procmon ucce cg1a ctisvr

>>>>clients

Session   Time Ver Flags   ClientID         AgentID AgentExt Signature

   Host

       5 67:44:42 16   AUX R Finesse     Finesse  (10.10.10.38:32990)

       7 66:53:16 16   AUX R Finesse     Finesse   (10.10.10.138:32995)

>>>>

Note: Detailed instructions on how to use Procmon can be found here.

Peripheral Error Code Descriptions

A quick way to obtain the description for UCCE Peripheral Error Codes is to log onto a UCCE system is to open a command prompt and navigate to C:\icm\bin directory and "checke where error code is the peripheral error code that you have identified.  In this example we would use c:\icm\bin>checke 12005

C:\icm\bin>checke 12005

Error 12005

   Symbol = PERERR_JTAPPLAY_ADDCALLOBSERVERFAILURE

   Level 1 = Login could not be performed - Possible causes are Invalid Instrument; Media Termination Problem o

r other CM issue

   Level 2 = AddCallObserver failed - Please see PIM log for more details

   Level 3 =

C:\icm\bin>

This command will provide a description of the error code and potential causes for the error.  In our example agent 2005 attempted to log-in with an Invalid Instrument.

Thanks,

Goran

[Jackson Braddock]

Goran, thanks for the great detailed answer!  One more question for you. If my CIM (Cisco Interaction Manager) is integrated with UCCE, what is the best point to start activity routing issues? Thanks again!

Jackson

[Goran Selthofer]

Hi Jackson,

You are more than welcome! Feel free to use rating system below each answer so that others can benefit from useful answers...

Now, my favorite topic - "routing issues in CIM-ICM environment" - thanks for asking!!!

OK, this one is very simple, let me explain:

- CIM can work standalone but also can work integrated with ICM

- in case CIM is a standalone, CIM will do all routing decisions

- in case CIM is integrated with ICM, then CIM will extended routing decisions to ICM. That is why the service from CIM side which makes this possible is called EAAS - EXTERNAL Agent Assignment Service.

- So, EAAS talks with PIM on MRPG side of ICM. Therefore, this makes life much easier as they are using MRI (MR Interface) hence they will have some standardized behavior.

- Now, exactly that PIM on your MRPG is THE Border Line to start troubleshooting routing issues.

- AND it is VERY SIMPLE - here is how you do it:

* First enable MR tracing on that MR PIM. Let's take example that ICM instance name is ACME and MRPG node is PG2A where PIM1 is mrpim talking to CIM.

So, open cmd line on MRPG box, procmon to that pim and enable all MR tracing:

> procmon acme pg2a pim1

>>>>ltrace (this will list traces currently enabled)

>>>>trace *.* /off (first you want to disable everything which is enabled currently to avoid noise in logs)

>>>>trace mr* /on (this enables ALL MR traces)

>>>>ltrace
...
...
mr_msg_config                         1       On
mr_msg_comm_session            2       On
mr_heartbeat_messages            3       On
mr_msg_incoming_mr                4       On
mr_msg_outgoing_mr                 5       On
mr_msg_incoming_inrc               6       On
mr_msg_outgoing_inrc                7       On
mr_msg_outgoing_csta               8       On
mr_function_call                         9       On
mr_ECC_variables                      10      On
>>>>

>>>>trace *heart* /off (this DIASABLES HEARBEATS as that will be too noisy in logs)

....
mr_heartbeat_messages            3       Off
....

>>>>

so in few seconds with above commands you enabled MR tracing.

To make long story short:

- when new email comes in, RX will pull it inside CIM and then after going via DB to generate ActivityID it will eventually hot CIM Workflow and there it will reach to integrated queue. That will make EAAS send activity to ICM for routing.

What exactly will happen is that EAAS will send NEW_TASK message to ICM via MRPIM:

08:56:30 pim1     Trace: Application->PG:
Message = NEW_TASK; Length = 73 bytes
   DialogueID = (1) Hex 00000001
   SendSeqNo = (1) Hex 00000001
   MRDomainID = (5002) Hex 0000138a
   PreviousTask = -1:-1:-1
   PreferredAgent = Undefined
   Service = (0) Hex 00000000
   CiscoReserved = (0) Hex 00000000
   ScriptSelector: EIM_SS
ECC Variable Name: user.cim.activity.id
Value: 1041
...

NEW_TASK needs to provide MRD ID and ActivityID.

If there are no available agents for this activity ICM will fail to route it and will send NEW_TASK_FAILURE_EVENT:

09:56:30 pim1     Trace: MR_Peripheral::On_Router_DialogFail:
DIALOG_FAIL  RCID=5001 PID=5001 FailureType=2 NumOfEvents=1 DID=1 DIDRelSeqNo=0 ReasonCode=11
09:56:30 pim1     Trace: Function==>MR_Peripheral::Send_ToApp_NewTaskFailureEvent
09:56:30 pim1     Trace: PG->Application:
Message = NEW_TASK_FAILURE_EVENT; Length = 12 bytes
   DialogueID = (1) Hex 00000001
   SendSeqNo = (1) Hex 00000001
   ReasonCode = (209) Hex 000000d1

However, if all is good, expected behavior is that ICM sends DO_THIS_WITH_TASK message with exact Agent ID back to the application.
After this CIM is in charge of assigning this actvity to that agent.

Here is the example: activity 1057 routed to agent 5010 (SkilTargetID from t_Agent table on ICM side) or that is 1004 from CIM side (USER ID from EGPL_USER table).

13:51:15 pim1     Trace: Application->PG:
Message = NEW_TASK; Length = 73 bytes
   DialogueID = (2) Hex 00000002
   SendSeqNo = (1) Hex 00000001
   MRDomainID = (5002) Hex 0000138a
   PreviousTask = -1:-1:-1
   PreferredAgent = Undefined
   Service = (0) Hex 00000000
   CiscoReserved = (0) Hex 00000000
   ScriptSelector: EIM_SS
ECC Variable Name: user.cim.activity.id
Value: 1057

...
...
...

13:51:15 pim1     Trace: PG->Application:
Message = DO_THIS_WITH_TASK; Length = 90 bytes
   DialogueID = (2) Hex 00000002
   SendSeqNo = (1) Hex 00000001
   IcmTaskID = 150881:202: 1
   SkillGroup = (5013) Hex 00001395
   Service = Undefined
   Agent = (5010) Hex 00001392
   AgentInfo: 1004
   Label:
   ApplicationString2:
   Call Variable 1:
   Call Variable 2:
   Call Variable 3:
   Call Variable 4:
   Call Variable 5:
   Call Variable 6:
   Call Variable 7:
   Call Variable 8:
   Call Variable 9:
   Call Variable 10:
ECC Variable Name: user.cim.activity.id
Value: 1057

So, bottom line, start from MR logs to see if NEW_TASK and DO_THIS_WITH_TASK are present for the same activity. If yes, then ICM job is done and issue is on CIM side. If DO_THIS_WITH_TASK is not there then it fails on ICM side.
This will point you furhter which side you need to investigate more.

I hope this helps!

Thanks,
Goran

[oabulaban]

Hi Goran,

I have a question about redundancy amd failures... how does UCCE handle it if connection between side A and B got disconnected ( both public and private WAN links) ? and how can one recover the data after they reconnect together...

also, how is the "RecoveryKey" calculated, and is it possible to reset it manually?


Sent from Cisco Technical Support Android App

[Goran Selthofer]

Hi!!!

Thanks for your questions!

I hope you find this event useful!

Ok, since there are more and more questions coming I will have to limit my answers to shorter ones

Ok, so your first question is answered in SRND guide in this chapter:

http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact/contact_center/icm_enterprise/icm_enterprise_10_0_1/Design/design_guide/UCCE_BK_UEA1023D_00_unified-cce-design-guide_chapter_011.html#UCCE_CN_UBD4EEF9_00

Check under "Response to failures of both networks"

However, I also invite you to read all other scenarios there as it really describes what happens when only Private network fails, or only Public network fails... or when only one Logger fails, or PG...

So, it will tell you that PG can buffer some data, also it tells you there that Logger can be 12 hours down and if more then you will need to do MANUAL sync of DBs....etc...

I am sure you will get very useful information from that document!

Now, your question about RecoveryKey calculation. I will use explanation I learned from BU while working on some case:

RecoveryKeys are numbers automatically generated by the CallRouter and once the data is replicated to the HDS are no longer used.
So, the system automatically generates the Recovery Key in all of ICM historical tables as they are written from the "temp" tables into the real ICM tables by the Recovery process.  This process generates the key based on the date/time of the Logger (down to the nano-second) and it figures out the number seconds that have passed since the starting date of 1/1/1995.  This gives us a Julian Date in Seconds from our starting point to be able to compare dates/times. 

This seconds value is multiplied by 1,000 to create the actual key -- based on the fact that we don't expect to be able to write more than 1,000 records per second into any one table.  Each time a table is
written to, the key is incremented for the next record within the same table.  This key is set when the Logger HistLogger process starts on the Logger -- it even spits out a message telling you that this is
the new recovery key for all historical data.

The recovery keys are kept unique by this concept on a table-by-table basis -- every table starts with the same key value --  then it is incremented in that table by one on every record written in that table.

When the data is then replicated from the Logger to the HDS, it is not written based on the recovery key, it is written based on the table itself--in alphabetical order taking each new item in their recovery key order within the table and writing to the HDS.

So, answer to your question - can you manually reset those keys - is ABSOLUTELY NO.

I hope this helps!

Thanks,

Goran

[Kishore Bhupal]

Hi Goran,

I've been tryin to get the finesse up and running, but i'm getting the invalid user id/ password error when i try to login..i've entered valid parameters in the finesse admin page..and have also verified to user mapping of the awdb  in the sql server management studio (i'm using windows authentication).

Is this related to awdb connection?

Thanks

Kishore B

[Goran Selthofer]

Hi Kishore!

Thanks for the question!

I had my colleague Zaid Salama preparing this while still being very busy with his own work so I want to thank him for that!

Ok, so here we go:

Regarding this question, I believe we will need more information on the Finesse version, UCCE version,..etc however from the first look on the description, I would expect that the issue is related to the AW, if you are sure that the username and password are correct, the user has the correct privileges, and the connectivity between both sides are good, it might be that you are using NTLMv2 authentication on the AW.

The Docuemnation defect "CSCuj95347 Document Finesse JDBC driver cannot authenticate using NTLMv2" confirms that NTLMv2 is not supported by Finesse as Finesse used a third party driver, that driver doesn't support NTLMV2. The resolution to this is to disable the NTLMv2 on AW, that can be done by running the following:

1) Disable NTLMv2 on the AW server hosting the AWDB and reboot the AW server.

2) Administrative Tools > Local Security Policies > Security Settings

>Local

Policies >Security Options

3) Network Security: LAN Manager authentication Level has "Send NTLMv2 response only" - Choose "Send LM & NTLM responses" or any option that requires only NTLMv2 reponses.

4) Network Security: Minimum session security for NTLM SSP based (including secure RPC) clients has "Require NTLMv2 session security" - UNCHECK this

5) Network Security: Minimum session security for NTLM SSP based (including secure RPC) servers has "Require NTLMv2 session security" - UNCHECK this

6) Reboot AW server.

Hope this helps!

[Omar Deen]

Hi Goran,

Thank you for doing this

I've always wondered... why doesnt TCD, or any other table within an ICM database hold the information as to who disconnected the call? I find it combersome to have to pull CDR logs and try to correlate it to TCD records just to find out who disconnected the call. I'm sure there's a good reason to this and I'd certainly like to hear your input on it.

Additionally, is there an easier way to correlate TCD and CDR records? Can RTMT play a role here?

Similar to what was asked here: https://supportforums.cisco.com/thread/2208349

Thanks

[Goran Selthofer]

Hi Omar!

Welcome to the party!

Yeah, "who disconnected call" investigation was always part of the call control side of things... historically, ICM worked with ACDs and never had to worry about call control hence that part was always left out since from ICM point of view it was not something to be concerned of as it is irrelevant for ICM calculations.

Well, times are changing so if there is real demand for this I believe that pressure will and should come from customer's side by contacting local Cisco Account Managers and asking them to open Product Enhancement Request (PER) for that. Of course, BU will need them to provide some business case / justification. As it is less of a technical than capacity and priority issue. Keep in mind that it will require developer hours and possible protocol changes. That is why there is no action on that side as risks are higher than gain (considering call control part already has that info).

OK, but now, your real question about how to correlate TCD and CDR.

Not sure if you have seen this, but more than 3 years ago I have already answered that question here:

https://supportforums.cisco.com/thread/2060220

I hope that will give the complete answer as I have outlined Cradle to Grave mapping there.

In case someone is not able to access that, i will paste it here:

Mapping CDR to TCD

-----------------------------

The UCManager CDR does have a mapping to ICM Termination Call Detail record, but you need to do some conversion.

The CDR globalCallID_CallManagerID and globalCallID_CallId is combined to create the TCD PeripheralCallKey.

The globalCallID_CallManagerID is moved into the high order byte. To shift it over the properly, multiply it by the hexadecimal value 1000000 and add it to the globalCallID_CallId.

So:

(globalCallID_CallId * 0x1000000) + globalCallID_CallId = PeripheralCallKey

These IDs are not unique because the same PeripheralCallKey and CallID are re-used in redirect, transfer and conference scenarios.

Also, this only works with in a single cluster. So in a multiple cluster environment, you need to map Cluster CDRs to a specific PeripheralID.

Cradle to Grave Call Tracking in ICM

----------------------------------------------------------

The RouterCallKeyDay, RouterCallKey, and RouterCallKeySequenceNumber will track a call from its first route until its final call leg.

The RouterCallKeyDay and RouterCallKey combine to provide common attribute across the calls.

The RouterCallKeySequenceNumber gives you some sense of order of when calls were created. (gselthof: so note, 'some sense' is not guaranteed order!!!)

In a multi-peripheral environment, this requires routing between peripherals. This means calls to the IVR need to be translation routed, and calls to other agent clusters need to be routed as well.

Identifying Routed Agent TCDs

----------------------------------------------------------

You will want to filter out the TCDs created for the CVP call legs, and calls are generated for agents for internal agent to agent calls.

Use the AgentSkillTargetID to identify agent, SkillGroupSkillTagetID to identify SkillGroup, and CallTypeID to identify Call Type / program.

If all three of these values are filled in, you know you got a call that was routed to an agent.

Sometimes more than one TCD will meet these three criteria for the same PeripheralCallKey In those cases, the one with the lowest RouterCallKeySequenceNumber will identify the first call answered by the agent.

CallDisposition

-----------------------------

The CallDispositionFlag is the best indicator to find out if a call was handled or not. There are a bunch of CallDispositions. The CallDispositionFlag distills the results down to 7 categories.

You can find details on what the CallDispositionFlags are in the schema help or schema guide.

I hope this helps!

Goran

[Maheshwar Tayal]

Hi Goran, Thanks for details, Can you please confirm if SIP refer is supported for call transfer with UCCE considering we have CUBE as well as 3rd party SBC

And My 2nd Question is for Blending support, inbound and outbound voice calls for same agent. Do we need specific license for same.

Thanks

[Goran Selthofer]

Hi Maheshwar!

Thanks for participating!

Let me try to address this by pointing to this link:

http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact/contact_center/customer_voice_portal/srnd/9_0/CCVP_BK_C7053373_00_cvp-srnd_chapter_01010.html#CCVP_TP_SAAE8C0F_00

That is where we confirm when SIP refer transfer is supported. Please check.

For the second question, if it is about Outbound Option (Dialer) then I believe yes, you need special licenses for that. You can check ordering guide here:

http://www.cisco.com/web/partners/downloads/partner/WWChannels/technology/ipc/downloads/CCBU_ordering_guide.pdf

I am not from Pre-Sales or Sales side but I can see this is being mentioned there:

5.1.1.2.2 Unified CCE Agent Licenses for Voice Applications with Outbound Option

Unified CCE Outbound Option requires purchase of at least Unified CCE Enhanced or Premium Agent voice application licenses and the appropriate number of Dialer Port Licenses.

...etc...

Have a great weekend!

Cheers,

Goran

[Omar Deen]

Thanks for the in-depth reply!

[oabulaban]

Thanks Goran for your clarification and links for failover and RecoveryKey

A couple of things related here:

- From my understanding about your explanation of how RecoveryKey is generated, does this mean that the RecoveryKey of ANY UCCE system is similar? As it depends on time, this means that today's RecoveryKey is always bigger than yesterday's, even if those were 2 different clusters at 2 different customers?

--> if that's the case, then why in a technology refresh system I can't migrate the HDS table AFTER the logger? This is based on the upgrade guide of UCCE, where all scenarios of upgrade have the HDS being upgraded first (or at the same time as the logger)

- In case side A and side B historical data (from loggers and also hds) are not equal due to some failure happening at some point (we checked icmdba and number of records is different), what is the best way to fix that?

[Goran Selthofer]

Hi!

Ok, let me get to the answers quickly:

- Yes. But they are independent between systems. One system will not create exact as other system. However, it is true that RC can only increase but looking from the own initial base on the system. RC between different customers should not even be discussed as it is not something which can or should be used anyway.

- Install/Upgrade Guide:

If you complete the upgrade of the main Administration & Data Server within the Logger purge window (usually 14 days), you can replace the temporary Administration & Data Servers with the upgraded Administration & Data Servers for reporting. The data replication process fills in any missing data.

So that means that since you would probably need your AW for some tasks during the upgrade, that you migrate it before/at the same time. However, if you don’t want then you can setup NEW TEMPORARY one for that and then migrate real AW/HDS later as said above.

- Recommendation is that you don’t bother with data holes as that is why you have two HDS on both sides. Since it can happen that you have data hole then you can just point and take reports from the side which has data. That action is anyhow just limited to the time you will need reports for that particular missing period. You collect reports and you are done. You don’t need to bother with that anymore. Not sure why you would need to really keep them in total sync as they are there to compensate for those data holes. If you want to keep them in total sync then stop icm services and do full backup of HDS1, start services on HDS1, copy that file over to HDS2, stop services on HDS2, delete HDS DB on HDS2 and recreate DATA and LOG size parts as for the HDS1 so that you can restore that HDS1 backup on HDS2 box (ICMDBA has limit to 32GB so once you create with ICMDBA then use SQL Studio to expand file parts to match HDS1 settings). Once you restore, truncate recovery table on HDS2 and start services on HDS2. Now you have both with same data.

Cheers,

Goran