01-06-2012 09:08 AM - edited 03-14-2019 09:08 AM
Hi,
I've a client with UCCX HA setup (System version: 8.0.2.10000-41).
I saw this alert generated yesterday in the RTMT under Alert Central. It came from Secondary UCCX server (It is running as Master). The client has some weird issues since yesterday, the agents started getting Licensing Error, CSD is not updating Agents statistics, Real time data in RT tables (for wallboard) is not getting updated etc. I bounced the LRM service and agents were able to login fine.
I looked at Bug: CSCtq91894 but not very helpful. Does anyone know what could have caused this alert and any possible fixes? I am attaching the MIVR logs covering this event.
Module Name : HISTORICAL_REPORTING_MANAGER
Module Failure Name : HISTORICAL_DATABASE
UNKNOWN_PARAMTYPE:Module Run-time Failure Cause : 3
Module Failure Message : Historical database queue is full. Historical data saved in files. Queue size= 0 Total number of lost records= 1 Please check dat
AppID : Cisco Unified CCX Engine
ClusterID :
NodeID : CELCC6802
TimeStamp : Thu Jan 05 09:50:11 AST 2012.
The alarm is generated on Thu Jan 05 08:50:11 EST 2012.Module Name : HISTORICAL_REPORTING_MANAGER
Module Failure Name : HISTORICAL_DATABASE
UNKNOWN_PARAMTYPE:Module Run-time Failure Cause : 3
Module Failure Message : Historical database queue is full. Historical data saved in files. Queue size= 0 Total number of lost records= 1 Please check dat
AppID : Cisco Unified CCX Engine
ClusterID :
NodeID : CELCC6802
TimeStamp : Thu Jan 05 09:50:11 AST 2012.
The alarm is generated on Thu Jan 05 08:50:11 EST 2012.
Thanks,
Kapil
01-08-2012 10:40 AM
On UCCX version 8.0.2 if you does not have SU4, its most recommended to have it. It has lots of bug fixes including the one you are likely to hit. http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtl93897
SU4 can be downloaded from here http://www.cisco.com/cisco/software/release.html?mdfid=270569179&flowid=5217&softwareid=280840578&release=8.0(2)_SU4&rellifecycle=&relind=AVAILABLE&reltype=latest
If your setup is already in production, I'd recommend to have Cisco TAC confirm it for you.
Pls rate the post if it helps.
GP.
04-10-2012 06:25 PM
We seem to be receiving these same alerts on UCCX 8.5(1)SU2 with COP file on HA.
04-10-2012 07:20 PM
Hi,
After searching for the error logs cam across a TAC case reference:617010385
Which is pointing to below 2 defects which have a fix for this issue.
---------------------------------------------------------
CSCtj88620: run time error while generating agent related historical reports
Description: HRC throws run time error while generating some agent related reports. HRC log shows the error "A subquery has returned not exactly one row".
Following is seen in the RTMT for the same time frame.
Module Name : HISTORICAL_REPORTING_MANAGER
Module Failure Name : HISTORICAL_DATABASE
UNKNOWN_PARAMTYPE:Module Run-time Failure Cause : 3
Module Failure Message : Historical database queue is full. Historical data saved in files. Queue size= 0 Total number of lost records= 300 Please check d AppID : Cisco Unified CCX Engine
ClusterID : NodeID : uccx1a
TimeStamp : Wed Nov 03 08:02:16 EDT 2010.
The alarm is generated on Wed Nov 03 08:02:16 EDT 2010.
To Be fixed: 8.0(2)SU2 8.5(1)SU3
-------------------------------------------------------------
CSCtl93897: UCCX Engine tries to update missing table, which causes RTMT alerts
Description:
When running a historical report, the customer's system generates RTMT alerts such as:
Subject: [RTMT-ALERT-StandAloneCluster] HistoricalDataWrittenToFiles
Module Name : HISTORICAL_REPORTING_MANAGER Module Failure Name : HISTORICAL_DATABASE UNKNOWN_PARAMTYPE:Module Run-time Failure Cause : 3
Module Failure Message : Historical database queue is full. Historical data saved in files. Queue size= 0 Total number of lost records= 1 Please check dat AppID : Cisco Unified CCX Engine ClusterID :
After further investigations, the cause of this issue was found to be the UCCX engine, which tries to write data to a table from a previous version, called the WorkflowTask table. Given that there is no longer a table within the database by this name, this causes the system to generate RTMT alerts.
While there is no direct impact on the system in a critical way, such as core functionality or critical data loss, this still needs to be addressed, as it causes concerns for the customer who may see these errors.
To Be fixed: 8.5(1)SU3 8.0(2)SU4
--------------------------------------------------------------
As I see you are currently running on UCCX 8.5(1)SU2 with COP file on HA, need to come to 8.5(1)SU3.
As of now I can only see 8.5 SU2 posted in CCO.
Please contact Cisco TAC for the ES or any other available COP file for this issue.
Hope it helps.
Anand
Please rate helpful posts by clicking on the stars below the right answers !!
04-10-2012 07:40 PM
You saved me from raising a TAC case
Thank you very much Anand.
-Yavuz
04-10-2012 07:51 PM
Hi Yavuz,
Glad to know that :-).
Please dont forget to rate the helpful posts by clicking on the stars below the right answer.
Thanks,
Anand
04-10-2012 08:26 PM
Anand, Do you have an exact date on when UCCX 8.5(1)SU3 be release on CCO downloads page? Thanks, Yavuz
04-10-2012 08:39 PM
Hi Yavuz,
The expected release date for UCCX 8.5(1)SU3 is April 18th, unless there are no major issues find during regression testing which might be in progress.
I hope there are no delays in this.
Hope it helps.
Anand
Please rate helpful posts !!
04-10-2012 09:03 PM
Fantastic news!
Would you be able to obtain the SU3 release notes from CCBU?
It will be nice to find out how many caveats SU3
04-11-2012 12:28 AM
Hi Yavuz,
Hope it helps.
Anand
Please rate helpful posts !!
10-21-2015 08:10 AM
Hi Gajanan,
I'm getting the same alert for UCCX 10.5.1.
Do you know if same bug (CSCtj88620) is affecting 10.5.1 as well or if there is another bug related to that ?
Also getting "CCXToCUICCVDSyncFailed" alert, do you know if this is related to a bug as well ?
=
HistoricalDataWrittenToFiles
Module Name : HISTORICAL_REPORTING_MANAGER
Module Failure Name : HISTORICAL_DATABASE
UNKNOWN_PARAMTYPE:Module Run-time Failure Cause : 3
Module Failure Message : Historical database queue is full. Historical data saved in files. Queue size= 0 Total number of lost records= 1 Please check dat
AppID : Cisco Unified CCX Engine
ClusterID :
==
CCXToCUICCVDSyncFailed
RTR : User with userID:mpoulter,sv025,sv021,sv013,sv024,sv029,sv022,ta031,sv032,sv020,sv011,sv018,sv030,sv003,sv031,ct14,sv036,rn064,sv
AppID : Cisco Unified CCX Cluster View Daemon
sharri
10-21-2015 08:58 AM
Hi Saima,
In your other below post, wherein I replied few minutes back you did not mention about the CCXToCUICCVDSyncFailed alert hence that response is categorically for "Historical database queue is full" issue. :
https://supportforums.cisco.com/discussion/11454016/uchrmgr-2-uccxhistoricaldatawrittentofiles
Now coming to this CCXToCUICCVDSyncFailed alert in conjunction with Historical database queue is full, you are definitely hitting the below defect wherein the alerts will immediately generate after the schedule purge is done. You can verify the same by going to Cisco Unified CCX Administration >> Tools >> Historical Reporting >> Purge Schedule Configuration and check the time for "Daily Purge At". Then match this time with the time of alerts and you will find it the same confirming the defect:
https://tools.cisco.com/bugsearch/bug/CSCur00452/?reffering_site=dumpcr
This defect is fixed in 10.6(1) train and since 10.6(1) SU1 is also out, it will be best to upgrade to that option in order to resolve the issue.
Little background on this defect CSCur00452:
In a HA environment, the sync request should only come from publisher UCCX but because of the issue in code, the sync request comes from the secondary node as well and since there is no mechanism in the code to entertain that, false alerts are generated.Starting version 10.6(1) onwards, there had been a fix implemented in the code itself wherein the sync request does not come from secondary node avoiding these false alerts to generate.
Regards
Deepak
10-21-2015 10:46 AM
Thanks Deepak for the detailed information.
I will try those and let you know the outcome
Sharri
10-21-2015 11:06 AM
Hi Deepak,
So that means if we are getting both alerts 'CCXToCUICCVDSyncFailed' & 'Historical database queue is full'
at the same time as 'Purging Schedule Daily' then we are hitting the defect below:
CSCur00452
Sharri
10-21-2015 12:46 PM
Hi Saima,
Yes that is correct.
Regards
Deepak
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide