PCCE 12.0 - Sudden Data Reset in Call-Type-Real-Time Table

[ln33147]

Hello,

This is regarding a PCCE environment version 12.0

We have noticed that the data in the Call_Type_Real_Time table is being reset under AWA and AWB randomly as you can see in the below screenshots.

However, the data is correct under the historical Call Type Interval table

Anyone faced a similar issue? any suggetsions on what to check?

Thanks in advance

Lara

[bill.king1]

I think we need some context for what the first screenshot is showing, unless I'm not following you. Are you saying you're running that throughout the day but then you run the historical one at 11pm and it shows completely different numbers? 
A better example to help[ show the issue would be to do the real time one in one screenshot at a fixed point in time (say 10am) and then do the exact same query for historical at the same time frame and compare those.

[ln33147]

Hello Bill,

Thank you for your time.

The issue initially appeared in the 2Ring Dashboard, where the Real-Time Data suddenly resets to zero during the day, consistently occurring at 6:00 PM. After some troubleshooting, we noticed that the issue lies with the Call_Type_Real_Time table and not on the 2Ring application.

The first screenshot shows that the Call_Type_Real_Time table only displays data from 6:00 PM to 11:59 PM, whereas the Historical table correctly reflects data for the entire day.

To note that the issue does not occur daily. However, when it does, it always happens at 6:00 PM so we need to investigate the possible causes of this behavior.

Any suggestions?

[bill.king1]

So you're saying this is what happens?
-From midnight to 6p ET (or whatever your time zone is), your call type real time calls offered and calls handled is 0
-At 601pm through to midnight, it starts incrementing, and then resets to 0 at midnight.
-But if you look at the historical call_type_interval report starting at midnight, it gradually increases throughout the day?

[ln33147]

Hi Bill,

- From midnight to 6:00 PM, the Call_Type_Real_Time shows correct data (Shown on the 2Ring dashboard). At 6:00 PM the values reset to zero. Then from 6:00 PM to 11:59 PM we have also correct data and it resets normally to 0 at midnight again.

- Yes The Call_Type_Interval shows the correct data throughout the day.

Regards,

[david.macias]

That's supposed to happen at midnight. Any chance your central controllers are on UTC and you're NOT?

david

[ln33147]

Hi David,

The issue doesn't occur daily, which is confusing. However, we checked the time on Call_Type_Real_Time and Agent_Real_Time (where we didn’t notice any resets), and the times are the same.

[bill.king1]

Is it perhaps that you didn't have traffic for the other call types and agents that's why you don't always see it? Perhaps test with another real time table that you know will always have traffic and then see if it is the midnight reset that David mentioned.

[ln33147]

It is a 24/7 call center that receives around 2,000 calls per day, so there is definitely data before 6:00 PM.
If it were a configuration issue, it would happen every single day.

[bill.king1]

Sounds like a TAC case then. Please let us know what happens.
One last thing might be worth checking, have you tried to see if you're seeing the same behavior on both A side central controller/HDS and B side as well? Just to eliminate that from the equation?

[ln33147 · Accepted solution]

I would like to provide an update on this thread.

The issue was caused by Group Policies/Windows Updates being applied to the CCE servers, which led to the reset issue. Below is a snippet from the AW/Router logs:

• A Windows group policy has been applied on Rogger A, it was successful and it disconnected the network
Information 1/27/2025 6:01:18 PM Microsoft-Windows-GroupPolicy 1502 None The Group Policy settings for the computer were processed successfully. New settings from 2 Group Policy objects were detected and applied.
Information 1/27/2025 6:01:18 PM Service Control Manager 7036 None The Network Connectivity Assistant service entered the stopped state.
Error 1/27/2025 6:01:12 PM Cisco Systems, Inc. ICM 32792 Message Delivery MDS going out of service.
Warning 1/27/2025 6:01:12 PM Cisco Systems, Inc. ICM 18 Message Delivery Synchronizer suspending operation.
Error 1/27/2025 6:01:11 PM Cisco Systems, Inc. ICM 34 Message Delivery Connectivity with duplexed partner has been lost due to a failure of the private network, or duplexed partner is out of service.
• Post this, MDS lost connection
18:01:11:440 ra-mds Connectivity with duplexed partner has been lost due to a failure of the private network, or duplexed partner is out of service.
Last EMT Error [121]: The semaphore timeout period has expired.
18:01:12:285 ra-rtr MDS has reported to the Router that it is out of service.

• ON AW, there was an update which failed
Information 1/27/2025 5:58:34 PM Windows Error Reporting 1001 None "Fault bucket , type 0
Event Name: WindowsUpdateFailure3
Response: Not available
Cab Id: 0

Problem signature:
P1: 10.0.14393.5127
P2: 80072ee2
P3: 00000000-0000-0000-0000-000000000000
P4: Scan
P5: 0
P6: 1
P7: 8024500b
P8: Windows Defender (77BDAF73-B396-481F-9042-AD358843EC24)
P9: {9482F4B4-E343-43B6-B170-9A65BC822C77}
P10: 0

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\NonCritical_10.0.14393.5127_9dee2ba4c6f49d8d7b4193c2afb1e9fc5c6a5fb9_00000000_3c70f71a

Analysis symbol:
Rechecking for solution: 0
Report Id: cef2b30c-c732-11ef-b84d-00505694b9c3
Report Status: 96
Hashed bucket: "
• After this, the following was noticed
Warning 1/27/2025 6:01:12 PM Cisco Systems, Inc. ICM 6 Real Time Feed Preferred feed from: roggera/40007, has been broken.
Warning 1/27/2025 6:01:12 PM Cisco Systems, Inc. ICM 10 Real Time Feed Attempting to establish backup feed from: roggera/41007.
Warning 1/27/2025 6:01:23 PM Cisco Systems, Inc. ICM 9 Real Time Feed Backup feed established from: roggera/41007.
• RTC real time client disconnection/clean-up of data
18:01:23:724 dis-rtc Trace: Real Time Feed switched to backup side of central controller.
18:01:23:724 dis-rtc Trace: Clean up the realtime information from the memory.
18:01:23:865 dis-rtc Trace: Successful Delete of old RealTime information from DB.
18:01:28:360 dis-rtc Received ConfigChange message. Latest config version: 10977345945138.

[bill.king1]

Thank you for posting the root cause. So some policy was causing the system to be disrupted some/most days at that time, which reset the counters basically?