finesse went down for approximately 5 mins - uccx 10.6(1)SU2

[ccg-collab1]

Hi All,

A customer experienced an outage in UCCX wherein they encountered an error "The Finesse Server is out of service" and then they cannot call the trunklines that are being routed to the trigger point in UCCX. As advised, the system or service is back to normal after approximately 5 mins. 

As agreed with the customer, we will check the system tomorrow since they already left the site. Appreciate if you can share us the what are the logs that we need to get to further check what really happened during the issue.

There is only one node for the UCCX. Kindly see below encountered error of the agensts:

[DJ Kulkarni]

Hi

Please check the cucm click to see if there was any core dump for cti.Also did he cti service restart that time ??

To take finesse logs for that time go via cli  

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/finesse/finesse_1001/installation/guide/CFIN_BK_CDE1E024_00_cisco-finesse-installation-and-upgrade/CFIN_BK_CDE1E024_00_cisco-finesse-installation-and-upgrade_appendix_0101.html#CFIN_RF_L47603BA_00

[ccg-collab1]

Hi DJ,

Done with the checking of the issue. We found out that UCCX Engine seems automatically restarted which is the cause of the issue on finesse. Further checking, we found an error "UnifiedCCXEngineMemoryUsageHigh" and from the MIVR logs "nvokeId=54479 }
13842138: Oct 28 17:58:13.880 PHT %MIVR-ICD_CTI-7-UNK:ClientConnMgr: Processing msg on socket:Socket[addr=127.0.0.1,port=43359,localport=12028,Client Id=Finesse] Msg is {length=4 type=HEARTBEAT_REQ,invokeId=78338 }
13842139: Oct 28 17:58:13.880 PHT %MIVR-SS_OB-7-UNK:ClusterMsgMgr.isMaster() returns true
13842140: Oct 28 17:58:13.880 PHT %MIVR-SS_OB-7-UNK:CheckCampaignStartStop:we are master and DB Publisher is up
13842141: Oct 28 17:58:13.880 PHT %MIVR-SS_OB-7-UNK:CheckCampaignStartStop:OB state: 2 and about to check RmCm state
13842142: Oct 28 17:58:13.880 PHT %MIVR-SS_OB-7-UNK:CheckCampaignStartStop:RmCm is up
13842143: Oct 28 17:58:13.880 PHT %MIVR-CFG_MGR-7-UNK:getAccessor(ConfigQuery):Try to get the default accessor
13842144: Oct 28 17:58:13.880 PHT %MIVR-CFG_MGR-7-UNK:DBAccessor.get(query))
13842145: Oct 28 17:58:13.880 PHT %MIVR-CFG_MGR-7-UNK:Schema class = com.cisco.crs.outbound.CampaignConfig
13842146: Oct 28 17:58:14.749 PHT %MIVR-MGR_MGR-1-HIGH_MEM_USAGE:JVM heap memory usage is high: Facility=MIVR,Current usage=499 MB,Maximum available=499 MB,percentage Old Gen Memory used=100.0
13842147: Oct 28 17:58:14.749 PHT %MIVR-ICD_CTI-7-UNK:ClientConnMgr: Processing msg on socket:Socket[addr=10.18.0.151,port=57809,localport=12028,Client Id=ClientID] Msg is {length=4 type=HEARTBEAT_REQ,invokeId=54480 }
13842148: Oct 28 17:58:14.749 PHT %MIVR-ICD_CTI-7-UNK:ClientConnMgr: Processing msg on socket:Socket[addr=10.18.0.151,port=57800,localport=12028,Client Id=ClientID] Msg is {length=4 type=HEARTBEAT_REQ,invokeId=54516 }
13842149: Oct 28 17:58:14.750 PHT %MIVR-MGR_MGR-3-THREAD_DEATH_EXCEPTION:An uncaught exception has be thrown in a thread: Thread name=SM Tomcat Poller Thread,Exception=java.lang.OutOfMemoryError: Java heap space
13842150: Oct 28 17:58:14.750 PHT %MIVR-SS_RM-7-UNK:RIMgrAddressCallObserver: TermConnDroppedEv received for call 19533714 [2756498/1] and agent brandereval
13842151: Oct 28 17:58:14.750 PHT %MIVR-SS_CM-7-UNK:ContactMgr.getRmCmContact(2756498/1) returns 19533714 [2756498/1]
13842152:

Tried searching and checking some information in the internet but there is not enough details about this issue. We found this bug id CSCuf79940 but it is from version 8 of uccx.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuf79940/?referring_site=bugquickviewredir

Appreciate if you know the resolution for this issue.

Thank you so much.

[DJ Kulkarni]

Hi,

I have not see it with 10.6 SU2 till now but if the JVM heap memory is more may be you are hitting the same bug.Can you check with TAC to see if they give any special patch for this ??

Regards,

DJ