UCCE v12.5 Courtesy Callback Error At CallbackEntry script

[SANJ21]

UCCE v12.5 system with Reporting server. Trying to get CCB up and running using the default 5 scripts Cisco has for CCB. Courtesy Callback is deployed in OPS Console to Call Servers, Reporting Server is deployed and licensed. Skillgroup/qname changed to reflect local skillgroup configured in ICM and EWT statically set to 300. All 5 VXML scripts are imported from OPS Console directory deployed and are all running. However, when I make calls to the sample ICM CCB script, I see that it fails in the CallbackEntry VXML script based on the VXML logs I see, specifically when it hits the "Enter Queue_01" node. This is prior to doing any of the validation steps. However, can't figure out why it's failing on the Entry node. See below snippet. Any ideas where I should look? TIA

10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,newcall,
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,ani,6024301111
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,areacode,602
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,exchange,430
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,dnis,6023042222
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,uui,NA
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,iidigits,NA
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,callid=7ECFE0ACC6E711EB9DC7E2AA2D66BBB6
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,_ccbServletReqTimeout=20
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,_ani=6024301111
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,ewt=300
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,queueapp=BillingQueue
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,qname=CCB_TEST
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,_ccbServlet=http://10.10.10.101:8000/cvp/CallbackServlet
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,_dnis=6023042222
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,_userCourtesyCallbackEnabled=1
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,,start,parameter,ani=6024301111
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.570,CVP Subdialog Start_01,enter,
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.617,CVP Subdialog Start_01,exit,done
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.617,Enter Queue_01,enter,
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.617,Enter Queue_01,custom,Callback_Enter_Queue,ELEMENT_ENTRY
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.617,Enter Queue_01,custom,thishost,10.10.10.100
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.617,Enter Queue_01,exit,
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.617,,element,error,
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.617,Return to ICM on Error,enter,
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.648,Return to ICM on Error,element,warning,A substitution representing Session data named "CallbackType" referred to non-existent information. An empty string was substituted instead.
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.648,Return to ICM on Error,exit,
10.10.10.100.1623084714570.700636.CallbackEntry,06/07/2021 11:51:54.648,,custom,Callback_Leave_Queue,ELEMENT_ENTRY

[SANJ21]

Had a delay working on this as I was waiting for CUBE configs to be completed but that didn't help/fix this issue. One thing I noticed is on the error logs for the CallbackEntry script, I was seeing the below errors. I searched online but couldn't find anything regarding this error. Any ideas?

10.10.10.101.1626280260161.1640948.CallbackEntry,07/14/2021 11:31:00.192,An error occurred in the on end call class com.cisco.cvp.vxml.custelem.callback.LeaveQueue. The error was: java.lang.IllegalArgumentException: SSL context may not be null
java.lang.IllegalArgumentException: SSL context may not be null
at org.apache.http.util.Args.notNull(Args.java:54)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(SSLConnectionSocketFactory.java:275)
at com.cisco.cvp.vxml.custelem.callback.HttpClientMgr.getPoolingConnectionManager(HttpClientMgr.java:157)
at com.cisco.cvp.vxml.custelem.callback.HttpClientMgr.<init>(HttpClientMgr.java:231)
at com.cisco.cvp.vxml.custelem.callback.LeaveQueue.sendHttpRequest(LeaveQueue.java:133)
at com.cisco.cvp.vxml.custelem.callback.LeaveQueue.onEndCall(LeaveQueue.java:82)
at com.audium.server.controller.Controller.runOnEndCall(Controller.java:5401)
at com.audium.server.controller.Controller.onEndCall(Controller.java:5264)
at com.audium.server.controller.Controller.doEarlySessionInvalidation(Controller.java:4910)
at com.audium.server.controller.Controller.doPost(Controller.java:1240)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

[bill.king1]

Did you also do the Courtesy Callback steps in CVP OAMP (like the number of call attempts, etc.)?

Can you include a screenshot of the script (not sure if it has changed much in v 12.5)?

[SANJ21]

Bill,

Yes, I have the configs setup in OAMP for CCB. It's set to 0 for now which is unlimited I believe. 

Don't believe it's changed for v12.5 but here's some screen shots of ICM and CVP script. Apologies if they're little hard to see but these are pretty much default except for the queue and CVP server. I did also change the ewt in ICM to "300" for now.

[bill.king1]

Hi, and the other settings in OAMP like Allow Unmatched Dialed Numbers: are checked? Are you doing anything unusual as far as secure connections with CVP/CVP Reporting server? And your reporting server shows up?

[SANJ21]

Bill,

Yes, I have allow unmatched dialed numbers checked. The secure connection  with courtesy callback db is left unchecked. Reporting server shows up as well.

[janinegraves]

1) I see in the Activity Log that you've changed the ICM script to pass "qname=CCB_TEST"  (instead of qname=billing).

Are you aware that you MUST match that qname in the SetQueuDefaults node in the CallbackEntry app? This might not be causing the 'error' you see, but if you do get past the error, your caller won't be offered a callback unless the Studio app executes a SetQueueDefaults node with the Queue Name setting matching CCB_TEST

2) The error in your error log is related to the end of call, not the EnterQueue. Is there any other error in the error log related to the EnterQueue?  The end of call error isn't giving you any useful info. It's caused because whenever a caller leaves the CallbackEntry app, some End of Call java code always runs to check whether to remove a caller from the CB Servlet's in-memory virtual queue. Since the caller wasn't successfully added to the in memory queue, the end of call code has a problem removing the caller from that queue. This will clear up once you fix the first error.

3) You might also check if the callback servlet is running by using a web browser and entering this URL:
http://10.10.10.101:8000/cvp/CallbackServlet?method=Diag     (upper case D)
you should see something that says "Servlet in memory queue"

4) I suspect you haven't exchanged all the Certificates required - that's a HUGE problem in 12.5 getting all the certs right.

HTH

[SANJ21]

Hello Janine,

Yes, the SetQueueDefaults_01 node has the CCB_TEST name updated to match. I can also get into the callback servlet URL and it shows similar metrics you have attached.

I will re-check the certificates and verify, as this was handled by someone else previously. Thank you!

[bill.king1]

Hi@SANJ21 what was the fix/root cause for this issue?

[SANJ21]

Bill,

Not yet. We have a TAC case opened on this and your peers (Rick and Margie) are working on this as well with a parallel TAC case opened.

[SANJ21]

So Janine was right on the money with the issue being certificates. Although accessing the Diag URL for each CVP server showed a padlock with valid cert path, this was not sufficient enough to validate that all certs were loaded as you need to check the keystore as well to make sure they're all imported.

To check on each CVP server:

1. In CMD, go to c:\cisco\cvp\jre\bin

2. Enter keytool.exe -list -v -storetype JCEKS -keystore c:\Cisco\CVP\conf\security\.keystore >cert.txt

3. Enter keystore pwd (found in registry HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2\CallServer\Parameters\Java and view Options 

4. Enter cert.txt

This should show all certs imported to keystore. For our case, the intermediate certs were not imported into keystore. One CVP server had it which I used to compare the rest to.

The fix was to import the missing certs and restart VXML service in OAMP.

[Paul_Milstone_01]

Hi there I'm currently on CVP 11.6 and I too am getting the exact same issue.  I cannot for the life of me find where that class file is for the LeaveQueue.  However, I can do all the steps up above and we aren't using any certs in my environment for the reporting servers.  The CVPs can hit it fine.  So if anyone has any idea on what could be causing this please let me know!

[bill.king1]

When you say the exact same issue, do you mean as far as the caller behavior or the entries in the logs. If you can post the application log and the error log of a test call and outline what caller heard, group may be able to offer suggestions.