com.audium.server.MultipleThreadException on LocalVar

[Isaac Asimov]

Hi there, I faced floating problem with com.audium.server.MultipleThreadException in application on Local Var element with piece of JS and I getting error.badfetch.

We have CVP 12.0.
I didn't dive deep, but in jconsole in peak we have around 4,1GB memory use and 250 thread. I think we get problem at this moment.

02/17/2023 17:12:08.124, The error was: Another thread is currently working on this session. This means the original thread took too long to complete what it was doing. To prevent abnormal behaviour the original thread will exit immediately. The call should not be affected.
com.audium.server.MultipleThreadException: Another thread is currently working on this session. This means the original thread took too long to complete what it was doing. To prevent abnormal behaviour the original thread will exit immediately. The call should not be affected.
at com.audium.server.session.ControllerData.isCurrentThreadOwner(ControllerData.java:5323)
at com.audium.server.voiceElement.ActionElementBase.service(ActionElementBase.java:395)
at com.audium.server.controller.Controller.goToAction(Controller.java:4446)
at com.audium.server.controller.Controller.goToElement(Controller.java:4082)
at com.audium.server.controller.Controller.continueCall(Controller.java:3762)
at com.audium.server.controller.Controller.goToElement(Controller.java:4178)
at com.audium.server.controller.Controller.continueCall(Controller.java:3762)
at com.audium.server.controller.Controller.goToElement(Controller.java:4178)
at com.audium.server.controller.Controller.continueCall(Controller.java:3762)
at com.audium.server.controller.Controller.goToElement(Controller.java:4178)
at com.audium.server.controller.Controller.continueCall(Controller.java:3762)
at com.audium.server.controller.Controller.goToElement(Controller.java:4178)
at com.audium.server.controller.Controller.continueCall(Controller.java:3762)
at com.audium.server.controller.Controller.goToElement(Controller.java:4178)
at com.audium.server.controller.Controller.continueCall(Controller.java:3762)
at com.audium.server.controller.Controller.goToElement(Controller.java:4178)
at com.audium.server.controller.Controller.continueCall(Controller.java:3762)
at com.audium.server.controller.Controller.doPost(Controller.java:1175)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:412)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:754)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1385)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Does root fetchtimeout is useful on this case?

Does fetchaudio can be set to VVB? (like from ICM with flash:/)

Any other ideas how to give time to JS execution on Local Var?

@janinegraves 

@ptindall 

BR, IA

[janinegraves]

I believe this is a known bug and that there's a patch to fix it.
I don't have access to bugs/patches, but  Bill King or Dave Macias could probably tell you more about the patch.

[Isaac Asimov]

Thanks, I will try to reach out to them!

@david.macias 

@bill.king1 

[bill.king1]

Hi @janinegraves when you say a known bug, in what sense? Something about LocalVar? A memory exhaustion issue? Something else?

[david.macias]

This one doesn't ring a bell. I would reach out to TAC and ask them what the recommendation ESes to have on a 12.0 system are. What is it that you're doing when you encounter this error? Does it happen after a period of time or because of call volume?

david

[Isaac Asimov]

Hi, thanks for answer.

Yes, it's looking like it's happen on call volume. Seems like LocalVar executes extra time, rather than usually. It's looks weird that community discuss same case for db/ftp/audio element, but not for localvar, although this features introduced since CVP 11.0

Does fetchtimeout on vxml app is useful for this case?

BR, IA

[bill.king1]

You may want to install ES8 (along with the required prerequisite ES before that) in case this is what you're running into. What does your regular log look like?
https://bst.cisco.com/bugsearch/bug/CSCvv90156

https://bst.cisco.com/bugsearch/bug/CSCvx14235

[janinegraves · Accepted solution]

Yes, CSCvx14235 was the one I was thinking of:

VXML applications failing after timeout randomly at Set Value node
CSCvx14235  

Customer Visible

Notifications

Save Bug

Open Support Case

Description

Symptom: VXML calls getting disconnected randomly at Set Value node. Conditions: With the below steps we can reproduce the issue with calls getting disconnected randomly at Set Value node of second application and later recovers automatically. 1610032247775.5408.Outbound_TAC_Test_2,01/07/2021 20:40:47.791,Set Audio Path,enter, 1610032247775.5408.Outbound_TAC_Test_2,01/07/2021 20:40:47.791,Set Audio Path,exit,done 1610032247775.5408.Outbound_TAC_Test_2,01/07/2021 20:40:47.791,Set Global Variables,enter, 1610032247775.5408.Outbound_TAC_Test_2,01/07/2021 20:40:47.791,Set Global Variables,custom,endpoin_GetContactDetail,Success 1610032247775.5408.Outbound_TAC_Test_2,01/07/2021 20:40:59.806,,end,how,disconnect 1610032247775.5408.Outbound_TAC_Test_2,01/07/2021 20:40:59.806,,end,result,normal 1610032247775.5408.Outbound_TAC_Test_2,01/07/2021 20:40:59.806,,end,duration,12 Workaround: NA Further Problem Description:

Was the description about this Bug Helpful?

(3)

Details

• Created Date
• Jan 25, 2021

• Last Modified
• Dec 15, 2022

• Status
• Fixed

• Severity
• 2 Severe

Product (1 of 1)

Cisco Unified Customer Voice Portal

Known Affected Releases (1 of 1)

12.0(1)

Known Fixed Releases (0)

No Records Found

[Isaac Asimov]

Thanks, missing this case when I looking at ES release notes yesterday.

[Isaac Asimov]

My issue looking the same, after 12 seconds of enter in LocalVar we catch error.badfetch by hotevent.

02/17/2023 13:00:30.975,SetValue,enter,
02/17/2023 13:00:42.051,SetValue,element,hotevent,Hotevent_Error_Badfetch

Does fetchtimeout is make any role in this case?

[janinegraves]

Setting fetchtimeout to something like 20s or 30s and setting fetchaudio to a url to play filler audio to the caller just before the slow app elements is always wise. It'll prevent the fetchtimeout. But you should also fix the bug of the set value element taking longer than it should.

[Isaac Asimov]

Thanks, do you know which a default fetchtimeout and fetchaudiodelay on CVP?

I'm installed ES8 situation seems better.