VXML Memory Leak Issue

[Roy_11]

Hi Experts ,

last week we had deployed on IVR application two xml file has been modifed and updated , it was working fine for 1 hour after that all appliaction are  impacted . there was call drop on the VRU node in ICM , there was no call hit on VXML application more than 4 hrs , we have taken all application backup and moved to side B , the side B started working and then in Side A we have removed all the application and added one by one then side A also started working on this , now currently few applications are running on side A , Here have attached the tomcat catalina logs for your reference , could you please review and tell us what went wrong on this , why so many threads are struck , what caused it happened .

Thanks!

[Omar Deen]

You need to verify that you actually do have a memory leak before you can go any further. Here's a good tutorial on how to use JConsole to find memory leaks: http://www.cleantutorials.com/jconsole/detecting-memory-leak-in-java-using-jconsole-with-example-code

You can also use Eclipse MAT: https://www.vogella.com/tutorials/EclipseMemoryAnalyzer/article.html

[rakeshjogi]

With the info you shared, it seems like the issue is related to blocked threads or may be any contention within the application after the changes you have done. Though application support in initial hours after the changes in XML, later seems to have some impact which suggests that issue happens gradually over period. Such things may happen because of blocked threads or due to resources getting exhausted.

So as per mentioned symptoms like call drops on the VRU node,  no traffic reaching the VXML application for several hours, all these indicate that application goes unresponsive.  This is because multiple threads will be in waiting state. So to check the exact root cause, it is better to check the stuck thread stack traces from the Catalina logs. This will give you an idea to see where the threads are getting blocked. Also try to review the following items like,  the XML changes, checking dependencies on external systems, and monitoring thread pool utilization under load which will help you to understand where the problem lies and would help in pinpointing what caused the threads to become stuck. To learn more about Java memory leak, identifying memory leaks by analyzing heap dumps,   check out this blog, From Symptoms to Solutions: Troubleshooting Java Memory Leaks & OutOfMemoryError.

[mbieyasid]

What CCE version are you using now?

I have experienced something similar. First, you must monitor for Java memory leaks. Then check the Java version installed on CVP. In my case, the leak was caused by the Java version. So when the issue occurs, we have to restart CVP, and it returns to normal. To fix this issue permanently, we need to upgrade the JDK on CVP, which is included in the ES patch

[bill.king1]

Hi @mbieyasid, what version of CVP and what ES did you need to apply to resolve this for you? And is there a Cisco defect tied to the issue?

[mbieyasid]

Hi @bill.king1 In my case, it was quite old. The issue happened on PCCE 12.5(1), specifically on a fresh version of CVP around 2021-22. To fix this issue, I patched it to ES 16, which — as you know — includes migrating from JRE to JDK (a higher version).

why the issue appear because we have the big app vxml on cvp include integration with core banking so the customer can transaction like transfer money on ivr 😅.

we already chalange the BU it can possible to increase heap size on CVP but it's not possible. the BU add some java trap on cvp also. So after a long check, we finally came to the conclusion that the problem was with the Java version.

first to thing the point is you must check behavior java on CVP.
hope it can help for you.