This document was generated from CDN thread
Created by: UMUT ARSLAN on 23-02-2010 09:43:04 AM
We are using CVP 4.1 and Websphere 6.0 in our VXML projects and we are monitoring websphere server's performance.
We have an observation of increase on the wesphere memory usage.
When websphere application server first starts and all the VXML applications (around 15 applications where 200.000 calls are processed daily) are up and running, websphere uses a memory of 400-500 megabaytes. The server memory performance continues like this for a while (up to 1-2 weeks). After 1-2 weeks servers memory usage constantly increases and becomes stable around 700-800 megabaytes, and after a while it uses 1 GB memory where we restart the VXML server..
Updating the application does not help in decreasing the memory usage, we have to restart CVP application server..
We have examined memory heapdump at different times according to growing memory size and find out that the objects which reside in memory are com/audium/logger/application/activity/CacheManager objects which are used to store the activity of the call and writes to activity log when the call ends.
We use different cache sizes for different applications according to appplication complexity and average activity log created for these applications.
The values are around 10kb and 50kb which is defined in ActivityLogConfig.xml file. Most used application has a cache size of 20 kb.
<cache>
<per_call kb_limit="20" allocate="once"/>
</cache>
Since there are activity cache objects in the memory which are not released without being sure we suspect that the calls belonging to these objects are ended and sessions for these calls are invalidated but the activities are not written to activity log file since it can not find the sessions of the calls. Is it possible under heavy load on the application ?
We use the standart values for the parameters related to session ending and activity logging.
<session_invalidation_delay>30</session_invalidation_delay>
<logger>
<minimum_thread_pool_size>1</minimum_thread_pool_size>
<maximum_thread_pool_size>500</maximum_thread_pool_size>
<keep_alive_time>30</keep_alive_time>
</logger>
Subject: RE: memory leak problem in CVP VXML server
Replied by: Jim Yang on 23-02-2010 05:17:15 PM
1. Did you do a full GC before you did the heapdump?
2. Does the number of Cache objects match to the number of active calls?
3. If the cache is disabled, do you still see the memory usage increases?
Subject: RE: memory leak problem in CVP VXML server
Replied by: Jim Yang on 23-02-2010 09:19:58 PM
For the time being, I suggest you disable the log cache. You could also check the activity log and reporting data to see if there are any abnormal call terminations. I would defnitely do a full GC before collecting a heapdump.
Subject: RE: memory leak problem in CVP VXML server
Replied by: UMUT ARSLAN on 23-02-2010 08:57:50 PM
When the memory reaches 1GB for example, we first set weight on the server to 0 from content server switch (load balancer), then we wait for the active calls to end and when there are no active calls on the server we get the heap dump.
When the heap dump is examined we see that there are many cache objects residing in the memory
1) Websphere does garbage collection constantly in every 2-3 minutes, we did not make an extra garbage collection
2) heap dump is taken when there is no active call on the server, so the answer to your question is no.
0 active calls -- many cache objects
3) we did not disable the cache
