AppDynamics Sandbox VMs appear to be resource starved

[jefcorco]

I've been attempting to use the AppDynamic sandbox occasionally over the past couple months. Each time I try to use it, I find that the services on the VMs are slow to start up or time out. For example, on the AppD controller, the published access for HTTP returns ERR_CONNECTION_REFUSED for the first hour that the sandbox is up (Main Controller UI - http://10.10.20.2:8090/controller/)

Similarly the Application VM has many services (7) that time out starting up:

appserver[centos]$ sudo systemctl status
appserver.localdomain
    State: starting
     Jobs: 7 queued
   Failed: 7 units
    Since: Thu 2023-03-23 09:38:34 PDT; 3h 10min ago
   CGroup: /

<snip>

This renders the server almost useless. It's been nearly impossible to get much done on these VMs. I don't have access to vSphere or the ESXi host to run any tools (like ESXTOP) to see if the VM is actually resource starved but it appears to be. Many standard commands that I've run on Linux that should be immediate take minutes to complete or time out.

[zowu]

Hi, the AppD controller is equipped with 8 vCPU and 16GB RAM, which should be sufficient for our sandbox non-production deployment. We will upgrade underlying linux kernel to see if better performance.  

[jefcorco]

Thanks, @ zowu. I saw those resources in the lab interface. You're right, that should be enough, but if the host or cluster is oversubscribed, the VM wouldn't be getting it's full share of 8 vCPUs and might be stalled waiting for vCPU time slices. I'm not sure why upgrading the kernel would help, unless there was a known issue. Have you checked in vSphere for performance issues? In the performance charts it's possible to plot CPU Usage, CPU ready time, memory usage, etc. and look for:

• High CPU usage with high CPU ready time
• High memory usage with high memory balloon or swapped memory
• High network or disk usage

In any case, can you tell me when you have made a change so I can test again? Please note also that it's both the App and Controller VM that seem to be having issues.

[zowu]

We've checked few reservations and the CPU peak only happens against App vm on load_test running.  

Meanwhile, the CPU overload is 3.86gHZ, which falls into our configured 8vCPU range.   

Regarding the VM controller, the CPU overload is only 285MHz. 

[jefcorco]

@ zowu You have not indicated that you made any change, but I reserved the lab again about 25 min ago, and in the past 25 min I have been able to progress to the same point that took 3-4 hours in previous attempts. It's very speedy now. I hope that this is a permanent improvement and not an intermittent one. Did you change anything?

[jefcorco]

Here's a performance update.

I had to take a break from the sandbox until about 5:00 MDT.

3/24/2023 5:02:45 PM Controller and App VM working fast.
3/24/2023 5:56:45 PM The controller is no longer responding to HTTP (http://10.10.20.2:8090/controller/)
3/24/2023 6:06:42 PM The webpage loads now, but is still too slow to use.
3/24/2023 6:20:21 PM Working ok now.
3/24/2023 6:44:12 PM Done working for the day.

So, I had some unresponsiveness on the Controller VM between 5:56 and 6:20 MDT.

[zowu]

@jefcorco Here is CPU spec for the controller VM, do you have recommend settings? Thanks, Bob

[jefcorco]

I think that this looks ok, but without a reservation and without seeing the other VMs that are configured to use this host/cluster, it's still possible that there is not enough resources at times for all the VMs. You would have to go into the reports as I mentioned in a prior post to see if there was any issue at the times that I specified in my last post (between 5:56 and 6:20 MDT).