Cisco AMP for Endpoints on Windows 2016 grabs more and more memory until it crashes

[AndrewCirel]

On our Windows 2016 Servers, Cisco AMP for Endpoints gradually takes more and more memory until the server crashes with memory exhaustion errors.  I looked at the memory usage on one of the servers just before it crashed and sfc.exe (Cisco AMP for Endpoints Connector) was using 18,661,428 Commit (KB), 227,656 Working Set (KB), 33,060 Shareable (KB) and 194,596 Private (KB).  So, Cisco AMP for Endpoints Connector has grabbed over 18GB of Memory.

I've upgraded to the latest version of Cisco AMP for Endpoints and it doesn't make a difference.

Any idea what could be causing this?

[Jim2k]

first question i have is what does your policy look like? I have version 7.1.5 on 41 2016 servers and i have not seen that issue. with that said i only have it on my servers to capture trajectory info for investigation purposes. when i installed it i used the /skiptetra 1 So Av part of Clamav would not get installed.

a suggestion i have is to put it in debug mode and then look at the logs to see what files it is hitting. you may need some exclusions

[AndrewCirel]

Thanks for the reply.

I've tried the debug and it hasn't showed any issues.

Can I ask what the sfc.exe memory usage (Commit / Working Set / Shareable / Private) is for one of your servers running Cisco Amp that hasn't been rebooted for a couple of months.

I'm going to temporarily disable some of the components in a test lab and see if that isolates what is causing the issue.

[Jim2k]

I actually now have a server with the same or similar issue as above.

amp version is 7.5.1  server is 2016. it is behind a firewall and is using a proxy. memory usage is normal but commit size just keeps growing

[AndrewCirel]

We escalated this problem with Cisco through our account manager and the final answer from Cisco was "Cisco Amp has historic memory problems".  We never did get a resolution and have to restart Cisco Amp when the monitoring software picks up memory issues.  If you find anything please let me know.

[Jim2k]

I found this in a Bug search  CSCwa86515 I am going to try a update the agent to 7.5.5

Symptom: Memory utilization spikes gradually, eventually leading to either an intermittent server reboot to alleviate the memory issue, or without this reboot, to a crash

Conditions: Affects Windows Servers (2012, 2016, 2019, 2022), uncertain what specific conditions cause the issue.

Workaround: The issue is generally seen as gradual (building memory usage over a number of weeks) and is alleviated by a reboot or Secure Endpoint connector restart (which does not require a reboot). Affected servers can be alleviated with a regular reboot *or* a restarting of the Secure Endpoint service on a cadence that aligns with the frequency that this issue is encountered. Some customers have reported that disabling Script Protection alleviates their issue.

Further Problem Description: A number of memory-usage issues are slated to be addressed in Secure Endpoint version 7.5.5. All affected customers are recommended to update to this version upon its release.

[AndrewCirel]

Thanks for the information - a good find.

I shall also try updating to the latest to see if it fixes it this time.

[Michael Schenck]

Did upgrading to 7.5.5 help?

[Jim2k]

the server with the issue was updated to 7.5.5 and it fixed the issue. I do have 3k 2016 servers that are on version 7.5.1 with no issues