Cisco Secure Endpoint 8.2.1 memory usage rises over time Win 10

[korniiva-flutterint]

My company uses Cisco Secure Endpoint and I have recently noticed that the memory consumption of sfc.exe rises over time up to about 2+Gb of RAM and I suspect it is starting to affect my Remote Desktop Connection at that point.

I have written a script to collect current memory usage of the sfc.exe and confirmed that it is going up. Sometimes it bounces back but never to the point of right after the system reboot. Log excerpt:

[Thu Nov 23 09:00:49 EST 2023] sfc.exe 4668 Services 0 694,124 K
[Thu Nov 23 09:01:10 EST 2023] sfc.exe 4668 Services 0 168,744 K
...
[Thu Nov 23 09:02:11 EST 2023] sfc.exe 4668 Services 0 357,884 K
...
[Thu Nov 23 09:19:30 EST 2023] sfc.exe 4668 Services 0 612,476 K
...
[Thu Nov 23 10:00:37 EST 2023] sfc.exe 4668 Services 0 778,676 K
[Thu Nov 23 10:00:58 EST 2023] sfc.exe 4668 Services 0 155,044 K
...
[Thu Nov 23 10:01:39 EST 2023] sfc.exe 4668 Services 0 385,624 K
...
[Thu Nov 23 12:00:36 EST 2023] sfc.exe 4668 Services 0 847,284 K

Can you, please, help me to resolve this issue?

Versions used:

Cisco Secure Client 5.0.00837  

(Thu Nov 23 14:13:01 2023)

Agent

Version: 8.2.1.21612

Windows version:
Edition Windows 10 Enterprise
Version 22H2
Installed on ‎Thu ‎02/‎11/‎2023
OS build 19045.2006
Experience Windows Feature Experience Pack 120.2212.4180.0

[korniiva-flutterint]

I have found a workaround. It is pretty strange.

I open Cisco Secure Client, press a cog icon (Advanced Window), go to Update tab and press "Update Now" button. After this the memory consumption resets.

Edit: The memory usage grows back after some time so I am still looking for a permanent solution instead of a workaround.

[Matthew Franks]

Glad you found a workaround, but is there any chance you could open a TAC case and provide some logs so we can investigate the issue further? 

Thanks,

-Matt

[korniiva-flutterint]

I would love to but I don't know how to open a TAC and I don't know where the sfc logs are located. Can you give me the instructions, please? I will then open a ticket and retrieve the logs.

[Matthew Franks]

You can open a TAC case here. To generate a Windows diagnostic bundle on your desktop, you can run the ipsupporttool.exe file in C:\Program Files\Cisco\AMP\YOUR_VERSION_NUMBER\ipsupporttool.exe. After running this, you should see a CiscoAMP_Support_Tool_2023_x_x.zip file on your desktop.

Thanks,

-Matt

[dustinn3]

We're having the same problem on at least one VM running Citrix.  It was using around 4 GB, but when I clicked update it drops to 0.  However after a few minutes it's already back up to a couple of GB. 

[dotran]

Been seeing this issue across all of our Windows 2019  Servers since 8.21.   I had already open 1 TAC case and added some exclusion but has not fixed the issue.   Will open another TAC case.  This looks to be a memory leak somewhere.

[dotran]

I believe the TAC case is private to our company.  After reviewing the debug information,  we believe the problem was due our use EDR/XDR agent use for vulnerability scanning.   I noticed after each vulnerability scan,  the memory would increase by 200-300KB after each scan, never fully releasing the memory.   Once we excluded the EDR/XDR agent,  I saw a BIG improvement in memory usage.  The memory still increased by 200-300KB after the 1st scan but it didn't increase after subsequent scans.   NOTE:  I didn't realize Cisco Secure Endpoint (AMP) requires 2GB just for their agent.  It's in the fineprint.   Also,  by doing a Policy Sync, I think it forces the agent to reload thereby clearing out any cache memory.   TLDR:  Submit a TAC case if you have a problem and submit a debug to determine what program may be causing the problem.

[dotran]

The problem returned so the fix above did not work.  This is definintely a memory leak.   If you update the Policy from the Amp console,  SFC.EXE will reload and clear the memory as a temporary fix (or restart the service).   There is a now CISCO BUG CASE on the issue.  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwi78497

[BobbyQuiason82578]

Hi, we are also seeing this issue on our Windows 2016 server running CSE 8.2.1. It is mainly running our Solarwinds environment. We have excluded only the Solarwinds directory for now. Any others to consider? Is the TAC case public and can be viewed by others or should I open a separate one?

Thanks.

[Roman Valenta]

Just a quick update on this issue.

Our DEV team was  able to identify the issue and also what seems to be successful replication in the LAB environment since we had a hard time to collect relevant logs due to its nature.

Currently they are in process of composing a new built that will then be tested internally for any crashes or anomalies, before we can provide the test release to customers that would like to verify  and test the patched built in their own environment.

I will keep you posted on the progress, those with active open TAC cases and once the availability is confirmed can request this test release through their TAC case.

[korniiva-flutterint]

Thanks for looking into it!

[Roman Valenta]

OK guys our DEV's  team worked their tail off on this one and we got test release available as of today. For those that have active ticket open with TAC for this issue regarding 8.2.1 release. You can now request that release which is CSE v8.2.3.30115 to be enabled under your ORG for TESTING purposes. Since this not official release please do not deploy in production but rather test on few troubled endpoints.

[Roman Valenta]

Another quick update. We start receiving positive feedback about the test release from customers that agreed to try it out. This release seem to fix the memory allocation issue. Expecting the patch connector to be released very soon , will report again once that happened.

[Roman Valenta]

Last update:

We are planning to release Cisco Secure Endpoint Windows Connector 8.2.3.30119 to production today. Please watch your portals for the updates.

Among other enhancements this release also includes fix for an issue where memory usage increases over time under certain conditions.