CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days uptime

[ac513]

A few months back, we deployed Cisco Secure Endpoint 8.0.1 to our pilot Windows group and then our production Windows group. In testing there were seemingly no issues; However, eventually we began to notice that after these upgrades, Windows 10 (mostly 21H2) and Windows 11 devices (mostly 21H2, some 22H2) would begin to show significant symptoms of slowness including (but not limited to):

* Window dragging/resizing becomes slow. As in, if all other system animations and video playback are displaying at 60fps as per usual, then window dragging/resizing looks like it's going at about 10-20fps. Just choppy and super unresponsive.

* Print jobs could sometimes take several minutes to process, sometimes fail due to vague memory problems in error messages.

* Opening emails in Outlook could take an entire second or two, rather than instant or mere milliseconds.

* Other LoB software at random taking a very long time to respond/work.

When these slowdowns are happening, there are no obvious resource constraints. For example on my own machine -- 20-30% of my i7-9700k CPU in use (not abnormal because I run tons of programs and VMs), 50% of 32GB RAM free, and marginal disk activity on a Samsung EVO 970 SSD. This is normal for my system, and I never see CPU/RAM/disk anywhere close to being maxed out when these slowdowns occur.

A simple reboot will eliminate the issues for some time, but once the machine has 4-5 or more days of uptime again, all slowdowns return.

All of these symptoms immediately cease and stay resolved if I uninstall Cisco Secure Endpoint.

As a test, I got a version 8.1.3 connector installer. I removed Secure Endpoint on an affected machine (mine, Win11 22H2), and then installed 8.1.3. After 4-5 days, the same slowdown symptoms begin again.

I saw that 8.1.3's release notes mentioned a few fixes related to performance/memory leaks, but none of them seemingly had any effect on these symptoms in my test case.  https://docs.amp.cisco.com/Release%20Notes.pdf

So far as policy options, here is what we configure. No functional changes since July 2021 when we enabled Behavioral Protection, everything we have has been in place for over a year with no issues on version 7 clients.

Is anyone else seeing this behavior with the version 8 clients for Secure Endpoint?

[Roman Valenta]

Please try this workaround to determine if this slowdown is related to the Secure Client UI component which is the graphical interface for Secure Endpoint.

Manually on one of your endpoints with Secure Endpoint 8.x.x installed that is experiencing this drastic slowdown  navigate to Windows Tool Bar and in the left bottom corner click on the connector UI icon and select QUIT.

or

Navigate to your policy under Management --- > Policies --- > Advanced Settings --- > Client User Interface (UI)  and uncheck Start Client User Interface. Please note that this will take affect on next reboot.

By quitting this graphical interface you are not affecting functionality of secure endpoint and everything remains the same with all services running and still protecting your endpoint.

If you see improvement or even complete recovery and no more issues please test one more thing by re-enabling the User UI and adding this ExPrev exclusion and apply this exclusion to your policy. Please make sure you sync the policy and either reboot your endpoint or restart the Secure Client for the ExPrev Policy to take the affect.

[ac513]

Thanks for the advice! I'll have to get the client back on my devices and wait some number of days, and get others to try this out as we get new reports.

[ac513]

Just heard back from a few test cases, and they claim that the slowdowns stopped as soon as the systray menu was closed out. Will have to add that exclusion next and see if it continues to help.

That's... a wildly problematic bug it sounds like. Surely this is a known bug and Cisco is working on it? (or at least I hope so, considering this suggestion was at the ready)

[Roman Valenta]

Yes we are currently actively working on this issue but so far we have very limited data collected as this issue seems to be sporadic and triggered differently. For example in your case you reported it takes couple days to get to that state,  some customers reported they can get to that state in few hours and we also have customers that have no issue at all.

If the exploit prevention exclusion will not work for you I would suggest keep the UI disabled in the policy to prevent this issue. You can also open TAC case and get update through the case notes or stay put and I will update this thread as soon as I hear back about any new progress on this issue.

[LudoD]

Hi,

Same problem here.

Try the exclusion but do not work on long term.
Just close the UI on a computer with the problem (and the exclusion), it solves the problem instantly.

I suppose the user haven't notifications when UI is closed...

Kind regards,
Ludo

[Roman Valenta]

Correct, NO UI = Notification.

BTW: Lots of customers have the notification disabled anyway to not disturbed the end user.

Now as far for the latest, we have so far great success with several customers where we implemented exclusion to Script Control via back end as that's the only solution for Script Control. And based on those test it seems that Script Control which is separate from Exploit Prevention causing this. If things goes well we should have global fix released with in a week. Again there is no set date on this and once I gain more inlet I will update this thread.

If you like to test this out you can try disable Script Control on small test group of computers to see the difference.

PS: Once the policy synchronize make sure to either restart the Secure Client or reboot the endpoint for the changes to take effect.

[ac513]

---

If you like to test this out you can try disable Script Control on small test group of computers to see the difference.

Would it need to be literally set to Disabled, or would Audit also provide the same results in a test case? (if unknown, that's fine)

Also as a heads up to everyone -- The Windows default Cisco-managed exclusions will be updated tomorrow as well. They include the system tray, plus a few extra things.

https://www.cisco.com/c/en/us/support/docs/security/amp-endpoints/214809-cisco-maintained-exclusion-list-changes.html

• C:\Windows\System32\omadmclient.exe
• .automaticDestinations-ms
• csc_ui.exe for Exploit Prevention

Lastly, had one very weird symptom. Using 8.1.3 and all scanning modes set to Audit, I noticed that Secure Endpoint was keeping a hold onto M365 Apps for enterprise. I was trying to manually update the package, and it kept saying "apps can't close". Removed Secure Endpoint, and it kicked right through and updated. Closing the systray menu did nothing for this one. First I've seen that, and I don't know if the above exclusions will account for this.

[Roman Valenta]

As I promised here is the newest update:

The Engineering Team would be adding a ExPrev and a Script Control Exclusion for csc_ui.exe for all the regions later today

EU, APJC, NAM are scheduled for 11:00am MDT

Patch release notes:

In policy.xml, csc_ui.exe is being added to exclude_app_list and script_control exclude list globally / by default in the exprev settings

This was identified as one of the possible reasons for the Windows Performance Issue related to the new Secure Client UI (8.x and above) Since this is a trusted process, we see no harm in excluding this.

This will still apply:

Once the policy synchronize make sure to either restart the Secure Client or reboot the endpoint for the changes to take effect.

[ac513]

We've pushed 8.1.3 to our production devices that were currently still on 8.0.1.  Devices should be rebooting soon (Windows Updates) so the new exclusions can take effect as well to go along with the newer client.

Here's hoping we see some better results out there! 

[ac513]

Unfortunately, not much change in behavior has been observed with both 1) client version 8.1.3, and 2) the new Cisco-maintained exclusions that included csc_ui.exe.

For example, I left my machine running over the weekend after our holiday. Coming in this morning with 4 days of uptime, window dragging was back to a choppy 20-ish fps and Outlook items were taking 2 or more entire seconds to open.  These problems immediately went away upon closing & reopening the Cisco Secure Client system tray icon.

[Roman Valenta]

At this point I would suggest to open TAC case as we will need to investigate this further and gather some additional logs. Diagnostic Bundle will be one of them, but there will be some Windows Internal Tools that we need to run as well to get idea what is going with the operating system. Procmon, RAMMap, vmmap will be one of those tools that we will need to utilize to gather more Intel on this.

[ac513]

I've opened a TAC case. Right now they're just looking at DART bundles for individual devices, not sure if our case is being linked up with others' in regards to this issue.

Has there been any other updates you can share from the engineering team? I see the ExPrev/Script Control global exclusions for csc_ui.exe were stated to be "one of the possible reasons", so are there other pieces still being actively investigated for this issue? Anything we can look at and try to rule out? (If you're able to divulge that info, that is)

[Roman Valenta]

Yes this will actually be investigated individually case by case. Majority customers with this issue reported that after we apply the fix globally their issue was mitigated. In such a cases like yours we will keep investigate and we will collect some additional information including recording of the issue for our DEV team. Also, I did found your case and already spoke with the engineer assigned to your case what will be the next steps. Thank you for you patience.

[ac513]

Appreciate that! Still running 3 test scenarios to report back in our TAC case, but for anyone else following this thread/situation and was curious, here's some smoking guns I've found:

1) The slowdowns are reproducible on a Win11 22H2 machine of mine at home, with nothing of significance installed/running on it in the background. Based on this, I'd feel confident in saying the slowdowns are not an interaction between Secure Endpoint and any of our organization's configurations, deployed apps, policies, etc.

2) On my primary company machine where this has been reproducible, the issue goes away entirely when moving it to a policy with the Secure Client UI disabled and all other settings matching our usual production policy.

So the issue is ongoing, and this pretty much solidifies the Secure Client UI (csc_ui.exe) being the common denominator to me. (I know previously in this thread, it was stated to be "suspected") Now the exact root cause, who knows...

Here's hoping a new client release may be on the horizon with some improvements. I was watching our console like a hawk, and noticed a version 8.1.5 client become available for our policies. However, it disappeared less than an hour later and never came back. Not sure if this was a glitch in automation, or if a planned release was delayed at the last minute?