topic Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up in Endpoint Security

CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days uptime

ac513 — Fri, 02 Dec 2022 23:01:31 GMT

A few months back, we deployed Cisco Secure Endpoint 8.0.1 to our pilot Windows group and then our production Windows group. In testing there were seemingly no issues; However, eventually we began to notice that after these upgrades, Windows 10 (mostly 21H2) and Windows 11 devices (mostly 21H2, some 22H2) would begin to show significant symptoms of slowness including (but not limited to):

* Window dragging/resizing becomes slow. As in, if all other system animations and video playback are displaying at 60fps as per usual, then window dragging/resizing looks like it's going at about 10-20fps. Just choppy and super unresponsive.

* Print jobs could sometimes take several minutes to process, sometimes fail due to vague memory problems in error messages.

* Opening emails in Outlook could take an entire second or two, rather than instant or mere milliseconds.

* Other LoB software at random taking a very long time to respond/work.

When these slowdowns are happening, there are no obvious resource constraints. For example on my own machine -- 20-30% of my i7-9700k CPU in use (not abnormal because I run tons of programs and VMs), 50% of 32GB RAM free, and marginal disk activity on a Samsung EVO 970 SSD. This is normal for my system, and I never see CPU/RAM/disk anywhere close to being maxed out when these slowdowns occur.

A simple reboot will eliminate the issues for some time, but once the machine has 4-5 or more days of uptime again, all slowdowns return.

All of these symptoms immediately cease and stay resolved if I uninstall Cisco Secure Endpoint.

As a test, I got a version 8.1.3 connector installer. I removed Secure Endpoint on an affected machine (mine, Win11 22H2), and then installed 8.1.3. After 4-5 days, the same slowdown symptoms begin again.

I saw that 8.1.3's release notes mentioned a few fixes related to performance/memory leaks, but none of them seemingly had any effect on these symptoms in my test case. https://docs.amp.cisco.com/Release%20Notes.pdf

So far as policy options, here is what we configure. No functional changes since July 2021 when we enabled Behavioral Protection, everything we have has been in place for over a year with no issues on version 7 clients.

Is anyone else seeing this behavior with the version 8 clients for Secure Endpoint?

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

Roman Valenta — Sat, 03 Dec 2022 16:06:13 GMT

Please try this workaround to determine if this slowdown is related to the Secure Client UI component which is the graphical interface for Secure Endpoint.

Manually on one of your endpoints with Secure Endpoint 8.x.x installed that is experiencing this drastic slowdown navigate to Windows Tool Bar and in the left bottom corner click on the connector UI icon and select QUIT.

Navigate to your policy under Management --- > Policies --- > Advanced Settings --- > Client User Interface (UI) and uncheck Start Client User Interface. Please note that this will take affect on next reboot.

By quitting this graphical interface you are not affecting functionality of secure endpoint and everything remains the same with all services running and still protecting your endpoint.

If you see improvement or even complete recovery and no more issues please test one more thing by re-enabling the User UI and adding this ExPrev exclusion and apply this exclusion to your policy. Please make sure you sync the policy and either reboot your endpoint or restart the Secure Client for the ExPrev Policy to take the affect.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Mon, 05 Dec 2022 15:21:29 GMT

Thanks for the advice! I'll have to get the client back on my devices and wait some number of days, and get others to try this out as we get new reports.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Tue, 06 Dec 2022 18:09:29 GMT

Just heard back from a few test cases, and they claim that the slowdowns stopped as soon as the systray menu was closed out. Will have to add that exclusion next and see if it continues to help.

That's... a wildly problematic bug it sounds like. Surely this is a known bug and Cisco is working on it? (or at least I hope so, considering this suggestion was at the ready)

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

Roman Valenta — Tue, 06 Dec 2022 22:34:13 GMT

Yes we are currently actively working on this issue but so far we have very limited data collected as this issue seems to be sporadic and triggered differently. For example in your case you reported it takes couple days to get to that state, some customers reported they can get to that state in few hours and we also have customers that have no issue at all.

If the exploit prevention exclusion will not work for you I would suggest keep the UI disabled in the policy to prevent this issue. You can also open TAC case and get update through the case notes or stay put and I will update this thread as soon as I hear back about any new progress on this issue.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

LudoD — Mon, 12 Dec 2022 23:28:29 GMT

Hi,

Same problem here.

Try the exclusion but do not work on long term.
Just close the UI on a computer with the problem (and the exclusion), it solves the problem instantly.

I suppose the user haven't notifications when UI is closed...

Kind regards,
Ludo

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

Roman Valenta — Tue, 13 Dec 2022 18:27:23 GMT

Correct, NO UI = Notification.

BTW: Lots of customers have the notification disabled anyway to not disturbed the end user.

Now as far for the latest, we have so far great success with several customers where we implemented exclusion to Script Control via back end as that's the only solution for Script Control. And based on those test it seems that Script Control which is separate from Exploit Prevention causing this. If things goes well we should have global fix released with in a week. Again there is no set date on this and once I gain more inlet I will update this thread.

If you like to test this out you can try disable Script Control on small test group of computers to see the difference.

PS: Once the policy synchronize make sure to either restart the Secure Client or reboot the endpoint for the changes to take effect.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Tue, 13 Dec 2022 19:04:40 GMT

If you like to test this out you can try disable Script Control on small test group of computers to see the difference.

Would it need to be literally set to Disabled, or would Audit also provide the same results in a test case? (if unknown, that's fine)

Also as a heads up to everyone -- The Windows default Cisco-managed exclusions will be updated tomorrow as well. They include the system tray, plus a few extra things.

https://www.cisco.com/c/en/us/support/docs/security/amp-endpoints/214809-cisco-maintained-exclusion-list-changes.html

C:\Windows\System32\omadmclient.exe
.automaticDestinations-ms
csc_ui.exe for Exploit Prevention

Lastly, had one very weird symptom. Using 8.1.3 and all scanning modes set to Audit, I noticed that Secure Endpoint was keeping a hold onto M365 Apps for enterprise. I was trying to manually update the package, and it kept saying "apps can't close". Removed Secure Endpoint, and it kicked right through and updated. Closing the systray menu did nothing for this one. First I've seen that, and I don't know if the above exclusions will account for this.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

Roman Valenta — Wed, 14 Dec 2022 17:14:48 GMT

As I promised here is the newest update:

The Engineering Team would be adding a ExPrev and a Script Control Exclusion for csc_ui.exe for all the regions later today

EU, APJC, NAM are scheduled for 11:00am MDT

Patch release notes:

In policy.xml, csc_ui.exe is being added to exclude_app_list and script_control exclude list globally / by default in the exprev settings

This was identified as one of the possible reasons for the Windows Performance Issue related to the new Secure Client UI (8.x and above) Since this is a trusted process, we see no harm in excluding this.

This will still apply:

Once the policy synchronize make sure to either restart the Secure Client or reboot the endpoint for the changes to take effect.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Wed, 14 Dec 2022 23:15:26 GMT

We've pushed 8.1.3 to our production devices that were currently still on 8.0.1. Devices should be rebooting soon (Windows Updates) so the new exclusions can take effect as well to go along with the newer client.

Here's hoping we see some better results out there!

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Tue, 03 Jan 2023 14:23:31 GMT

Unfortunately, not much change in behavior has been observed with both 1) client version 8.1.3, and 2) the new Cisco-maintained exclusions that included csc_ui.exe.

For example, I left my machine running over the weekend after our holiday. Coming in this morning with 4 days of uptime, window dragging was back to a choppy 20-ish fps and Outlook items were taking 2 or more entire seconds to open. These problems immediately went away upon closing & reopening the Cisco Secure Client system tray icon.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

Roman Valenta — Wed, 04 Jan 2023 18:52:50 GMT

At this point I would suggest to open TAC case as we will need to investigate this further and gather some additional logs. Diagnostic Bundle will be one of them, but there will be some Windows Internal Tools that we need to run as well to get idea what is going with the operating system. Procmon, RAMMap, vmmap will be one of those tools that we will need to utilize to gather more Intel on this.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Thu, 12 Jan 2023 16:36:48 GMT

I've opened a TAC case. Right now they're just looking at DART bundles for individual devices, not sure if our case is being linked up with others' in regards to this issue.

Has there been any other updates you can share from the engineering team? I see the ExPrev/Script Control global exclusions for csc_ui.exe were stated to be "one of the possible reasons", so are there other pieces still being actively investigated for this issue? Anything we can look at and try to rule out? (If you're able to divulge that info, that is)

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

Roman Valenta — Fri, 13 Jan 2023 22:39:54 GMT

Yes this will actually be investigated individually case by case. Majority customers with this issue reported that after we apply the fix globally their issue was mitigated. In such a cases like yours we will keep investigate and we will collect some additional information including recording of the issue for our DEV team. Also, I did found your case and already spoke with the engineer assigned to your case what will be the next steps. Thank you for you patience.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Sat, 21 Jan 2023 23:56:17 GMT

Appreciate that! Still running 3 test scenarios to report back in our TAC case, but for anyone else following this thread/situation and was curious, here's some smoking guns I've found:

1) The slowdowns are reproducible on a Win11 22H2 machine of mine at home, with nothing of significance installed/running on it in the background. Based on this, I'd feel confident in saying the slowdowns are not an interaction between Secure Endpoint and any of our organization's configurations, deployed apps, policies, etc.

2) On my primary company machine where this has been reproducible, the issue goes away entirely when moving it to a policy with the Secure Client UI disabled and all other settings matching our usual production policy.

So the issue is ongoing, and this pretty much solidifies the Secure Client UI (csc_ui.exe) being the common denominator to me. (I know previously in this thread, it was stated to be "suspected") Now the exact root cause, who knows...

Here's hoping a new client release may be on the horizon with some improvements. I was watching our console like a hawk, and noticed a version 8.1.5 client become available for our policies. However, it disappeared less than an hour later and never came back. Not sure if this was a glitch in automation, or if a planned release was delayed at the last minute?

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

MakoWish — Fri, 27 Jan 2023 15:52:37 GMT

Just wanted to chime in and say we are also experiencing this issue with 8.1.3. In my case, I have a very beefy Dell Precision T5810 with an nVidia Quadro M4000, so to see choppiness when dragging a window is quite shocking. The issue starts for me after only a couple hours of uptime. I can confirm killing the tray UI does resolve the issue.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

Roman Valenta — Fri, 27 Jan 2023 16:27:52 GMT

For any issues related to this I would highly suggest open a TAC case. I would also recommend to check for the exclusion and just verify that it is presented which you should be able to get from the policy.xml (only Admin will be able to do that unless it's otherwise allowed).

You should see this in the policy.xml

<exclude>csc_ui.exe</exclude> <<< ----------------------------------- Global Exclusion

additionally I would gather some other helpful info such as, how many computers are being impacted, full procmon logs with time stamps before and after turning the UI off. Recording during WebEx session with your engineer will be also helpful.

This issue will be now investigated on a separate manner individually case by case to figured out what is causing this slow down that is related to the User Interface (UI)

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Tue, 07 Feb 2023 15:01:27 GMT

Thanks for confirming it's being seen elsewhere, so hopefully this gets additional traction. Still working with TAC and collecting procmon captures, but what I've found on my own since my last post:

1) Issue continues to be consistently reproducible with Secure Endpoint 8.1.3 on a clean, unmanaged Windows 11 device. (not on our org network, no org software/configs, etc)

2) Issue also reproducible with 8.1.5 (released days ago), which I suspected as the Secure Client UI version did not change (5.0.00622).

Same as you've seen and as I've mentioned before, closing the Secure Client UI/tray resolves all issues.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

ac513 — Wed, 15 Mar 2023 14:00:04 GMT

Update from TAC:

Just wanted to let you know that Internal Team contacted to Devs and they are now working together on this issue. Devs are working deeper with the Exploit Prevention engine and its interaction with the csc_ui.exe process, so that they can isolate it and check if this is independent of this engine or not and perform the next internal tests.
Fortunately, the setup of the internal lab was successfully, this is now reproducible for the Devs and Internal team requests and the ongoing tests.
Thank you for your time while the Internal Team and Devs are managing this ongoing investigation.

Re: CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days up

LudoD — Wed, 15 Mar 2023 14:36:19 GMT

Thank you for the follow-up