cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
12577
Views
28
Helpful
31
Replies

CSE 8.0.1 and 8.1.3 causing significant slowdown after 4-5 days uptime

ac513
Level 1
Level 1

A few months back, we deployed Cisco Secure Endpoint 8.0.1 to our pilot Windows group and then our production Windows group. In testing there were seemingly no issues; However, eventually we began to notice that after these upgrades, Windows 10 (mostly 21H2) and Windows 11 devices (mostly 21H2, some 22H2) would begin to show significant symptoms of slowness including (but not limited to):

* Window dragging/resizing becomes slow. As in, if all other system animations and video playback are displaying at 60fps as per usual, then window dragging/resizing looks like it's going at about 10-20fps. Just choppy and super unresponsive.

* Print jobs could sometimes take several minutes to process, sometimes fail due to vague memory problems in error messages.

* Opening emails in Outlook could take an entire second or two, rather than instant or mere milliseconds.

* Other LoB software at random taking a very long time to respond/work.

When these slowdowns are happening, there are no obvious resource constraints. For example on my own machine -- 20-30% of my i7-9700k CPU in use (not abnormal because I run tons of programs and VMs), 50% of 32GB RAM free, and marginal disk activity on a Samsung EVO 970 SSD. This is normal for my system, and I never see CPU/RAM/disk anywhere close to being maxed out when these slowdowns occur.

A simple reboot will eliminate the issues for some time, but once the machine has 4-5 or more days of uptime again, all slowdowns return.

All of these symptoms immediately cease and stay resolved if I uninstall Cisco Secure Endpoint.

As a test, I got a version 8.1.3 connector installer. I removed Secure Endpoint on an affected machine (mine, Win11 22H2), and then installed 8.1.3. After 4-5 days, the same slowdown symptoms begin again.

I saw that 8.1.3's release notes mentioned a few fixes related to performance/memory leaks, but none of them seemingly had any effect on these symptoms in my test case.  https://docs.amp.cisco.com/Release%20Notes.pdf

So far as policy options, here is what we configure. No functional changes since July 2021 when we enabled Behavioral Protection, everything we have has been in place for over a year with no issues on version 7 clients.

ac513_0-1670021931186.png

 

Is anyone else seeing this behavior with the version 8 clients for Secure Endpoint?

 

31 Replies 31

MakoWish
Level 1
Level 1

Just wanted to chime in and say we are also experiencing this issue with 8.1.3. In my case, I have a very beefy Dell Precision T5810 with an nVidia Quadro M4000, so to see choppiness when dragging a window is quite shocking. The issue starts for me after only a couple hours of uptime. I can confirm killing the tray UI does resolve the issue. 

For any issues related to this I would highly suggest open a TAC case. I would also recommend to check for the exclusion and just verify that it is presented which you should be able to get from the policy.xml (only Admin will be able to do that unless it's otherwise allowed).

 

You should see this in the policy.xml

 

<exclude>csc_ui.exe</exclude> <<< ----------------------------------- Global Exclusion

 

additionally I would gather some other helpful info such as, how many computers are being impacted, full procmon logs with time stamps before and after turning the UI off. Recording during WebEx session with your engineer will be also helpful.

 

This issue will be now investigated on a separate manner individually case by case to figured out what is causing this slow down that is related to the User Interface (UI)

 

 

Thanks for confirming it's being seen elsewhere, so hopefully this gets additional traction. Still working with TAC and collecting procmon captures, but what I've found on my own since my last post:

1) Issue continues to be consistently reproducible with Secure Endpoint 8.1.3 on a clean, unmanaged Windows 11 device. (not on our org network, no org software/configs, etc)

2) Issue also reproducible with 8.1.5 (released days ago), which I suspected as the Secure Client UI version did not change (5.0.00622).

Same as you've seen and as I've mentioned before, closing the Secure Client UI/tray resolves all issues.

ac513
Level 1
Level 1

Update from TAC:

 

Just wanted to let you know that Internal Team contacted to Devs and they are now working together on this issue. Devs are working deeper with the Exploit Prevention engine and its interaction with the csc_ui.exe process, so that they can isolate it and check if this is independent of this engine or not and perform the next internal tests.

Fortunately, the setup of the internal lab was successfully, this is now reproducible for the Devs and Internal team requests and the ongoing tests.

Thank you for your time while the Internal Team and Devs are managing this ongoing investigation.


 

Thank you for the follow-up

Roman Valenta
Cisco Employee
Cisco Employee

This is correct we successfully reproduced this issue and currently working with developers to get this fixed. For anyone that is still experiencing this issue while the UI is active, please verify this filed in Task Manager.

 

Screenshot_2529.png


What we noticed when endpoint was experiencing this slowness was that GDI Objects was hovering in thousands. The other similarity was that most burden was captured while working with MS apps like Outlook, Word Excel.

I will post to this room once again when I have more details on this ongoing investigation.

 

What should the GDI objects be sitting around on a known good system? We are having similar issues with our clients but when I look at a couple of clients they are around 970. Also is it still recommended to not show the UI then?

KingJ23_0-1678971706758.png

 



Matthew Franks
Cisco Employee
Cisco Employee

Ideally the number of GDI objects would be around 200.  From what we've seen, slowness is experienced when it gets to around 6,000.  We have identified the issue and are working on a release containing the fix. Should be in the next release (8.1.7) unless further obstacles are encountered during testing.

-Matt

Thanks for that info Matt. We are starting to see clients reach 5,900 GDI objects and slowing down. I thought I read that stopping the UI from loading helps this, is that still true as a workaround till 8.1.7 comes out?

@KingJ-23 That is correct.  The workaround is to stop the UI.

-Matt

ac513
Level 1
Level 1

The few I've checked start in the 900s with a clean start of csc_ui.exe.

Glad to hear about the impending release of 8.1.7 including a fix, fingers crossed it puts a pin in this!

 

Roman Valenta
Cisco Employee
Cisco Employee

Just a quick update:

Now more officially the permanent fix is going out in the 8.1.7 release as Matthew mentioned above, which is currently planed for release in mid April.   

Also for more official statement if anyone is looking the externally facing bug can be found:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwe72861 

 

 

Thank You for the update. I can't wait for mid April to approach.

Also excited for the production release of 8.1.7. Currently internally testing a pre-release 8.1.7 through our TAC case, and with 2 days of uptime I'm seeing csc_ui.exe's GDI object count still at 148 and not increasing. Hoping this continues and that it's a solid fix!

Thanks for being the tip of the spear on that mess! I'm waiting to roll my full population to 8.x for this very reason.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: