topic Re: Cisco Secure Endpoint Service Stopped in Endpoint Security

Cisco Secure Endpoint Service Stopped

mandrews — Tue, 21 Jun 2022 16:34:37 GMT

Good day all!

From time to time, I find that there are several of our machines that have their service stopped with Secure Endpoint. I haven't found what has been stopping it, but has anyone seen this and know what has been causing this? And is there a way to detect machines whose service has been stopped from the console?

Thank you,

Maurice

Re: Cisco Secure Endpoint Service Stopped

JennieZhang — Mon, 04 Jul 2022 15:18:15 GMT

hello,

do you mean from time to time you find 'Cisco Secure Endpoint' service was in a 'stopped' status?

have you ticked the checkbox of 'enable connector protection'? you can find this option under 'advanced settings' -> 'administrative features' of your policy.

This feature can prevent malware, application or user from disabling secure endpoint service.

Re: Cisco Secure Endpoint Service Stopped

mandrews — Tue, 05 Jul 2022 21:22:25 GMT

Hey Jennie! Thanks for responding. Yes, that's exactly what I mean on the services being stopped. I've found a few machines in the environment where it was stopped and it wasn't in positioned to be scanned or anything. In regards to your question about the connector protection, yes, we do have that turned on as well. I'm not sure if there's anything that's stopping it outside of that or if anyone has experienced this happening consistently. Outside of having people open CSE on their computers, I'm not sure if there's a way to check from the console if the services have stopped.

Re: Cisco Secure Endpoint Service Stopped

newberntac — Wed, 13 Jul 2022 15:27:11 GMT

Is there any update on this? We have the same issue, to the tune of 25% of our systems at a time. And having to connect to EACH one just to restart the service is a time killer.

Re: Cisco Secure Endpoint Service Stopped

mandrews — Wed, 13 Jul 2022 17:30:35 GMT

Hey! Unfortunately, the closest solution that has been recommended was the connector protection. Unfortunately, that doesn't keep the connector services from stopping. I'm still not sure what is happening to cause the service to stop, but it's something I'd like to get to the bottom of to make sure that our environment is secured.

Re: Cisco Secure Endpoint Service Stopped

DaphneG — Wed, 13 Jul 2022 19:23:22 GMT

@mandrews, I suggest checking the Secure Endpoint directory for crash dumps then opening a TAC case. If you have an open SR, feel free to PM me the number so I can review/follow up with the TAC engineer.

Re: Cisco Secure Endpoint Service Stopped

mandrews — Wed, 13 Jul 2022 19:54:28 GMT

Hey Daphne!

Do you know the file path or the file name for the file that would have the crash dump? I'm assuming that it would be a temp file.

Re: Cisco Secure Endpoint Service Stopped

newberntac — Wed, 13 Jul 2022 20:11:23 GMT

I found this on setting the client up to collect debug info. We're going to try and gather debugs too.

https://www.cisco.com/c/en/us/support/docs/security/amp-endpoints/216035-collect-debug-logs-file-in-amp-for-endpo.html

Re: Cisco Secure Endpoint Service Stopped

mandrews — Wed, 13 Jul 2022 21:22:19 GMT

I normally use debugs when CSE is using a lot of CPU utilization from scanning. I don't know if it'll reveal what is kicking off the connector, unless it scanning something too much is what stops the service.

Re: Cisco Secure Endpoint Service Stopped

DaphneG — Wed, 13 Jul 2022 21:31:06 GMT

You should see it under Program Files > Cisco > AMP. However, I forgot to mention earlier that there's a setting that'll determine if the dump will be written and saved locally or sent to the cloud. This setting, called "Automatic Crash Dump Uploads", can be found under Policy > Advanced Settings > Administrative Features. The "Connector Log Level" will also be in the same page so I recommend setting the log level to Debug then disabling the Automated Crash Dump Upload and see if there's a .dmp file created under the AMP directory once the issue reoccurs.

Re: Cisco Secure Endpoint Service Stopped

DaphneG — Wed, 13 Jul 2022 23:44:51 GMT

With the debugging enabled, it provides us more context and enables us to correlate things with the dump.

Btw, if after following the recommendations below you're still not seeing any .dmp files, please open a TAC case and use this discussion as reference.

Re: Cisco Secure Endpoint Service Stopped

mandrews — Thu, 14 Jul 2022 17:32:37 GMT

That feature makes sense now. Assuming that we have crash dump logs sent to the cloud, do our organization have access to that? Also, is there a way to detect a stopped service with CSE from the console?

Re: Cisco Secure Endpoint Service Stopped

Vibhor Amrodia — Wed, 07 Sep 2022 18:22:11 GMT

Hello Maurice,

I wanted to know if you were able to file a TAC Case for this issue so that we can investigate this issue further.

To answer your queries above:

1) No, unfortunately, these logs(Crash Dumps) are not to the customers

2) You should be able to detect stopped services on the endpoint using Orbital as a probe. On the portal, the "Last Seen" Date would be the only indicator to help you detect stopped service on the Endpoints.

Thanks,

Vibhor

Re: Cisco Secure Endpoint Service Stopped

mandrews — Wed, 21 Sep 2022 21:57:09 GMT

Hey Vibhor!

No, I haven't opened a TAC case for this yet. I haven't spotted any lately, but maybe your suggestion using Orbital may help. What query can I use to detect the stopped CSE service? If that returns any results, then I'll use that evidence and submit a TAC case with it.

Re: Cisco Secure Endpoint Service Stopped

UMontero — Wed, 21 Sep 2022 22:12:06 GMT

To further add on Vibhor's reply, you can also use Powershell to get a list of Services and their status

Get-Service -Name Cisco* | ft -auto

If you want the results in a txt file, please use Get-Service -Name Cisco* | ft -auto | Out-File "C:\Users\insertusernamehere\Desktop\cisco.txt"

Re: Cisco Secure Endpoint Service Stopped

joljol — Thu, 12 Jan 2023 22:42:46 GMT

Any update on this? We are seeing similar things. In December we noticed the service was stopped on a bunch of (Windows) servers. Services seemed to never have started after an automatic server restart (windows update) and we only noticed because we have monitoring of services on this particular customers servers. So it really stood out on our monitoring-dashboard that the services were not running on all these servers.

A few days ago, another customer also noted stopped AMP services on some of his servers and also on some Windows clients.

Both customers are running v. 7.5.x. Maybe version 8 is better?

Re: Cisco Secure Endpoint Service Stopped

mandrews — Fri, 13 Jan 2023 21:29:20 GMT

Not really. I haven't been discovering many stopped services anymore. It helps to audit the console every month or so to see if there are computers consistently not being seen for over a week or a month. The script that was mentioned earlier in the thread is helpful if you can find a way to run it across your network. We have been updating our versions, so maybe it was the 7.5.5 version that wasn't performing. Either way, if I stumble across something else, then I'll let you know.

Re: Cisco Secure Endpoint Service Stopped

joljol — Mon, 16 Jan 2023 19:58:26 GMT

Thank you for replying. I will try and convince our customer to update to version 8 of the connector, and instruct him on how to setup and gather debug logs, in case it happens again.

Re: Cisco Secure Endpoint Service Stopped

crockbot — Tue, 17 Jan 2023 17:08:50 GMT

Well I found a clue to why Cisco Secure Endpoint is periodically stopped or disabled. In the system event logs, I found both the CSE service and its companion service, Cisco SCMS, failed to start after a system reboot, because the network service hadnt started yet (really? <anger emoji>):

Event# 7009, Source: Service Control Manager: A timeout was reached (30000 milliseconds) while waiting for the CiscoAMP service to connect.
Event# 7000, Source: Service Control Manager: The CiscoAMP service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.
Event# 7009, Source: Service Control Manager: A timeout was reached (30000 milliseconds) while waiting for the CiscoSCMS service to connect.
Event# 7000, Source: Service Control Manager: The CiscoSCMS service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.
1. Eight seconds later, the network starts.
Event# 7036, Source: Service Control Manager: The Network Setup Service service entered the running state.

Not gonna lie, this is maddening. Isnt the whole point of the Cisco SCMS (Security Connector Monitoring Service) to identify when the service is down and restart it? (and look at that, the Crowdstrike rep is calling me again.)

Re: Cisco Secure Endpoint Service Stopped

crockbot — Tue, 17 Jan 2023 17:13:03 GMT

Event# 7009, Source: Service Control Manager: A timeout was reached (30000 milliseconds) while waiting for the CiscoAMP service to connect.
Event# 7000, Source: Service Control Manager: The CiscoAMP service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.
Event# 7009, Source: Service Control Manager: A timeout was reached (30000 milliseconds) while waiting for the CiscoSCMS service to connect.
Event# 7000, Source: Service Control Manager: The CiscoSCMS service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.
1. Eight seconds later, the network starts.
Event# 7036, Source: Service Control Manager: The Network Setup Service service entered the running state.