cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
8622
Views
15
Helpful
29
Replies

Cisco Secure Endpoint Service Stopped

mandrews
Level 1
Level 1

Good day all!

 

From time to time, I find that there are several of our machines that have their service stopped with Secure Endpoint. I haven't found what has been stopping it, but has anyone seen this and know what has been causing this? And is there a way to detect machines whose service has been stopped from the console?

Thank you,

Maurice

29 Replies 29

joljol
Level 1
Level 1

Any update on this? We are seeing similar things. In December we noticed the service was stopped on a bunch of (Windows) servers. Services seemed to never have started after an automatic server restart (windows update) and we only noticed because we have monitoring of services on this particular customers servers. So it really stood out on our monitoring-dashboard that the services were not running on all these servers.

A few days ago, another customer also noted stopped AMP services on some of his servers and also on some Windows clients.

Both customers are running v. 7.5.x. Maybe version 8 is better?

Not really. I haven't been discovering many stopped services anymore. It helps to audit the console every month or so to see if there are computers consistently not being seen for over a week or a month. The script that was mentioned earlier in the thread is helpful if you can find a way to run it across your network. We have been updating our versions, so maybe it was the 7.5.5 version that wasn't performing. Either way, if I stumble across something else, then I'll let you know.

Well I found a clue to why Cisco Secure Endpoint is periodically stopped or disabled.  In the system event logs, I found both the CSE service and its companion service, Cisco SCMS, failed to start after a system reboot, because the network service hadnt started yet (really? <anger emoji>):

  1. Event# 7009, Source: Service Control Manager:  A timeout was reached (30000 milliseconds) while waiting for the CiscoAMP service to connect.
  2. Event# 7000, Source: Service Control Manager:  The CiscoAMP service failed to start due to the following error:  The service did not respond to the start or control request in a timely fashion.

  3. Event# 7009, Source: Service Control Manager:  A timeout was reached (30000 milliseconds) while waiting for the CiscoSCMS service to connect.
  4. Event# 7000, Source: Service Control Manager:  The CiscoSCMS service failed to start due to the following error:  The service did not respond to the start or control request in a timely fashion.
    1. Eight seconds later, the network starts.
  5. Event# 7036, Source: Service Control Manager:  The Network Setup Service service entered the running state.

joljol
Level 1
Level 1

Thank you for replying. I will try and convince our customer to update to version 8 of the connector, and instruct him on how to setup and gather debug logs, in case it happens again. 

crockbot
Level 1
Level 1

Well I found a clue to why Cisco Secure Endpoint is periodically stopped or disabled.  In the system event logs, I found both the CSE service and its companion service, Cisco SCMS, failed to start after a system reboot, because the network service hadnt started yet (really? <anger emoji>):

  1. Event# 7009, Source: Service Control Manager:  A timeout was reached (30000 milliseconds) while waiting for the CiscoAMP service to connect.
  2. Event# 7000, Source: Service Control Manager:  The CiscoAMP service failed to start due to the following error:  The service did not respond to the start or control request in a timely fashion.

  3. Event# 7009, Source: Service Control Manager:  A timeout was reached (30000 milliseconds) while waiting for the CiscoSCMS service to connect.
  4. Event# 7000, Source: Service Control Manager:  The CiscoSCMS service failed to start due to the following error:  The service did not respond to the start or control request in a timely fashion.
    1. Eight seconds later, the network starts.
  5. Event# 7036, Source: Service Control Manager:  The Network Setup Service service entered the running state.

Not gonna lie, this is maddening.  Isnt the whole point of the Cisco SCMS (Security Connector Monitoring Service) to identify when the service is down and restart it?   (and look at that, the Crowdstrike rep is calling me again.)

That's interesting to see. Whenever I was auditing machines that haven't responded in a month, it was normally re-imaged machines for us. The next time we run into this issue, I'll see if I can recover those logs to see if this is the issue we're having as well. Thanks for that!

joljol
Level 1
Level 1

I would also think the Recovery settings on the service itself would take care of restarting the service if it failed to start? But alas...

Below are the default settings.

joljol_0-1673982520407.pngjoljol_1-1673982559770.png

 

I appreciate you bringing that to light. Was this in the user guide? I wonder if you set the "Run Program" to the path of the connector if that would help with keeping that loop from happening of the service not being restarted. Either way, thanks for reporting this!

Mel Chandler
Level 1
Level 1

Any updates to this issue?  We're experiencing similar issues.

newberntac
Level 1
Level 1

We found that updating the connector seemed to resolve it.

I've had this problem on many of our machines, most noticeable after windows updates. All connector versions, even the latest.8.1.5. Probably wont renew this product, this is a security product that has problem even starting.

itguy1024
Level 1
Level 1

I'm also having this issue. Also seeing that it's most noticeable after windows updates.
Any feedback from Cisco?

Updating the connector fixed it once for us, but we also set a password for the service and that seemed to work as well.  BUT be warned, you'll have to enter it anytime you want to stop the service for other reasons


@newberntac wrote:

Updating the connector fixed it once for us, but we also set a password for the service and that seemed to work as well.  BUT be warned, you'll have to enter it anytime you want to stop the service for other reasons


What version were you on and what did you go to? I'm on 8.1.7.21417 and the latest seems to be 8.1.7.21585

Roman Valenta
Cisco Employee
Cisco Employee

Hi,

 

We do have cases from time to time  related to this Issue but the reason could be different for each scenario so opening TAC case will be the best for us to Investigate. The most common are due to resources.

After Windows Updates -- > Restarts -- >Windows Processes may not be ready in time which causes the Secure Endpoint Service to fail. This can happen to other services as well. This can also happen due to all the services trying to start around the same time and causing resources to be busy causing some services not to be able to start correctly.

 

Unfortunately we don’t have a more solid work-around as this is an issue with Microsoft Windows, not so much with Cisco Secure Endpoint. This can affect many services not just Secure Endpoint.

 

Due to the Service not crashing but timing out, the service doesn't recover on its own and stays stuck in the "Stopped" State.

Delayed start is sometimes a resolution, when it’s not we can also utilize this method:


Configure Scheduled Tasks to listen to Event ID: 7009 and then try to restart the service using a PowerShell script such as this one bellow. Please note that this is not officially supported by Cisco so test first.

 

 

#Requires -RunAsAdministrator

Write-Host "Checking status of Secure Endpoint services"

function CheckAndFixService
{
    Param(
        [string]$serviceName = $(throw "Please specify a service name, -service")
        )
    $retryCount = 0
    $stopLoop = $false
    $GrabbedService = Get-Service -name $serviceName
    if($GrabbedService.Status -eq "Running"){
        Write-Host "$serviceName already started"
    }
    while($GrabbedService.Status -ne "Running" -or $stopLoop -eq $true){
        try{
            Start-Service $serviceName
            Write-Host "Attempting to start $serviceName service"
            Start-Sleep -seconds 5
            $GrabbedService.Refresh()
            if($GrabbedService.Status -eq "Running"){
                Write-Host "$serviceName started successfully"
            }
        } catch{
            if($retryCount -gt 12){
                Write-Host "Could not start $serviceName"
                $stopLoop = $true
            }
        }
    }
}

CheckAndFixService "CiscoSCMS"
CheckAndFixService "CiscoAMP"

Write-Host "All Cisco Secure Endpoint services started"

 

 

Regards,

Roman