AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

[Jbuttle]

Hi there,

I have run into a problem with the latest update to endpoints. Essentially what I'm seeing is the management console shows that the connector has not been seen for 7+ days, however the service is running on the endpoint. I have verified this one several affected systems. A select few systems show the service is stopped, but this could be a result of other users not signing out all the way. I know the tray icon shows the service is stopped when more than one user is on.

Rebooting the endpoint restores the connection and restarts the service. Stopping and restarting the service does not fix the problem.

I could be losing my mind here too, but it does appear that this is reoccurring. As in a week or so after rebooting, the endpoint will show up as having not communicated with the management console for over 7 days again. The endpoint will have the service stopped once again. The reason why I think I'm losing my mind is that I've done the reboot process for over 100 machines and I'm not entirely too sure if any of those were repeats. I'll have to track that better. Mine machine was the latest to show up in the list, so if it comes back in a couple weeks, I'll know for sure.

I've looked into the release notes and search the community here, haven't found anything in regards to the Endpoints behavior. Nothing in our environment has changed and not all equipment is being affected, so I'm scratching my head with this. Has anyone else experienced this?

UPDATE:

Just out of curiosity/troubleshooting sake, I attempted to uninstall AMP on one of the affected endpoints in order to reinstall it to address the problem. The process hung on trying to stop the service, so there seems to be an issue with the application... I should probably enable debugging but I'm not entirely sure that this would work effectively based on symptoms.

[DaphneG]

We're seeing this mostly on servers but there were reports on workstations too. This is caused by a deadlock which happens whenever there's a change with the connection state while there's scanning going on or the connector is busy. When that deadlock happens, the event processors get stuck and cloud lookup doesn't complete. The service shutdown event go through the event processor as well so that explains why the service hangs on shutdown. 

I don't think un/reinstalling will fix this. As you mentioned, you'll probably encounter the issue again after a few days. 

If you're interested to try the 7.3.7 test build when it becomes available, please open a TAC case so we can share it with you. 

[Jbuttle]

So this could happen say... during times of high bandwidth use such as a back up job running over night pushing high amounts of data possibly interrupting connections to the cloud?

Also, does this affect the application's ability to protect the endpoint?

[DaphneG]

That's possible. Yes, the machine will be in an unprotected state. The other option is to downgrade to 7.3.1 while waiting for 7.3.7. 

The engineering team is looking to get the 7.3.7 test build ready by next week. 

[Jbuttle]

Do you know if the engineering team has an approximate time line for the full release?

[DaphneG]

The initial plan is to release 7.3.7 by end of November but with this issue, we might get it out earlier than planned. Also, the timeline will depend on the performance of the test build. The more customers available to validate the test build the better. 

[DaphneG]

The preview build for 7.3.7, which has the fix for the bug discussed here, is now available. Please open a TAC case if you're interested to test it. 

[WilliamBrown95047]

We are seeing the same issue with 7.3.5, specifically related to AnyConnect VPN.  No need to wait 7 days.  We have AnyConnect set to time out at 24 hours.  If I try to do a right-click AMP scan on a file after AnyConnect has timed out, I get the pop-up, but only the timer is populated.  It never completes; I can cancel the scan and the window closes.  If I try to do another right-click scan, nothing at all happens - no pop-up scan window.

Like others, I cannot stop the AMP connector service (even with the password).

The console shows my endpoint last seen 2 days ago.  My tray application reports 'Connected'.

Luckily we were testing with a handful of endpoints; we will downgrade those to 7.3.1 and leave the rest on 7.2.7 (which I know is unsupported, but seems to work).  May rise a TAC to get the 7.3.7 test client if I can figure out how.