topic Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping in Endpoint Security

AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Thu, 29 Oct 2020 14:25:26 GMT

Hi there,

I have run into a problem with the latest update to endpoints. Essentially what I'm seeing is the management console shows that the connector has not been seen for 7+ days, however the service is running on the endpoint. I have verified this one several affected systems. A select few systems show the service is stopped, but this could be a result of other users not signing out all the way. I know the tray icon shows the service is stopped when more than one user is on.

Rebooting the endpoint restores the connection and restarts the service. Stopping and restarting the service does not fix the problem.

I could be losing my mind here too, but it does appear that this is reoccurring. As in a week or so after rebooting, the endpoint will show up as having not communicated with the management console for over 7 days again. The endpoint will have the service stopped once again. The reason why I think I'm losing my mind is that I've done the reboot process for over 100 machines and I'm not entirely too sure if any of those were repeats. I'll have to track that better. Mine machine was the latest to show up in the list, so if it comes back in a couple weeks, I'll know for sure.

I've looked into the release notes and search the community here, haven't found anything in regards to the Endpoints behavior. Nothing in our environment has changed and not all equipment is being affected, so I'm scratching my head with this. Has anyone else experienced this?

UPDATE:

Just out of curiosity/troubleshooting sake, I attempted to uninstall AMP on one of the affected endpoints in order to reinstall it to address the problem. The process hung on trying to stop the service, so there seems to be an issue with the application... I should probably enable debugging but I'm not entirely sure that this would work effectively based on symptoms.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Thu, 29 Oct 2020 14:30:00 GMT

In response to my update, I had to open task manager and kill the service in there in order for the uninstall process to complete. I can update later if reinstalling fixes the issue.

I would rather not do this for 100+ PCs, so if anyone has any insight into this, I would love feedback.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Troja007 — Thu, 29 Oct 2020 18:41:19 GMT

Hello @Jbuttle,

sound weird to me. So this is not a common problem i heard of. It is hard to determine what is really happening when having no Logs, no info hot the endpoint was installed, if there have been some updates, old/new version, other security software installed on your endpoint and so on.

From my point of experience, it looks like the best way is to open a TAC case, so an Engineer can do a deeper look into the logs.

Greetings,
Thorsten

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Fri, 30 Oct 2020 20:42:04 GMT

I'll see about doing that. I'm going to try and update the connector version first to see if there's any improvment.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jim2k — Sat, 31 Oct 2020 21:15:44 GMT

i have similar if the same issue. i am about to roll out 7.3.3 to fix an issue with a recent rollout of 7.2.11 Now i am thinking the 7.x is bugged. 6.x worked fine

My issue. i have. in the amp console it showed my machine stopped sending trajectory data to the console. on the 29th in the am. on the 30th is when i notice this so i run test and see the drivers are loaded. the amp service is started the amp agent fron the sys tray is showing a green check. I try to stop the service and it hangs. so i kill it with taskkill and then start the service. data started flowing to the console from the time i started the service. so it is missing data from the time it stopped until then. I do have a TAC case open

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Tue, 03 Nov 2020 17:14:51 GMT

I'm not convinced 7.x as a whole is the problem. Everything prior to 7.3.3 was working just fine for me. I'm working on rolling out the 7.3.5 update this week to see if anything improves. It's just unfortunate I have to wait more than a week to find out.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Wed, 04 Nov 2020 14:20:04 GMT

@Troja007 Could this be an issue related to licensing? If we have a surplus of PCs and not enough licenses, does the connector stop reporting into the management console?

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Troja007 — Thu, 05 Nov 2020 08:37:59 GMT

Hello @Jbuttle,
i´m not aware of any technical feature which prevents connector communication when you overdraw your license count.
Greetings,
Thorsten

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

ChiefSec-SF — Thu, 05 Nov 2020 20:45:55 GMT

Seeing the same problem after going 7.2.11 -> 7.3.3

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Thu, 05 Nov 2020 21:21:31 GMT

@ChiefSec-SF out of curiosity, when you upgraded did you upgrade from the console or did you push out the updates manually? I feel like most of mine were done via console and are having issues. It feels like those I updated manually have been fine.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Thu, 05 Nov 2020 21:25:46 GMT

Just an update for those who are curious, it does not appear that rebooting the PCs fixes the problem on the affected 7.3.3. I can confirm that a PC I rebooted back on October 28th has now shown up back in the "Over 7 days" list.

I'm currently working on updating all of my machines to 7.3.5. Will update once I can establish that this fixes the issue.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

DaphneG — Thu, 05 Nov 2020 21:43:13 GMT

@Jbuttle, you're probably hitting this https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw34067.

Unfortunately, 7.3.5 is also affected. The engineering team is currently working on a fix and planning to get it included with 7.3.7 release. You can subscribe to that bug to get updates.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Thu, 05 Nov 2020 21:58:16 GMT

This certainly seems like what I'm experiencing! I'll be sure to get this info off to my team. I appreciate the feedback!

Do you happen to know if this is affecting all endpoints? Or are there specific circumstances where this could happen to an endpoint?

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

ChiefSec-SF — Thu, 05 Nov 2020 22:07:53 GMT

@Jbuttle We did our upgrades manually and it does appear to be the same 7day interval before they stop reporting.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Thu, 05 Nov 2020 22:45:58 GMT

Yep, a Cisco employee referenced a bug in 7.3.3 and 7.3.5 to be addressed in 7.3.7

The work around is what we have been doing, reboot the endpoint.

Just in case you missed the post below, here's a link to the bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw34067

What I'm going to try next is uninstall completely and then reinstall to see if this at least corrects the problem as it doesn't seem to be wide spread. I suspect the bug happens during the upgrade process regardless of the method.

I can report back what I find in trying this out until this is addressed later by the engineering team.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

DaphneG — Thu, 05 Nov 2020 23:52:07 GMT

We're seeing this mostly on servers but there were reports on workstations too. This is caused by a deadlock which happens whenever there's a change with the connection state while there's scanning going on or the connector is busy. When that deadlock happens, the event processors get stuck and cloud lookup doesn't complete. The service shutdown event go through the event processor as well so that explains why the service hangs on shutdown.

I don't think un/reinstalling will fix this. As you mentioned, you'll probably encounter the issue again after a few days.

If you're interested to try the 7.3.7 test build when it becomes available, please open a TAC case so we can share it with you.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Fri, 06 Nov 2020 15:13:27 GMT

So this could happen say... during times of high bandwidth use such as a back up job running over night pushing high amounts of data possibly interrupting connections to the cloud?

Also, does this affect the application's ability to protect the endpoint?

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

DaphneG — Fri, 06 Nov 2020 17:06:45 GMT

That's possible. Yes, the machine will be in an unprotected state. The other option is to downgrade to 7.3.1 while waiting for 7.3.7.

The engineering team is looking to get the 7.3.7 test build ready by next week.

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

Jbuttle — Fri, 06 Nov 2020 17:23:06 GMT

Do you know if the engineering team has an approximate time line for the full release?

Re: AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

DaphneG — Fri, 06 Nov 2020 18:46:45 GMT

The initial plan is to release 7.3.7 by end of November but with this issue, we might get it out earlier than planned. Also, the timeline will depend on the performance of the test build. The more customers available to validate the test build the better.