AMP for Endpoints 7.3.3 disconnecting from management console/services stopping

[Jbuttle]

Hi there,

I have run into a problem with the latest update to endpoints. Essentially what I'm seeing is the management console shows that the connector has not been seen for 7+ days, however the service is running on the endpoint. I have verified this one several affected systems. A select few systems show the service is stopped, but this could be a result of other users not signing out all the way. I know the tray icon shows the service is stopped when more than one user is on.

Rebooting the endpoint restores the connection and restarts the service. Stopping and restarting the service does not fix the problem.

I could be losing my mind here too, but it does appear that this is reoccurring. As in a week or so after rebooting, the endpoint will show up as having not communicated with the management console for over 7 days again. The endpoint will have the service stopped once again. The reason why I think I'm losing my mind is that I've done the reboot process for over 100 machines and I'm not entirely too sure if any of those were repeats. I'll have to track that better. Mine machine was the latest to show up in the list, so if it comes back in a couple weeks, I'll know for sure.

I've looked into the release notes and search the community here, haven't found anything in regards to the Endpoints behavior. Nothing in our environment has changed and not all equipment is being affected, so I'm scratching my head with this. Has anyone else experienced this?

UPDATE:

Just out of curiosity/troubleshooting sake, I attempted to uninstall AMP on one of the affected endpoints in order to reinstall it to address the problem. The process hung on trying to stop the service, so there seems to be an issue with the application... I should probably enable debugging but I'm not entirely sure that this would work effectively based on symptoms.

[Jbuttle]

In response to my update, I had to open task manager and kill the service in there in order for the uninstall process to complete. I can update later if reinstalling fixes the issue.

I would rather not do this for 100+ PCs, so if anyone has any insight into this, I would love feedback.

[Troja007]

Hello @Jbuttle,

sound weird to me. So this is not a common problem i heard of. It is hard to determine what is really happening when having no Logs, no info hot the endpoint was installed, if there have been some updates, old/new version, other security software installed on your endpoint and so on.

From my point of experience, it looks like the best way is to open a TAC case, so an Engineer can do a deeper look into the logs.

Greetings,
Thorsten

[Jbuttle]

I'll see about doing that. I'm going to try and update the connector version first to see if there's any improvment.

[Jbuttle]

@Troja007 Could this be an issue related to licensing? If we have a surplus of PCs and not enough licenses, does the connector stop reporting into the management console?

[Troja007]

Hello @Jbuttle,
i´m not aware of any technical feature which prevents connector communication when you overdraw your license count.
Greetings,
Thorsten

[Jim2k]

i have similar if the same issue. i am about to roll out 7.3.3 to fix an issue with a recent rollout of 7.2.11  Now i am thinking the 7.x is bugged. 6.x worked fine

My issue. i have. in the amp console it showed my machine stopped sending trajectory data to the console. on the 29th in the am. on the 30th is when i notice this so i run test and see the drivers are loaded. the amp service is started the amp agent fron the sys tray is showing a green check. I try to stop the service and it hangs. so i kill it with taskkill and then start the service. data started flowing to the console from the time i started the service.  so it is missing data from the time it stopped until then. I do have a TAC case open

[Jbuttle]

I'm not convinced 7.x as a whole is the problem. Everything prior to 7.3.3 was working just fine for me. I'm working on rolling out the 7.3.5 update this week to see if anything improves. It's just unfortunate I have to wait more than a week to find out.

[ChiefSec-SF]

Seeing the same problem after going 7.2.11 -> 7.3.3

[Jbuttle]

@ChiefSec-SF out of curiosity, when you upgraded did you upgrade from the console or did you push out the updates manually? I feel like most of mine were done via console and are having issues. It feels like those I updated manually have been fine.

[ChiefSec-SF]

@Jbuttle We did our upgrades manually and it does appear to be the same 7day interval before they stop reporting.

[Jbuttle]

Yep, a Cisco employee referenced a bug in 7.3.3 and 7.3.5 to be addressed in 7.3.7

The work around is what we have been doing, reboot the endpoint.

Just in case you missed the post below, here's a link to the bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw34067

What I'm going to try next is uninstall completely and then reinstall to see if this at least corrects the problem as it doesn't seem to be wide spread. I suspect the bug happens during the upgrade process regardless of the method.

I can report back what I find in trying this out until this is addressed later by the engineering team.

[Jbuttle]

Just an update for those who are curious, it does not appear that rebooting the PCs fixes the problem on the affected 7.3.3. I can confirm that a PC I rebooted back on October 28th has now shown up back in the "Over 7 days" list.

I'm currently working on updating all of my machines to 7.3.5. Will update once I can establish that this fixes the issue.

[DaphneG · Accepted solution]

@Jbuttle, you're probably hitting this https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw34067.

Unfortunately, 7.3.5 is also affected. The engineering team is currently working on a fix and planning to get it included with 7.3.7 release. You can subscribe to that bug to get updates.

[Jbuttle]

This certainly seems like what I'm experiencing! I'll be sure to get this info off to my team. I appreciate the feedback!

Do you happen to know if this is affecting all endpoints? Or are there specific circumstances where this could happen to an endpoint?