Resolving AMP for Endpoints TETRA Definition Issues

[aledipas]

Hi Everyone,

Recently we were made aware of a TETRA AV definition update which caused the Windows AMP for

Endpoints service to crash. 

Note:  Customers who do NOT have TETRA enabled are not affected by this issue. 

While we have already removed the problematic definition set, which was available for ~30 minutes (see further notes below), affected systems will need to be fixed manually by uninstalling/re-installing the Connector (instructions below). Once the connector has been re-installed, a non-affected definition set will be downloaded and resolve the issue.

How to determine if you are impacted:

The issue causes the AMP for Endpoints service to crash or hang. The best way to determine if you have an affected system is to determine if any Connectors have been offline since the bad definition set was published.

To get the Last Seen Timestamp from the AMP Console, go to the Management tab and select Computers. From here you can download a CSV file using the "Export to CSV" option. The CSV will contain the Last Seen Timestamp. You can sort and filter on Connectors that have not been seen since 16:00 UTC February 06 2018 – these are likely Connectors that have been affected by this issue.

Resolution:

We urge all customers who are affected by this issue to open a TAC case immediately.

Resolving this issue does involve uninstalling and reinstalling the Connector.

Uninstall via Add/Remove Programs:

a) Uninstall the connector (choose "No" when asked if you plan to install the Connector again)
b) Re-install connector

Uninstall via Command Line:

<installer> /R /S /stopservicecoe 1 /remove 1

Uninstall via Command Line with Connector Protection Enabled:

<installer> /R /S /stopservicecoe 1 /remove 1 /uninstallpassword <INSERT YOUR PASSWORD>

Affected Software Versions:

All Windows Connector versions with TETRA enabled are affected on both 32bit and 64bit versions of Windows 7/8/10, Windows Server 2008R2 and Server 2012

Notes:

TETRA Definition Sets:

Faulty TETRA definition revision (16:20 UTC)

32bit = 101032, 64bit = 70876

Updated TETRA definition revision (16:50 UTC)

32bit = 101034, 64bit = 70878

A Root Cause Analysis (RCA) document will be prepared and shared with affected customers. 

[rvargo120]

This does not work on protected clients. Neither do other command switches provided by TAC.

[Todd Everett]

Which version of the Connector are you running?

Are you running the Command Prompt as Administrator?

1. The top of the thread is updated with
   

   Uninstall via Command Line:

   <installer path> /r /S /stopservicecoe 1 /remove 1

   Uninstall via Command Line with Connector Protection Enabled:

   <installer path> /r /S /stopservicecoe 1 /remove 1 /uninstallpassword <password>

   If those do not work, try running <path, likely C:\Program Files\Cisco\AMP\>sfc.exe -k <password>

[rvargo120]

Thanks for your reply,
Here is what I am finding.

The policy all affected clients are placed in was still set to v5.1.13, however, over half (around 2/3 or 600) clients had been updated to v6.0.5. The remaining portion of clients are still running v5.1.13 matching the policy configuration.

Additionally, NONE of the command line options are working to remove protected clients in either version (yes CMD/PowerShell is elevated). TAC reported the troubleshooting file contains entries about a bad password. I can, however, use the password to uninstall the connector with the GUI every time. I have tried so many combinations of commands so many times on so many different clients, I'm getting dizzy just thinking about it.

Regardless, it seems that there is still a lot up in the air right now. Hoping Cisco can wrangle this beast and provide an effective and scalable solution.

[s22crocker]

Can I get a clarification please?  You said they are set to the 5.1.13 but most were running the 6.0.5.  Can you confirm the version that the affected clients were actually running at the time of issue?  We are looking to update to the 5.1.13 so we can see the clients information correctly in the console but if you can confirm even the 5.1.13 connectors are potentially compromised we will probably give this longer to settle down.  Thank you for putting your information out there.  Communities like this work when people contribute.

[rvargo120]

Based on my sampling from this morning (maybe 30 clients), all of the affected clients were running 6.0.5. Now all of these clients should have been running 5.1.13 since that is what the policy is configured for. I updated this policy between 1/22 and 1/29 from 5.1.11.

Based on our ticketing system, approx 65 are affected, but this number seems to grow by the hour as users notice, or get fed up with the error message every 5-10 min.

Thanks,

[s22crocker]

So after a bit of digging on our part, we found that only clients with the 6.0.5.10636 version of the connector are affected at our locations.   Can anyone else out there confirm if this is the case for you as well?

[chrisloyd@uniongeneral.org]

That seems to be the case for us as well. So far we do not have a resolution. Once uninstalling, and trying to reinstall it states the service is running, when it is not running. It no longer exists.

[rvargo120]

I can second this. Although, I'd have to double check which of our policies have TETRA enabled. So far only 6.0.5 clients are affected.

[JSUll]

I have found, with Password protected connectors, the only process that works is running the connector install executable with the following switches to uninstall the entire client, reboot, then re-install, and reboot again. 

 %INST%\install\XXXX_Detection_Group_FireAMPSetup.exe /S /remove 1 /uninstallpassword XXXXXXXXX

This uninstall scrip seems to work, as long as you reboot after. 

I've deployed 2 SCCM scripts, one to remove the connector and reboot, one to install it all over again and reboot, beginning at 2:00 am.  We found using a rough guide of any connector who hasn't checked in since the Feb 6, 2018 00:016 UTC gave us a rough number, but we expect it to be off +/- at least 10-20 pc's.  

[Maverick HURLEY]

Has anybody been to remediate this all thru a huge corporate environment?    We have been trying to find big companies that have adopted AMP that has moved from McAfee.   We are not having good vibes about going to AMP.   We got with this update as well and still no real solution to fix these machines on a bulk basis.   Basically we are running exposed because AMP broke itself. 

[pascal.bourbonnais@mcgill.ca]

Anyone knows the name of the file responsible for the crash (corrupted update file) ?

[chrisloyd@uniongeneral.org]

We were able to correct with a login batch file, check if the file c:\windows\temp\BitDefender Threat Scanner.dmp exists, if it does, then the machine has the problem.

Delete the amp file and then

Run the following

call FireAMPSetup.exe /S /remove 1 /uninstallpassword yourpasswordhere

The Fireampsetup.exe is the installation file, your name may be different.

then have the script do a shutdown /r command. 

This will uninstall and then you can reinstall. Our script installs automatically

if %PROGRAMS_HOME%\Cisco\AMP\local.xml does not exist then install.

[chrisloyd@uniongeneral.org]

Sorry, delete the .dmp file, not amp file so that it is no longer there the next time your script runs.

[rvargo120]

I was finally able to make some progress and have an effective automated solution for password PROTECTED clients.

Here are the steps
1) Copy the INSTALL file (the one you download from the AMP cloud) to the client's C: drive.
2) Execute [ "C:\FireAMPSetup.exe" /r /S /remove 1 /uninstallpassword <Password(InDoubleQuotes)> ]
3) Remove the installer (Optional, but you may need it again)
4) Reboot the device
5) Install AMP as normal
6) Reboot

This process has been deployed successfully to over 50 devices in nearly 20 facilities so far today.

TAC kept telling me to use the "uninstall.exe" file in the AMP program files directory. This does NOT work for protected clients. ONLY the use of the INSTALLER worked in this scenario. This part needs more emphasis than has been provided as it is easily over looked and can make for a major PITA.

Good luck everyone, it's a headache, but doable.

[szielonka1]

Cisco gave me this window batch file to use:

@echo off
:: replace CiscoAMP%% with immunetprotect%% for earlier versions of A4E
:: adjust your installation directory if necessary as well for older connector versions using the sourcefire branding
:: run this as admin
:: this script won't be able to stop the A4E service if connector protection is enabled

wmic service where "name like 'CiscoAMP%%'" call stopservice > nul
timeout 10 > nul
del "C:\Program Files\Cisco\AMP\tetra\Plugins\*" /q
del "C:\Program Files\Cisco\AMP\update\Plugins\*" /q
timeout 5 > nul
wmic service where "name like 'CiscoAMP%%'" call startservice > nul