Resolving AMP for Endpoints TETRA Definition Issues

[aledipas]

Hi Everyone,

Recently we were made aware of a TETRA AV definition update which caused the Windows AMP for

Endpoints service to crash. 

Note:  Customers who do NOT have TETRA enabled are not affected by this issue. 

While we have already removed the problematic definition set, which was available for ~30 minutes (see further notes below), affected systems will need to be fixed manually by uninstalling/re-installing the Connector (instructions below). Once the connector has been re-installed, a non-affected definition set will be downloaded and resolve the issue.

How to determine if you are impacted:

The issue causes the AMP for Endpoints service to crash or hang. The best way to determine if you have an affected system is to determine if any Connectors have been offline since the bad definition set was published.

To get the Last Seen Timestamp from the AMP Console, go to the Management tab and select Computers. From here you can download a CSV file using the "Export to CSV" option. The CSV will contain the Last Seen Timestamp. You can sort and filter on Connectors that have not been seen since 16:00 UTC February 06 2018 – these are likely Connectors that have been affected by this issue.

Resolution:

We urge all customers who are affected by this issue to open a TAC case immediately.

Resolving this issue does involve uninstalling and reinstalling the Connector.

Uninstall via Add/Remove Programs:

a) Uninstall the connector (choose "No" when asked if you plan to install the Connector again)
b) Re-install connector

Uninstall via Command Line:

<installer> /R /S /stopservicecoe 1 /remove 1

Uninstall via Command Line with Connector Protection Enabled:

<installer> /R /S /stopservicecoe 1 /remove 1 /uninstallpassword <INSERT YOUR PASSWORD>

Affected Software Versions:

All Windows Connector versions with TETRA enabled are affected on both 32bit and 64bit versions of Windows 7/8/10, Windows Server 2008R2 and Server 2012

Notes:

TETRA Definition Sets:

Faulty TETRA definition revision (16:20 UTC)

32bit = 101032, 64bit = 70876

Updated TETRA definition revision (16:50 UTC)

32bit = 101034, 64bit = 70878

A Root Cause Analysis (RCA) document will be prepared and shared with affected customers. 

[pascal.bourbonnais@mcgill.ca]

Your could change the resolution script like this:

    # remediation:
   
    $sfc = @(Get-ChildItem -Path "$env:ProgramFiles\Cisco\AMP" -Recurse -Include "sfc.exe")
    $sfc | foreach { .$($_.fullname) -k <YOUR PASSWORD> }
   
    $svc = Get-Service | Where-Object {$_.name -match "ciscoamp"}
    if ($svc.status -eq "running") {
        Stop-Service $svc -Force
    }

    Remove-Item "$env:ProgramFiles\Cisco\AMP\tetra\Plugins\*" -Force
    Remove-Item "$env:ProgramFiles\Cisco\AMP\update\Plugins\*" -Force
    Start-Service $svc

[szielonka1]

Same Problem

[akton112233]

Trying to modify the provided script in the OP using the documentation here ( https://www.cisco.com/c/en/us/support/docs/security/sourcefire-fireamp-endpoints/118588-technote-fireamp-00.html ) since we have connector protection enabled. Currently not working (access denied) but still trying.

[ianbuch]

1. Try this addition to the script to get it to disable the connector protection.

3. :: START cmd.exe /c "cd C:\Program Files\Cisco\AMP\<version you're running > & sfc.exe -k <connector protection password>" 
   
   START cmd.exe /c "cd C:\Program Files\Cisco\AMP\5.1.13 & sfc.exe -k Cisco123456" 
   timeout 10 > nul
   del "C:\Program Files\Cisco\AMP\tetra\Plugins\*" /q
   del "C:\Program Files\Cisco\AMP\update\Plugins\*" /q
   timeout 5 > nul
   wmic service where "name like 'CiscoAMP%%'" call startservice > nul

[tnhp]

That one solved the problem - thanks.

Uninstall.

Choose no - (all files including quarantined are deleted).

Reboot.

Download and install connector.

[damelton]

Fixed my problem and the Window's security definition error, and the very slow response my PC was having.  All is well now thankfully.

[UnUser]

It is not possible to delete definition files after stopping the Service.

Looks like the Kerneldriver is preventing changes in AMP-Directory

thanks Cisco for this great Product

[tnhp]

I haven't encountered duplicates in AMP by using the 'say no'-solution.

Advise: Try a few machines and check AMP before continuing.

[UnUser]

you are right

changed my reply

[Jens Nygaard]

Are there any "Mass deployment" solution to this mess? this is really bad for a corporate environment to Uninstall the connector, Force a reboot, Install the connector and force another reboot..

[Jeffsoh]

Need a way to manage definitions through the console. Also need status information in the console on whether a connector is connected or not. Not really an enterprise solution with so little control over the endpoints. 

[CE_Tristan]

The delete_defs.bat script does not work for mass deployment purposes as it relies on manual elevation of the command prompt to delete the problematic files.

Alternatively, I'm unaware of a CLI parameter that can be set with the execution of the uninstall.exe executable in the local Cisco AMP version folder that would allow for the scripting of the AMP's uninstallation that includes removal of the definitions as well.

Cisco AMP customers need a better solution than has been provided to accommodate the large scope of machines that have been impacted. A manual machine-by-machine resolution is not optimal for companies managing hundreds or thousands of computers.

[s22crocker]

Can we get more details on this issue?  Are all versions of connector afffected?  What is the TETRA definition version number that is the issue?  Does the connector stop working completely until the fix is applied or does a reboot temporarily get it going again with the possibility of another crash? 
That information would greatly help us determine our exposure and assist in our remediation planning.

[ESherlock]

I would really like to get some explanation of the 0016 timestamp. Are you saying anything that has checked in from 0017 on is okay? Is it anything in a certain window?

[Todd Everett]

Rather than manually touching each endpoint, please remember you can push uninstalls and reinstalls via SCCM or any other deployment  tool using Command Line Switches. These are listed in the latest Deployment Strategy Guide on page 26.

 https://docs.amp.cisco.com/en/A4E/AMP%20for%20Endpoints%20Deployment%20Strategy.pdf

/remove 1       Uninstalls the Connector and removes all associated file

So you can use these commands:

<installer> /r /S /stopservicecoe 1 /remove 1

 <installer> /r /S /stopservicecoe 1 /remove 1 /uninstallpassword <INSERT YOUR PASSWORD>

 to remove the application. The <installer> should be the original installation file for the AMP for Endpoints Connector.

Todd