topic Re: Resolving AMP for Endpoints TETRA Definition Issues in Endpoint Security

Resolving AMP for Endpoints TETRA Definition Issues

aledipas — Sat, 09 Mar 2019 01:46:31 GMT

Hi Everyone,

Recently we were made aware of a TETRA AV definition update which caused the Windows AMP for

Endpoints service to crash.

Note: Customers who do NOT have TETRA enabled are not affected by this issue.

While we have already removed the problematic definition set, which was available for ~30 minutes (see further notes below), affected systems will need to be fixed manually by uninstalling/re-installing the Connector (instructions below). Once the connector has been re-installed, a non-affected definition set will be downloaded and resolve the issue.

How to determine if you are impacted:

The issue causes the AMP for Endpoints service to crash or hang. The best way to determine if you have an affected system is to determine if any Connectors have been offline since the bad definition set was published.

To get the Last Seen Timestamp from the AMP Console, go to the Management tab and select Computers. From here you can download a CSV file using the "Export to CSV" option. The CSV will contain the Last Seen Timestamp. You can sort and filter on Connectors that have not been seen since 16:00 UTC February 06 2018 – these are likely Connectors that have been affected by this issue.

Resolution:

We urge all customers who are affected by this issue to open a TAC case immediately.

Resolving this issue does involve uninstalling and reinstalling the Connector.

Uninstall via Add/Remove Programs:

a) Uninstall the connector (choose "No" when asked if you plan to install the Connector again)
b) Re-install connector

Uninstall via Command Line:

<installer> /R /S /stopservicecoe 1 /remove 1

Uninstall via Command Line with Connector Protection Enabled:

<installer> /R /S /stopservicecoe 1 /remove 1 /uninstallpassword <INSERT YOUR PASSWORD>

Affected Software Versions:

All Windows Connector versions with TETRA enabled are affected on both 32bit and 64bit versions of Windows 7/8/10, Windows Server 2008R2 and Server 2012

Notes:

TETRA Definition Sets:

Faulty TETRA definition revision (16:20 UTC)

32bit = 101032, 64bit = 70876

Updated TETRA definition revision (16:50 UTC)

32bit = 101034, 64bit = 70878

A Root Cause Analysis (RCA) document will be prepared and shared with affected customers.

Re: Resolving AMP for Endpoints TETRA Definition Issues

unfairbeef — Tue, 06 Feb 2018 21:08:55 GMT

When was the update available? How can we tell through the console which systems may be exhibiting the behavior? We have around 4,500 connectors in our environment.

Re: Resolving AMP for Endpoints TETRA Definition Issues

jlinder — Tue, 06 Feb 2018 21:20:47 GMT

We also have 20k+ connectors with tetra offline scanning enabled. Would be nice to know which ones were affected.

Re: Resolving AMP for Endpoints TETRA Definition Issues

Vivek Mathur — Tue, 06 Feb 2018 21:40:24 GMT

While this fix is for the endpoints, can something be done on the backend to prevent this from spreading any further?

With this deployed across all endpoints in our network, working on every machine individually is going to be extremely tedious to say the least......

Re: Resolving AMP for Endpoints TETRA Definition Issues

gba18 — Tue, 06 Feb 2018 21:43:01 GMT

Affected clients are getting this error.

Affected clients are getting this error. If you have connector protection enabled like we do it looks like resolution is going to be a nightmare.

Re: Resolving AMP for Endpoints TETRA Definition Issues

pascal.bourbonnais@mcgill.ca — Tue, 06 Feb 2018 21:48:22 GMT

It seems that only 32bits OS would requires the fix to be executed. We are using a powershell script to detect if the event is present in the windows logs if so clear the content of the update directory. The script is to be deployed with SCCM and the compliance check is that the service is present and not running. AND FINALLY we will disable TETRA and deploy SCEP: 3rd strike, we are not beta tester.

Re: Resolving AMP for Endpoints TETRA Definition Issues

Vivek Mathur — Tue, 06 Feb 2018 21:55:14 GMT

Pascal,

All our machines that have so far been affected are all Windows 7 64 bit. For the most part we have no 32 bit machines deployed.

The batch file works but, only when the machine is in booted in safe mode.

This is going to be a real pain in the ***

Re: Resolving AMP for Endpoints TETRA Definition Issues

trippin315 — Tue, 06 Feb 2018 22:11:13 GMT

What is the name of the event that you are searching for in the Windows logs?

Re: Resolving AMP for Endpoints TETRA Definition Issues

pascal.bourbonnais@mcgill.ca — Tue, 06 Feb 2018 22:17:05 GMT

Event ID 7031 that matches cisco AMP

$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7003;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if ($ampLog.Count -gt 0) {
    $svc = Get-Service | Where-Object {$_.name -match "amp"}
    if ($svc.status -eq "running") {
        Stop-Service $svc -Force
    }
    # clean the **bleep**...
}

Re: Resolving AMP for Endpoints TETRA Definition Issues

szielonka1 — Tue, 06 Feb 2018 22:18:00 GMT

Same Problem

Re: Resolving AMP for Endpoints TETRA Definition Issues

pascal.bourbonnais@mcgill.ca — Tue, 06 Feb 2018 22:18:06 GMT

Correction: 7031

$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

Re: Resolving AMP for Endpoints TETRA Definition Issues

akton112233 — Tue, 06 Feb 2018 22:32:00 GMT

Trying to modify the provided script in the OP using the documentation here ( https://www.cisco.com/c/en/us/support/docs/security/sourcefire-fireamp-endpoints/118588-technote-fireamp-00.html ) since we have connector protection enabled. Currently not working (access denied) but still trying.

Re: Resolving AMP for Endpoints TETRA Definition Issues

pascal.bourbonnais@mcgill.ca — Tue, 06 Feb 2018 22:42:13 GMT

FINAL VERSION FOR SCCM THAT APPLY TO 32-bit only

___________________________

TO APPLY ON ALL : just remove

-and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" )

___________________________

# compliance check:
$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if (($ampLog.Count -gt 0) -and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" ) ) {
return $false
} else {
return $true
}

# remediation:
$svc = Get-Service | Where-Object {$_.name -match "ciscoamp"}
if ($svc.status -eq "running") {
Stop-Service $svc -Force
}
Remove-Item "$env:ProgramFiles\Cisco\AMP\tetra\Plugins\*"
Remove-Item "$env:ProgramFiles\Cisco\AMP\update\Plugins\*"
Start-Service $svc

Re: Resolving AMP for Endpoints TETRA Definition Issues

ianbuch — Tue, 06 Feb 2018 23:18:27 GMT

Try this addition to the script to get it to disable the connector protection.

:: START cmd.exe /c "cd C:\Program Files\Cisco\AMP\<version you're running > & sfc.exe -k <connector protection password>" 

START cmd.exe /c "cd C:\Program Files\Cisco\AMP\5.1.13 & sfc.exe -k Cisco123456" 
timeout 10 > nul
del "C:\Program Files\Cisco\AMP\tetra\Plugins\*" /q
del "C:\Program Files\Cisco\AMP\update\Plugins\*" /q
timeout 5 > nul
wmic service where "name like 'CiscoAMP%%'" call startservice > nul

Re: Resolving AMP for Endpoints TETRA Definition Issues

tnhp — Wed, 07 Feb 2018 10:39:24 GMT

That one solved the problem - thanks.

Uninstall.

Choose no - (all files including quarantined are deleted).

Reboot.

Download and install connector.

Re: Resolving AMP for Endpoints TETRA Definition Issues

UnUser — Wed, 07 Feb 2018 12:52:27 GMT

It is not possible to delete definition files after stopping the Service.

Looks like the Kerneldriver is preventing changes in AMP-Directory

thanks Cisco for this great Product

Re: Resolving AMP for Endpoints TETRA Definition Issues

tnhp — Wed, 07 Feb 2018 12:39:33 GMT

I haven't encountered duplicates in AMP by using the 'say no'-solution.

Advise: Try a few machines and check AMP before continuing.

Re: Resolving AMP for Endpoints TETRA Definition Issues

UnUser — Wed, 07 Feb 2018 12:53:20 GMT

you are right

changed my reply

Re: Resolving AMP for Endpoints TETRA Definition Issues

Jens Nygaard — Wed, 07 Feb 2018 14:19:49 GMT

Are there any "Mass deployment" solution to this mess? this is really bad for a corporate environment to Uninstall the connector, Force a reboot, Install the connector and force another reboot..

Re: Resolving AMP for Endpoints TETRA Definition Issues

Jeffsoh — Wed, 07 Feb 2018 18:19:01 GMT

Need a way to manage definitions through the console. Also need status information in the console on whether a connector is connected or not. Not really an enterprise solution with so little control over the endpoints.