Resolving AMP for Endpoints TETRA Definition Issues

[aledipas]

Hi Everyone,

Recently we were made aware of a TETRA AV definition update which caused the Windows AMP for

Endpoints service to crash. 

Note:  Customers who do NOT have TETRA enabled are not affected by this issue. 

While we have already removed the problematic definition set, which was available for ~30 minutes (see further notes below), affected systems will need to be fixed manually by uninstalling/re-installing the Connector (instructions below). Once the connector has been re-installed, a non-affected definition set will be downloaded and resolve the issue.

How to determine if you are impacted:

The issue causes the AMP for Endpoints service to crash or hang. The best way to determine if you have an affected system is to determine if any Connectors have been offline since the bad definition set was published.

To get the Last Seen Timestamp from the AMP Console, go to the Management tab and select Computers. From here you can download a CSV file using the "Export to CSV" option. The CSV will contain the Last Seen Timestamp. You can sort and filter on Connectors that have not been seen since 16:00 UTC February 06 2018 – these are likely Connectors that have been affected by this issue.

Resolution:

We urge all customers who are affected by this issue to open a TAC case immediately.

Resolving this issue does involve uninstalling and reinstalling the Connector.

Uninstall via Add/Remove Programs:

a) Uninstall the connector (choose "No" when asked if you plan to install the Connector again)
b) Re-install connector

Uninstall via Command Line:

<installer> /R /S /stopservicecoe 1 /remove 1

Uninstall via Command Line with Connector Protection Enabled:

<installer> /R /S /stopservicecoe 1 /remove 1 /uninstallpassword <INSERT YOUR PASSWORD>

Affected Software Versions:

All Windows Connector versions with TETRA enabled are affected on both 32bit and 64bit versions of Windows 7/8/10, Windows Server 2008R2 and Server 2012

Notes:

TETRA Definition Sets:

Faulty TETRA definition revision (16:20 UTC)

32bit = 101032, 64bit = 70876

Updated TETRA definition revision (16:50 UTC)

32bit = 101034, 64bit = 70878

A Root Cause Analysis (RCA) document will be prepared and shared with affected customers. 

[unfairbeef]

When was the update available? How can we tell through the console which systems may be exhibiting the behavior? We have around 4,500 connectors in our environment. 

[Todd Everett]

As stated above, the Replacement definition was posted within 30 minutes of the original problematic definition. This happened before 17:00 UTC on February 6. The versions are also referenced above.

[jlinder]

We also have 20k+ connectors with tetra offline scanning enabled. Would be nice to know which ones were affected.

[gba18]

Affected clients are getting this error.

Affected clients are getting this error.  If you have connector protection enabled like we do it looks like resolution is going to be a nightmare.  

[damelton]

I had this problem also, and my PC was really slow. I removed AMP (with no plan to reinstalled) and then re-installed as suggested and both problems went away and APM is now working.

[Vivek Mathur]

While this fix is for the endpoints, can something be done on the backend to prevent this from spreading any further?

With this deployed across all endpoints in our network, working on every machine individually is going to be extremely tedious to say the least......

[Todd Everett]

Please see below
Todd

[pascal.bourbonnais@mcgill.ca]

It seems that only 32bits OS would requires the fix to be executed. We are using a powershell script to detect if the event is present in the windows logs if so clear the content of the update directory. The script is to be deployed with SCCM and the compliance check is that the service is present and not running.  AND FINALLY we will disable TETRA and deploy SCEP: 3rd strike, we are not beta tester.

[Vivek Mathur]

Pascal,

All our machines that have so far been affected are all Windows 7 64 bit. For the most part we have no 32 bit machines deployed.

The batch file works but, only when the machine is in booted in safe mode.

This is going to be  a real pain in the ***

[trippin315]

What is the name of the event that you are searching for in the Windows logs? 

[pascal.bourbonnais@mcgill.ca]

Event ID 7031 that matches cisco AMP

$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7003;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if ($ampLog.Count -gt 0) {
    $svc = Get-Service | Where-Object {$_.name -match "amp"}
    if ($svc.status -eq "running") {
        Stop-Service $svc -Force
    }
    # clean the **bleep**...
}

[pascal.bourbonnais@mcgill.ca]

Correction: 7031

$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

[pascal.bourbonnais@mcgill.ca]

FINAL VERSION FOR SCCM THAT APPLY TO 32-bit only

___________________________

TO APPLY ON ALL : just remove

-and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" )

___________________________

# compliance check:
$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if (($ampLog.Count -gt 0) -and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" ) ) {
    return $false
} else {
    return $true
}

# remediation:
$svc = Get-Service | Where-Object {$_.name -match "ciscoamp"}
if ($svc.status -eq "running") {
    Stop-Service $svc -Force
}
Remove-Item "$env:ProgramFiles\Cisco\AMP\tetra\Plugins\*"
Remove-Item "$env:ProgramFiles\Cisco\AMP\update\Plugins\*"
Start-Service $svc

[JSUll]

Does this sccm script work for password protected instances?

---

pascal.bourbonnais@mcgill.ca wrote:

FINAL VERSION FOR SCCM THAT APPLY TO 32-bit only

___________________________

TO APPLY ON ALL : just remove

-and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" )

___________________________

# compliance check:
@$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if (($ampLog.Count -gt 0) -and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" ) ) {
    return $false
} else {
    return $true
}

# remediation:
$svc = Get-Service | Where-Object {$_.name -match "ciscoamp"}
if ($svc.status -eq "running") {
    Stop-Service $svc -Force
}
Remove-Item "$env:ProgramFiles\Cisco\AMP\tetra\Plugins\*"
Remove-Item "$env:ProgramFiles\Cisco\AMP\update\Plugins\*"
Start-Service $svc

---