cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Cognitive Intelligence Update, May 2019: Machine Learning Backend Improved

479
Views
15
Helpful
0
Comments
Cisco Employee

Throughout March, April and May 2019 we improved Machine Learning backend infrastructures and processes to accelerate discovery of new Indicators of Compromise (IoC). Higher number of discovered IoCs has two benefits: 1. higher percentage of Cognitive incidents will fall into Confirmed Threat category, 2. more IoCs used for (re)training of Machine Learned predictors means extended ability to get net-new detections. The improvemens include:

  • Optimized RAndom Forests (OraF) - we implemented and open-sourced super-fast Random Forest training code (see http://github.com/cisco/oraf)
  • Confirmed Threat IoC automated discovery - we implemented automation of new IoC discovery for existing Confirm Threats. The technique reduced manual confirmation load by half without compromising the precision of Confirmed Threat incidents (see IoC statistics below)
  • Talos-integration - we built a Machine Learning pipeline and process for improved exchange of intelligence between Talos and Cognitive Intelligence (see Confirmed Threat IDs CTALxxxx in Coverage of Confirmed Threats Extended blog)
  • Threat confirmation pipelines - we optimized infrastructures to take maximum effect of each verdict provided by analyst. IoC candidates come through Active Learning (see next item). Every manually confirmed or disproved IoC candidate gets automatically included in fresh Machine Learning ground truth, prediction models get re-trained, re-evaluated and queued for push-to-prod; existing Incident pool gets re-evaluated and Incidents related to the newly confirmed IoC move from Detected to Confirmed.
  • Active Learning - we built clever prioritization of threat analytics tasks for human analysts - the Machine Learning system identifies the most promising potential new IoCs for confirmation by human, among millions of suspicious - previously unknown - findings each week. Note that we build on top of A.I. Loop, first introduced in Closing one learning loop blog. Note that we also improved robustness of Machine Learning evaluation pipelines (see our NeurIPS paper: Bad practices in evaluation methodology relevant to class-imbalanced problems)

 

The cumulative effect of these updates can be seen in Coverage of Confirmed Threats Extended blog, see net-new Confirmed Threats and increase in IoC numbers in updated Confirmed Threats.

IoCs High Risk AutoUpdate.pngIoCs Medium Risk AutoUpdate.pngIoCs Low Risk AutoUpdate.pngThe increase of the number of maintained IoCs is seen in absolute nubers per week above. Note the difference between end of 2018 and 2019 so far. Note also the solid line - the percentage of Confirmed Threats covered by fully-automated Machine Learned update process. Higher percentage means less regular update work by human analyst, thus more time available for net new exploration.