Machine Learning And the Peak of Inflated Expectations. TK Keanini Distinguished Engineer June 2018

Transcription

1 Machine Learning And the Peak of Inflated Expectations TK Keanini Distinguished Engineer June 2018

2 Gartner Hype Cycle for Emerging Technologies 2017 Expectations M A C H I N E L E A R N I N G Source: Gartner (July, 2017) Time Innovation Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity

3 Vendors Got Us Here Advanced Threats are no match for A.I. Our machines detect threats others cannot 100% predictive

4 How We Disservice Machine Learning Silver Bullet Marketing No Explanation or Discussion Limited Guidance 2018 Cisco and/or its affiliates. All rights reserved. Cisco Pulbic

5 M A C H INE L E A RNING What it is

6 Field of study that gives computers the ability to learn without being explicitly programmed. Arthur Samuel s definition of machine learning in 1959

7 clustering bayesian clustering bayesian ground truth ground truth instance based instance based ensemble ensemble machine learning algorithms machine learning algorithms regularization regularization rule system rule system classifier classifier deep learning deep learning regression regression neural network neural network decision tree decision tree dimensionality reduction dimensionality reduction N E R D A L E R T Let s define the helpful data science terms

8 Machine Learning The Big Picture Artificial Intelligence Machine Learning Supervised Learning Unsupervised Learning Reinforcement Learning

9 Machine Learning Common Techniques Supervised Learning When you know the question you are trying to ask and have examples of it being asked and answered correction Unsupervised Learning You don't have answers and may not fully know the questions Reinforcement Learning The other category Trial and error behavior effective in game scenarios

10 75% 15% 10% E Supervised Learning Unsupervised Learning Other (Reinforcement Learning, etc.)

11 What did we do before Machine Learning? Use in combination with Machine Learning a Simple Pattern Matching Statistical Methods Rules and First Order Logic (FoL)

12 M A C H INE L E A RNING Techniques

13 Field of study that gives computers the ability to learn without being explicitly programmed. Translation Field of study that gives computers the ability to be implicitly programmed.

14 Training Classifiers Training Data Machine Learning Algorithm New Data Classifier Prediction

15 Ground Truth Used in Supervised Learning The 'Ground Truth' is the pairing of example questions and answers. If you can phrase a problem as 'we know this is right, learn a way to answer more questions of this type'. Success depends greatly on the dataset expressing the Question -> Answer mapping.

16 M A C H INE L E A RNING Pitfalls

17 One Size Does Not Fit All Other ML Application Security N E R D A L E R T Warning: Success in one domain does not guarantee success in another

18 What Is At Stake Matters Because you watched Deadpool, you might like Deadpool X-Men: First Class The Flash Captain America: The First Avenger

19 How did you come to that conclusion? The Explainability Problem Normal Workflow CFO daily calendar Irregular Activity ML detects suspicious activity and suggests remediation Quarantined However, ML cannot articulate *why* it wants to remediate Loss of time and resources

20 M A C H INE L E A RNING For Security

21 How We Know Machine Learning is Working Accuracy How often does my classifier give me the correct answer? Precision When my classifier predicts an instance in a certain class, how often does the instance belong to that class? N E R D A L E R T Root mean square error & Logical Regression Translation: On average, how far away are my predictions from what we later know to be true values?

22 Why is Machine Learning so useful in Security? Static With limited variability or is well-understood Evolving Security The security domain is always evolving, has a large amount of variability, and is not well-understood

23 Insider Threats and Behavioral Security Analytics Attackers They re not breaking in, they are logging in Detecting Through novelty and outliers Events Turn weak signals into a strong one

24 Classify the Observable World and Infer the Rest Threat Actor Activity Weird Stuff (but not threat related) Normal Activity

25 Multi-layer Analytical Pipeline Cascade of specialized layers of Machine Learning algorithms Billions of connections Anomaly Detection and Trust Modeling Event Classification and Entity Modeling Relationship Modeling Multiple-Instance Learning Probabilistic Threat Propagation Neural Networks Graph-Statistical Methods Statistical Methods Information-Theoretical Methods 70+ Unsupervised Anomaly Detectors Dynamic Adaptive Ensemble Creation Rule Mining Random Forests Boosting ML: Supervised Learning Random Graphs Graph Methods Supervised Classifier Training

26 New Oct. 3 Oct. 4 C&C url Oct. 15 Anomalous http Oct. 16 Heavy uploader Dropbox.com Oct. 25 Oct. 28 Malicious http Recurring Security that Shows its Work Malware: sality Dec days 3 Spam tracking #CSPM02 8 Information Stealer #CDCH01

27 Measure the Right Things Efficacy of the Assertions True/False Positive True/False Negative Overfitting/Undefitting Root Mean Squared Error

28 Measure the Right Things

29 Conclusion

30 What to Ask Your Vendor How are you applying Machine Learning in your product and why? How do you measure its effectiveness? Regarding supervised learning, what are you using for ground truth? What non-machine learning are you using and why? What papers or open-source have you published regarding your analytics? For the ML based assertions, what entailments are provided? 2018 Cisco and/or its affiliates. All rights reserved. Cisco Pulbic

31 A Good Machine Learning Approach Be Pragmatic Entailments Analytical pipeline, over single technique Success is Domain Specific Measure helpfulness, not mathematical accuracy

32 N E R D A L E R T Thank you!