2017 Predictive Analytics Symposium

Transcription

1 2017 Predictive Analytics Symposium Session 35, Kaggle Contests--Tips From Actuaries Who Have Placed Well Moderator: Kyle A. Nobbe, FSA, MAAA Presenters: Thomas DeGodoy Shea Kee Parkes, FSA, MAAA SOA Antitrust Compliance Guidelines SOA Presentation Disclaimer

2 Predictive Modeling Contests Tom de Godoy

3 Tom de Godoy CTO & Co-founder, DataRobot DataRobot is an Automated Machine Learning Platform 15 years of experience in Insurance Analytics Previously, Director of Research & Modeling at Travelers Insurance Advisor to the DataRobot Insurance practice that works with a large number of insurance companies Founded in 2012, Funding over $100 million Experts with 70+ years of insurance analytics experience DataRobot s insurance portfolio includes Fortune 100, Regional players, global players and InsurTechs This session is based on my experience working with leading insurers and then, founding a Machine Learning company that is helping hundreds of companies in their Machine Learning journey.

4 Why Kaggle? - Money prizes? Glory? - Learn Machine Learning by doing it! - Be part of a large community of data scientists

5 Why Learn Machine Learning? Traditional data Unstructured data 90% of the data in the world today has been created in the last two years alone open source programming democratization Avalanche of new data Big Data environment: Velocity, Volume, Variety Open-Source Innovations ML driven by open-source and academics Better Product Better Service Optimised Operations Low-cost computing Disruptive Competition New business models around data

6 Predictive Analytics is a Competition Keys to Building a Competitive Advantage: 1. The ability to identify opportunities 2. The ability to execute on these opportunities 3. Better predictive models than your competitor

7 Keys to Winning This Competition Develop Models Better and Faster Than Your Competitors 1. Knowledge of the data and of the business problem 2. Large and diverse set of algorithms 3. Robust model validation 4. Speed

8 Know Your Data Most useful insights about the data come from machine learning models. Simple ways to know your data : - Data dictionary - Simple profiles & summaries - Interactive queries Insights from machine learning models: - Identify important features - Visualize partial dependencies - Discover non-linear effects and interactions - Discover prediction outliers and their reasons

9 Quick Prototype & Rapid Iterations Rapid iteration and early socialization are the key! rapid iteration socialize feedback loop prototype

10 Leverage a Large and Diverse Set of Algorithms For each particular method there are situations for which it is particularly well suited, and others where it performs badly compared to the best that can be done with that data Source:

11 How to Leverage More Algorithms? For each new algorithm, you need to figure out... What library/implementation should you use? How do you tune the model? How do you prepare the data for the model? How do you score new data with this model? How do you run it faster and less costly?

12 How to Leverage More Algorithms? Automated Machine Learning Platform Having a diverse set of algorithms is key to maximizing accuracy.

13 Robust Validation Don t trust the leaderboard. Trust your own cross validation. On your own cross-validation framework, evaluate your models using: - Ranking & accuracy metrics (AUC, Gini, R-Squared, MSE etc) - Lift charts & dual-lift charts - Feature importance plots - Partial dependency plots - Reason codes This cross-validation framework should be used only for evaluation and not for tuning

14 A Lesson from Kaggle

15 Trust Your Own Cross-Validation

16 Speed Speed is a limiting factor for: You Must be Faster than Your Competitors - Leveraging a large number of features (and data sources) - Modeling complex types of data - Using many models to discover the best solution - Maximizing model accuracy - Doing robust validation of any model

17 The #1 Barrier: The Traditional Approach is Hard! R Python Spark Hadoop Hacking & Coding Skills DATA SCIENCE Math & Stats Logistic Regression GLM GBM Random Forest Decision Trees Neural Nets Deep Learning Text Mining Feature Engineering Blending Cross Validation Domain Expertise

18 Advantages of Automated Machine Learning 1. Time to Value: 10x faster to build and deploy predictive models. 2. Accuracy: Unprecedented accuracy of models out-of-the-box. 3. Transparency: Easy to know your model and collaborate on projects. 4. Pervasiveness: Simple UI and workflow for people of various backgrounds to leverage machine learning. 5. Democratization: Not limited to data scientists. 6. Consistency: Best practices in model building, validation and deployment applied consistently in every project.

19 Summary - Know your data (with multivariate model insights) - Leverage a large and diverse set of algorithms - Apply robust model validation - Speed is critical! - Leverage automation as much as possible

20 Questions?

21 How to do well at a Kaggle contest Shea Parkes, FSA MAAA

22 Limitations The views expressed in this presentation are those of the presenter, and not those of Milliman or the Society of Actuaries. Nothing in this presentation is intended to represent a professional opinion or be an interpretation of actuarial standards of practice. 2

23 3

24 Focus on a single contest 4

25 Join a contest when it begins 5

26 Read all contest information 6

27 Participate in forums 7

28 Participate in notebooks and kernels 8

29 Join a team 9

30 Spend a ton of time feature engineering 10

31 Setup appropriate validation framework 11

32 Use existing implementations of common algorithms 12

33 Write code 13

34 Use GitHub 14