Machine Learning and Development Policy

Transcription

1 Machine Learning and Development Policy Sendhil Mullainathan (joint papers with Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Ziad Obermeyer)

2

3

4

5 Magic? Hard not to be wowed But what makes it tick? Could that be used elsewhere? In my own work?

6

7 AI Approach We do it perfectly. Introspect how Program that up

8 Programming For each review make a vector of words Figure out whether it has positive words and negative words Count

9 Trying to Copy Humans Brilliant Dazzling Cool Gripping Moving 60% Bad Suck Cliched Slow Awful

10 This Approach Stalled Trivial problems proved impossible Marvin Minsky once assigned "the problem of computer vision" as a summer project Forget about the more complicated problems like language

11

12 What is the magic trick? Make this an empirical exercise Collect some data Look at what combination of words predicts being a good review Example dataset: 2000 movie reviews 1000 good and 1000 bad reviews

13 Learning not Programming Love Superb STILL 95% Bad Stupid Great? Worst! Pang, Lee and Vaithyanathan

14 Machine learning Turn any intelligence task into an empirical learning task Specify what is to be predicted Specify what is used to predict it Y = {0, 1} {z } Positive? X = {0, 1} k {z } Word Vector ˆf = argmin f E[L(f(x),y)]

15 Machine learning Turn any intelligence task into an empirical learning task Specify what is to be predicted Specify what is used to predict it Underneath most machine intelligence you see Not a coincidence that ML and big data arose together

16 Wonderful Great that engineers discovered the 100+ year old field of statistics! We ve been estimating functions from data for a long time In part true But far from the full story

17 Pang, Lee and Vaithyanathan

18 Machine Learning High dimensional statistical procedure Despite this machine learning can do well It s no surprise we fit well We fit well out of sample

19 So Estimation Fit Y with X Machine Learning Fit Y with X out of sample Low dimensional High dimensional JUST BETTER?

20 Data S n =(y i,x i ) iid Reviews Function Class F Vector of Words Estimation ˆ Estimates ˆf [f, f] Unbiased functions E S [ ˆf A,S ]=f = E[y x] {z } Right ˆf Sen9ment Predictor Data size Information going in Thousands? Estimates Information coming out Tens of Thousands How can we get more information out than we re putting in?

21 Data S n =(y i,x i ) iid Reviews Function Class F Vector of Words Estimation ˆ Estimates ˆf [f, f] Unbiased functions E S [ ˆf A,S ]=f = E[y x] {z } Right ˆf Sen9ment Predictor Do we need this?

22 Unbiased functions E S [ ˆf A,S ]=f = E[y x] {z } Right ˆf Gets more out? Put more in

23 Prediction and Estimation Estimation Adjudicate between variables Confidence intervals around coefficients Prediction Do not adjudicate Arbitrary choices to deal with covariance

24 Estimation vs Prediction Estimation Strict assumptions about data generating process Prediction Allow for flexible functional forms Back out parameters Get individual predictions Low dimensional Do not adjudicate between observably similar functions (variables) ˆβ ŷ i

25 Great for Engineers. Much of what we do is inference of coefficiens In fact we fret about causal inference What use is a procedure where even the coefficients aren t meaningful?

26 Applications of Machine Learning New Data Prediction in Policy

27 Applications of Machine Learning New Data Prediction in Policy

28 An Example New Data

29

30 Xie et. al. (2016)

31 What does this have to do with ML? Processing of data requires machine learning How do you relate luminosity to income levels?

32 Crop Yield

33

34 Cell Phone Data Blumenstock et. al. 2015

35 Blumenstock et. al. (2015)

36

37 New Kinds of Data Measurement has always played a central role in development These new data give us a new way to measure Not the depth that Morduch will discuss tomorrow But breadth And a very different look at life.

38 Applications of Machine Learning New Data Prediction in Policy

39 Applications of Machine Learning New Data Prediction in Policy

40 Question Can prediction be directly useful in policy? These decisions seem inherently causal Should we do policy X? What will X do? What happens with and without X? In fact decisions seem inherently causal

41 Two Toy Policy Decisions Rain Dance Causa9on ˆ Umbrella Predic9on ŷ Common Elements Both are decisions with payoffs Both rely on data of the type: Y = rain, X = variables correlated with rain Both use data to estimate function y = f(x)

42 Framework X Atmospheric Condi9ons Decision X 0 Rain Dance Y Rain Causa9on

43 Framework X Atmospheric Condi9ons Decision X 0 Umbrella Y Rain Predic9on

44 Atmospheric Condi9ons Atmospheric Condi9ons X X X 0 X 0 Rain Dance Y Rain Umbrella Y Rain Causa9on Predic9on Experiments Machine Learning

45 X X 0 Causation Y dπ = Π (Y ) + Π dx 0 X 0 Y Y X 0 Prediction Prediction Causation

46 Are there Umbrella Problems? Decisions where predictions matter Where we can have big social impact And with enough data Prediction policy problems

47 Prediction

48 A Policy Problem in the US Each year police make over 12 million arrests Where do people wait for trial? Release vs. detain high stakes Pre-trial detention spells avg. 2-3 months (can be up to 9-12 months) Nearly 750,000 people in jails in US Consequential for jobs, families as well as crime Kleinberg Lakkaraju Leskovec Ludwig and Mullainathan

49 Judge s Problem Judge must decide whether to release or not (bail) Defendant when out on bail can behave badly: Fail to appear at case Commit a crime The judge is making a prediction

50 Prediction Policy Problem Large dataset of decisions Build a prediction algorithm

51 Build a Decision Aid? Simplest aid: safeguard

52 Build a Decision Aid? Simplest aid: safeguard Re-ranking?

53 Bail Not Unique Pure prediction problems: Poverty targeting (Adelmen et. al. 2016) Retail crystal ball: Weather and yield prediction (Rosenzweig and Udry 2013) Teacher selection (Predict non-attendance?) Pseudo-prediction problems Treatment effects depend on risk Predict risk Target high risk pregnancies for hospital delivery

54 Key Inputs Problem: Prediction affects deicision Individual, micro decisions Inputs Reasonable individual data Large samples (10,000+?)

55 Conclusion Fortunate enough to see two large changes in policy: RCTs Behavioral economics I think this will be the next one Three papers I ve drawn on: Machine Learning: An Econometric Approach, wih Jann Spiess, Journal of Economic Perspectives, forthcoming. Human Decisions and Machine Predictions, with Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, and Jens Ludwig Targeting Poverty by Predicting Poverty: Using Machine Learning in Targeted Transfer Program, with Melissa Adelman, Jonathan Glidden, Paul Niehaus, and Jack Willis.