Optimization for Data Science

Transcription

1 Optimization for Data Science Master 2 Data Science, Univ. Paris Saclay Robert M. Gower & Alexandre Gramfort

2 Core Info Where : Telecom ParisTech Location : Amphi Estaunié or B312 ECTS : 5 ECTS Volume : 40h When : 12 weeks (including one week break for holidays + one week for exam) Online: All teaching materials on moodle: Students upload their projects / reports via moodle too. All students **must** be registered on moodle.

3 Who am I? Robert M. Gower Assistant Prof at Telecom Research topics: Stochastic algorithms for optimization, numerical linear algebra, quasi-newton methods and automatic differentiation (backpropagation).

4 Introduction to Optimization in Machine Learning Robert M. Gower Master 2 Data Science, Univ. Paris Saclay Optimisation for Data Science

5 An Introduction to Supervised Learning

6 References for this class Chapter 1 Understanding Machine Learning: From Theory to Algorithms Pages 67 to 79 Convex Optimization

7 Is There a Cat in the Photo? Yes No

8 Is There a Cat in the Photo? Yes

9 Is There a Cat in the Photo? Yes

10 Is There a Cat in the Photo? No

11 Is There a Cat in the Photo? Yes

12 Is There a Cat in the Photo? Yes No x: Input/Feature y: Output/Target Find mapping h that assigns the correct target to each input

13 Labeled Data: The training set

14 Labeled Data: The training set y= -1 means no/false

15 Labeled Data: The training set y= -1 means no/false Learning Algorithm

16 Labeled Data: The training set y= -1 means no/false Learning Algorithm

17 Labeled Data: The training set y= -1 means no/false Learning Algorithm -1

18 Example: Linear Regression for Height Labeled data Sex Male Sex Female Age 30 Age 70 Height 1,72 cm Height 1,52 cm

19 Example: Linear Regression for Height Labeled data Sex Male Sex Female Age 30 Age 70 Height 1,72 cm Height 1,52 cm Example Hypothesis: Linear Model

20 Example: Linear Regression for Height Labeled data Sex Male Sex Female Age 30 Age 70 Height 1,72 cm Height 1,52 cm Example Hypothesis: Linear Model Example Training Problem:

21 Linear Regression for Height H e i g h t Age

22 Linear Regression for Height H e i g h t The Training Algorithm Age

23 Linear Regression for Height H e i g h t The Training Algorithm Other options aside from linear? Age

24 Parametrizing the Hypothesis Linear: Polinomial: Neural Net: H e i g h t H e i g h t Age Age

25 Loss Functions Why a Squared Loss?

26 Loss Functions Why a Squared Loss? Loss Functions The Training Problem

27 Loss Functions Why a Squared Loss? Loss Functions The Training Problem Typically a convex function

28 Choosing the Loss Function Quadratic Loss Binary Loss Hinge Loss

29 Choosing the Loss Function Quadratic Loss Binary Loss Hinge Loss y=1 in all figures

30 Choosing the Loss Function Quadratic Loss Binary Loss Hinge Loss EXE: Plot the binary and hinge loss function in when y=1 in all figures

31 Loss Functions Is a notion of Loss enough? What happens when we do not have enough data?

32 Loss Functions The Training Problem Is a notion of Loss enough? What happens when we do not have enough data?

33 Overfitting and Model Complexity Fitting 1st order polynomial

34 Overfitting and Model Complexity Fitting 1st order polynomial

35 Overfitting and Model Complexity Fitting 3rd order polynomial

36 Overfitting and Model Complexity Fitting 9th order polynomial

37 Regularization Regularizor Functions General Training Problem

38 Regularization Regularizor Functions General Training Problem Goodness of fit, fidelity term...etc

39 Regularization Regularizor Functions General Training Problem Goodness of fit, fidelity term...etc Penlizes complexity

40 Regularization Regularizor Functions Controls tradeoff between fit and complexity General Training Problem Goodness of fit, fidelity term...etc Penlizes complexity

41 Regularization Regularizor Functions Controls tradeoff between fit and complexity General Training Problem Goodness of fit, fidelity term...etc Exe: Penlizes complexity

42 Overfitting and Model Complexity Fitting kth order polynomial

43 Overfitting and Model Complexity For λ big enough, the solution is a 2nd order polynomial Fitting kth order polynomial

44 Exe: Ridge Regression Linear hypothesis L2 loss Ridge Regression L2 regularizor

45 Exe: Support Vector Machines Linear hypothesis Hinge loss SVM with soft margin L2 regularizor

46 Exe: Logistic Regression Linear hypothesis Logistic loss Logistic Regression L2 regularizor

47 The Machine Learners Job

48 The Machine Learners Job

49 The Machine Learners Job

50 The Machine Learners Job

51 The Machine Learners Job

52 The Machine Learners Job

53 The Statistical Learning Problem: The hard truth Do we really care if the loss is small on the known labelled data paris (xi,yi)? Nope We really want to have a small loss on new unlabelled Observations! Assume data sampled distribution where is an unknown

54 The Statistical Learning Problem: The hard truth The statistical learning problem: Minimize the expected loss over an unknown expectation Variance of sample mean: