Computational Machine Learning

Transcription

1 Computational Machine Learning Zaid Harchaoui NYU Fall 2015 Zaid Harchaoui (NYU) CSCI-GA Fall / 40

2 Outline 1 Overview of machine learning Unsupervised learning Supervised learning Learning Machines Feature representation 2 Course logistics Zaid Harchaoui (NYU) CSCI-GA Fall / 40

3 Outline 1 Overview of machine learning Unsupervised learning Supervised learning Learning Machines Feature representation 2 Course logistics Zaid Harchaoui (NYU) CSCI-GA Fall / 40

4 Statistical learning : a tentative big picture Unsupervised learning (learning without a teacher) Find structure of x X, given observations x i, i = 1,..., n Supervised learning (learning with a teacher) Predict y Y from x X, given observations (x i, y i ), i = 1,..., n Zaid Harchaoui (NYU) CSCI-GA Fall / 40

5 Statistical learning : a tentative big picture Applications in many fields Computer vision Bioinformatics Audio/speech processing Text mining Computational astronomy etc. Interplays interplay between computer science and data science, towards ai interplay between theory, algorithms, and real-world applications Zaid Harchaoui (NYU) CSCI-GA Fall / 40

6 A blend of scientific disciplines A blend of scientific disciplines computer science : algorithmics, complexity, information theory data science : statistics, experimental design applied mathematics : matrix analysis, convex optimization, probability, Application in other scientific fields bioinformatics computer vision speech/audio processing neuroscience finance Zaid Harchaoui (NYU) CSCI-GA Fall / 40

7 Unsupervised learning Unsupervised learning Dimension reduction face images Zambian President Levy Mwanawasa has won a second term in office in an election his challenger Michael Sata accused him of rigging, official results showed on Monday. documents According to media reports, a pair of hackers said on Saturday that the Firefox Web browser, commonly perceived as the safer and more customizable alternative to market leader Internet Explorer, is critically flawed. A presentation on the flaw was shown during the ToorCon hacker conference in San Diego. gene expression data MEG readings Zaid Harchaoui (NYU) CSCI-GA Fall / 40

8 Unsupervised learning Unsupervised learning Dimension reduction Computational efficiency : space and time savings Statistical performance : fewer dimensions regularization Visualization : discover underlying structure of the data PCA and KPCA Zaid Harchaoui (NYU) CSCI-GA Fall / 40

9 Unsupervised learning Unsupervised learning Feature extraction z ϕ x (x) ϕ y (y) x: y: A view from Idyllwild, California, with pine trees and snow capped Marion Mountain under a blue sky. Learn kernelized projections that relate both spaces Zaid Harchaoui (NYU) CSCI-GA Fall / 40

10 Unsupervised learning Unsupervised learning Feature extraction Multimodality : leverage the correlation between the modalities Statistical performance : take advantage of both views of the data Putting in relation : discover underlying relations between the modalities CCA and KCCA Zaid Harchaoui (NYU) CSCI-GA Fall / 40

11 Unsupervised learning Unsupervised learning Clustering Zaid Harchaoui (NYU) CSCI-GA Fall / 40

12 Unsupervised learning Unsupervised learning Clustering Semantics : grouping datapoints in meaningful clusters Statistical performance : intrinsic degrees of freedom of the data Visualization : discover groupings between datapoints spectral clustering, temporal segmentation, and regularized clustering (DIFFRAC) Zaid Harchaoui (NYU) CSCI-GA Fall / 40

13 Unsupervised learning Unsupervised learning Detection problems Zaid Harchaoui (NYU) CSCI-GA Fall / 40

14 Unsupervised learning Unsupervised learning Detection problems Balance risks : control detection rate with a guaranteed false alarm probability Power : detect differences not only in mean or covariance homogeneity testing, change detection Zaid Harchaoui (NYU) CSCI-GA Fall / 40

15 Supervised learning Supervised learning Human action recognition Zaid Harchaoui (NYU) CSCI-GA Fall / 40

16 Supervised learning Supervised learning Image classification and scene understanding Variance : high intra-class variability Structure : spatial and temporal structure Unknowns : unknown localization of the object of interest Kernel ridge regression, Kernel logistic regression, Support vector machine Zaid Harchaoui (NYU) CSCI-GA Fall / 40

17 Supervised learning Image categorization/classification Zaid Harchaoui (NYU) CSCI-GA Fall / 40

18 Supervised learning Image categorization/classification Zaid Harchaoui (NYU) CSCI-GA Fall / 40

19 Learning Machines Artificial Learning Machines ( s) Zaid Harchaoui (NYU) CSCI-GA Fall / 40

20 Learning Machines Perceptron (Rosenblatt, 1957) Zaid Harchaoui (NYU) CSCI-GA Fall / 40

21 Learning Machines Perceptron (Rosenblatt, 1957) Zaid Harchaoui (NYU) CSCI-GA Fall / 40

22 Learning Machines Convolutional Neural Nets : origins Zaid Harchaoui (NYU) CSCI-GA Fall / 40

23 Learning Machines Neocognitron (Fukushima, 1980) Zaid Harchaoui (NYU) CSCI-GA Fall / 40

24 Learning Machines Convolutional Neural Net (LeCun, 1988) Figure : Picture from (LeCun, 1998) CNNs perform simple operations such as convolutions, point-wise non-linearities and subsampling. for most successful applications of CNNs, training is supervised. Zaid Harchaoui (NYU) CSCI-GA Fall / 40

25 Feature representation Visual recognition : traditional approach Zaid Harchaoui (NYU) CSCI-GA Fall / 40

26 Feature representation Visual recognition : traditional features Zaid Harchaoui (NYU) CSCI-GA Fall / 40

27 Feature representation Parallel approaches in vision, speech, and NLP Zaid Harchaoui (NYU) CSCI-GA Fall / 40

28 Feature representation Challenges in vision, speech, and NLP Zaid Harchaoui (NYU) CSCI-GA Fall / 40

29 Feature representation Recent popular strategy : learning all the way through Recent popular strategy : learning all the way through 1 Low-level feature representation is learnt from raw data 2 Mid-level feature representation is learnt from data 3 High-level feature representation should be learned from data 4 Downstream learning classifier is learnt from data Zaid Harchaoui (NYU) CSCI-GA Fall / 40

30 Course logistics Outline 1 Overview of machine learning Unsupervised learning Supervised learning Learning Machines Feature representation 2 Course logistics Zaid Harchaoui (NYU) CSCI-GA Fall / 40

31 Course logistics Computational Machine Learning, CSCI-GA Class webpage : Syllabus on the website Piazza : Ask questions here Zaid Harchaoui (NYU) CSCI-GA Fall / 40

32 Course logistics Evaluation About 4 to 5 homeworks Midterm Exam Project Extra Credit Opportunities Machine learning flashcards Optional problems or competitions on the homework Zaid Harchaoui (NYU) CSCI-GA Fall / 40

33 Course logistics Homework First assignment out after the 3rd course Due Fri. Sep. 25th noon Late homework : Accepted with a 10% penalty per hour late. Zaid Harchaoui (NYU) CSCI-GA Fall / 40

34 Course logistics Homework First assignment out after the 3rd course Due Fri. Sep. 25th noon Late homework : Accepted with a 10% penalty per hour late. Collaboration is fine, but Write up solutions and code on your own List names of who you talked to about each problem Zaid Harchaoui (NYU) CSCI-GA Fall / 40

35 Course logistics Homework First assignment out after the 3rd course Due Fri. Sep. 25th noon Late homework : Accepted with a 10% penalty per hour late. Collaboration is fine, but Write up solutions and code on your own List names of who you talked to about each problem Zaid Harchaoui (NYU) CSCI-GA Fall / 40

36 Course logistics Midterm In class during lecture Zaid Harchaoui (NYU) CSCI-GA Fall / 40

37 Course logistics Projects Find some new data or new approach to old data Project philosophy the same as in these courses : projects.html Zaid Harchaoui (NYU) CSCI-GA Fall / 40

38 Course logistics Projects Find some new data or new approach to old data Project philosophy the same as in these courses : projects.html Logistics : 2 students per group or Start meeting asap with instructor to discuss potential ideas for project Project proposal due on Week 6 Zaid Harchaoui (NYU) CSCI-GA Fall / 40

39 Course logistics Prerequisites Introduction to Data Science (DS-GA 1001) or equivalent data science-ish course Math Multivariate Calculus Linear Algebra Probability Theory Statistics Python programming (numpy) Zaid Harchaoui (NYU) CSCI-GA Fall / 40

40 Course logistics General Philosophy Mastery vs Performance (understanding vs getting the grade ) Zaid Harchaoui (NYU) CSCI-GA Fall / 40

41 Course logistics General Philosophy Mastery vs Performance (understanding vs getting the grade ) Don t confuse kind of understanding with actual understanding Zaid Harchaoui (NYU) CSCI-GA Fall / 40

42 Course logistics General Philosophy Mastery vs Performance (understanding vs getting the grade ) Don t confuse kind of understanding with actual understanding Can you explain this picture? Zaid Harchaoui (NYU) CSCI-GA Fall / 40

43 Course logistics Course Topics Week 1 : Overview of Machine Learning Week 2 : Perceptron, Neocognitron, Performance metrics, Capacity, Regularization Week 3 : Linear Predictors, Ridge Regression, Logistic Regression, Linear Support Vector Machines, Boosting Week 4 : Convex Learning Problems, Stochastic Gradient Descent Week 5 : Faster Stochastic Gradient Descent, Model Selection, Validation Week 6 : Kernel-based Methods, Boosting Week 7 : Decision Trees, Random Forests, Ensemble Methods Zaid Harchaoui (NYU) CSCI-GA Fall / 40

44 Course logistics Course Topics Week 8 : Deep Neural Networks, I Week 9 : Quantization, Clustering, Compression Week 10 : Dimensionality Reduction Week 11 : Latent Variable Models Week 12 : Feature Selection. Feature Design Week 13 : Deep Neural Networks, II Week 14 : The Art of Machine Learning Modelling Zaid Harchaoui (NYU) CSCI-GA Fall / 40

45 Course logistics Questions? What are you looking to get out of the course? Questions for me? Zaid Harchaoui (NYU) CSCI-GA Fall / 40