Computational Machine Learning Zaid Harchaoui NYU Fall 2015 Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 1 / 40
Outline 1 Overview of machine learning Unsupervised learning Supervised learning Learning Machines Feature representation 2 Course logistics Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 2 / 40
Outline 1 Overview of machine learning Unsupervised learning Supervised learning Learning Machines Feature representation 2 Course logistics Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 3 / 40
Statistical learning : a tentative big picture Unsupervised learning (learning without a teacher) Find structure of x X, given observations x i, i = 1,..., n Supervised learning (learning with a teacher) Predict y Y from x X, given observations (x i, y i ), i = 1,..., n Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 4 / 40
Statistical learning : a tentative big picture Applications in many fields Computer vision Bioinformatics Audio/speech processing Text mining Computational astronomy etc. Interplays interplay between computer science and data science, towards ai interplay between theory, algorithms, and real-world applications Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 5 / 40
A blend of scientific disciplines A blend of scientific disciplines computer science : algorithmics, complexity, information theory data science : statistics, experimental design applied mathematics : matrix analysis, convex optimization, probability, Application in other scientific fields bioinformatics computer vision speech/audio processing neuroscience finance Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 6 / 40
Unsupervised learning Unsupervised learning Dimension reduction face images Zambian President Levy Mwanawasa has won a second term in office in an election his challenger Michael Sata accused him of rigging, official results showed on Monday. documents According to media reports, a pair of hackers said on Saturday that the Firefox Web browser, commonly perceived as the safer and more customizable alternative to market leader Internet Explorer, is critically flawed. A presentation on the flaw was shown during the ToorCon hacker conference in San Diego. gene expression data MEG readings Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 7 / 40
Unsupervised learning Unsupervised learning Dimension reduction Computational efficiency : space and time savings Statistical performance : fewer dimensions regularization Visualization : discover underlying structure of the data PCA and KPCA Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 8 / 40
Unsupervised learning Unsupervised learning Feature extraction z ϕ x (x) ϕ y (y) x: y: A view from Idyllwild, California, with pine trees and snow capped Marion Mountain under a blue sky. Learn kernelized projections that relate both spaces Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 9 / 40
Unsupervised learning Unsupervised learning Feature extraction Multimodality : leverage the correlation between the modalities Statistical performance : take advantage of both views of the data Putting in relation : discover underlying relations between the modalities CCA and KCCA Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 10 / 40
Unsupervised learning Unsupervised learning Clustering Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 11 / 40
Unsupervised learning Unsupervised learning Clustering Semantics : grouping datapoints in meaningful clusters Statistical performance : intrinsic degrees of freedom of the data Visualization : discover groupings between datapoints spectral clustering, temporal segmentation, and regularized clustering (DIFFRAC) Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 12 / 40
Unsupervised learning Unsupervised learning Detection problems Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 13 / 40
Unsupervised learning Unsupervised learning Detection problems Balance risks : control detection rate with a guaranteed false alarm probability Power : detect differences not only in mean or covariance homogeneity testing, change detection Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 14 / 40
Supervised learning Supervised learning Human action recognition Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 15 / 40
Supervised learning Supervised learning Image classification and scene understanding Variance : high intra-class variability Structure : spatial and temporal structure Unknowns : unknown localization of the object of interest Kernel ridge regression, Kernel logistic regression, Support vector machine Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 16 / 40
Supervised learning Image categorization/classification Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 17 / 40
Supervised learning Image categorization/classification Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 18 / 40
Learning Machines Artificial Learning Machines (1940-1950s) Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 19 / 40
Learning Machines Perceptron (Rosenblatt, 1957) Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 20 / 40
Learning Machines Perceptron (Rosenblatt, 1957) Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 21 / 40
Learning Machines Convolutional Neural Nets : origins Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 22 / 40
Learning Machines Neocognitron (Fukushima, 1980) Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 23 / 40
Learning Machines Convolutional Neural Net (LeCun, 1988) Figure : Picture from (LeCun, 1998) CNNs perform simple operations such as convolutions, point-wise non-linearities and subsampling. for most successful applications of CNNs, training is supervised. Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 24 / 40
Feature representation Visual recognition : traditional approach Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 25 / 40
Feature representation Visual recognition : traditional features Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 26 / 40
Feature representation Parallel approaches in vision, speech, and NLP Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 27 / 40
Feature representation Challenges in vision, speech, and NLP Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 28 / 40
Feature representation Recent popular strategy : learning all the way through Recent popular strategy : learning all the way through 1 Low-level feature representation is learnt from raw data 2 Mid-level feature representation is learnt from data 3 High-level feature representation should be learned from data 4 Downstream learning classifier is learnt from data Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 29 / 40
Course logistics Outline 1 Overview of machine learning Unsupervised learning Supervised learning Learning Machines Feature representation 2 Course logistics Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 30 / 40
Course logistics Computational Machine Learning, CSCI-GA-3033-012 Class webpage : http://cs.nyu.edu/courses/fall15/csci-ga.3033-012/ Syllabus on the website Piazza : https://piazza.com/nyu/fall2015/csciga3033012/ Ask questions here Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 31 / 40
Course logistics Evaluation About 4 to 5 homeworks Midterm Exam Project Extra Credit Opportunities Machine learning flashcards Optional problems or competitions on the homework Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 32 / 40
Course logistics Homework First assignment out after the 3rd course Due Fri. Sep. 25th noon Late homework : Accepted with a 10% penalty per hour late. Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 33 / 40
Course logistics Homework First assignment out after the 3rd course Due Fri. Sep. 25th noon Late homework : Accepted with a 10% penalty per hour late. Collaboration is fine, but Write up solutions and code on your own List names of who you talked to about each problem Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 33 / 40
Course logistics Homework First assignment out after the 3rd course Due Fri. Sep. 25th noon Late homework : Accepted with a 10% penalty per hour late. Collaboration is fine, but Write up solutions and code on your own List names of who you talked to about each problem Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 33 / 40
Course logistics Midterm In class during lecture Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 34 / 40
Course logistics Projects Find some new data or new approach to old data Project philosophy the same as in these courses : http://cs.nyu.edu/~dsontag/courses/ml14/assignments/ projects.html http://web.stanford.edu/class/cs221/project.html Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 35 / 40
Course logistics Projects Find some new data or new approach to old data Project philosophy the same as in these courses : http://cs.nyu.edu/~dsontag/courses/ml14/assignments/ projects.html http://web.stanford.edu/class/cs221/project.html Logistics : 2 students per group or Start meeting asap with instructor to discuss potential ideas for project Project proposal due on Week 6 Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 35 / 40
Course logistics Prerequisites Introduction to Data Science (DS-GA 1001) or equivalent data science-ish course Math Multivariate Calculus Linear Algebra Probability Theory Statistics Python programming (numpy) Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 36 / 40
Course logistics General Philosophy Mastery vs Performance (understanding vs getting the grade ) Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 37 / 40
Course logistics General Philosophy Mastery vs Performance (understanding vs getting the grade ) Don t confuse kind of understanding with actual understanding Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 37 / 40
Course logistics General Philosophy Mastery vs Performance (understanding vs getting the grade ) Don t confuse kind of understanding with actual understanding Can you explain this picture? Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 37 / 40
Course logistics Course Topics Week 1 : Overview of Machine Learning Week 2 : Perceptron, Neocognitron, Performance metrics, Capacity, Regularization Week 3 : Linear Predictors, Ridge Regression, Logistic Regression, Linear Support Vector Machines, Boosting Week 4 : Convex Learning Problems, Stochastic Gradient Descent Week 5 : Faster Stochastic Gradient Descent, Model Selection, Validation Week 6 : Kernel-based Methods, Boosting Week 7 : Decision Trees, Random Forests, Ensemble Methods Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 38 / 40
Course logistics Course Topics Week 8 : Deep Neural Networks, I Week 9 : Quantization, Clustering, Compression Week 10 : Dimensionality Reduction Week 11 : Latent Variable Models Week 12 : Feature Selection. Feature Design Week 13 : Deep Neural Networks, II Week 14 : The Art of Machine Learning Modelling Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 39 / 40
Course logistics Questions? What are you looking to get out of the course? Questions for me? Zaid Harchaoui (NYU) CSCI-GA-3033-012 Fall 2015 40 / 40