ID2223 Lecture 2: Distributed ML and Linear Regression

ID2223 Lecture 2: Distributed ML and Linear Regression

Terminology Observations. Entities used for learning/evaluation Features. Attributes (typically numeric) used to represent an observation Labels. Values/categories assigned to observations Model. Parameters/weights that are adjusted by training to predict label(s) given observations. Training, Validation, and Test Data. Observations for training and evaluating a learning algorithm - Training data is given to the algorithm for training - Validation data is withheld from training and is used to measure the performance of training - Test data is withheld at train time 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 2/77

Supervised Learning Labelled Observations Input Data Supervised Learning Prediction Learn from labeled observations. Labels teach the algorithm to learn a mapping from observations to labels 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 3

Supervised Learning Classification. Assign a category to each item - Categories are discrete [Image from Sze et al, 2017] 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 4/77

Supervised Learning Regression. Predict a real value for each item - Labels are continuous - Can define closeness when comparing prediction with label 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 5/77

Unsupervised Learning List of Data Points Unsupervised Learning List of Cluster Labels Unsupervised Learning Clustering. Partition observations into homogeneous regions Dimensionality Reduction. Transform an initial feature representation into a more concise representation 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 6

Non-Distributed Representations Not Compositional x x x x x x x Regions defined by learned Prototypes - Nearest Neighbour - Decision Trees (DTs) Random Forests Gradient Boosted DTs - Clustering x [Bengio, BayArea DL School, 16] 2017-11-02 7/58

Parametric Learning A learning model that summarizes data with a set of parameters of fixed size (independent of the number of training examples) is called a parametric model. No matter how much data you throw at a parametric model, it won t change its mind about how many parameters it needs. Artificial Intelligence: A Modern Approach, page 737 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 8/77

Parametric vs Non Parametric Learning Parametric = bounded number of parameters Non-parametric = unbounded number of parameters Parametric or non-parametric models affect - the way the learning is stored - the method for learning Possible Combinations: Supervised Parametric Unsupervised Parametric Supervised Non-Parametric Unsupervised Non-Parametric 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 9

Supervised Parametric Learning Imagine a machine where you can input some data, turn some dials and observe its performance - Output data is a prediction 2017-11-02 [Grokking Deep Learning, Manning 16]` 10

Supervised Parametric Learning Algorithm 1. Predict 2. Compare to actual result (true pattern) 3. Adjust the dials (parameters) to improve predictions 2017-11-02 [Grokking Deep Learning, Manning 16] 11

Machine Learning Pipeline 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 12/77

Machine Learning Hierarchy of Needs Prediction DDL (Distributed Deep Learning) Deep Learning, RL, Automated ML Labeled Data, ML Experimentation Analytics B.I. Analytics, Metrics, Aggregates, Features, Training/Test Data Reliable Data Pipelines, ETL, Unstructured and Structured Data Storage, Real-Time Data Ingestion 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 13/77

Importing Data Import the raw data (observations) from some source - Real-time data (Kafka) - Data-at-rest (HDFS) Different data formats likely Data may have duplicate, missing columns, invalid values - Clean and wrangle the data Store the data in a format that is efficient for querying (partitioned) Acquire Raw Data Clean/Partition Data 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 14/77

Feature Extraction Extract features to represent the observations - Exploit domain knowledge Acquire Raw Data Clean/Partition Data Nearly always want numeric features Feature Extraction Choice of features is crucial to the success of the entire pipeline 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 15/77

Supervised Learning Train a supervised model using labeled data Classification or Regression model Acquire Raw Data Clean/Partition Data Feature Extraction Supervised Learning 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 16/77

Learning Q: How do we determine the quality of the model we ve just trained? A: We can evaluate it on test / hold-out data, i.e., labeled data not used for training If we don t like the results, we iterate Acquire Raw Data Clean/Partition Data Feature Extraction Supervised Learning Evaluation 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 17/77

Predict Once we re happy with our model, we can use it to make predictions on future observations, i.e., data without a known label Acquire Raw Data Clean/Partition Data Feature Extraction Supervised Learning Evaluation Predict 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 18/77

Classification Goal: Learn a mapping from observations to discrete labels given a set of training examples (supervised learning) 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 19/77

Classification Examples Spam Classification - Emails {spam, ham} Anomaly detection - Network activity {anomoly, not anomoly} Fraud detection - Shopping activity {fraud, not fraud} Clickthrough rate prediction - User viewing an ad {click, no click} 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 20/77

Classification Example 20 Newsgroups 1000s of documents from 20 Usenet Newsgroups - alt.atheism, soc.religion.christian, talk.politics.guns - comp.sys.ibm.pc.hardware, comp.sys.mac.hardware Train a model to classify documents from the 20 Newsgroups data set into two categories according to whether or not the documents are computer related. Goal: classify each document as: IS_COMPUTERS or NOT_COMPUTERS. 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 21/77

Classification Pipeline Newsgroups20 Acquire Raw Data Raw data consists of a set of labeled training observations > >>I have been at a shooting range where >>gang members were "practicing" shooting. Feature Extraction Supervised Learning Evaluation Predict In article <C5qsBF.IEK@ms.uky.edu> billq@ms.uky.edu (Billy Quinn) writes: >I built a little project using the radio shack 5vdc relays to switch > 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 22/77

Classification Pipeline Observation talk.politics.guns/54279 > >>I have been at a shooting range where >>gang members were "practicing" shooting. Label NOT_COMPUTERS Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict sci.electronics/53909 In article <C5qsBF.IEK@ms.uky.edu> billq@ms.uky.edu (Billy Quinn) writes: >I built a little project using the radio shack 5vdc relays to switch >. IS_COMPUTERS.. 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 23/77

0 0 1 0 0 1 1 1 0 2 1 2 0 3 1 3 0 4 1 4 7830 7831 7832 7833 7834 7835 7836 7837 7838 7839 0 5 1 5 0 6 1 6 0 7 1 7 0 8 1 8 0 9 1 9 Classification Pipeline > >>I have been at a shooting range where >>gang members were "practicing" shooting. Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict Feature extraction usually converts each observation into a vector of real numbers (features) Choosing a good description for an observation will have huge baring on the success or failure of a classifier. 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 24/77

Classification Pipeline case class newsgroupscaseclass (id: String, text: String, topic: String) Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 25/77

Classification Pipeline Show only documents that are related to computer topics Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 26/77

0 0 1 0 0 1 1 1 0 2 1 2 0 3 1 3 0 4 1 4 7830 7831 7832 7833 7834 7835 7836 7837 7838 7839 0 5 1 5 0 6 1 6 0 7 1 7 0 8 1 8 0 9 1 9 Classification Pipeline > >>I have been at a shooting range where >>gang members were "practicing" shooting. Classifier Acquire Raw Data Feature Extraction Supervised Learning Evaluation Supervised Learning: Train classifier using training data - Common classifiers include Logistic Regression, SVMs, Decision Trees, Random Forests, etc. Training (especially at scale) often involves iterative computations, e.g., gradient descent Predict 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 27/77

Logistic Regression Goal: Find linear decision boundary - Parameters to learn are feature weights and offset - Nice probabilistic interpretation - Covered in more detail later in course Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict [Distributed Machine Learning with Apache Spark, Berkeley 16 ] 2017-11-02 28/77

Evaluation How can we evaluate the quality of our classifier? We want good predictions on unobserved data - Generalization ability Accuracy on training data is overly optimistic since classifier has already learned from it - We might be overfitting Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 29/77

Overfitting and Generalization Fitting training data does not guarantee generalization, e.g., lookup table Which figure below is a better classifier? - Left: perfectly fits training samples, but it is complex and overfits the data Favour simple models over complex ones, ceteris paribus Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict 2017-11-02 30/77 [Distributed Machine Learning with Apache Spark, Berkeley 16 ]

Classification Pipeline How can we evaluate the quality of our classifier? Idea: Create test set to simulate unobserved data Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict Evaluation: Split dataset into training / testing datasets - Various ways to compare predicted and true labels - Evaluation criterion is called a loss function - Accuracy (or 0-1 loss) is common for classification 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 31/77

Classification Pipeline Split data set into separate training (70%) and test (30%) data sets Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict [Distributed Machine Learning with Apache Spark, Berkeley 16 ] 2017-11-02 32/77

Classification Pipeline Predict: Final classifier can then be used to make predictions on future observations Acquire Raw Data Feature Extraction Supervised Learning Evaluation Predict [Distributed Machine Learning with Apache Spark, Berkeley 16 ] 2017-11-02 33/77

Linear Regression 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 34/77

Regression Goal: Learn a mapping from observations (features) to continuous values/labels given a training set (supervised learning) 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 35/77

Linear Least Squares Regression For each observation we have a feature vector, x, and label, y x T = x 1 x 2 x d We assume a linear mapping between features and label: y w o + w 1 x 1 + w d 1 x d 1 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 36/77

Linear Least Squares Regression We can augment the feature vector to incorporate offset: x T = 1 x 1 x 2 x d We can then rewrite this linear mapping as a scalar product: y y = d i=0 w i x i = w T x 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 37/77

Least Squares Regression Given n training samples with d features, we define: X R n d : matrix storing points y R n : real-valued labels y R n : predicted labels, where y = Xw w R d : model parameters 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 38/77

Evaluating Predictions What is an appropriate evaluation metric or loss function? Absolute loss: Squared loss: y y y y 2 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 39/77

Learning a Model an Optimization Problem Assume we have n training points, where x (i) denotes the ith point Idea: Find the model w that minimizes squared loss over the training points: min w n i=1 w T x i y i 2 where y = w T x i 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 40/77

Least Squares Regression Least Squares Regression: Learn the mapping (w) from features to labels that minimizes residual sum of squares: min w Xw y 2 2 Equivalent by definition of the Euclidean norm: min w n i=1 w T x i y i 2 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 41/77

Closed Form Solution Solve by setting derivative to zero - (left as exercise) - Hint: find the minimum by differentiating the loss (error) function and setting to zero. Then you will need two linear algebra steps. Solution: w = ( X T X ) 1 X T y ( X T X ) 1 should not be zero. 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 42/77

Overfitting and Generalization We want good predictions on new data, i.e., generalization Least squares regression minimizes training error, and could overfit - Simpler models are more likely to generalize (Occam s razor) - Can we change the problem to penalize for model complexity? - Intuitively, models with smaller weights are simpler Can we penalize overly complex models? Ridge Regression does this by adding a regularization term 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 43/77

Ridge Regression Ridge Regression: Learn mapping (w) that minimizes the residual sum of squares along with a regularization term: Training Error Model Complexity min w Xw y 2 2 + λ w 2 2 Free parameter that trades off training error and model complexity 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 44/77

Hyperparameters λ w 2 2 How do find good values for this hyperparameter? - Need to tune hyperparameters Problem: If we try out different hyperparameter values and evaluate them using the test set, there is a danger of overfitting the data. Why?? - Problem arises as the test set should reflect unobserved data Solution: Use a 2 nd hold out dataset to evaluate different tunings for hyperparameters 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 45/77

Evaluating with the Validation Set Hyperparameter tuning - Training: train different models - Validation: evaluate different models - Test: evaluate the accuracy of the final model [Distributed Machine Learning with Apache Spark, Berkeley 16 ] 2017-11-02 46/77

Hyperparameter Search Grid Search: Exhaustively search through hyperparameter space - Define and discretize search space (linear or log scale) 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 47/77 [Distributed Machine Learning with Apache Spark, Berkeley 16 ]

Evaluation Evaluate final model - Training set: train various models - Validation set: evaluate various models - Test set: evaluate final model s accuracy 2017-11-02 [Distributed Machine Learning with Apache Spark, Berkeley 16 ] 49/77

Predict The trained model can then be used to make predictions on unseen observations [Distributed Machine Learning with Apache Spark, Berkeley 16 ] 2017-11-02 50/77

Distributed Machine Learning 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 51/77

Computational Complexity - Review 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 52/77

Time Complexity The (worst-case) time complexity of a problem is the time complexity of the fastest algorithm that solves the problem. Ο(n): there is an algorithm that solves the problem faster than this time - Has to be true for all possible inputs, that is, the worst-case What is the time complexity of the following code snippet? for ( i=0 ; i<n ; i++ ) for( j=0 ; j<n ; j++ ) for( k=0 ; k<n ; k++ ) sum[i][j] += entry[i][j][k]; 53

Orders-of-Growth 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 54/77

Common Orders-of-Growth O(1) constant O(log(n)) logarithmic time O(n) linear O(n log(n)) linearithmic time O(n 2 ) quadratic time O(2 n ) exponential time 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 55/77

Orders-of-Growth Example 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 56/77

f x = O g x Big-Oh (O) - ignore constant and lower-order terms Big-Oh determines the upper bound on a function f n = O g n iff c > 0 and n 0 > 0 where f n cg n n n 0 2017-11-02 57/77

Space Complexity The space complexity of an algorithm is an expression for the worst-case amount of memory that the algorithm will use. 58/77

Linear Regression:Computational Complexity Can we count the number of arithmetic operations for? - Assume we have d features and n samples w = ( X T X ) 1 X T y 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 59/77

Linear Regression:Computational Complexity If we count the number of arithmetic operations for: w = ( X T X ) 1 X T y Cost of operations (n is #samples, d is #features): - X T X : O(nd 2 ) - Matrix inverse of X T X: O(d 3 ) - X T y : O(nd) - Product X T y and Inverse Matrix: O(d 2 ) The Time Complexity is: O(nd 2 + d 3 ) Ignore lower-order terms 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 60/77

Linear Regression: Space Complexity Storage costs: X T X and its inverse: O(d 2 ) floats X : O(nd) floats Space complexity: O(nd + d 2 ) floats 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 61/77

Big n and Small d Assume O(d 3 ) computation and O(d 2 ) memory is feasible on a single machine Storing X and computing X T X are the bottlenecks How can we distribute storage and computation? - Partition rows of X over hosts - Compute X T X as a sum of outer products 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 62/77

Matrix Multiplication using Inner Products Each entry of output matrix is result of inner product of inputs matrices 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 63/77

Matrix Multiplication using Inner Products 2 3 3 4 6 2 3 1 1 1 4 5 = 34 35 11 8 3 2 + 4 1 + 6 4 = 34 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 64/77

Matrix Multiplication using Inner Products 2 3 3 4 6 2 3 1 1 1 4 5 = 34 35 11 8 3 3 + 4 1 + 6 5 = 35 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 65/77

Matrix Multiplication using Inner Products 2 3 3 4 6 2 3 1 1 1 4 5 = 34 35 11 8 2 2 + 3 1 + 1 4 = 11 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 66/77

Matrix-Matrix Multiplication 2 3 3 4 6 2 3 1 1 1 4 5 = 34 35 11 8 2 3 + 3 1 + 1 5 = 8 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 67/77

Matrix Multiplication via Outer Products Output matrix is the sum of outer products between corresponding rows and columns of input matrices 2 3 3 4 6 2 3 1 1 1 4 5 = 6 9 4 6 + 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 68/77

Matrix Multiplication via Outer Products Output matrix is the sum of outer products between corresponding rows and columns of input matrices 2 3 3 4 6 2 3 1 1 1 4 5 = 6 9 4 6 + 4 4 3 3 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 69/77

Matrix Multiplication via Outer Products Output matrix is the sum of outer products between corresponding rows and columns of input matrices 2 3 3 4 6 2 3 1 1 1 4 5 = 34 35 11 8 6 9 4 6 + 4 4 3 3 + 24 30 4 5 = 34 35 11 8 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 70/77

Computational Complexity with Map-Reduce 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 71/77 [Distributed Machine Learning with Apache Spark, Berkeley 16 ]

Computational Complexity with Map-Reduce 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 72/77 [Distributed Machine Learning with Apache Spark, Berkeley 16 ]

Big n and Big d With big d, storing and computing are still the bottlenecks O(d 3 ) computation in the reduce step is not easily parallelized

Big n and Big d not Computationally Feasible Distributing storage and processing doesn t change the cubic computational complexity Options to scale include: 1. Exploit data sparsity to reduce dimensionality or 2. Make computation (and storage) linear in (n, d) 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 74/77

Iterative Algorithms are more Efficient We need methods that are linear in time and space Gradient descent is an iterative algorithm that requires O(nd) computation and O(d) local storage per iteration 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 75/77

Gradient Descent [Distributed Machine Learning with Apache Spark, Berkeley 16 ] 2017-11-02 76/77

References Distributed Machine Learning with Apache Spark, UCLA/Berkeley Course 2016 Andrew Ng, Machine Learning CS 229, Stanford Deep Learning Book, Section 4.5 Sze et Al, Efficient Processing of Deep Neural Networks: A Tutorial and Survey, https://arxiv.org/abs/1703.09039 2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 77/77