# COMP 551 Applied Machine Learning Lecture 6: Performance evaluation. Model assessment and selection.

Save this PDF as:

Size: px
Start display at page:

Download "COMP 551 Applied Machine Learning Lecture 6: Performance evaluation. Model assessment and selection."

## Transcription

1 COMP 551 Applied Machine Learning Lecture 6: Performance evaluation. Model assessment and selection. Instructor: Class web page: Unless otherwise noted, all material posted for this course are copyright of the instructor, and cannot be reused or reposted without the instructor s written permission.

2 Today s quiz (on mycourses) 1. Name one advantage of LDA over Naive Bayes. 2. Name one disadvantage of LDA over Naive Bayes. 3. True or False: Generative learning typically requires learning more parameters than discriminative learning (assuming the same number of features and examples). 4. Why? 2

4 Evaluating performance Different objectives: Selecting the right model for a problem. Testing performance of a new algorithm. Evaluating impact on a new application. 4

5 Overfitting Adding more degrees of freedom (more features) always seems to improve the solution! 5

6 Minimizing the error Find the low point in the validation error: Prediction Error High Bias Low Variance Low Bias High Variance Validation error Train error Model Complexity (df) 6

7 Performance metrics for classification Not all errors have equal impact! There are different types of mistakes, particularly in the classification setting. 7

8 Example 1 8

9 Example 1 Accuracy = True positives + True Negatives / Total number of examples Sensitivity = True positives / Total number of actual positives Specificity = True negatives / Total number of actual negatives 9

10 Performance metrics for classification Not all errors have equal impact! There are different types of mistakes, particularly in the classification setting. E.g. Consider the diagnostic of a disease. Two types of mis-diagnostics: Patient does not have disease but received positive diagnostic (Type I error); Patient has disease but it was not detected (Type II error). 10

11 Performance metrics for classification Not all errors have equal impact! There are different types of mistakes, particularly in the classification setting. E.g. Consider the diagnostic of a disease. Two types of mis-diagnostics: Patient does not have disease but received positive diagnostic (Type I error); Patient has disease but it was not detected (Type II error). E.g. Consider the problem of spam classification: A message that is not spam is assigned to the spam folder (Type I error); A message that is spam appears in the regular folder (Type II error). 11

12 Performance metrics for classification Not all errors have equal impact! There are different types of mistakes, particularly in the classification setting. E.g. Consider the diagnostic of a disease. Two types of mis-diagnostics: Patient does not have disease but received positive diagnostic (Type I error); Patient has disease but it was not detected (Type II error). E.g. Consider the problem of spam classification: A message that is not spam is assigned to the spam folder (Type I error); A message that is spam appears in the regular folder (Type II error). How many Type I errors are you willing to tolerate, for a reasonable rate of Type II errors? 12

13 Example 2 13

14 Example 3 14

15 Terminology Type of classification outputs: True positive (m11): Example of class 1 predicted as class 1. False positive (m01): Example of class 0 predicted as class 1. Type 1 error. True negative (m00): Example of class 0 predicted as class 0. False negative (m10): Example of class 1 predicted as class 0. Type II error. Total number of instances: m = m00 + m01 + m10 + m11 15

16 Terminology Type of classification outputs: True positive (m11): Example of class 1 predicted as class 1. False positive (m01): Example of class 0 predicted as class 1. Type 1 error. True negative (m00): Example of class 0 predicted as class 0. False negative (m10): Example of class 1 predicted as class 0. Type II error. Total number of instances: m = m00 + m01 + m10 + m11 Error rate: (m01 + m10) / m If the classes are imbalanced (e.g. 10% from class 1, 90% from class 0), one can achieve low error (e.g. 10%) by classifying everything as coming from class 0! 16

17 Confusion matrix Many software packages output this matrix. apple m00 m 01 m 10 m 11 17

18 Confusion matrix Many software packages output this matrix. apple m00 m 01 m 10 m 11 Be careful! Sometimes the format is slightly different (E.g. 18

19 Common measures Accuracy = (TP+ TN) / (TP + FP + FN + TN) Precision = True positives / Total number of declared positives = TP / (TP+ FP) Recall = True positives / Total number of actual positives = TP / (TP + FN) 19

20 Common measures Accuracy = (TP+ TN) / (TP + FP + FN + TN) Precision = True positives / Total number of declared positives Text classification = TP / (TP+ FP) Recall = True positives / Total number of actual positives = TP / (TP + FN) Medicine Sensitivity is the same as recall. Specificity = True negatives / Total number of actual negatives = TN / (FP + TN) 20

21 Common measures Accuracy = (TP+ TN) / (TP + FP + FN + TN) Precision = True positives / Total number of declared positives Text classification = TP / (TP+ FP) Recall = True positives / Total number of actual positives = TP / (TP + FN) Medicine Sensitivity is the same as recall. Specificity = True negatives / Total number of actual negatives = TN / (FP + TN) False positive rate = FP / (FP + TN) 21

22 Common measures Accuracy = (TP+ TN) / (TP + FP + FN + TN) Precision = True positives / Total number of declared positives Text classification = TP / (TP+ FP) Recall = True positives / Total number of actual positives = TP / (TP + FN) Medicine Sensitivity is the same as recall. Specificity = True negatives / Total number of actual negatives = TN / (FP + TN) False positive rate = FP / (FP + TN) F1 measure 22

23 Trade-off Often have a trade-off between false positives and false negatives. E.g. Consider 30 different classifiers trained on a class. Classify a new sample as positive if K classifiers output positive. Vary K between 0 and

24 Receiver-operator characteristic (ROC) curve Characterizes the performance of a binary classifier over a range of classification thresholds Data from 4 prediction results: ROC curve: Example from: 24

25 Understanding the ROC curve Consider a classification problem where data is generated by 2 Gaussians (blue = negative class; red = positive class). Consider the decision boundary (shown as a vertical line on the left figure), where you predict Negative on the left of the boundary and predict Positive on the right of the boundary. Changing that boundary defines the ROC curve on the right. Predict negative Predictive positive Figures from: 25

26 Building the ROC curve In many domains, the empirical ROC curve will be non-convex (red line). Take the convex hull of the points (blue line). 26

27 Using the ROC curve To compare 2 algorithms over a range of classification thresholds, consider the Area Under the Curve (AUC). A perfect algorithm has AUC=1. A random algorithm has AUC=0.5. Higher AUC doesn t mean all performance measures are better. 27

28 K-fold cross-validation Single test-train split: Estimation test error with high variance. 4-fold test-train splits: Better estimation of the test error, because it is averaged over four different test-train splits. 28

29 K-fold cross-validation K=1: High variance estimate of Err(). Fast to compute. K>1: Improved estimate of Err(); wastes 1/K of the data. K times more expensive to compute. 29

30 K-fold cross-validation K=1: High variance estimate of Err(). Fast to compute. K>1: Improved estimate of Err(); wastes 1/K of the data. K times more expensive to compute. K=N: Lowest variance estimate of Err(). Doesn t waste data. N times slower to compute than single train/validate split. 30

31 Brief aside: Bootstrapping Basic idea: Given a dataset D with N examples. Randomly draw (with replacement) B datasets of size N from D. Estimate the measure of interest on each of the B datasets. Take the mean of the estimates. Err 1 Err 2 Err B D 1 D 2 D B Is this a good measure for estimating the error? D True data distribution 31

32 Bootstrapping the error Use a dataset b to fit a hypothesis f b. Use the original dataset D to evaluate the error. Average over all bootstrap sets b in B. Êrr boot = 1 B 1 N Problem: Some of the same samples are used for training the learning and validation. B b=1 N L(y i, ˆf b (x i )). i=1 32

33 Bootstrapping the error Use a dataset b to fit a hypothesis f b. Use the original dataset D to evaluate the error. Average over all bootstrap sets b in B. Êrr boot = 1 1 B N L(y i, B N ˆf b (x i )). b=1 i=1 Problem: Some of the same samples are used for training the learning and validation. Better idea: Include the error of a data sample i only over classifiers trained with those bootstrap sets b in which i isn t included (denoted C -i ). Êrr (1) = 1 N 1 N C i L(y i, ˆf b (x i )). i=1 b C i (Note: Bootstrapping is a very general ideal, which can be applied for empirically estimating many different quantities.) 33

34 Strategy #1 Consider a classification problem with a large number of features, greater than the number of examples (m>>n). Consider the following strategies to avoid over-fitting in such a problem. Strategy 1: 1. Check for correlation between each feature (individually) and the output. Keep a small set of features showing strong correlation. 2. Divide the examples into k groups at random. 3. Using the features from step 1 and the examples from k-1 groups from step 2, build a classifier. 4. Use this classifier to predict the output for the examples in group k and measure the error. 5. Repeat steps 3-4 for each group to produce the cross-validation estimate of the error. 34

35 Strategy #2 Consider a classification problem with a large number of features, greater than the number of examples (m>>n). Consider the following strategies to avoid over-fitting in such a problem. Strategy 2: 1. Divide the examples into k groups at random. 2. For each group, find a small set of features showing strong correlation with the output. 3. Using the features and examples from k-1 groups from step 1, build a classifier. 4. Use this classifier to predict the output for the examples in group k and measure the error. 5. Repeat 2-4 for each group to produce the cross-validation estimate of the error. 35

36 Strategy #3 Consider a classification problem with a large number of features, greater than the number of examples (m>>n). Consider the following strategies to avoid over-fitting in such a problem. Strategy 3: 1. Randomly sample n examples. 2. For the sampled data, find a small set of features showing strong correlation with the outptut 3. Using the examples from step 1 and features from step 2, build a classifier. 4. Use this classifier to predict the output for those examples in the dataset that are not in n and measure the error. 5. Repeat steps 1-4 k times to produce the cross-validation estimate of the error. 36

37 Strategy 1: Summary of 3 strategies 1. Check for correlation between each feature (individually) and the output. Keep a small set of features showing strong correlation. 2. Divide the examples into k groups at random. 3. Using the features from step 1 and the examples from k-1 groups from step 2, build a classifier. 4. Use this classifier to predict the output for the examples in group k and measure the error. 5. Repeat steps 3-4 for each group to produce the cross-validation estimate of the error. Strategy 2: 1. Divide the examples into k groups at random. 2. For each group, find a small set of features showing strong correlation with the output. 3. Using the features and examples from k-1 groups from step 1, build a classifier. 4. Use this classifier to predict the output for the examples in group k and measure the error. 5. Repeat 2-4 for each group to produce the cross-validation estimate of the error. Strategy 3: 1. Randomly sample n examples. 2. For the sampled data, find a small set of features showing strong correlation with the ouptut 3. Using the examples from step 1 and features from step 2, build a classifier. 4. Use this classifier to predict the output for those examples in the dataset that are not in n and measure the error. 5. Repeat steps 1-4 k times to produce the cross-validation estimate of the error. 37

38 Discussion Strategy 1 is prone to overfitting, because the full dataset is considered in step 1, to select the features. Thus we do not get an unbiased estimate of the generalization error in step 5. Strategy 2 is closest to standard k-fold cross-validation. One can view the joint procedure of selecting the features and building the classifier as the training step, to be applied (separately) on each training fold. Strategy 3 is closer to a bootstrap estimate. It can give a good estimate of the generalization error, but the estimate will possibly have higher variance than the one obtained using Strategy 2. 38

39 A word of caution Intensive use of cross-validation can overfit! E.g. Given a dataset with 50 examples and 1000 features. Consider 1000 linear regression models, each built with a single feature. The best of those 1000 will look very good! But it would have looked good even if the output was random! What should we do about this? 39

40 To avoid overfitting to the validation set When you need to optimize many parameters of your model or learning algorithm. Use three datasets: The training set is used to estimate the parameters of the model. The validation set is used to estimate the prediction error for the given model. The test set is used to estimate the generalization error once the model is fixed. Train Validation Test 40

41 Kaggle 41

42 Lessons for evaluating ML algorithms Always compare to a simple baseline: In classification: Classify all samples as the majority class. Classify with a threshold on a single variable. In regression: Predict the average of the output for all samples. Compare to a simple linear regression. Use K-fold cross validation to properly estimate the error. If necessary, use a validation set to estimate hyper-parameters. Consider appropriate measures for fully characterizing the performance: Accuracy, Precision, Recall, F1, AUC. 42

43 What you should know Understand the concepts of loss, error function, bias, variance. Commit to correctly applying cross-validation. Understand the common measures of performance. Know how to produce and read ROC curves. Understand the use of bootstrapping. Be concerned about good practices for machine learning! Read this paper today! K. Wagstaff, Machine Learning that Matters, ICML

### Course 395: Machine Learning - Lectures

Course 395: Machine Learning - Lectures Lecture 1-2: Concept Learning (M. Pantic) Lecture 3-4: Decision Trees & CBC Intro (M. Pantic & S. Petridis) Lecture 5-6: Evaluating Hypotheses (S. Petridis) Lecture

### Machine Learning with Weka

Machine Learning with Weka SLIDES BY (TOTAL 5 Session of 1.5 Hours Each) ANJALI GOYAL & ASHISH SUREKA (www.ashish-sureka.in) CS 309 INFORMATION RETRIEVAL COURSE ASHOKA UNIVERSITY NOTE: Slides created and

### Introduction to Classification, aka Machine Learning

Introduction to Classification, aka Machine Learning Classification: Definition Given a collection of examples (training set ) Each example is represented by a set of features, sometimes called attributes

### Evaluation and Comparison of Performance of different Classifiers

Evaluation and Comparison of Performance of different Classifiers Bhavana Kumari 1, Vishal Shrivastava 2 ACE&IT, Jaipur Abstract:- Many companies like insurance, credit card, bank, retail industry require

### TOWARDS DATA-DRIVEN AUTONOMICS IN DATA CENTERS

TOWARDS DATA-DRIVEN AUTONOMICS IN DATA CENTERS ALINA SIRBU, OZALP BABAOGLU SUMMARIZED BY ARDA GUMUSALAN MOTIVATION 2 MOTIVATION Human-interaction-dependent data centers are not sustainable for future data

### Introduction to Classification

Introduction to Classification Classification: Definition Given a collection of examples (training set ) Each example is represented by a set of features, sometimes called attributes Each example is to

### Linear Models Continued: Perceptron & Logistic Regression

Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Linear Models for Classification Feature function

### Big Data Analytics Clustering and Classification

E6893 Big Data Analytics Lecture 4: Big Data Analytics Clustering and Classification Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science September 28th, 2017 1

### Arrhythmia Classification for Heart Attack Prediction Michelle Jin

Arrhythmia Classification for Heart Attack Prediction Michelle Jin Introduction Proper classification of heart abnormalities can lead to significant improvements in predictions of heart failures. The variety

### Random Under-Sampling Ensemble Methods for Highly Imbalanced Rare Disease Classification

54 Int'l Conf. Data Mining DMIN'16 Random Under-Sampling Ensemble Methods for Highly Imbalanced Rare Disease Classification Dong Dai, and Shaowen Hua Abstract Classification on imbalanced data presents

### Learning Imbalanced Data with Random Forests

Learning Imbalanced Data with Random Forests Chao Chen (Stat., UC Berkeley) chenchao@stat.berkeley.edu Andy Liaw (Merck Research Labs) andy_liaw@merck.com Leo Breiman (Stat., UC Berkeley) leo@stat.berkeley.edu

### Session 1: Gesture Recognition & Machine Learning Fundamentals

IAP Gesture Recognition Workshop Session 1: Gesture Recognition & Machine Learning Fundamentals Nicholas Gillian Responsive Environments, MIT Media Lab Tuesday 8th January, 2013 My Research My Research

### Classifying Breast Cancer By Using Decision Tree Algorithms

Classifying Breast Cancer By Using Decision Tree Algorithms Nusaibah AL-SALIHY, Turgay IBRIKCI (Presenter) Cukurova University, TURKEY What Is A Decision Tree? Why A Decision Tree? Why Decision TreeClassification?

### Cost-Sensitive Learning and the Class Imbalance Problem

To appear in Encyclopedia of Machine Learning. C. Sammut (Ed.). Springer. 2008 Cost-Sensitive Learning and the Class Imbalance Problem Charles X. Ling, Victor S. Sheng The University of Western Ontario,

### 1. Subject. 2. Dataset. Resampling approaches for prediction error estimation.

1. Subject Resampling approaches for prediction error estimation. The ability to predict correctly is one of the most important criteria to evaluate classifiers in supervised learning. The preferred indicator

### Don t Get Kicked - Machine Learning Predictions for Car Buying

STANFORD UNIVERSITY, CS229 - MACHINE LEARNING Don t Get Kicked - Machine Learning Predictions for Car Buying Albert Ho, Robert Romano, Xin Alice Wu December 14, 2012 1 Introduction When you go to an auto

### A study of the NIPS feature selection challenge

A study of the NIPS feature selection challenge Nicholas Johnson November 29, 2009 Abstract The 2003 Nips Feature extraction challenge was dominated by Bayesian approaches developed by the team of Radford

### Lecture 1: Machine Learning Basics

1/69 Lecture 1: Machine Learning Basics Ali Harakeh University of Waterloo WAVE Lab ali.harakeh@uwaterloo.ca May 1, 2017 2/69 Overview 1 Learning Algorithms 2 Capacity, Overfitting, and Underfitting 3

### Analytical Study of Some Selected Classification Algorithms in WEKA Using Real Crime Data

Analytical Study of Some Selected Classification Algorithms in WEKA Using Real Crime Data Obuandike Georgina N. Department of Mathematical Sciences and IT Federal University Dutsinma Katsina state, Nigeria

### Performance Analysis of Various Data Mining Techniques on Banknote Authentication

International Journal of Engineering Science Invention ISSN (Online): 2319 6734, ISSN (Print): 2319 6726 Volume 5 Issue 2 February 2016 PP.62-71 Performance Analysis of Various Data Mining Techniques on

### Additional file 3. Class balancing Both datasets used in this work for training the classifiers are characterized by strong

Additional file 3 Class balancing Both datasets used in this work for training the classifiers are characterized by strong class imbalance. Specifically, in the obligate/non- obligate dataset the fraction

### INLS 613 Text Data Mining Homework 2 Due: Monday, October 10, 2016 by 11:55pm via Sakai

INLS 613 Text Data Mining Homework 2 Due: Monday, October 10, 2016 by 11:55pm via Sakai 1 Objective The goal of this homework is to give you exposure to the practice of training and testing a machine-learning

### 36-350: Data Mining. Fall Lectures: Monday, Wednesday and Friday, 10:30 11:20, Porter Hall 226B

36-350: Data Mining Fall 2009 Instructor: Cosma Shalizi, Statistics Dept., Baker Hall 229C, cshalizi@stat.cmu.edu Teaching Assistant: Joseph Richards, jwrichar@stat.cmu.edu Lectures: Monday, Wednesday

### A COMPARATIVE ANALYSIS OF META AND TREE CLASSIFICATION ALGORITHMS USING WEKA

A COMPARATIVE ANALYSIS OF META AND TREE CLASSIFICATION ALGORITHMS USING WEKA T.Sathya Devi 1, Dr.K.Meenakshi Sundaram 2, (Sathya.kgm24@gmail.com 1, lecturekms@yahoo.com 2 ) 1 (M.Phil Scholar, Department

### IMBALANCED data sets (IDS) correspond to domains

Diversity Analysis on Imbalanced Data Sets by Using Ensemble Models Shuo Wang and Xin Yao Abstract Many real-world applications have problems when learning from imbalanced data sets, such as medical diagnosis,

### Cross-Validation. By: Huaicheng Liu Jiaxin Deng

Cross-Validation By: Huaicheng Liu Jiaxin Deng 1 2 Overviews 1.Model Assessment and Selection The Application of Cross-Validation 2.Cross-Validation 3.K-Fold Cross- Validation (1)What value should we choose

### Dudon Wai Georgia Institute of Technology CS 7641: Machine Learning Atlanta, GA

Adult Income and Letter Recognition - Supervised Learning Report An objective look at classifier performance for predicting adult income and Letter Recognition Dudon Wai Georgia Institute of Technology

### I400 Health Informatics Data Mining Instructions (KP Project)

I400 Health Informatics Data Mining Instructions (KP Project) Casey Bennett Spring 2014 Indiana University 1) Import: First, we need to import the data into Knime. add CSV Reader Node (under IO>>Read)

(Sub)Gradient Descent CMSC 422 MARINE CARPUAT marine@cs.umd.edu Figures credit: Piyush Rai Logistics Midterm is on Thursday 3/24 during class time closed book/internet/etc, one page of notes. will include

### Modelling Student Knowledge as a Latent Variable in Intelligent Tutoring Systems: A Comparison of Multiple Approaches

Modelling Student Knowledge as a Latent Variable in Intelligent Tutoring Systems: A Comparison of Multiple Approaches Qandeel Tariq, Alex Kolchinski, Richard Davis December 6, 206 Introduction This paper

### Similarity-Weighted Association Rules for a Name Recommender System

Similarity-Weighted Association Rules for a Name Recommender System Benjamin Letham Operations Research Center Massachusetts Institute of Technology Cambridge, MA, USA bletham@mit.edu Abstract. Association

### Machine Learning 2nd Edition

INTRODUCTION TO Lecture Slides for Machine Learning 2nd Edition ETHEM ALPAYDIN, modified by Leonardo Bobadilla and some parts from http://www.cs.tau.ac.il/~apartzin/machinelearning/ The MIT Press, 2010

### M3 - Machine Learning for Computer Vision

M3 - Machine Learning for Computer Vision Traffic Sign Detection and Recognition Adrià Ciurana Guim Perarnau Pau Riba Index Correctly crop dataset Bootstrap Dataset generation Extract features Normalization

### P(A, B) = P(A B) = P(A) + P(B) - P(A B)

AND Probability P(A, B) = P(A B) = P(A) + P(B) - P(A B) P(A B) = P(A) + P(B) - P(A B) Area = Probability of Event AND Probability P(A, B) = P(A B) = P(A) + P(B) - P(A B) If, and only if, A and B are independent,

### Admission Prediction System Using Machine Learning

Admission Prediction System Using Machine Learning Jay Bibodi, Aasihwary Vadodaria, Anand Rawat, Jaidipkumar Patel bibodi@csus.edu, aaishwaryvadoda@csus.edu, anandrawat@csus.edu, jaidipkumarpate@csus.edu

### Utility Theory, Minimum Effort, and Predictive Coding

Utility Theory, Minimum Effort, and Predictive Coding Fabrizio Sebastiani (Joint work with Giacomo Berardi and Andrea Esuli) Istituto di Scienza e Tecnologie dell Informazione Consiglio Nazionale delle

### WEKA tutorial exercises

WEKA tutorial exercises These tutorial exercises introduce WEKA and ask you to try out several machine learning, visualization, and preprocessing methods using a wide variety of datasets: Learners: decision

### Cross-Validation TOM STEVENSON 24 OCTOBER 2016

Cross-Validation TOM STEVENSON T.J.STEVENSON@QMUL.AC.UK MOTIVATION AND THE ISSUE Cross-Validation in TMVA Need confidence that the trained MVA is robust: Performance on unseen samples accurately predicted.

### 6.034 Notes: Section 13.1

6.034 Notes: Section 13.1 Slide 13.1.1 Now that we have looked at the basic mathematical techniques for minimizing the training error of a neural net, we should step back and look at the whole approach

### Speech Accent Classification

Speech Accent Classification Corey Shih ctshih@stanford.edu 1. Introduction English is one of the most prevalent languages in the world, and is the one most commonly used for communication between native

### Scaling Quality On Quora Using Machine Learning

Scaling Quality On Quora Using Machine Learning Nikhil Garg @nikhilgarg28 @Quora @QconSF 11/7/16 Goals Of The Talk Introducing specific product problems we need to solve to stay high-quality Describing

### Cross-Domain Video Concept Detection Using Adaptive SVMs

Cross-Domain Video Concept Detection Using Adaptive SVMs AUTHORS: JUN YANG, RONG YAN, ALEXANDER G. HAUPTMANN PRESENTATION: JESSE DAVIS CS 3710 VISUAL RECOGNITION Problem-Idea-Challenges Address accuracy

### Python Machine Learning

Python Machine Learning Unlock deeper insights into machine learning with this vital guide to cuttingedge predictive analytics Sebastian Raschka [ PUBLISHING 1 open source I community experience distilled

### Analysis of Different Classifiers for Medical Dataset using Various Measures

Analysis of Different for Medical Dataset using Various Measures Payal Dhakate ME Student, Pune, India. K. Rajeswari Associate Professor Pune,India Deepa Abin Assistant Professor, Pune, India ABSTRACT

### Session 7: Face Detection (cont.)

Session 7: Face Detection (cont.) John Magee 8 February 2017 Slides courtesy of Diane H. Theriault Question of the Day: How can we find faces in images? Face Detection Compute features in the image Apply

### Machine Learning for Beam Based Mobility Optimization in NR

Master of Science Thesis in Communication Systems Department of Electrical Engineering, Linköping University, 2017 Machine Learning for Beam Based Mobility Optimization in NR Björn Ekman Master of Science

### CS Machine Learning

CS 478 - Machine Learning Projects Data Representation Basic testing and evaluation schemes CS 478 Data and Testing 1 Programming Issues l Program in any platform you want l Realize that you will be doing

### Comprehensible Data Mining: Gaining Insight from Data

Comprehensible Data Mining: Gaining Insight from Data Michael J. Pazzani Information and Computer Science University of California, Irvine pazzani@ics.uci.edu http://www.ics.uci.edu/~pazzani Outline UC

### CPSC 340: Machine Learning and Data Mining. Course Review/Preview Fall 2015

CPSC 340: Machine Learning and Data Mining Course Review/Preview Fall 2015 Admin Assignment 6 due now. We will have office hours as usual next week. Final exam details: December 15: 8:30-11 (WESB 100).

### Machine Learning. Basic Concepts. Joakim Nivre. Machine Learning 1(24)

Machine Learning Basic Concepts Joakim Nivre Uppsala University and Växjö University, Sweden E-mail: nivre@msi.vxu.se Machine Learning 1(24) Machine Learning Idea: Synthesize computer programs by learning

### Machine Learning and Data Mining. Ensembles of Learners. Prof. Alexander Ihler

Machine Learning and Data Mining Ensembles of Learners Prof. Alexander Ihler Ensemble methods Why learn one classifier when you can learn many? Ensemble: combine many predictors (Weighted) combina

### Paper Examining Higher Education Performance Metrics with SAS Enterprise Miner and SAS Visual Analytics

ABSTRACT Paper 788-2017 Examining Higher Education Performance Metrics with SAS Enterprise Miner and SAS Visual Analytics Taylor Blaetz, M.S., Western Kentucky University; Bowling Green, KY Tuesdi Helbig,

### Biomedical Research 2016; Special Issue: S87-S91 ISSN X

Biomedical Research 2016; Special Issue: S87-S91 ISSN 0970-938X www.biomedres.info Analysis liver and diabetes datasets by using unsupervised two-phase neural network techniques. KG Nandha Kumar 1, T Christopher

### Cost-Sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs?

Cost-Sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs? Gary M. Weiss, Kate McCarthy, and Bibi Zabar Department of Computer and Information Science

### Classification with class imbalance problem: A Review

Int. J. Advance Soft Compu. Appl, Vol. 7, No. 3, November 2015 ISSN 2074-8523 Classification with class imbalance problem: A Review Aida Ali 1,2, Siti Mariyam Shamsuddin 1,2, and Anca L. Ralescu 3 1 UTM

### Predictive Analysis of Text: Concepts, Features, and Instances

of Text: Concepts, Features, and Instances Jaime Arguello jarguell@email.unc.edu August 26, 2015 of Text Objective: developing and evaluating computer programs that automatically detect a particular concept

### CS545 Machine Learning

Machine learning and related fields CS545 Machine Learning Course Introduction Machine learning: the construction and study of systems that learn from data. Pattern recognition: the same field, different

### Assignment 6 (Sol.) Introduction to Machine Learning Prof. B. Ravindran

Assignment 6 (Sol.) Introduction to Machine Learning Prof. B. Ravindran 1. Assume that you are given a data set and a neural network model trained on the data set. You are asked to build a decision tree

### Bird Species Identification from an Image

Bird Species Identification from an Image Aditya Bhandari, 1 Ameya Joshi, 2 Rohit Patki 3 1 Department of Computer Science, Stanford University 2 Department of Electrical Engineering, Stanford University

### Capacity, Learning, Teaching

Capacity, Learning, Teaching Xiaojin Zhu Department of Computer Sciences University of Wisconsin-Madison jerryzhu@cs.wisc.edu 23 Machine learning human learning Learning capacity and generalization bounds

### A Few Useful Things to Know about Machine Learning. Pedro Domingos Department of Computer Science and Engineering University of Washington" 2012"

A Few Useful Things to Know about Machine Learning Pedro Domingos Department of Computer Science and Engineering University of Washington 2012 A Few Useful Things to Know about Machine Learning Machine

### Linear Regression: Predicting House Prices

Linear Regression: Predicting House Prices I am big fan of Kalid Azad writings. He has a knack of explaining hard mathematical concepts like Calculus in simple words and helps the readers to get the intuition

### Overview COEN 296 Topics in Computer Engineering Introduction to Pattern Recognition and Data Mining Course Goals Syllabus

Overview COEN 296 Topics in Computer Engineering to Pattern Recognition and Data Mining Instructor: Dr. Giovanni Seni G.Seni@ieee.org Department of Computer Engineering Santa Clara University Course Goals

### Inductive Learning and Decision Trees

Inductive Learning and Decision Trees Doug Downey EECS 349 Spring 2017 with slides from Pedro Domingos, Bryan Pardo Outline Announcements Homework #1 was assigned on Monday (due in five days!) Inductive

### Note that although this feature is not available in IRTPRO 2.1 or IRTPRO 3, it has been implemented in IRTPRO 4.

TABLE OF CONTENTS 1 Fixed theta estimation... 2 2 Posterior weights... 2 3 Drift analysis... 2 4 Equivalent groups equating... 3 5 Nonequivalent groups equating... 3 6 Vertical equating... 4 7 Group-wise

### Predicting Disengagement in Free-To-Play Games with Highly Biased Data

Player Analytics: Papers from the AIIDE Workshop AAAI Technical Report WS-16-23 Predicting Disengagement in Free-To-Play Games with Highly Biased Data Hanting Xie and Sam Devlin and Daniel Kudenko Department

### INTRODUCTION TO DATA SCIENCE

DATA11001 INTRODUCTION TO DATA SCIENCE EPISODE 6: MACHINE LEARNING TODAY S MENU 1. WHAT IS ML? 2. CLASSIFICATION AND REGRESSSION 3. EVALUATING PERFORMANCE & OVERFITTING WHAT IS MACHINE LEARNING? Definition:

### Machine Learning for NLP

Natural Language Processing SoSe 2014 Machine Learning for NLP Dr. Mariana Neves April 30th, 2014 (based on the slides of Dr. Saeedeh Momtazi) Introduction Field of study that gives computers the ability

### Predicting Student Risks Through Longitudinal Analysis

Predicting Student Risks Through Longitudinal Analysis Ashay Tamhane, IBM Research, Bangalore, India Shajith Ikbal, IBM Research, Bangalore, India Bikram Sengupta, IBM Research, Bangalore, India Mayuri

### Introduction to Ensemble Learning Featuring Successes in the Netflix Prize Competition

Introduction to Ensemble Learning Featuring Successes in the Netflix Prize Competition Todd Holloway Two Lecture Series for B551 November 20 & 27, 2007 Indiana University Outline Introduction Bias and

### Lecture 1. Introduction Bastian Leibe Visual Computing Institute RWTH Aachen University

Advanced Machine Learning Lecture 1 Introduction 20.10.2015 Bastian Leibe Visual Computing Institute RWTH Aachen University http://www.vision.rwth-aachen.de/ leibe@vision.rwth-aachen.de Organization Lecturer

### Jurnal Teknologi TACKLING IMBALANCED CLASS IN SOFTWARE DEFECT PREDICTION USING TWO-STEP CLUSTER BASED RANDOM UNDERSAMPLING AND STACKING TECHNIQUE

Jurnal Teknologi TACKLING IMBALANCED CLASS IN SOFTWARE DEFECT PREDICTION USING TWO-STEP CLUSTER BASED RANDOM UNDERSAMPLING AND STACKING TECHNIQUE Adi Wijaya a,c*, Romi Satria Wahono b a Informatics Engineering

### Homework III Using Logistic Regression for Spam Filtering

Homework III Using Logistic Regression for Spam Filtering Introduction to Machine Learning - CMPS 242 By Bruno Astuto Arouche Nunes February 14 th 2008 1. Introduction In this work we study batch learning

### Feature Selection Using Decision Tree Induction in Class level Metrics Dataset for Software Defect Predictions

, October 20-22, 2010, San Francisco, USA Feature Selection Using Decision Tree Induction in Class level Metrics Dataset for Software Defect Predictions N.Gayatri, S.Nickolas, A.V.Reddy Abstract The importance

### Combating the Class Imbalance Problem in Small Sample Data Sets

Combating the Class Imbalance Problem in Small Sample Data Sets Michael Wasikowski Submitted to the Department of Electrical Engineering & Computer Science and the Graduate Faculty of the University of

### June 30, Relating STAR Reading and STAR Math to the Florida Standards Assessments (FSA) Performance

June 30, 2016 Relating STAR Reading and STAR Math to the Florida Standards Assessments (FSA) Performance Quick reference guide to the STAR Assessments STAR Reading used for screening and progress-monitoring

### Word Sense Disambiguation with Semi-Supervised Learning

Word Sense Disambiguation with Semi-Supervised Learning Thanh Phong Pham 1 and Hwee Tou Ng 1,2 and Wee Sun Lee 1,2 1 Department of Computer Science 2 Singapore-MIT Alliance National University of Singapore

### Evaluating and Comparing Classifiers: Review, Some Recommendations and Limitations

Evaluating and Comparing Classifiers: Review, Some Recommendations and Limitations Katarzyna Stapor (B) Institute of Computer Science, Silesian Technical University, Gliwice, Poland katarzyna.stapor@polsl.pl

### Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement George Forman, Martin Scholz

Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement George Forman, Martin Scholz HP Laboratories HPL-29-359 Keyword(s): AUC,, machine learning, ten-fold cross-validation,

### Data Mining: A Prediction for Academic Performance Improvement of Science Students using Classification

Data Mining: A Prediction for Academic Performance Improvement of Science Students using Classification I.A Ganiyu Department of Computer Science, Ramon Adedoyin College of Science and Technology, Oduduwa

### TANGO Native Anti-Fraud Features

TANGO Native Anti-Fraud Features Tango embeds an anti-fraud service that has been successfully implemented by several large French banks for many years. This service can be provided as an independent Tango

### Linear Regression. Chapter Introduction

Chapter 9 Linear Regression 9.1 Introduction In this class, we have looked at a variety of di erent models and learning methods, such as finite state machines, sequence models, and classification methods.

### CUSBoost: Cluster-based Under-sampling with Boosting for Imbalanced Classification

CUSBoost: Cluster-based Under-sampling with Boosting for Imbalanced Classification Farshid Rayhan, Sajid Ahmed, Asif Mahbub, Md. Rafsan Jani, Swakkhar Shatabda, and Dewan Md. Farid Department of Computer

### Adaptive Quality Estimation for Machine Translation

Adaptive Quality Estimation for Machine Translation Antonis Advisors: Yanis Maistros 1, Marco Turchi 2, Matteo Negri 2 1 School of Electrical and Computer Engineering, NTUA, Greece 2 Fondazione Bruno Kessler,

### !"#\$%#&'()\$*#+','()#\$(-+,./01)

Questions!"#\$%#&'()\$*#+','()#\$(-+,./01) Since induction is fallible, it is necessary to be able to assess its reliability!! Typical questions:! AgroParisTech! What is the true performance of my (learned)

### Predicting Student Performance by Using Data Mining Methods for Classification

BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 13, No 1 Sofia 2013 Print ISSN: 1311-9702; Online ISSN: 1314-4081 DOI: 10.2478/cait-2013-0006 Predicting Student Performance

About This Specialization The 5 courses in this University of Michigan specialization introduce learners to data science through the python programming language. This skills-based specialization is intended

### Learning From the Past with Experiment Databases

Learning From the Past with Experiment Databases Joaquin Vanschoren 1, Bernhard Pfahringer 2, and Geoff Holmes 2 1 Computer Science Dept., K.U.Leuven, Leuven, Belgium 2 Computer Science Dept., University

### Machine Learning and Applications in Finance

Machine Learning and Applications in Finance Christian Hesse 1,2,* 1 Autobahn Equity Europe, Global Markets Equity, Deutsche Bank AG, London, UK christian-a.hesse@db.com 2 Department of Computer Science,

### Improving Classifier Utility by Altering the Misclassification Cost Ratio

Improving Classifier Utility by Altering the Misclassification Cost Ratio Michelle Ciraco, Michael Rogalewski and Gary Weiss Department of Computer Science Fordham University Rose Hill Campus Bronx, New

### Baseline Methods for Active Learning

JMLR: Workshop and Conference Proceedings 6 (0) 47 57 Workshop on Active Learning and Experimental Design Baseline Methods for Active Learning Gavin C. Cawley School of Computing Sciences University of

### Australian Journal of Basic and Applied Sciences

AENSI Journals Australian Journal of Basic and Applied Sciences ISSN:1991-8178 Journal home page: www.ajbasweb.com Feature Selection Technique Using Principal Component Analysis For Improving Fuzzy C-Mean

### Applied Machine Learning Lecture 1: Introduction

Applied Machine Learning Lecture 1: Introduction Richard Johansson January 16, 2018 welcome to the course! machine learning is getting increasingly popular among students our courses are full! many thesis

### (-: (-: SMILES :-) :-)

(-: (-: SMILES :-) :-) A Multi-purpose Learning System Vicent Estruch, Cèsar Ferri, José Hernández-Orallo, M.José Ramírez-Quintana {vestruch, cferri, jorallo, mramirez}@dsic.upv.es Dep. de Sistemes Informàtics

### One-Shot Learning of Faces

One-Shot Learning of Faces Luke Johnston William Chen Department of Computer Science, Stanford University Introduction The ability to learn and generalize from single or few examples is often cited as

### Scheduling Tasks under Constraints CS229 Final Project

Scheduling Tasks under Constraints CS229 Final Project Mike Yu myu3@stanford.edu Dennis Xu dennisx@stanford.edu Kevin Moody kmoody@stanford.edu Abstract The project is based on the principle of unconventional

### BGS Training Requirement in Statistics

BGS Training Requirement in Statistics All BGS students are required to have an understanding of statistical methods and their application to biomedical research. Most students take BIOM611, Statistical

### Classification of Arrhythmia Using Machine Learning Techniques

Classification of Arrhythmia Using Machine Learning Techniques THARA SOMAN PATRICK O. BOBBIE School of Computing and Software Engineering Southern Polytechnic State University (SPSU) 1 S. Marietta Parkway,