!"#$%#&'()$*#+','()#$(-+,./01)

Size: px
Start display at page:

Download "!"#$%#&'()$*#+','()#$(-+,./01)"

Transcription

1 Questions!"#$%#&'()$*#+','()#$(-+,./01) Since induction is fallible, it is necessary to be able to assess its reliability!! Typical questions:! AgroParisTech! What is the true performance of my (learned) classification rule Is my learning algorithm better than this other one? (based in part on Sebastian Thrun CMU class " and on the tutorial of Padraic Cunningham at ECML-09)! Evaluating ML algorithms 2 Outline 1. Measuring the error rate 2. Confusion matrices and various performance criteria!1&0#&'()./*).+%*)*++-+)+#.*) 3. The ROC curve Evaluating ML algorithms 3 Evaluating ML algorithms 4

2 Evaluating classification rules Various sets of data The whole available data set Large data sample Very small data sample Learning set Validation set Test set Illimited sample Evaluating ML algorithms 5 Evaluating ML algorithms 6 Asymptotic behaviour (ideal case) Over-fitting (over-learning) Erreur erreur sur base de test Sur-apprentissage! Useful for very large data sets! erreur sur base d'apprentissage Arrêt de l'apprentissage t Evaluating ML algorithms 7 Evaluating ML algorithms 8

3 Over-fitting (NNs) Why using a test set?! The control parameters of the learning algorithm E.g.: number of hidden layers, number of neurons,... Are tuned in order to reduce the error on the validation set ))2-%+3*1)4-%+)5)666)*7*04$*1)! In order to have a non optimistically biased estimate of the error, one must measure it on an independent data set: the test set!!"#$%&'(!)#$%!*!+++!','-).'(!/! Evaluating ML algorithms 9 Evaluating ML algorithms 10 Evaluating classification rules Evaluating the error rate! True error:! (Real risk) e =! y " f ( x, # ) p( x, y) dx, y D D D = the true distribution! Test error:! (Empirical risk) 1 eˆ S =! y # f ( x, $ ) m x, y " ST A lot Few m = # of test examples T = test data Evaluating ML algorithms 11 Evaluating ML algorithms 12

4 Example: Confidence intervals! We want to estimate error D (h).!! The learned hypothesis incorrectly classifies 12 out of 40 examples in the test set T.! Q : What will be the true error rate?! R :???! We estimate it by using error T (h) which follows a binomial law! With mean! And standard error!!! They are estimated using the normal law with: Mean: Standard deviation: Evaluating ML algorithms 13 Evaluating ML algorithms 14 Confidence intervals Confidence intervals! The normal law!! The normal law! With probability N%, the true error error D lies in the interval:! N% 50% 68% 80% 90% 95% 98% 99% z N Evaluating ML algorithms 15 Evaluating ML algorithms 16

5 Confidence intervals (cf. Mitchell 97) Example: If T contains m examples independently sampled m! 30 Then With probability 95%, the true error e D is within: eˆ S ± 1.96 eˆ S (1! eˆ S ) m! The learned hypothesis incorrectly classifies 12 out of 40 test examples in T.! Q: What will be the true error on unseen examples?! A: With 95% confidence, the true error will lie within [0.16;0.44] " eˆ S ± 1.96 eˆ S (1! eˆ S ) m m = eˆ ˆ eˆ S (1 " es ) S = = ! m Evaluating ML algorithms 17 Evaluating ML algorithms 18 95% confidence intervals Performance curves 95% confidence intervals Erreur de test Erreur d apprentissage Evaluating ML algorithms 19 Evaluating ML algorithms 20

6 Evaluating learned hypotheses Various sets Data Lot of data Few Learning test " error Evaluating ML algorithms 21 Evaluating ML algorithms 22 Small data sets: a dilemma Small data sets: a dilemma Evaluating ML algorithms 23 Evaluating ML algorithms 24

7 Cross validation (k-fold) Data Learn on yellow, test on rose " error 1 Learn on yellow, test on rose " error 2 Learn on yellow, test on rose " error 3 k-way split Learn on yellow, test on rose " error 4 Learn on yellow, test on rose " error 5 Learn on yellow, test on rose " error 6 The leave-one-out procedure Data! Low bias! Highvariance! Tends to underestimate the error if the data are not fully i.i.d. Learn on yellow, test on rose " error 7 Learn on yellow, test on rose " error 8 [Guyon & Elisseeff, jmlr, 03]! error = # error i / k Evaluating ML algorithms 25 Evaluating ML algorithms 26 The Bootstrap estimate Problem Data! The calculation of the confidence interval supposes the independence of the estimations.! But our estimations are not independent. # " Learn on yellow, test on rose " error " Repeat and compute the mean Estimation of the true risk for the final h Mean of the risks On the k test samples Mean of the risk on whole data set Evaluating ML algorithms 27 Evaluating ML algorithms 28

8 Types of performance criteria 2-'8%1,-')0#.+,9*1) #':)"#+,-%1)4*+8-+0#'9*)9+,.*+,#) Evaluating ML algorithms 29 Evaluating ML algorithms 30 Confusion matrix Confusion matrix 14% of the butterflies are recognized as fishes Réel! Estimé! +! -! +! VP! FP! -! FN! VN! Evaluating ML algorithms 31 Evaluating ML algorithms 32

9 Types of performance criteria Types of performance criteria Evaluating ML algorithms 33 Evaluating ML algorithms 34 Types of performance criteria Types of performance criteria Evaluating ML algorithms 35 Evaluating ML algorithms 36

10 Types of performance measures Performance measures! Sensitivity! VP FN + VP! Recall! VP VP + FN! Specificity! VN VN + FP! Precision! VP VP + FP Réel! Estimé! +! -! +! VP! FP! -! FN! VN! Evaluating ML algorithms 37 Evaluating ML algorithms 38 Performance measures Performance measures! FN-rate! FN VP + FN! FP-rate! FP FP + VN! F-measure! 2 x recall x precision Recall + precision = 2 VP 2 VP + FP + FN Réel! Estimé! +! -! +! VP! FP! -! FN! VN! Evaluating ML algorithms 39 Evaluating ML algorithms 40

11 Performance measures Performance measures Evaluating ML algorithms 41 Evaluating ML algorithms 42 Performance measures H/*)IJ2)9%+"*)!!!!!!!!!!!!"#$%! &'()#! *++,! -.,! (--:) 6;<=>) 3#:) 6;56>) 6;A<A) Evaluating ML algorithms 43 Evaluating ML algorithms 44

12 The ROC curve Types of errors Evaluating ML algorithms 45 Evaluating ML algorithms 46 The ROC curve The ROC curve ROC = Receiver Operating Characteristic Probabilité de la classe Classe '+' Faux négatifs Vrais positifs Probabilité de la classe Classe '-' Classe '+' (10%) (90%) Critère de décision Probabilité de la classe Classe '-' Vrais négatifs Faux positifs Critère de décision (50%) (50%) Evaluating ML algorithms 47 Critère de décision Evaluating ML algorithms 48

13 Classe '+' Faux négatifs Classe '- ' Vrais négatifs Faux positifs Vrais positifs Critère de décision Critère de décision Classe '+' Faux négatifs (10%) Classe '- ' Vrais négatifs (50%) (50%) Faux positifs Vrais positifs (90%) Critère de décision Critère de décision The ROC curve The ROC curve PROPORTION DE VRAIS NEGATIFS 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0 1,0 1,0 0,9 0,9 PROPORTION DE VRAIS POSITIFS 0,8 0,7 0,6 0,5 0,4 0,3 Courbe ROC (pertinence = 0,90) Ligne de hasard (pertinence = 0,5) 0,8 0,7 0,6 0,5 0,4 0,3 PROPORTION DE FAUX NEGATIFS 0,2 0,2 0,1 0, ,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0 0 PROPORTION DE FAUX POSITIFS Evaluating ML algorithms 49 Evaluating ML algorithms 50 The ROC curve The ROC curve PROPORTION DE VRAIS NEGATIFS PROPORTION DE VRAIS NEGATIFS 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0 1,0 1,0 1,0 1,0 0,9 0,9 0,9 Seuil "laxiste" 0,9 PROPORTION DE VRAIS POSITIFS 0,8 0,7 0,6 0,5 0,4 0,3 Courbe ROC (pertinence = 0,90) Ligne de hasard (pertinence = 0,5) 0,8 0,7 0,6 0,5 0,4 0,3 PROPORTION DE FAUXNEGATIFS PROPORTION DE VRAIS POSITIFS 0,8 0,7 0,6 0,5 0,4 0,3 Seuil "sévère" Probabilité delaclase Probabilité delaclase Probabilité delaclase Probabilité delaclase 0,8 0,7 0,6 0,5 0,4 0,3 PROPORTION DE FAUXNEGATIFS 0,2 0,2 0,2 0,2 0,1 0,1 0,1 0, ,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1, ,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0 PROPORTION DE FAUX POSITIFS PROPORTION DE FAUX POSITIFS Evaluating ML algorithms 51 Evaluating ML algorithms 52

14 The ROC curve Comparaison of learning algorithms! Résumé!! Comparison on a single data sets [Dietterich, 1998] recommends using: 5 x 2 cross-validation Paired t-test The McNemar test on a validation set! Comparison on multiples (different) data sets [Demsar, 2006] recommends using: Wilcoxon Signed Ranks Test The Friedman test Evaluating ML algorithms 53 Evaluating ML algorithms 54 Résumé Specific problems! Attention à votre fonction de coût : qu est-ce qui importe pour la mesure de performance?! Données en nombre fini: calculez les intervalles de confiance! Données rares : Attention à la répartition entre données d apprentissage et données test. Validation croisée.! N oubliez pas l ensemble de validation! The distribution of the classes is very unbalanced (e.g. 1% ou 1%O for one of the two classes)! Gray zone (uncertain labels)! Multi-valued functions! L évaluation est très importante Ayez l esprit critique Convainquez-vous vous même! Evaluating ML algorithms 55 Evaluating ML algorithms 56

15 Other evaluation criteria References! Intelligibility of the learned decision function E.g. SVMs or boosting are not good! Performances in generalization Often not correlated to the previous performance criterion! Various costs Data preparation Computational cost Cost of the ML expertise Cost of the domain expertise! Dietterich, T. G., (1998). Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 10 (7) !! JapKowicz N. & Shah M. (2011). Evaluating Learning Algorithms. A classification perspective. Cambridge University Press, (An interesting book)! Evaluating ML algorithms 57 Evaluating ML algorithms 58 The Weka ML toolkit The Weka ML toolkit! Evaluating ML algorithms 59 Evaluating ML algorithms 60

Python Machine Learning

Python Machine Learning Python Machine Learning Unlock deeper insights into machine learning with this vital guide to cuttingedge predictive analytics Sebastian Raschka [ PUBLISHING 1 open source I community experience distilled

More information

Evaluating and Comparing Classifiers: Review, Some Recommendations and Limitations

Evaluating and Comparing Classifiers: Review, Some Recommendations and Limitations Evaluating and Comparing Classifiers: Review, Some Recommendations and Limitations Katarzyna Stapor (B) Institute of Computer Science, Silesian Technical University, Gliwice, Poland katarzyna.stapor@polsl.pl

More information

Introduction to Ensemble Learning Featuring Successes in the Netflix Prize Competition

Introduction to Ensemble Learning Featuring Successes in the Netflix Prize Competition Introduction to Ensemble Learning Featuring Successes in the Netflix Prize Competition Todd Holloway Two Lecture Series for B551 November 20 & 27, 2007 Indiana University Outline Introduction Bias and

More information

Lecture 1: Machine Learning Basics

Lecture 1: Machine Learning Basics 1/69 Lecture 1: Machine Learning Basics Ali Harakeh University of Waterloo WAVE Lab ali.harakeh@uwaterloo.ca May 1, 2017 2/69 Overview 1 Learning Algorithms 2 Capacity, Overfitting, and Underfitting 3

More information

Learning From the Past with Experiment Databases

Learning From the Past with Experiment Databases Learning From the Past with Experiment Databases Joaquin Vanschoren 1, Bernhard Pfahringer 2, and Geoff Holmes 2 1 Computer Science Dept., K.U.Leuven, Leuven, Belgium 2 Computer Science Dept., University

More information

Active Learning. Yingyu Liang Computer Sciences 760 Fall

Active Learning. Yingyu Liang Computer Sciences 760 Fall Active Learning Yingyu Liang Computer Sciences 760 Fall 2017 http://pages.cs.wisc.edu/~yliang/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven,

More information

Predicting Student Attrition in MOOCs using Sentiment Analysis and Neural Networks

Predicting Student Attrition in MOOCs using Sentiment Analysis and Neural Networks Predicting Student Attrition in MOOCs using Sentiment Analysis and Neural Networks Devendra Singh Chaplot, Eunhee Rhim, and Jihie Kim Samsung Electronics Co., Ltd. Seoul, South Korea {dev.chaplot,eunhee.rhim,jihie.kim}@samsung.com

More information

CSL465/603 - Machine Learning

CSL465/603 - Machine Learning CSL465/603 - Machine Learning Fall 2016 Narayanan C Krishnan ckn@iitrpr.ac.in Introduction CSL465/603 - Machine Learning 1 Administrative Trivia Course Structure 3-0-2 Lecture Timings Monday 9.55-10.45am

More information

Laboratorio di Intelligenza Artificiale e Robotica

Laboratorio di Intelligenza Artificiale e Robotica Laboratorio di Intelligenza Artificiale e Robotica A.A. 2008-2009 Outline 2 Machine Learning Unsupervised Learning Supervised Learning Reinforcement Learning Genetic Algorithms Genetics-Based Machine Learning

More information

Acquisition vs. Learning of a Second Language: English Negation

Acquisition vs. Learning of a Second Language: English Negation Interculturalia Acquisition vs. Learning of a Second Language: English Negation Oana BADEA Key-words: acquisition, learning, first/second language, English negation General Remarks on Theories of Second/

More information

QuickStroke: An Incremental On-line Chinese Handwriting Recognition System

QuickStroke: An Incremental On-line Chinese Handwriting Recognition System QuickStroke: An Incremental On-line Chinese Handwriting Recognition System Nada P. Matić John C. Platt Λ Tony Wang y Synaptics, Inc. 2381 Bering Drive San Jose, CA 95131, USA Abstract This paper presents

More information

Laboratorio di Intelligenza Artificiale e Robotica

Laboratorio di Intelligenza Artificiale e Robotica Laboratorio di Intelligenza Artificiale e Robotica A.A. 2008-2009 Outline 2 Machine Learning Unsupervised Learning Supervised Learning Reinforcement Learning Genetic Algorithms Genetics-Based Machine Learning

More information

Probability and Statistics Curriculum Pacing Guide

Probability and Statistics Curriculum Pacing Guide Unit 1 Terms PS.SPMJ.3 PS.SPMJ.5 Plan and conduct a survey to answer a statistical question. Recognize how the plan addresses sampling technique, randomization, measurement of experimental error and methods

More information

Evolutive Neural Net Fuzzy Filtering: Basic Description

Evolutive Neural Net Fuzzy Filtering: Basic Description Journal of Intelligent Learning Systems and Applications, 2010, 2: 12-18 doi:10.4236/jilsa.2010.21002 Published Online February 2010 (http://www.scirp.org/journal/jilsa) Evolutive Neural Net Fuzzy Filtering:

More information

EMC Publishing s C est à toi! Level 3, 2 nd edition Correlated to the Oregon World Language Content Standards

EMC Publishing s C est à toi! Level 3, 2 nd edition Correlated to the Oregon World Language Content Standards EMC Publishing s C est à toi! Level 3, 2 nd edition Correlated to the Oregon World Language Content Standards Oregon Correlation C est à toi! Level 1, 2 nd edition to the: Oregon World Language Content

More information

CS Machine Learning

CS Machine Learning CS 478 - Machine Learning Projects Data Representation Basic testing and evaluation schemes CS 478 Data and Testing 1 Programming Issues l Program in any platform you want l Realize that you will be doing

More information

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur Module 12 Machine Learning 12.1 Instructional Objective The students should understand the concept of learning systems Students should learn about different aspects of a learning system Students should

More information

Iterative Cross-Training: An Algorithm for Learning from Unlabeled Web Pages

Iterative Cross-Training: An Algorithm for Learning from Unlabeled Web Pages Iterative Cross-Training: An Algorithm for Learning from Unlabeled Web Pages Nuanwan Soonthornphisaj 1 and Boonserm Kijsirikul 2 Machine Intelligence and Knowledge Discovery Laboratory Department of Computer

More information

Using dialogue context to improve parsing performance in dialogue systems

Using dialogue context to improve parsing performance in dialogue systems Using dialogue context to improve parsing performance in dialogue systems Ivan Meza-Ruiz and Oliver Lemon School of Informatics, Edinburgh University 2 Buccleuch Place, Edinburgh I.V.Meza-Ruiz@sms.ed.ac.uk,

More information

Algebra 1, Quarter 3, Unit 3.1. Line of Best Fit. Overview

Algebra 1, Quarter 3, Unit 3.1. Line of Best Fit. Overview Algebra 1, Quarter 3, Unit 3.1 Line of Best Fit Overview Number of instructional days 6 (1 day assessment) (1 day = 45 minutes) Content to be learned Analyze scatter plots and construct the line of best

More information

System Implementation for SemEval-2017 Task 4 Subtask A Based on Interpolated Deep Neural Networks

System Implementation for SemEval-2017 Task 4 Subtask A Based on Interpolated Deep Neural Networks System Implementation for SemEval-2017 Task 4 Subtask A Based on Interpolated Deep Neural Networks 1 Tzu-Hsuan Yang, 2 Tzu-Hsuan Tseng, and 3 Chia-Ping Chen Department of Computer Science and Engineering

More information

Knowledge Transfer in Deep Convolutional Neural Nets

Knowledge Transfer in Deep Convolutional Neural Nets Knowledge Transfer in Deep Convolutional Neural Nets Steven Gutstein, Olac Fuentes and Eric Freudenthal Computer Science Department University of Texas at El Paso El Paso, Texas, 79968, U.S.A. Abstract

More information

Lecture 1: Basic Concepts of Machine Learning

Lecture 1: Basic Concepts of Machine Learning Lecture 1: Basic Concepts of Machine Learning Cognitive Systems - Machine Learning Ute Schmid (lecture) Johannes Rabold (practice) Based on slides prepared March 2005 by Maximilian Röglinger, updated 2010

More information

ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY DOWNLOAD EBOOK : ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY PDF

ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY DOWNLOAD EBOOK : ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY PDF Read Online and Download Ebook ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY DOWNLOAD EBOOK : ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY PDF Click link bellow and free register to download

More information

MULTIPLE CHOICE. Choose the one alternative that best completes the statement or answers the question.

MULTIPLE CHOICE. Choose the one alternative that best completes the statement or answers the question. Ch 2 Test Remediation Work Name MULTIPLE CHOICE. Choose the one alternative that best completes the statement or answers the question. Provide an appropriate response. 1) High temperatures in a certain

More information

Disambiguation of Thai Personal Name from Online News Articles

Disambiguation of Thai Personal Name from Online News Articles Disambiguation of Thai Personal Name from Online News Articles Phaisarn Sutheebanjard Graduate School of Information Technology Siam University Bangkok, Thailand mr.phaisarn@gmail.com Abstract Since online

More information

Issues in the Mining of Heart Failure Datasets

Issues in the Mining of Heart Failure Datasets International Journal of Automation and Computing 11(2), April 2014, 162-179 DOI: 10.1007/s11633-014-0778-5 Issues in the Mining of Heart Failure Datasets Nongnuch Poolsawad 1 Lisa Moore 1 Chandrasekhar

More information

Knowledge based expert systems D H A N A N J A Y K A L B A N D E

Knowledge based expert systems D H A N A N J A Y K A L B A N D E Knowledge based expert systems D H A N A N J A Y K A L B A N D E What is a knowledge based system? A Knowledge Based System or a KBS is a computer program that uses artificial intelligence to solve problems

More information

OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS

OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS Václav Kocian, Eva Volná, Michal Janošek, Martin Kotyrba University of Ostrava Department of Informatics and Computers Dvořákova 7,

More information

OCR for Arabic using SIFT Descriptors With Online Failure Prediction

OCR for Arabic using SIFT Descriptors With Online Failure Prediction OCR for Arabic using SIFT Descriptors With Online Failure Prediction Andrey Stolyarenko, Nachum Dershowitz The Blavatnik School of Computer Science Tel Aviv University Tel Aviv, Israel Email: stloyare@tau.ac.il,

More information

STA 225: Introductory Statistics (CT)

STA 225: Introductory Statistics (CT) Marshall University College of Science Mathematics Department STA 225: Introductory Statistics (CT) Course catalog description A critical thinking course in applied statistical reasoning covering basic

More information

Exemplar for Internal Achievement Standard French Level 1

Exemplar for Internal Achievement Standard French Level 1 Exemplar for internal assessment resource French for Achievement Standard 90882 Exemplar for Internal Achievement Standard French Level 1 This exemplar supports assessment against: Achievement Standard

More information

Probability Therefore (25) (1.33)

Probability Therefore (25) (1.33) Probability We have intentionally included more material than can be covered in most Student Study Sessions to account for groups that are able to answer the questions at a faster rate. Use your own judgment,

More information

A Case Study: News Classification Based on Term Frequency

A Case Study: News Classification Based on Term Frequency A Case Study: News Classification Based on Term Frequency Petr Kroha Faculty of Computer Science University of Technology 09107 Chemnitz Germany kroha@informatik.tu-chemnitz.de Ricardo Baeza-Yates Center

More information

Probability estimates in a scenario tree

Probability estimates in a scenario tree 101 Chapter 11 Probability estimates in a scenario tree An expert is a person who has made all the mistakes that can be made in a very narrow field. Niels Bohr (1885 1962) Scenario trees require many numbers.

More information

Chapter 2 Rule Learning in a Nutshell

Chapter 2 Rule Learning in a Nutshell Chapter 2 Rule Learning in a Nutshell This chapter gives a brief overview of inductive rule learning and may therefore serve as a guide through the rest of the book. Later chapters will expand upon the

More information

Applications of data mining algorithms to analysis of medical data

Applications of data mining algorithms to analysis of medical data Master Thesis Software Engineering Thesis no: MSE-2007:20 August 2007 Applications of data mining algorithms to analysis of medical data Dariusz Matyja School of Engineering Blekinge Institute of Technology

More information

Evolution of Symbolisation in Chimpanzees and Neural Nets

Evolution of Symbolisation in Chimpanzees and Neural Nets Evolution of Symbolisation in Chimpanzees and Neural Nets Angelo Cangelosi Centre for Neural and Adaptive Systems University of Plymouth (UK) a.cangelosi@plymouth.ac.uk Introduction Animal communication

More information

For Jury Evaluation. The Road to Enlightenment: Generating Insight and Predicting Consumer Actions in Digital Markets

For Jury Evaluation. The Road to Enlightenment: Generating Insight and Predicting Consumer Actions in Digital Markets FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO The Road to Enlightenment: Generating Insight and Predicting Consumer Actions in Digital Markets Jorge Moreira da Silva For Jury Evaluation Mestrado Integrado

More information

Semi-Supervised Face Detection

Semi-Supervised Face Detection Semi-Supervised Face Detection Nicu Sebe, Ira Cohen 2, Thomas S. Huang 3, Theo Gevers Faculty of Science, University of Amsterdam, The Netherlands 2 HP Research Labs, USA 3 Beckman Institute, University

More information

PROJECT 1 News Media. Note: this project frequently requires the use of Internet-connected computers

PROJECT 1 News Media. Note: this project frequently requires the use of Internet-connected computers 1 PROJECT 1 News Media Note: this project frequently requires the use of Internet-connected computers Unit Description: while developing their reading and communication skills, the students will reflect

More information

Assignment 1: Predicting Amazon Review Ratings

Assignment 1: Predicting Amazon Review Ratings Assignment 1: Predicting Amazon Review Ratings 1 Dataset Analysis Richard Park r2park@acsmail.ucsd.edu February 23, 2015 The dataset selected for this assignment comes from the set of Amazon reviews for

More information

1. Share the following information with your partner. Spell each name to your partner. Change roles. One object in the classroom:

1. Share the following information with your partner. Spell each name to your partner. Change roles. One object in the classroom: French 1A Final Examination Study Guide January 2015 Montgomery County Public Schools Name: Before you begin working on the study guide, organize your notes and vocabulary lists from semester A. Refer

More information

Paper: Collaborative Information Behaviour of Engineering Students

Paper: Collaborative Information Behaviour of Engineering Students Nasser Saleh, Andrew Large McGill University, Montreal, Quebec Paper: Collaborative Information Behaviour of Engineering Students Abstract: Collaborative information behaviour is an emerging area in information

More information

THE ROLE OF DECISION TREES IN NATURAL LANGUAGE PROCESSING

THE ROLE OF DECISION TREES IN NATURAL LANGUAGE PROCESSING SISOM & ACOUSTICS 2015, Bucharest 21-22 May THE ROLE OF DECISION TREES IN NATURAL LANGUAGE PROCESSING MarilenaăLAZ R 1, Diana MILITARU 2 1 Military Equipment and Technologies Research Agency, Bucharest,

More information

Version Space. Term 2012/2013 LSI - FIB. Javier Béjar cbea (LSI - FIB) Version Space Term 2012/ / 18

Version Space. Term 2012/2013 LSI - FIB. Javier Béjar cbea (LSI - FIB) Version Space Term 2012/ / 18 Version Space Javier Béjar cbea LSI - FIB Term 2012/2013 Javier Béjar cbea (LSI - FIB) Version Space Term 2012/2013 1 / 18 Outline 1 Learning logical formulas 2 Version space Introduction Search strategy

More information

Predicting Students Performance with SimStudent: Learning Cognitive Skills from Observation

Predicting Students Performance with SimStudent: Learning Cognitive Skills from Observation School of Computer Science Human-Computer Interaction Institute Carnegie Mellon University Year 2007 Predicting Students Performance with SimStudent: Learning Cognitive Skills from Observation Noboru Matsuda

More information

Optimizing to Arbitrary NLP Metrics using Ensemble Selection

Optimizing to Arbitrary NLP Metrics using Ensemble Selection Optimizing to Arbitrary NLP Metrics using Ensemble Selection Art Munson, Claire Cardie, Rich Caruana Department of Computer Science Cornell University Ithaca, NY 14850 {mmunson, cardie, caruana}@cs.cornell.edu

More information

Detecting Wikipedia Vandalism using Machine Learning Notebook for PAN at CLEF 2011

Detecting Wikipedia Vandalism using Machine Learning Notebook for PAN at CLEF 2011 Detecting Wikipedia Vandalism using Machine Learning Notebook for PAN at CLEF 2011 Cristian-Alexandru Drăgușanu, Marina Cufliuc, Adrian Iftene UAIC: Faculty of Computer Science, Alexandru Ioan Cuza University,

More information

A Neural Network GUI Tested on Text-To-Phoneme Mapping

A Neural Network GUI Tested on Text-To-Phoneme Mapping A Neural Network GUI Tested on Text-To-Phoneme Mapping MAARTEN TROMPPER Universiteit Utrecht m.f.a.trompper@students.uu.nl Abstract Text-to-phoneme (T2P) mapping is a necessary step in any speech synthesis

More information

Linking the Ohio State Assessments to NWEA MAP Growth Tests *

Linking the Ohio State Assessments to NWEA MAP Growth Tests * Linking the Ohio State Assessments to NWEA MAP Growth Tests * *As of June 2017 Measures of Academic Progress (MAP ) is known as MAP Growth. August 2016 Introduction Northwest Evaluation Association (NWEA

More information

9779 PRINCIPAL COURSE FRENCH

9779 PRINCIPAL COURSE FRENCH CAMBRIDGE INTERNATIONAL EXAMINATIONS Pre-U Certificate MARK SCHEME for the May/June 2014 series 9779 PRINCIPAL COURSE FRENCH 9779/03 Paper 1 (Writing and Usage), maximum raw mark 60 This mark scheme is

More information

(Sub)Gradient Descent

(Sub)Gradient Descent (Sub)Gradient Descent CMSC 422 MARINE CARPUAT marine@cs.umd.edu Figures credit: Piyush Rai Logistics Midterm is on Thursday 3/24 during class time closed book/internet/etc, one page of notes. will include

More information

Practical Research. Planning and Design. Paul D. Leedy. Jeanne Ellis Ormrod. Upper Saddle River, New Jersey Columbus, Ohio

Practical Research. Planning and Design. Paul D. Leedy. Jeanne Ellis Ormrod. Upper Saddle River, New Jersey Columbus, Ohio SUB Gfittingen 213 789 981 2001 B 865 Practical Research Planning and Design Paul D. Leedy The American University, Emeritus Jeanne Ellis Ormrod University of New Hampshire Upper Saddle River, New Jersey

More information

FUZZY EXPERT. Dr. Kasim M. Al-Aubidy. Philadelphia University. Computer Eng. Dept February 2002 University of Damascus-Syria

FUZZY EXPERT. Dr. Kasim M. Al-Aubidy. Philadelphia University. Computer Eng. Dept February 2002 University of Damascus-Syria FUZZY EXPERT SYSTEMS 16-18 18 February 2002 University of Damascus-Syria Dr. Kasim M. Al-Aubidy Computer Eng. Dept. Philadelphia University What is Expert Systems? ES are computer programs that emulate

More information

Rule Learning With Negation: Issues Regarding Effectiveness

Rule Learning With Negation: Issues Regarding Effectiveness Rule Learning With Negation: Issues Regarding Effectiveness S. Chua, F. Coenen, G. Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX Liverpool, United

More information

Course Outline. Course Grading. Where to go for help. Academic Integrity. EE-589 Introduction to Neural Networks NN 1 EE

Course Outline. Course Grading. Where to go for help. Academic Integrity. EE-589 Introduction to Neural Networks NN 1 EE EE-589 Introduction to Neural Assistant Prof. Dr. Turgay IBRIKCI Room # 305 (322) 338 6868 / 139 Wensdays 9:00-12:00 Course Outline The course is divided in two parts: theory and practice. 1. Theory covers

More information

Name of Course: French 1 Middle School. Grade Level(s): 7 and 8 (half each) Unit 1

Name of Course: French 1 Middle School. Grade Level(s): 7 and 8 (half each) Unit 1 Name of Course: French 1 Middle School Grade Level(s): 7 and 8 (half each) Unit 1 Estimated Instructional Time: 15 classes PA Academic Standards: Communication: Communicate in Languages Other Than English

More information

Using Web Searches on Important Words to Create Background Sets for LSI Classification

Using Web Searches on Important Words to Create Background Sets for LSI Classification Using Web Searches on Important Words to Create Background Sets for LSI Classification Sarah Zelikovitz and Marina Kogan College of Staten Island of CUNY 2800 Victory Blvd Staten Island, NY 11314 Abstract

More information

1. REFLEXES: Ask questions about coughing, swallowing, of water as fast as possible (note! Not suitable for all

1. REFLEXES: Ask questions about coughing, swallowing, of water as fast as possible (note! Not suitable for all Human Communication Science Chandler House, 2 Wakefield Street London WC1N 1PF http://www.hcs.ucl.ac.uk/ ACOUSTICS OF SPEECH INTELLIGIBILITY IN DYSARTHRIA EUROPEAN MASTER S S IN CLINICAL LINGUISTICS UNIVERSITY

More information

Using focal point learning to improve human machine tacit coordination

Using focal point learning to improve human machine tacit coordination DOI 10.1007/s10458-010-9126-5 Using focal point learning to improve human machine tacit coordination InonZuckerman SaritKraus Jeffrey S. Rosenschein The Author(s) 2010 Abstract We consider an automated

More information

Multi-label classification via multi-target regression on data streams

Multi-label classification via multi-target regression on data streams Mach Learn (2017) 106:745 770 DOI 10.1007/s10994-016-5613-5 Multi-label classification via multi-target regression on data streams Aljaž Osojnik 1,2 Panče Panov 1 Sašo Džeroski 1,2,3 Received: 26 April

More information

Forget catastrophic forgetting: AI that learns after deployment

Forget catastrophic forgetting: AI that learns after deployment Forget catastrophic forgetting: AI that learns after deployment Anatoly Gorshechnikov CTO, Neurala 1 Neurala at a glance Programming neural networks on GPUs since circa 2 B.C. Founded in 2006 expecting

More information

Activity Recognition from Accelerometer Data

Activity Recognition from Accelerometer Data Activity Recognition from Accelerometer Data Nishkam Ravi and Nikhil Dandekar and Preetham Mysore and Michael L. Littman Department of Computer Science Rutgers University Piscataway, NJ 08854 {nravi,nikhild,preetham,mlittman}@cs.rutgers.edu

More information

Australian Journal of Basic and Applied Sciences

Australian Journal of Basic and Applied Sciences AENSI Journals Australian Journal of Basic and Applied Sciences ISSN:1991-8178 Journal home page: www.ajbasweb.com Feature Selection Technique Using Principal Component Analysis For Improving Fuzzy C-Mean

More information

Model Ensemble for Click Prediction in Bing Search Ads

Model Ensemble for Click Prediction in Bing Search Ads Model Ensemble for Click Prediction in Bing Search Ads Xiaoliang Ling Microsoft Bing xiaoling@microsoft.com Hucheng Zhou Microsoft Research huzho@microsoft.com Weiwei Deng Microsoft Bing dedeng@microsoft.com

More information

Approaches for analyzing tutor's role in a networked inquiry discourse

Approaches for analyzing tutor's role in a networked inquiry discourse Lakkala, M., Muukkonen, H., Ilomäki, L., Lallimo, J., Niemivirta, M. & Hakkarainen, K. (2001) Approaches for analysing tutor's role in a networked inquiry discourse. In P. Dillenbourg, A. Eurelings., &

More information

Experiment Databases: Towards an Improved Experimental Methodology in Machine Learning

Experiment Databases: Towards an Improved Experimental Methodology in Machine Learning Experiment Databases: Towards an Improved Experimental Methodology in Machine Learning Hendrik Blockeel and Joaquin Vanschoren Computer Science Dept., K.U.Leuven, Celestijnenlaan 200A, 3001 Leuven, Belgium

More information

The Lexicalization of Acronyms in English: The Case of Third Year E.F.L Students, Mentouri University- Constantine

The Lexicalization of Acronyms in English: The Case of Third Year E.F.L Students, Mentouri University- Constantine The Lexicalization of Acronyms in English: The Case of Third Year E.F.L Students, Mentouri University- Constantine Yamina BENNANE Université Frères Mentouri. Constantine 1. Algérie Abstract: The present

More information

Machine Learning from Garden Path Sentences: The Application of Computational Linguistics

Machine Learning from Garden Path Sentences: The Application of Computational Linguistics Machine Learning from Garden Path Sentences: The Application of Computational Linguistics http://dx.doi.org/10.3991/ijet.v9i6.4109 J.L. Du 1, P.F. Yu 1 and M.L. Li 2 1 Guangdong University of Foreign Studies,

More information

What Different Kinds of Stratification Can Reveal about the Generalizability of Data-Mined Skill Assessment Models

What Different Kinds of Stratification Can Reveal about the Generalizability of Data-Mined Skill Assessment Models What Different Kinds of Stratification Can Reveal about the Generalizability of Data-Mined Skill Assessment Models Michael A. Sao Pedro Worcester Polytechnic Institute 100 Institute Rd. Worcester, MA 01609

More information

METHODS FOR EXTRACTING AND CLASSIFYING PAIRS OF COGNATES AND FALSE FRIENDS

METHODS FOR EXTRACTING AND CLASSIFYING PAIRS OF COGNATES AND FALSE FRIENDS METHODS FOR EXTRACTING AND CLASSIFYING PAIRS OF COGNATES AND FALSE FRIENDS Ruslan Mitkov (R.Mitkov@wlv.ac.uk) University of Wolverhampton ViktorPekar (v.pekar@wlv.ac.uk) University of Wolverhampton Dimitar

More information

Evidence for Reliability, Validity and Learning Effectiveness

Evidence for Reliability, Validity and Learning Effectiveness PEARSON EDUCATION Evidence for Reliability, Validity and Learning Effectiveness Introduction Pearson Knowledge Technologies has conducted a large number and wide variety of reliability and validity studies

More information

Stopping rules for sequential trials in high-dimensional data

Stopping rules for sequential trials in high-dimensional data Stopping rules for sequential trials in high-dimensional data Sonja Zehetmayer, Alexandra Graf, and Martin Posch Center for Medical Statistics, Informatics and Intelligent Systems Medical University of

More information

Notes on The Sciences of the Artificial Adapted from a shorter document written for course (Deciding What to Design) 1

Notes on The Sciences of the Artificial Adapted from a shorter document written for course (Deciding What to Design) 1 Notes on The Sciences of the Artificial Adapted from a shorter document written for course 17-652 (Deciding What to Design) 1 Ali Almossawi December 29, 2005 1 Introduction The Sciences of the Artificial

More information

Question 1 Does the concept of "part-time study" exist in your University and, if yes, how is it put into practice, is it possible in every Faculty?

Question 1 Does the concept of part-time study exist in your University and, if yes, how is it put into practice, is it possible in every Faculty? Name of the University Country Univerza v Ljubljani Slovenia Tallin University of Technology (TUT) Estonia Question 1 Does the concept of "part-time study" exist in your University and, if yes, how is

More information

How to Judge the Quality of an Objective Classroom Test

How to Judge the Quality of an Objective Classroom Test How to Judge the Quality of an Objective Classroom Test Technical Bulletin #6 Evaluation and Examination Service The University of Iowa (319) 335-0356 HOW TO JUDGE THE QUALITY OF AN OBJECTIVE CLASSROOM

More information

Rule Learning with Negation: Issues Regarding Effectiveness

Rule Learning with Negation: Issues Regarding Effectiveness Rule Learning with Negation: Issues Regarding Effectiveness Stephanie Chua, Frans Coenen, and Grant Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX

More information

Machine Learning and Development Policy

Machine Learning and Development Policy Machine Learning and Development Policy Sendhil Mullainathan (joint papers with Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Ziad Obermeyer) Magic? Hard not to be wowed But what makes

More information

Linking Task: Identifying authors and book titles in verbose queries

Linking Task: Identifying authors and book titles in verbose queries Linking Task: Identifying authors and book titles in verbose queries Anaïs Ollagnier, Sébastien Fournier, and Patrice Bellot Aix-Marseille University, CNRS, ENSAM, University of Toulon, LSIS UMR 7296,

More information

CAVE LANGUAGES KS2 SCHEME OF WORK LANGUAGE OVERVIEW. YEAR 3 Stage 1 Lessons 1-30

CAVE LANGUAGES KS2 SCHEME OF WORK LANGUAGE OVERVIEW. YEAR 3 Stage 1 Lessons 1-30 CAVE LANGUAGES KS2 SCHEME OF WORK LANGUAGE OVERVIEW AUTUMN TERM Stage 1 Lessons 1-8 Christmas lessons 1-4 LANGUAGE CONTENT Greetings Classroom commands listening/speaking Feelings question/answer 5 colours-recognition

More information

Twitter Sentiment Classification on Sanders Data using Hybrid Approach

Twitter Sentiment Classification on Sanders Data using Hybrid Approach IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661,p-ISSN: 2278-8727, Volume 17, Issue 4, Ver. I (July Aug. 2015), PP 118-123 www.iosrjournals.org Twitter Sentiment Classification on Sanders

More information

Calibration of Confidence Measures in Speech Recognition

Calibration of Confidence Measures in Speech Recognition Submitted to IEEE Trans on Audio, Speech, and Language, July 2010 1 Calibration of Confidence Measures in Speech Recognition Dong Yu, Senior Member, IEEE, Jinyu Li, Member, IEEE, Li Deng, Fellow, IEEE

More information

Memory-based grammatical error correction

Memory-based grammatical error correction Memory-based grammatical error correction Antal van den Bosch Peter Berck Radboud University Nijmegen Tilburg University P.O. Box 9103 P.O. Box 90153 NL-6500 HD Nijmegen, The Netherlands NL-5000 LE Tilburg,

More information

Computerized Adaptive Psychological Testing A Personalisation Perspective

Computerized Adaptive Psychological Testing A Personalisation Perspective Psychology and the internet: An European Perspective Computerized Adaptive Psychological Testing A Personalisation Perspective Mykola Pechenizkiy mpechen@cc.jyu.fi Introduction Mixed Model of IRT and ES

More information

Comparison of EM and Two-Step Cluster Method for Mixed Data: An Application

Comparison of EM and Two-Step Cluster Method for Mixed Data: An Application International Journal of Medical Science and Clinical Inventions 4(3): 2768-2773, 2017 DOI:10.18535/ijmsci/ v4i3.8 ICV 2015: 52.82 e-issn: 2348-991X, p-issn: 2454-9576 2017, IJMSCI Research Article Comparison

More information

Teachers response to unexplained answers

Teachers response to unexplained answers Teachers response to unexplained answers Ove Gunnar Drageset To cite this version: Ove Gunnar Drageset. Teachers response to unexplained answers. Konrad Krainer; Naďa Vondrová. CERME 9 - Ninth Congress

More information

Agenda Montreal, Quebec October 17 19

Agenda Montreal, Quebec October 17 19 Wednesday, October 17 6:30 8:00 a.m. 8:00 9:45 a.m. 9:45 10:00 a.m. Break Agenda Montreal, Quebec October 17 19 Registration 10:00 11:30 a.m. Breakouts Continental breakfast 11:30 a.m. 1:00 p.m. Lunch

More information

arxiv: v1 [cs.lg] 15 Jun 2015

arxiv: v1 [cs.lg] 15 Jun 2015 Dual Memory Architectures for Fast Deep Learning of Stream Data via an Online-Incremental-Transfer Strategy arxiv:1506.04477v1 [cs.lg] 15 Jun 2015 Sang-Woo Lee Min-Oh Heo School of Computer Science and

More information

An Empirical and Computational Test of Linguistic Relativity

An Empirical and Computational Test of Linguistic Relativity An Empirical and Computational Test of Linguistic Relativity Kathleen M. Eberhard* (eberhard.1@nd.edu) Matthias Scheutz** (mscheutz@cse.nd.edu) Michael Heilman** (mheilman@nd.edu) *Department of Psychology,

More information

Using computational modeling in language acquisition research

Using computational modeling in language acquisition research Chapter 8 Using computational modeling in language acquisition research Lisa Pearl 1. Introduction Language acquisition research is often concerned with questions of what, when, and how what children know,

More information

Word Segmentation of Off-line Handwritten Documents

Word Segmentation of Off-line Handwritten Documents Word Segmentation of Off-line Handwritten Documents Chen Huang and Sargur N. Srihari {chuang5, srihari}@cedar.buffalo.edu Center of Excellence for Document Analysis and Recognition (CEDAR), Department

More information

A Version Space Approach to Learning Context-free Grammars

A Version Space Approach to Learning Context-free Grammars Machine Learning 2: 39~74, 1987 1987 Kluwer Academic Publishers, Boston - Manufactured in The Netherlands A Version Space Approach to Learning Context-free Grammars KURT VANLEHN (VANLEHN@A.PSY.CMU.EDU)

More information

Large-Scale Web Page Classification. Sathi T Marath. Submitted in partial fulfilment of the requirements. for the degree of Doctor of Philosophy

Large-Scale Web Page Classification. Sathi T Marath. Submitted in partial fulfilment of the requirements. for the degree of Doctor of Philosophy Large-Scale Web Page Classification by Sathi T Marath Submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy at Dalhousie University Halifax, Nova Scotia November 2010

More information

STAT 220 Midterm Exam, Friday, Feb. 24

STAT 220 Midterm Exam, Friday, Feb. 24 STAT 220 Midterm Exam, Friday, Feb. 24 Name Please show all of your work on the exam itself. If you need more space, use the back of the page. Remember that partial credit will be awarded when appropriate.

More information

Human Emotion Recognition From Speech

Human Emotion Recognition From Speech RESEARCH ARTICLE OPEN ACCESS Human Emotion Recognition From Speech Miss. Aparna P. Wanare*, Prof. Shankar N. Dandare *(Department of Electronics & Telecommunication Engineering, Sant Gadge Baba Amravati

More information

B. How to write a research paper

B. How to write a research paper From: Nikolaus Correll. "Introduction to Autonomous Robots", ISBN 1493773070, CC-ND 3.0 B. How to write a research paper The final deliverable of a robotics class often is a write-up on a research project,

More information

Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling

Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling Notebook for PAN at CLEF 2013 Andrés Alfonso Caurcel Díaz 1 and José María Gómez Hidalgo 2 1 Universidad

More information

Cooperative evolutive concept learning: an empirical study

Cooperative evolutive concept learning: an empirical study Cooperative evolutive concept learning: an empirical study Filippo Neri University of Piemonte Orientale Dipartimento di Scienze e Tecnologie Avanzate Piazza Ambrosoli 5, 15100 Alessandria AL, Italy Abstract

More information