CS 510: Lecture 8. Deep Learning, Fairness, and Bias


 Arthur Stevens
 8 months ago
 Views:
Transcription
1 CS 510: Lecture 8 Deep Learning, Fairness, and Bias
2 Next Week All Presentations, all the time Upload your presentation before class if using slides Sign up for a timeslot google doc, if you haven t already done so
3 Artificial Neural Networks: History Belief that it was necessary to model underlying brain architecture for AI In contrast to encoded symbolic knowledge (best represented by expert systems) Hebb  learning is altering strength of synaptic connections
4 Neural Networks Attempt to build a computation system based on the parallel architecture of brains Characteristics: Many simple processing elements Many connections Simple messages Adaptive interaction
5 Brains neurons of > 20 types, synapses, 1ms 10ms cycle time Signals are noisy spike trains of electrical potential Axonal arborization Synapse Axon from another cell Dendrite Axon Nucleus Synapses Cell body or Soma Chapter 20, Section 5 3
6 Benefits of NN User friendly (well, reasonably) Nonlinear Noise tolerant Many applications Credit fraud/assignment Robotic Control
7 Neurons Inputs (either from outside or other neurons ) Weighted connections that correspond to synaptic efficiency Threshold values to weight the inputs Passed through activation function to determine output
8 Example Unit Binary input/output Rule 1 if w0*i0 + w1*i1 +wb > 0 0 if w0*i0 + w1*i1 +wb <= 0
9 Activation functions g(in i ) g(in i ) (a) in i (b) in i (a) is a step function or threshold function (b) is a sigmoid function 1/(1 + e x ) Changing the bias weight W 0,i moves the threshold location Note similarity to logistic regression... Chapter 20, Section 5 5
10 0.06 W12.5 W2 f(x) W3 1.4
11 f(x) x = =
12 How to Adapt? Perceptron Learning Rule change the weight by an amount proportional to the difference between the desired output and the actual output. As an equation: ΔWi = η * (D  Y)Ii, where D is desired output and Y is actual output Stop when converges
13 Limits of Perceptrons Minsky and Papert 1969 Fails on linearly inseparable instances XOR linearly separable  pattern space can be separated by single hyperplane
14 Perceptrons vs Decision Trees
15 Multilayer Perceptrons (MLP)
16 Back Propagation Start with a set of known examples (supervised approach) Assign random initial weights Run examples through and calculate the meansquared error Propagate the error by making small changes to the weights at each level Use chain rule to calculate the gradient efficiently Lather, rinse, repeat
17 Gradient Descent Algorithm Have some function Want Outline: Start with some Keep changing to reduce until we hopefully end up at a minimum
18 The gradient of J ( J) at a point can be thought of as a vector indicating which way is uphill J(θ 0,θ 1 ) If J is an error function we want to move downhill  opposite to the gradient
19 Gradient descent algorithm Have function J Want to produce vectors s.t. J(θ1)>J(θ2)>... start w/ θ0 θi+1 = θi  ɑi J(θi) ɑ(alpha) is the learning rate
20 Stochastic Gradient Descent Update J every time you look at a training example
21 Some nonlinear activation functions
22 Most common activation function
23 A dataset Fields class etc
24 Training the neural network Fields class etc
25 Training data Fields class etc Initialise with random weights
26 Training data Fields class etc Present a training pattern
27 Training data Fields class etc 1.4 Feed it through to get output
28 Training data Fields class etc 1.4 Compare with target output error 0.8
29 Training data Fields class etc 1.4 Adjust weights based on error error 0.8
30 Training data Fields class etc Present a training pattern
31 Training data Fields class etc 6.4 Feed it through to get output
32 Training data Fields class etc 6.4 Compare with target output error 0.1
33 Training data Fields class etc 6.4 Adjust weights based on error error 0.1
34 Training data Fields class etc 6.4 And so on error 0.1 Repeat this thousands, maybe millions of times each time taking a random training instance, and making slight weight adjustments Algorithms for weight adjustment are designed to make changes that will reduce the error
35 The decision boundary perspective Initial random weights
36 The decision boundary perspective Present a training instance / adjust the weights
37 The decision boundary perspective Present a training instance / adjust the weights
38 The decision boundary perspective Present a training instance / adjust the weights
39 The decision boundary perspective Present a training instance / adjust the weights
40 The decision boundary perspective Eventually.
41 The point I am trying to make weightlearning algorithms for NNs are dumb they work by making thousands and thousands of tiny adjustments, each making the network do better at the most recent pattern, but perhaps a little worse on many others but, by dumb luck, eventually this tends to be good enough to learn effective classifiers for many real applications
42 Some other points Detail of a standard NN weight learning algorithm later If f(x) is nonlinear, a network with 1 hidden layer can, in theory, learn perfectly any classification problem. A set of weights exists that can produce the targets from the inputs. The problem is finding them.
43 Some other by the way points If f(x) is linear, the NN can only draw straight decision boundaries (even if there are many layers of units)
44 Some other by the way points NNs use nonlinear f(x) so they can draw complex boundaries, but keep the data unchanged
45 Some other by the way points NNs use nonlinear f(x) so they can draw complex boundaries, but keep the data unchanged SVMs only draw straight lines, but they transform the data first in a way that makes that OK
46 Neural network vocabulary Neuron = logistic regression or similar function Input layer = input training/test vector Bias unit = intercept term/always on feature Activation = response Activation function is a logistic (or similar sigmoid nonlinearity) Backpropagation = running stochastic gradient descent across a multilayer network Weight decay  regularization or Bayesian prior
47 Deep Learning Most current machine learning works well because of humandesigned representations and input features Machine learning becomes just optimizing weights to best make a final prediction Representation learning attempts to automatically learn good features or representations Deep learning algorithms attempt to learn multiple levels of representation of increasing complexity/abstraction
48 Deep Architecture
49 Deep Learning Overview Train networks with many layers (vs. shallow nets with just a couple of layers) Multiple layers work to build an improved feature space First layer learns 1st order features (e.g. edges ) 2nd layer learns higher order features (combinations of first layer features, combinations of edges, etc.) In current models layers often learn in an unsupervised mode and discover general features of the input space serving multiple tasks related to the unsupervised instances (image recognition, etc.) Then final layer features are fed into supervised layer(s) And entire network is often subsequently tuned using supervised training of the entire net, using the initial weightings learned in the unsupervised phase Could also do fully supervised versions, etc. (early BP attempts)
50 Why Deep Learning?
51 Learning Representations Handcrafting features is timeconsuming incomplete domain/... The features are often both overspecified and The work has to be done again for each task/ We must move beyond handcrafted features and simple ML Humans develop representations for learning and reasoning Our computers should do the same
52 The Curse of Dimensionality
53 Unsupervised Feature and Weight Learning Today, most practical, good NLP& ML methods require labeled training data (i.e., supervised learning) But almost all data is unlabeled Most information must be acquired unsupervised Fortunately, a good model of observed data can really help you learn classification decisions
54 Learning Multiple Levels of Representation
55 Successive Layers Learn Deeper Representations object models object parts (combination of edges) edges pixels
56 Impressive Results Especially on Large Datasets Object Recognition  better than anything out there Speech Recognition (google voice search) Many other perceptual tasks in vision and NLP
57 Why now? Bigger Data  deep learning works best Better Hardware  multicore CPUs and GPUs Better Algorithms  autoencoders, deep belief networks, etc Let us train multiple inner layers well
58 Breakthrough: Unsupervised Pretraining
59 Difficulties with Supervised Networks Early layers of MLP do not get trained well Diffusion of Gradient error attenuates as it propagates to earlier layers Leads to very slow training Exacerbated since top couple layers can usually learn any task "pretty well" and thus the error to earlier layers drops quickly as the top layers "mostly" solve the task lower layers never get the opportunity to use their capacity to improve results, they just do a random feature map Need a way for early layers to do effective work Often not enough labeled data available while lots of unlabeled data Can we use unsupervised/semisupervised approaches to take advantage of the unlabeled data Deep networks tend to have more local minima problems than shallow networks during supervised training
60 Semisupervised Learning
61 Semisupervised Learning
62 Training Deep Networks Build a feature space Note that this is what we do with SVM kernels, or trained hidden layers in BP, etc., but now we will build the feature space using deep architectures Unsupervised training between layers can decompose the problem into distributed subproblems (with higher levels of abstraction) to be further decomposed at subsequent layers
63 Greedy Layerwise Training Train first layer using your data without the labels (unsupervised) Since there are no targets at this level, labels don't help. Could also use the more abundant unlabeled data which is not part of the training set (i.e. selftaught learning). Freeze the first layer parameters and start training the second layer using the output of the first layer as the unsupervised input to the second layer Repeat this for as many layers as desired This builds our set of robust features Use the outputs of the final layer as inputs to a supervised layer/model and train the last supervised layer(s) (leave early weights frozen) Unfreeze all weights and fine tune the full network by training with a supervised approach, given the preprocessed weight settings
64 Greedy Layerwise Training Greedy layerwise training avoids many of the problems of trying to train a deep net in a supervised fashion Each layer gets full learning focus in its turn since it is the only current "top" layer Can take advantage of the unlabeled data When you finally tune the entire network with supervised training the network weights have already been adjusted so that you are in a good error basin and just need fine tuning. This helps with problems of Ineffective early layer learning Deep network local minima We will discuss the two most common approaches Stacked AutoEncoders Deep Belief Networks
65 The new way to train multilayer NNs Train this layer first
66 The new way to train multilayer NNs Train this layer first then this layer
67 The new way to train multilayer NNs Train this layer first then this layer then this layer
68 The new way to train multilayer NNs Train this layer first then this layer then this layer then this layer
69 The new way to train multilayer NNs Train this layer first then this layer then this layer then this layer finally this layer
70 The new way to train multilayer NNs EACH of the (nonoutput) layers is trained to be an autoencoder Basically, it is forced to learn good features that describe what comes from the previous layer
71 an autoencoder is trained, with an absolutely standard weightadjustment algorithm to reproduce the input
72 an autoencoder is trained, with an absolutely standard weightadjustment algorithm to reproduce the input By making this happen with (many) fewer units than the inputs, this forces the hidden layer units to become good feature detectors
73 One Autoencoder 73
74 Stacked Autoencoders Stack sparse autoencoders on top of each other, drop decode layer each time 74
75 Stacked autoencoders Do supervised training on last layer Then do supervised training on whole network to fine tune the weights 75
76 Manifold Learning Hypothesis 76
77 Caveats Prevent the layers from just learning the identity (learn Features instead) Undercomplete  middle layer smaller than input Sparsity  penalize hidden unit activations Use regularization to keep most nodes at or near 0 Denoising  Stochastically corrupt training instance, but train encoder to decode uncorrupted instance Contractive  force encoder to have small derivatives (stay on manifold) 77
78 Fairness and Learning Going to show video of Aylin Link on course website to her talk (may be easier for online students if there is feedback). 78
Deep Learning. Early Work Why Deep Learning Stacked Auto Encoders Deep Belief Networks. l l l l. CS 678 Deep Learning 1
Deep Learning Early Work Why Deep Learning Stacked Auto Encoders Deep Belief Networks CS 678 Deep Learning 1 Deep Learning Overview Train networks with many layers (vs. shallow nets with just a couple
More informationIntroduction to Deep Learning
Introduction to Deep Learning M S Ram Dept. of Computer Science & Engg. Indian Institute of Technology Kanpur Reading of Chap. 1 from Learning Deep Architectures for AI ; Yoshua Bengio; FTML Vol. 2, No.
More informationClassification with Deep Belief Networks. HussamHebbo Jae Won Kim
Classification with Deep Belief Networks HussamHebbo Jae Won Kim Table of Contents Introduction... 3 Neural Networks... 3 Perceptron... 3 Backpropagation... 4 Deep Belief Networks (RBM, Sigmoid Belief
More informationArtificial Neural Networks. Andreas Robinson 12/19/2012
Artificial Neural Networks Andreas Robinson 12/19/2012 Introduction Artificial Neural Networks Machine learning technique Learning from past experience/data Predicting/classifying novel data Biologically
More informationIntroduction of connectionist models
Introduction of connectionist models Introduction to ANNs Markus Dambek Uni Bremen 20. Dezember 2010 Markus Dambek (Uni Bremen) Introduction of connectionist models 20. Dezember 2010 1 / 66 1 Introduction
More informationCS 2750: Machine Learning. Neural Networks. Prof. Adriana Kovashka University of Pittsburgh February 28, 2017
CS 2750: Machine Learning Neural Networks Prof. Adriana Kovashka University of Pittsburgh February 28, 2017 HW2 due Thursday Announcements Office hours on Thursday: 4:15pm5:45pm Talk at 3pm: http://www.sam.pitt.edu/arc
More informationDEEP LEARNING AND ITS APPLICATION NEURAL NETWORK BASICS
DEEP LEARNING AND ITS APPLICATION NEURAL NETWORK BASICS Argument on AI 1. Symbolism 2. Connectionism 3. Actionism Kai Yu. SJTU Deep Learning Lecture. 2 Argument on AI 1. Symbolism Symbolism AI Origin Cognitive
More informationLecture 5: 21 September 2016 Intro to machine learning and singlelayer neural networks. Jim Tørresen This Lecture
This Lecture INF3490  Biologically inspired computing Lecture 5: 21 September 2016 Intro to machine learning and singlelayer neural networks Jim Tørresen 1. Introduction to learning/classification 2.
More informationArtificial Neural Networks
Artificial Neural Networks Outline Introduction to Neural Network Introduction to Artificial Neural Network Properties of Artificial Neural Network Applications of Artificial Neural Network Demo Neural
More informationIntelligent Systems. Neural Networks. Copyright 2009 Dieter Fensel and Reto Krummenacher
Intelligent Systems Neural Networks Copyright 2009 Dieter Fensel and Reto Krummenacher 1 Where are we? # Title 1 Introduction 2 Propositional Logic 3 Predicate Logic 4 Theorem Proving, Description Logics
More informationMachine Learning and Artificial Neural Networks (Ref: Negnevitsky, M. Artificial Intelligence, Chapter 6)
Machine Learning and Artificial Neural Networks (Ref: Negnevitsky, M. Artificial Intelligence, Chapter 6) The Concept of Learning Learning is the ability to adapt to new surroundings and solve new problems.
More informationEvolution of Neural Networks. October 20, 2017
Evolution of Neural Networks October 20, 2017 Single Layer Perceptron, (1957) Frank Rosenblatt 1957 1957 Single Layer Perceptron Perceptron, invented in 1957 at the Cornell Aeronautical Laboratory by Frank
More informationLecture 6: Course Project Introduction and Deep Learning Preliminaries
CS 224S / LINGUIST 285 Spoken Language Processing Andrew Maas Stanford University Spring 2017 Lecture 6: Course Project Introduction and Deep Learning Preliminaries Outline for Today Course projects What
More informationPython Machine Learning
Python Machine Learning Unlock deeper insights into machine learning with this vital guide to cuttingedge predictive analytics Sebastian Raschka [ PUBLISHING 1 open source I community experience distilled
More informationDeep Neural Networks for Acoustic Modelling. Bajibabu Bollepalli Hieu Nguyen Rakshith Shetty Pieter Smit (Mentor)
Deep Neural Networks for Acoustic Modelling Bajibabu Bollepalli Hieu Nguyen Rakshith Shetty Pieter Smit (Mentor) Introduction Automatic speech recognition Speech signal Feature Extraction Acoustic Modelling
More informationIndepth: Deep learning (one lecture) Applied to both SL and RL above Code examples
Introduction to machine learning (two lectures) Supervised learning Reinforcement learning (lab) Indepth: Deep learning (one lecture) Applied to both SL and RL above Code examples 20170930 2 1 To enable
More informationDeep learning for music genre classification
Deep learning for music genre classification Tao Feng University of Illinois taofeng1@illinois.edu Abstract In this paper we will present how to use Restricted Boltzmann machine algorithm to build deep
More informationCPSC 340: Machine Learning and Data Mining. Course Review/Preview Fall 2015
CPSC 340: Machine Learning and Data Mining Course Review/Preview Fall 2015 Admin Assignment 6 due now. We will have office hours as usual next week. Final exam details: December 15: 8:3011 (WESB 100).
More informationLarge Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Introduction to Deep Learning U Kang Seoul National University U Kang 1 In This Lecture Overview of deep learning History of deep learning and its recent advances
More informationArticle from. Predictive Analytics and Futurism December 2015 Issue 12
Article from Predictive Analytics and Futurism December 2015 Issue 12 The Third Generation of Neural Networks By Jeff Heaton Neural networks are the phoenix of artificial intelligence. Right now neural
More informationLecture 1: Machine Learning Basics
1/69 Lecture 1: Machine Learning Basics Ali Harakeh University of Waterloo WAVE Lab ali.harakeh@uwaterloo.ca May 1, 2017 2/69 Overview 1 Learning Algorithms 2 Capacity, Overfitting, and Underfitting 3
More informationExploration vs. Exploitation. CS 473: Artificial Intelligence Reinforcement Learning II. How to Explore? Exploration Functions
CS 473: Artificial Intelligence Reinforcement Learning II Exploration vs. Exploitation Dieter Fox / University of Washington [Most slides were taken from Dan Klein and Pieter Abbeel / CS188 Intro to AI
More informationPerspective on HPCenabled AI Tim Barr September 7, 2017
Perspective on HPCenabled AI Tim Barr September 7, 2017 AI is Everywhere 2 Deep Learning Component of AI The punchline: Deep Learning is a High Performance Computing problem Delivers benefits similar
More informationEra of AI (Deep Learning) and harnessing its true potential
Era of AI (Deep Learning) and harnessing its true potential Artificial Intelligence (AI) AI Augments our brain with infallible memories and infallible calculators Humans and Computers have become a tightly
More informationMachine Learning for SAS Programmers
Machine Learning for SAS Programmers The Agenda Introduction of Machine Learning Supervised and Unsupervised Machine Learning Deep Neural Network Machine Learning implementation Questions and Discussion
More informationLinear Models Continued: Perceptron & Logistic Regression
Linear Models Continued: Perceptron & Logistic Regression CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Linear Models for Classification Feature function
More informationThe Generalized Delta Rule and Practical Considerations
The Generalized Delta Rule and Practical Considerations Introduction to Neural Networks : Lecture 6 John A. Bullinaria, 2004 1. Training a Single Layer Feedforward Network 2. Deriving the Generalized
More informationArtifi ifi i c l a Neur l a Networks Mohamed M. El Wakil t akil.ne 1
Artificial i lneural lnetworks Mohamed M. El Wakil mohamed@elwakil.net 1 Agenda Natural Neural Networks Artificial Neural Networks XOR Example Design Issues Applications Conclusion 2 Artificial Neural
More informationA study of the NIPS feature selection challenge
A study of the NIPS feature selection challenge Nicholas Johnson November 29, 2009 Abstract The 2003 Nips Feature extraction challenge was dominated by Bayesian approaches developed by the team of Radford
More informationProgramming Assignment2: Neural Networks
Programming Assignment2: Neural Networks Problem :. In this homework assignment, your task is to implement one of the common machine learning algorithms: Neural Networks. You will train and test a neural
More informationLoad Forecasting with Artificial Intelligence on Big Data
1 Load Forecasting with Artificial Intelligence on Big Data October 9, 2016 Patrick GLAUNER and Radu STATE SnT  Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg 2
More informationCS519: Deep Learning 1. Introduction
CS519: Deep Learning 1. Introduction Winter 2017 Fuxin Li With materials from Pierre Baldi, Geoffrey Hinton, Andrew Ng, Honglak Lee, Aditya Khosla, Joseph Lim 1 Cutting Edge of Machine Learning: Deep Learning
More informationContents  MLPs & Pose/Expression Classification 1. Contents
Contents  MLPs & Pose/Expression Classification 1 Contents Contents... 1 Abstract...3 Acknowledgements... 4 1. Introduction... 5 2. Possible Realworld Applications... 7 3. Facial Expression Analysis...
More informationMachine Learning: Neural Networks. Junbeom Park Radiation Imaging Laboratory, Pusan National University
Machine Learning: Neural Networks Junbeom Park (pjb385@gmail.com) Radiation Imaging Laboratory, Pusan National University 1 Contents 1. Introduction 2. Machine Learning Definition and Types Supervised
More informationDeep (Structured) Learning
Deep (Structured) Learning Yasmine Badr 06/23/2015 NanoCAD Lab UCLA What is Deep Learning? [1] A wide class of machine learning techniques and architectures Using many layers of nonlinear information
More informationDeep Dictionary Learning vs Deep Belief Network vs Stacked Autoencoder: An Empirical Analysis
Target Target Deep Dictionary Learning vs Deep Belief Network vs Stacked Autoencoder: An Empirical Analysis Vanika Singhal, Anupriya Gogna and Angshul Majumdar Indraprastha Institute of Information Technology,
More informationDeep Learning for AI Yoshua Bengio. August 28th, DS3 Data Science Summer School
Deep Learning for AI Yoshua Bengio August 28th, 2017 @ DS3 Data Science Summer School A new revolution seems to be in the work after the industrial revolution. And Machine Learning, especially Deep Learning,
More informationA Review on Classification Techniques in Machine Learning
A Review on Classification Techniques in Machine Learning R. Vijaya Kumar Reddy 1, Dr. U. Ravi Babu 2 1 Research Scholar, Dept. of. CSE, Acharya Nagarjuna University, Guntur, (India) 2 Principal, DRK College
More informationarxiv: v3 [cs.lg] 9 Mar 2014
Learning Factored Representations in a Deep Mixture of Experts arxiv:1312.4314v3 [cs.lg] 9 Mar 2014 David Eigen 1,2 Marc Aurelio Ranzato 1 Ilya Sutskever 1 1 Google, Inc. 2 Dept. of Computer Science, Courant
More informationIntroduction: Convolutional Neural Networks for Visual Recognition.
Introduction: Convolutional Neural Networks for Visual Recognition boris.ginzburg@intel.com 1 Acknowledgments This presentation is heavily based on: http://cs.nyu.edu/~fergus/pmwiki/pmwiki.php http://deeplearning.net/readinglist/tutorials/
More information545 Machine Learning, Fall 2011
545 Machine Learning, Fall 2011 Final Project Report Experiments in Automatic Text Summarization Using Deep Neural Networks Project Team: Ben King Rahul Jha Tyler Johnson Vaishnavi Sundararajan Instructor:
More informationIntroduction to the Theories of Machine Learning
Introduction to the Theories of Machine Learning with FeedForward Artificial Neural Networks and Evolving with Genetic Algorithms Second Research Paper Bachelor course on Media Technology at St. Pölten
More information62 Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall
Business Intelligence and Decision Support Systems (9 th Ed., Prentice Hall) Chapter 6: Artificial Neural Networks for Data Mining Learning Objectives Understand the concept and definitions of artificial
More informationDeep Learning in Customer Churn Prediction: Unsupervised Feature Learning on Abstract Company Independent Feature Vectors
1 Deep Learning in Customer Churn Prediction: Unsupervised Feature Learning on Abstract Company Independent Feature Vectors Philip Spanoudes, Thomson Nguyen Framed Data Inc, New York University, and the
More informationProgramming Social Robots for Human Interaction. Lecture 4: Machine Learning and Pattern Recognition
Programming Social Robots for Human Interaction Lecture 4: Machine Learning and Pattern Recognition ZhengHua Tan Dept. of Electronic Systems, Aalborg Univ., Denmark zt@es.aau.dk, http://kom.aau.dk/~zt
More informationComputer Vision for Card Games
Computer Vision for Card Games Matias Castillo matiasct@stanford.edu Benjamin Goeing bgoeing@stanford.edu Jesper Westell jesperw@stanford.edu Abstract For this project, we designed a computer vision program
More informationCSC321 Lecture 1: Introduction
CSC321 Lecture 1: Introduction Roger Grosse Roger Grosse CSC321 Lecture 1: Introduction 1 / 26 What is machine learning? For many problems, it s difficult to program the correct behavior by hand recognizing
More informationDay 2 Lecture 5. Transfer learning and domain adaptation
Day 2 Lecture 5 Transfer learning and domain adaptation Semisupervised and transfer learning Myth: you can t do deep learning unless you have a million labelled examples for your problem. Reality You
More informationReinforcement Learning with Deep Architectures
000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050
More informationEvolving Artificial Neural Networks
Evolving Artificial Neural Networks Christof Teuscher Swiss Federal Institute of Technology Lausanne (EPFL) Logic Systems Laboratory (LSL) http://lslwww.epfl.ch christof@teuscher.ch http://www.teuscher.ch/christof
More information(Sub)Gradient Descent
(Sub)Gradient Descent CMSC 422 MARINE CARPUAT marine@cs.umd.edu Figures credit: Piyush Rai Logistics Midterm is on Thursday 3/24 during class time closed book/internet/etc, one page of notes. will include
More informationNeural Networks and Learning Machines
Neural Networks and Learning Machines Third Edition Simon Haykin McMaster University Hamilton, Ontario, Canada Upper Saddle River Boston Columbus San Francisco New York Indianapolis London Toronto Sydney
More informationDeep MultiTask Learning with evolving weights
Deep MultiTask Learning with evolving weights Soufiane Belharbi1, Romain He rault1, Cle ment Chatelain1 and Se bastien Adam2 1 INSA de Rouen  LITIS EA 4108 Saint E tienne du Rouvray 76800  France 2
More informationCourse Outline. Course Grading. Where to go for help. Academic Integrity. EE589 Introduction to Neural Networks NN 1 EE
EE589 Introduction to Neural Assistant Prof. Dr. Turgay IBRIKCI Room # 305 (322) 338 6868 / 139 Wensdays 9:0012:00 Course Outline The course is divided in two parts: theory and practice. 1. Theory covers
More informationInventor Chung T. Nguyen NOTTCE. The above identified patent application is available for licensing. Requests for information should be addressed to:
Serial No. 802.572 Filing Date 3 February 1997 Inventor Chung T. Nguyen NOTTCE The above identified patent application is available for licensing. Requests for information should be addressed to: OFFICE
More informationIntroducing Deep Learning with MATLAB
Introducing Deep Learning with MATLAB What is Deep Learning? Deep learning is a type of machine learning in which a model learns to perform classification tasks directly from images, text, or sound. Deep
More informationCOMP150 DR Final Project Proposal
COMP150 DR Final Project Proposal Ari Brown and Julie Jiang October 26, 2017 Abstract The problem of sound classification has been studied in depth and has multiple applications related to identity discrimination,
More informationDNN Low Level Reinitialization: A Method for Enhancing Learning in Deep Neural Networks through Knowledge Transfer
DNN Low Level Reinitialization: A Method for Enhancing Learning in Deep Neural Networks through Knowledge Transfer Lyndon White (20361362) Index Terms Deep Belief Networks, Deep Neural Networks, Neural
More informationSession 4: Regularization (Chapter 7)
Session 4: Regularization (Chapter 7) Tapani Raiko Aalto University 30 September 2015 Tapani Raiko (Aalto University) Session 4: Regularization (Chapter 7) 30 September 2015 1 / 27 Table of Contents Background
More informationDeep Learning and its application to CV and NLP. Fei Yan University of Surrey June 29, 2016 Edinburgh
Deep Learning and its application to CV and NLP Fei Yan University of Surrey June 29, 2016 Edinburgh Overview Machine learning Motivation: why go deep Feedforward networks: CNN Recurrent networks: LSTM
More informationMachine Learning : Hinge Loss
Machine Learning Hinge Loss 16/01/2014 Machine Learning : Hinge Loss Recap tasks considered before Let a training dataset be given with (i) data and (ii) classes The goal is to find a hyper plane that
More informationIntroduction to Machine Learning
Introduction to Machine Learning Hamed Pirsiavash CMSC 678 http://www.csee.umbc.edu/~hpirsiav/courses/ml_fall17 The slides are closely adapted from Subhransu Maji s slides Course background What is the
More informationArtificial Neural Networks written examination
1 (8) Institutionen för informationsteknologi Olle Gällmo Universitetsadjunkt Adress: Lägerhyddsvägen 2 Box 337 751 05 Uppsala Artificial Neural Networks written examination Monday, May 15, 2006 9 0014
More informationSynaptic Weight Noise During MLP Learning Enhances FaultTolerance, Generalisation and Learning Trajectory
Synaptic Weight Noise During MLP Learning Enhances FaultTolerance, Generalisation and Learning Trajectory Alan F. Murray Dept. of Electrical Engineering Edinburgh University Scotland Peter J. Edwards
More informationA Shallow Introduction to Deep Learning by Rafael Espericueta
Traditional AI vs Deep Learning A Shallow Introduction to Deep Learning by Rafael Espericueta Deep learning is one form of machine learning which is part of the field of artificial intelligence. Basically
More informationLearning General Features From Images and Audio With Stacked Denoising Autoencoders
Portland State University PDXScholar Dissertations and Theses Dissertations and Theses Fall 1232014 Learning General Features From Images and Audio With Stacked Denoising Autoencoders Nathaniel H. Nifong
More informationDocument Classification using Neural Networks Based on Words
Volume 6, No. 2, MarchApril 2015 International Journal of Advanced Research in Computer Science RESEARCH PAPER Available Online at www.ijarcs.info Document Classification using Neural Networks Based on
More informationA Distributional Representation Model For Collaborative
A Distributional Representation Model For Collaborative Filtering Zhang Junlin,Cai Heng,Huang Tongwen, Xue Huiping Chanjet.com {zhangjlh,caiheng,huangtw,xuehp}@chanjet.com Abstract In this paper, we propose
More informationKnowledge Matters: Importance of Prior Information for Optimization
Journal of Machine Learning Research 17 (2016) 132 Submitted 7/13; Revised 10/14; Published 3/16 Knowledge Matters: Importance of Prior Information for Optimization Çağlar Gülçehre gulcehrc@iro.umontreal.ca
More informationStudies in Deep Belief Networks
Studies in Deep Belief Networks Jiquan Ngiam jngiam@cs.stanford.edu Chris Baldassano chrisb33@cs.stanford.edu Abstract Deep networks are able to learn good representations of unlabelled data via a greedy
More informationDeep Learning: An Overview. Bradley J Erickson, MD PhD Mayo Clinic, Rochester
Deep Learning: An Overview Bradley J Erickson, MD PhD Mayo Clinic, Rochester Medical Imaging Informatics and Teleradiology Conference 1:302:05pm June 17, 2016 Disclosures Relationships with commercial
More informationReverse Dictionary Using Artificial Neural Networks
International Journal of Research Studies in Science, Engineering and Technology Volume 2, Issue 6, June 2015, PP 1423 ISSN 23494751 (Print) & ISSN 2349476X (Online) Reverse Dictionary Using Artificial
More informationConnectionism (Artificial Neural Networks) and Dynamical Systems
COMP 40260 Connectionism (Artificial Neural Networks) and Dynamical Systems Part 2 Read Rethinking Innateness, Chapters 1 & 2 Let s start with an old neural network, created before training from data was
More informationArtificial Neural Networks
Artificial Neural Networks Andres Chavez Math 382/L T/Th 2:003:40 April 13, 2010 Chavez2 Abstract The main interest of this paper is Artificial Neural Networks (ANNs). A brief history of the development
More informationArtificial Neural Networks for Storm Surge Predictions in NC. DHS Summer Research Team
Artificial Neural Networks for Storm Surge Predictions in NC DHS Summer Research Team 1 Outline Introduction; Feedforward Artificial Neural Network; Design questions; Implementation; Improvements; Conclusions;
More informationDeep Learning in Music Informatics
Deep Learning in Music Informatics Demystifying the Dark Art, Part III Practicum Eric J. Humphrey 04 November 2013 Outline In this part of the talk, we ll touch on the following: Recap: What is deep learning
More informationPattern Classification and Clustering Spring 2006
Pattern Classification and Clustering Time: Spring 2006 Room: Instructor: Yingen Xiong Office: 621 McBryde Office Hours: Phone: 2314212 Email: yxiong@cs.vt.edu URL: http://www.cs.vt.edu/~yxiong/pcc/ Detailed
More informationUNIVERSITY OF OSLO. Faculty of Mathematics and Natural Sciences
Page 1 of 7 UNIVERSITY OF OSLO Faculty of Mathematics and Natural Sciences Exam in INF3490/4490 iologically Inspired omputing ay of exam: ecember 9th, 2015 Exam hours: 09:00 13:00 This examination paper
More informationLinear Regression. Chapter Introduction
Chapter 9 Linear Regression 9.1 Introduction In this class, we have looked at a variety of di erent models and learning methods, such as finite state machines, sequence models, and classification methods.
More informationLearning facial expressions from an image
Learning facial expressions from an image Bhrugurajsinh Chudasama, Chinmay Duvedi, Jithin Parayil Thomas {bhrugu, cduvedi, jithinpt}@stanford.edu 1. Introduction Facial behavior is one of the most important
More informationNeural Networks. CSC 4504 : Langages formels et applications. J Paul Gibson, D311.
CSC 4504 : Langages formels et applications J Paul Gibson, D311 paul.gibson@telecomsudparis.eu /~gibson/teaching/csc4504/problem11neuralnetworks.pdf Neural Networks 1 2 The following slides are a summary
More informationCS519: Deep Learning. Winter Fuxin Li
CS519: Deep Learning Winter 2017 Fuxin Li Course Information Instructor: Dr. Fuxin Li KEC 2077, lif@eecs.oregonstate.edu TA: Mingbo Ma: mam@oregonstate.edu Xu Xu: xux@oregonstate.edu My office hour: TBD
More informationMaturaarbeit Oktober A neural network learns to play Mortal Kombat 3
Maturaarbeit Oktober 2016 A neural network learns to play Mortal Kombat 3 Author, class: Carlo Hartmann, M4a Supervising teacher: Andreas Umbach Contents 1 Abstract 1 2 Foreword 2 2.1 Motivation...................................
More informationIntroduction to Computational Neuroscience A. The Brain as an Information Processing Device
Introduction to Computational Neuroscience A. The Brain as an Information Processing Device Jackendoff (Consciousness and the Computational Mind, Jackendoff, MIT Press, 1990) argues that we can put off
More informationData Mining. CS57300 Purdue University. Bruno Ribeiro. February 15th, 2018
Data Mining CS573 Purdue University Bruno Ribeiro February 15th, 218 1 Today s Goal Ensemble Methods Supervised Methods Metalearners Unsupervised Methods 215 Bruno Ribeiro Understanding Ensembles The
More informationEVOLVING NEURAL NETWORKS WITH HYPERNEAT AND ONLINE TRAINING. Shaun M. Lusk, B.S.
EVOLVING NEURAL NETWORKS WITH HYPERNEAT AND ONLINE TRAINING by Shaun M. Lusk, B.S. A thesis submitted to the Graduate Council of Texas State University in partial fulfillment of the requirements for the
More informationDeep Learning of Representations for Unsupervised and Transfer Learning
JMLR: Workshop and Conference Proceedings 7 (2011) 1 20 Workshop on Unsupervised and Transfer Learning Deep Learning of Representations for Unsupervised and Transfer Learning Yoshua Bengio yoshua.bengio@umontreal.ca
More informationSOFTCOMPUTING IN MODELING & SIMULATION
SOFTCOMPUTING IN MODELING & SIMULATION 9th July, 2002 Faculty of Science, Philadelphia University Dr. Kasim M. AlAubidy Computer & Software Eng. Dept. Philadelphia University The only way not to succeed
More informationThe Health Economics and Outcomes Research Applications and Valuation of Digital Health Technologies and Machine Learning
The Health Economics and Outcomes Research Applications and Valuation of Digital Health Technologies and Machine Learning Workshop W29  Session V 3:00 4:00pm May 25, 2016 ISPOR 21 st Annual International
More informationARTIFICIAL NEURAL NETWORKS
ARTIFICIAL NEURAL NETWORKS Amrender Kumar Indian Agricultural Statistics Research Institute, New Delhi11012 akjha@iasri.res.in 1. Introduction Neural networks, more accurately called Artificial Neural
More informationDudon Wai Georgia Institute of Technology CS 7641: Machine Learning Atlanta, GA
Adult Income and Letter Recognition  Supervised Learning Report An objective look at classifier performance for predicting adult income and Letter Recognition Dudon Wai Georgia Institute of Technology
More informationAdvances in Music Information Retrieval using Deep Learning Techniques  Sid Pramod
Advances in Music Information Retrieval using Deep Learning Techniques  Sid Pramod Music Information Retrieval (MIR) Science of retrieving information from music. Includes tasks such as Query by Example,
More informationCS81: Learning words with Deep Belief Networks
CS81: Learning words with Deep Belief Networks George Dahl gdahl@cs.swarthmore.edu Kit La Touche kit@cs.swarthmore.edu Abstract In this project, we use a Deep Belief Network (Hinton et al., 2006) to learn
More informationPsychology 452 Week 1: Connectionism and Association
Psychology 452 Week 1: Connectionism and Association Course Overview Properties Of Connectionism Building Associations Into Networks The Hebb Rule The Delta Rule Michael R.W. Dawson PhD from University
More information4 Feedforward Neural Networks, Binary XOR, Continuous XOR, Parity Problem and Composed Neural Networks.
4 Feedforward Neural Networks, Binary XOR, Continuous XOR, Parity Problem and Composed Neural Networks. 4.1 Objectives The objective of the following exercises is to get acquainted with the inner working
More informationScaling Quality On Quora Using Machine Learning
Scaling Quality On Quora Using Machine Learning Nikhil Garg @nikhilgarg28 @Quora @QconSF 11/7/16 Goals Of The Talk Introducing specific product problems we need to solve to stay highquality Describing
More informationA Few Useful Things to Know about Machine Learning. Pedro Domingos Department of Computer Science and Engineering University of Washington" 2012"
A Few Useful Things to Know about Machine Learning Pedro Domingos Department of Computer Science and Engineering University of Washington 2012 A Few Useful Things to Know about Machine Learning Machine
More informationMachine Learning L, T, P, J, C 2,0,2,4,4
Subject Code: Objective Expected Outcomes Machine Learning L, T, P, J, C 2,0,2,4,4 It introduces theoretical foundations, algorithms, methodologies, and applications of Machine Learning and also provide
More informationA SELFLEARNING NEURAL NETWORK
769 A SELFLEARNING NEURAL NETWORK A. Hartstein and R. H. Koch IBM  Thomas J. Watson Research Center Yorktown Heights, New York ABSTRACf We propose a new neural network structure that is compatible with
More information