Deep Learning Mohammad Ali Keyvanrad Lecture 1:Introduction
OUTLINE Recent success with Deep Learning Deep Learning definition History Course plan Resources Grading Policy 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 2
OUTLINE Recent success with Deep Learning Deep Learning definition History Course plan Resources Grading Policy 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 3
Recent success with Deep Learning Over the last few years Deep Learning was applied to hundreds of problems. Computer vision and pattern recognition Speech recognition and speech synthesis Natural language processing Computer games, robots & self-driving cars In many problems they have established the state of the art Often exceeding previous benchmarks by large margins 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 4
Learning Lip Sync from Audio (University of Washington, 2017) 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 5
Restore colors in B&W photos (Waseda University, 2016) Input [Larsson et al. 2016] [Zhang et al. 2016a] [Zhang et al. 2016b] 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 6
Pixel restoration (Google Brain, 2017) Take very low resolution images and predict what each image most likely looks like. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 7
Describing photos (Stanford University, 2015) Computers can automatically classify our photos Facebook can automatically tag your friends Deep Learning not only learned to classify the elements in the photo, but to actually describe them. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 8
Translation (Google Translate, 2015) Google Translate app now does real-time visual translation of 20 more languages. A photo taken by the phone, and Google Translate "reads" the text and replaces it with a text in English in real-time. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 9
Create new images (University of California, 2017) Deep Learning network to create other types of new images 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 10
Reading text in the Wild (University of Oxford, 2014) An attempt to read text from photos and videos Search for text from BBC News videos 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 11
Teach a computer to play (DeepMind, 2015) Google's DeepMind used a Deep Learning technique to teach a computer to play Control of the keyboard while watching the score, and its goal was to maximize the score 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 12
Beating people in dozens of computer games Computer program playing Doom using only raw pixel data. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 13
Self-driving cars (Tesla, 2016) A Tesla electric vehicle drives without human intervention Notice how it distinguishes different type of objects, including people and road signs. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 14
Robotics (BostonDynamics, 2016) Deep Learning is also heavily used in robotics these days SpotMini and Atlas 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 15
Voice generation (Google WaveNet, 2016) Deep Learning is taking us a step closer to giving computers the ability to speak like humans do. Google released WaveNet and Tacotron Baidu released Deep Speech 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 16
Voice generation (Google Tacotron, 2017) Tacotron learns pronunciations based on phrase semantics. He has read the whole thing. He reads books. Tacotron is sensitive to punctuation. This is your personal assistant, Google Home. Tacotron learns stress and intonation. The buses aren't the problem, they actually provide a solution. The buses aren't the PROBLEM, they actually provide a SOLUTION. Tacotron's prosody changes in a question. The quick brown fox jumps over the lazy dog. Does the quick brown fox jump over the lazy dog? 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 17
Restoring sound in videos (MIT, Berkeley, Google, 2016) Deep Learning network was trained on videos in which people were hitting and scratching objects After several iterations learning, the scientists asked the computer to regenerate the sound 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 18
LIPNET (Oxford, DeepMind, 2016) LipNet reached 93% success in reading people's lips where an average lipreader succeeds 52% of the time. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 19
Automatically writing (Stanford University, 2016) Let a Deep Learning network "read" Shakespeare, Wikipedia, math papers and computer code. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 20
Handwriting (University of Toronto, 2014) Today the computer can also handwrite. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 21
OUTLINE Recent success with Deep Learning Deep Learning definition History Course plan Resources Grading Policy 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 22
Deep Learning definition Deep learning is a class of machine learning algorithms that: Definition 1: They use a cascade of many layers of nonlinear processing units for feature extraction and transformation Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised. Applications include pattern analysis (unsupervised) and classification (supervised). Definition 2: They are part of the broader machine learning field of learning representations of data. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 23
Deep Learning definition Definition 3: These are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher level features are derived from lower level features to form a hierarchical representation. Definition 4: They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 24
Deep Learning definition Common in definitions Models consisting of multiple layers or stages of nonlinear information processing. The supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 25
Deep Learning definition Deep or Shallow? Credit Assignment Path (CAP) A chain of transformations from input to output. CAPs describe potentially causal connections between input and output. CAP depth Number of hidden layers plus one as the output layer is also parameterized For recurrent neural networks the CAP depth is potentially unlimited. a signal may propagate through a layer more than once. 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 26
Deep Learning definition Deep/shallow? No universally agreed upon threshold of depth divides shallow learning from deep learning Most researchers agree that deep learning has multiple nonlinear layers (CAP > 2). 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 27
OUTLINE Recent success with Deep Learning Deep Learning definition History Course plan Resources Grading Policy 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 28
History 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 29
Evolution of Depth LeNet 1998 AlexNet 2012 GoogLeNet 2014 ResNet 2016 7 Layers 8 Layers 22 Layers 152 Layers 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 30
Evolution of Depth 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 31
OUTLINE Recent success with Deep Learning Deep Learning definition History Course plan Resources Grading Policy 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 32
Course plan Introduction A Review of Artificial Neural Networks Perceptron Stochastic Gradient Descent Backpropagation Rectified Linear Function Root Mean Square Propagation Dropout L1 and L2 Regularization 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 33
Course plan Deep Belief Network PGM MRF Sampling RBM Auto-Encoder Linear Auto-Encoder Denoising Auto-Encoder Computational Network 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 34
Course plan Selected Applications in Object Recognition and Computer Vision Convolutional Neural Networks Region Based CNN Generative Adversarial Network GoogLeNet and Microsoft ResNet Selected Applications in Language Modeling and Natural Language Processing Word2Vec Recurrent Neural Networks and Language Models Machine translation and advanced recurrent LSTMs and GRUs 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 35
Course plan Selected Applications in Speech and Audio Processing Speech recognition and bi-directional RNN Speech synthesis and WaveNet or Tacotron Deep Reinforcement Learning 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 36
OUTLINE Recent success with Deep Learning Deep Learning definition History Course plan Resources Grading Policy 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 37
Resources Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning (Adaptive Computation and Machine Learning series), MIT Press, 2016. Dong Yu, Li Deng, Automatic Speech Recognition: A Deep Learning Approach, Springer, 2015 L. Deng and D. Yu, Deep Learning: Methods and Applications, Now Publishers Inc, 2014. Pattern Recognition and Machine Learning, Christopher M. Bishop, 2006. Stanford (CS224n: Natural Language Processing with Deep Learning, 2017) Stanford (CS231n: Convolutional Neural Networks for Visual Recognition, 2017) Related papers 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 38
OUTLINE Recent success with Deep Learning Deep Learning definition History Course plan Resources Grading Policy 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 39
Grading Policy Assignments: 20% Presentation: 15% Final Exam: 35% Final Project: 30% 9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 40
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 41
9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 42