Demystifying Deep Learning Paola.Jaramillo@mathworks.nl Matthias.Sommer@mathworks.ch Application Engineer, MathWorks Benelux Application Engineer, MathWorks Switzerland 2015 The MathWorks, Inc. 1
Agenda What s Deep Learning and why should I care? A practical approach to Deep Learning (for images) Transfer the learning from an expert model to your own application Building a Deep Learning network from scratch Deep Learning for time series and text data Key learnings of the session and cool features 2
Deep learning with MATLAB is easy and accessible! Use Less Data and Less Time with Transfer Learning Automatically generate code! Image Pre-processing CNN Image Post-processing Prototype and productize your entire workflow on embedded GPUs Apply Deep Learning not only on images but also on text and time series data 3
Why Machine Learning or Deep Learning? Transformational technology Close (better) than human accuracy for specific tasks Performance scales with data It is hard to use, it is challenging Enables engineers, researchers and other domain experts to create products and applications with more built-in intelligence 4
Application Breadth Artificial Intelligence, Machine Learning and Deep Learning Artificial Intelligence Machine Learning Deep Learning 1950s 1997 2016 Timeline Today 5
Machine Learning vs Deep Learning Machine Learning Features Training Data Deep Learning Classification Task End-to-end Learning is an iterative process 6
Deep Learning Common Workflow ACCESS AND EXPLORE LABEL AND PREPROCESS DEVELOP PREDICTIVE INTEGRATE MODELS WITH DATA DATA MODELS SYSTEMS Files Data Augmentation/ Transformation Hardware-Accelerated Training Desktop Apps Databases Labeling Automation Hyperparameter Tuning Enterprise Scale Systems Sensors Import Reference Models Network Visualization Embedded Devices and Hardware 7
Deep learning is usually implemented using a neural network architecture The term deep refers to the number of layers in the network the more layers, the deeper the network. Data flows through network in layers, which provide transformation of data Convolutional Neural Network 8
Convolutional Neural Network is a popular Deep Learning architecture Shallow Neural Network Convolutional Neural Network Input Layer Hidden Layer Output Layer Every input neuron Connects to every neuron in the hidden layer Local receptive fields connect to neurons in the hidden layer Translate across an image create a feature map efficiently with convolution Input a b c d e f g h i j k l Output aw + bx + ey + fz Kernel w y x z https://nl.mathworks.com/videos/series/introduction-to-deep-learning.html 9
Deep Learning: different types of network architectures Type Series Network Directly Acyclic Graph Network (DAG) Recurrent Neural Network (RNN) Basic Architecture Network Example AlexNet, VGG R-CNN (fast, faster), GoogLeNet, SegNet LSTM Application Object Recognition: Cars, Lane, Pedestrian Object Detection: Cars, Traffic Signs, Scene (Semantic Segmentation) Sequential data: time series, signals 10
Agenda What s Deep Learning and why should I care? A practical approach to Deep Learning (for images) Transfer the learning from an expert model to your own application Building a Deep Learning network from scratch Deep Learning for time series and text data Key learnings of the session and cool features 11
Deep Learning can be complex and challenging to apply Design Deep Learning & Vision Algorithms Main Challenges Handle large image sets Image labeling is tedious Have access to models Accelerate and Scale Training Main Challenges Capability of training with multiple GPUs Capability of training in the cloud High Performance Deployment Main Challenges Convert models to CUDA code Compress models to fit into embedded GPUs. 12
Object Recognition using Deep Learning in MATLAB Supervised Learning Stapler CoffeCoaster BLACK BOX Highlighter RubikCube AlexNet PRETRAINED MODEL Transfer Knowledge OfficeNet NEW MODEL MATLAB Support Package for USB Webcams 13
Where to start? Two Approaches for Deep Learning Train from Scratch Deep Neural Network, e.g. CNN Transfer Learning Fine - Tune 14
Why Perform Transfer Learning? Less data Less training time Leverage best network types from top researchers AlexNet PRETRAINED MODEL VGG-16 PRETRAINED MODEL ResNet PRETRAINED MODEL Reference models are great feature extractors Caffe M O D E L S GoogLeNet PRETRAINED MODEL TensorFlow/Keras M O D E L S 15
Transfer Learning Workflow Replace final layers Load pretrained network Early layers that learned low-level features (edges, blobs, colors) Last layers that learned task specific features New layers learn features specific to your data 1 million images 1000s classes Fewer classes Learn faster 16
Transfer Learning Workflow Load pretrained network Early layers that learned low-level features (edges, blobs, colors) Last layers that learned task specific features Replace final layers New layers to learn features specific to your data Training images Train network Training options 1 million images 1000s classes Fewer classes Learn faster 100s images 10s classes 17
Transfer Learning Workflow Deploy results Load pretrained network Early layers that learned low-level features (edges, blobs, colors) Last layers that learned task specific features Replace final layers New layers to learn features specific to your data Training images Train network Training options Predict and assess network accuracy Test images Car Probability Truck 1 million images 1000s classes Fewer classes Learn faster 100s images 10s classes Trained Network 98 % Bike 18
Transfer Learning Workflow Load pretrained network Replace final layers Train network Training images Predict and assess network accuracy Deploy results Early layers that learned low-level features (edges, blobs, colors) Last layers that learned task specific features New layers to learn features specific to your data Training options Test images Probability Car Truck 1 million images 1000s classes Fewer classes Learn faster 100s images 10s classes Trained Network 98 % Bike 19
Accelerate training and prediction! Deep Learning End-to-end Prediction Learning is an iterative process Alexnet vs Squeezenet More GPUs Whitepaper: Deep Learning in the Cloud 20
Frames per second Alexnet Inference on NVIDIA Titan Xp GPU Coder + TensorRT (3.0.1, int8) GPU Coder + TensorRT (3.0.1) GPU Coder + cudnn mxnet (1.1.0) TensorFlow (1.6.0) Batch Size CPU Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz Testing platform GPU cudnn Pascal Titan Xp v7 21
Deep Learning is easy and accessible with MATLAB! IN OUT MKL-DNN TensorRT & cudnn Neon Design Deep Learning & Vision Algorithms Accelerate and Scale Training High Performance Deployment Highlights Highlights Highlights Datastores for large image sets Single line of code to: Automate compilation with Automate image labeling Accelerate training with GPU Coder Direct access to models within MATLAB with multiple GPUs or 1.4x speedup over C++ Caffe support packages Scale to clusters on Jetson TX2 Import Tensor Flow Keras and Caffe networks 22
Agenda What s Deep Learning and why should I care? A practical approach to Deep Learning (for images) Transfer the learning from an expert model to your own application Building a Deep Learning network from scratch Deep Learning for time series and text data Key learnings of the session and cool features 23
Deep Learning for Time Series, Sequences and Text Sentiment Analysis Speech-to-text Music Recognition Predictive Maintenance Text-to-Speech Movie Ranking Prediction Instant Translation 24
Deep Learning for Time Series Example: Seizure prediction (time series classification) Goal: Predict seizures in long-term Dataset: iegg time series Data size: 20GB Output: Classification before or between seizure Link for melbourne-university-seizure-prediction 25
Types of Datasets Numeric Data Time Series/ Text Data Image Data Long Short Term Memory (LSTM) LSTM or CNN Convolutional Neural Networks (CNN) Directed acyclic graph networks (DAG) 26
Deep Learning for Time Series CNN: Data for Time series = Pixel for Images 0.2-0.5-1 -2.1-1.3 0.8 1.1-2 1.2 0 1.2 1.6 0.8 0.7-0.2-0.4 1 0 5 4 3 4 8 3 1 4 6 5 2 5 4 1 27
Deep Learning for Time Series Long Short Term Memory (LSTM) Network 28
Text Analytics Access and Explore Data Preprocess Data Develop Predictive Models Clean-up Text Convert to Numeric Media reported two trees blown down along I-40 in the Old Fort area. cat dog run two doc1 1 0 1 0 media report two tree blown down i40 old fort area doc2 1 1 0 1 Word Docs PDF s Text Files Stop Words Stemming Tokenization Bag of Words TF-IDF Word Embeddings LSTM Latent Dirichlet Allocation Latent semantic analysis 29
Key Takeaways: Deep Learning for Time Series and Text Applications Time series Text forecasting classification (Predictive Maintenance) classification (Sentiment Analysis, Tagging) clustering (Topic Modelling) Text Analytics: Prepend Text preprocessing Conversion to numeric 30
Agenda What s Deep Learning and why should I care? A practical approach to Deep Learning (for images) Transfer the learning from an expert model to your own application Building a Deep Learning network from scratch Deep Learning for time series and text data Key learnings of the session and cool features 31
Deep learning with MATLAB is easy and accessible! Use Less Data and Less Time with Transfer Learning Automatically generate code! Image Pre-processing CNN Image Post-processing Prototype and productize your entire workflow on embedded GPUs Apply Deep Learning not only on images but also on text and time series data 32
Algorithm Development, Testing & Verification Neural Network Toolbox GPU Coder Images 14x with MEX ~66 Fps (Tegra X1) 2x with MEX Semantic Scene Segmentation Acceleration (Classification) Deployment (Embedded) Time Series 33
Mini-batchable datastore Have a small number of large high-res images To train, need a large number of pairs of images 34
Ground truth labelling: attributes and sublabels 35
Visualize and understand the network architecture 19-Apr-2018 10:00:00 Detect problems before wasting time training! Missing or disconnected layers, Mismatching or incorrect sizes of layer inputs, Incorrect number of layer inputs, Invalid graph structures. Download Support Package 36
Semantic Segmentation Fully convolutional networks (FCN) Segmentation Networks (SegNet) Other directed acyclic graph (DAG) Manage connections, add and remove layers Manage label data and evaluate performance >>help semanticseg 37
Algorithm Development, Testing & Verification Applicable to other sensors, e.g. LiDar Automated Driving System Toolbox Computer Vision System Toolbox Neural Network Toolbox Access Data Velodyne FileReader Preprocess and Label Object Classification PointNet Deep Neural Network Preprocessing Clustering LiDAR Annotation Ground Truth [Under Request] 38
Input LSTM Fully Connected Regression Layer LSTM for both Classification and Regression Time to failure Feature extraction Classification 40
Define new operations for deep networks with Custom Layers 41
THANK YOU! Use Less Data and Less Time with Transfer Learning Automatically generate code! Image Pre-processing CNN Image Post-processing Prototype and productize your entire workflow on embedded GPUs Apply Deep Learning not only on images but also on time series data 42