Artificial Intelligence with MATLAB Sarah Hung Application Engineer 2018/05/02
Outline AI, machine learning and deep learning overview Essential tools for machine learning Deep learning for computer vision applications 2
AI, Machine Learning and Deep Learning 1950 Alan Turing introduces Turing Test 1966 ELIZA chatbot is introduced 1968 Kubrick s 2001: A Space Odyssey 1981 DEC deploys first Expert System 1997 IBM Deep Blue beats Kasparov at Chess 2005 Stanford wins DARPA grand challenge 2011 IBM Watson wins Jeopardy! 2012 Deep Learning starts winning computer vision challenges 2016 AlphaGo beats human pro at Go Source: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/ 3
Machine Learning Basic Concepts Start with an initial set of data Learn from this data Train your algorithm with this data Use the resulting model to predict outcomes for new data sets 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8-0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 4
Machine Learning Characteristics and Examples Characteristics Lots of data (many variables) System too complex to know the governing equation (e.g., black-box modeling) Examples Pattern recognition (speech, images) Financial algorithms (credit scoring, algo trading) Energy forecasting (load, price) Biology (tumor detection, drug discovery) AAA 93.68% 5.55% 0.59% 0.18% 0.00% AA A BBB BB B CCC 2.44% 0.14% 0.03% 0.03% 0.00% 0.00% 92.60% 4.18% 0.23% 0.12% 0.00% 0.00% 4.03% 91.02% 7.49% 0.73% 0.11% 0.00% 0.73% 3.90% 87.86% 8.27% 0.82% 0.37% 0.15% 0.60% 3.78% 86.74% 9.64% 1.84% 0.00% 0.00% 0.00% 0.00% 0.08% 0.00% 0.39% 0.06% 3.28% 0.18% 85.37% 2.41% 6.24% 81.88% 0.00% 0.06% 0.08% 0.16% 0.64% 1.64% 9.67% D 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% AAA AA A BBB BB B CCC D 5
Different Types of Learning Type of Learning Categories of Algorithms Supervised Learning Classification Output is a choice between classes (True, False) (Red, Blue, Green) Machine Learning Develop predictive model based on both input and output data Regression Output is a real number (temperature, stock prices) Unsupervised Learning Clustering No output - find natural groups and patterns from input data only Discover an internal representation from input data only 6
Different Types of Learning Type of Learning Categories of Algorithms Supervised Learning Classification Support Vector Machines Discriminant Analysis Naive Bayes Nearest Neighbor Machine Learning Develop predictive model based on both input and output data Regression Linear Regression GLM SVR, GPR Ensemble Methods Regression Trees Neural Networks Unsupervised Learning Clustering kmeans, kmedoids Fuzzy C-Means Hierarchical Gaussian Mixture Discover an internal representation from input data only Neural Networks Hidden Markov Model 7
Machine Learning Workflow Typically accounts for 80% effort Access & Explore Preprocess Data Develop Predictive Models Integrate Analytics to Systems Embedded Systems Machine Learning with Big Data Selecting Useful Features Quickly Comparing Models Fine-tuning Model Parameters Sharing & Deploying Models 8
Machine Learning vs Deep Learning Deep learning performs end-to-end learning by learning features, representations and tasks directly from images, text and sound Machine Learning Deep Learning 9
Deep Learning: Convolutional Neural Networks (CNN/ConvNet) Multi-layered neural network car truck van bicycle Input Convolution + ReLu Pooling Convolution + ReLu Pooling Flatten Fully Connected Softmax Feature Learning Classification 10
Convolution Activation Pooling Convolution Activation Pooling Convolution Activation Pooling Convolution Activation Pooling Fully Connected Layers CNN Feature Learning Probability Flower Car Tree Dog Filters 11
What problem does Deep Learning solve? Training Prediction Millions of images from 1000 different categories Real-time object recognition using a webcam connected to a laptop 12
Why is Deep Learning So Popular Now? Human Accuracy Source: ILSVRC Top-5 Error on ImageNet 13
Deep Learning Enablers Increased GPU acceleration Labeled public datasets World-class models AlexNet PRETRAINED MODEL Caffe M O D E L S VGG-16 PRETRAINED MODEL GoogLeNet PRETRAINED MODEL ResNet PRETRAINED MODEL TensorFlow/Keras M O D E L S 14
Outline AI, machine learning and deep learning overview Essential tools for machine learning Deep learning for computer vision applications 15
Objective Solve real-world problems using Computer Vision and Machine Learning Introduction Workflows Common Challenges #1 Measuring Drug Effectiveness from Motion Detection Object recognition, and tracking Quantification of movement to position (x,y) and velocity #2 Classify parasite type in blood smear images Handle large sets of images Develop and evaluate classifiers 16
Image Processing & Computer Vision What are the differences? Image Processing Micro level analysis Enhance & Analyze images Down to pixel level Static images Computer Vision Macro level analysis Find & Track objects Human scale Video Images over time Typical Applications: Medical AeroDef Mapping Typical Applications: Security AeroDef Robotics Bio/medical 17
Pixels Features Object Detection / Tracking / Classification Object Detection Image Pixels Feature Extraction Object Tracking Object Classification 18
#1 Measuring Drug Effectiveness from Motion Detection Videos of zebra fish movement Use computer vision techniques to detect distinct objects Identify motion tracks and quantify movement capture position, velocity plot and analyze movement quantify outliers Video courtesy: Thierry Ralph, FMI, Switzerland 19
Using Vision System Objects in MATLAB >> demo % Create Objects >> vreader = vision.videofilereader ('ZebraFish.avi'); Functions >> vplayer = vision.videoplayer('position', [20, 400, 700, 400]); [output1, output2, ] = function_name (input1, input2, input3) % Loop over frames: % Examples: acquire, process, display % Hundreds of advanced built-in library (or build your own functions) A = imread( myimgfile.jpg ); % Read from a jpeg file >> while ~isdone(vreader) >> frame = vreader(); >> % do some processing here % example object detector >> detector = tf = affine(a) % A is the input image, tf is the affine transformed output image vision.foregrounddetector... >> vplayer(frame); ('NumGaussians', 3,... >> end >> release(vreader) 'NumTrainingFrames', 100,... 'MinimumBackgroundRatio', 0.7); >> release(vplayer) 20
Pixels Features Object Detection / Tracking / Classification Object Detection Image Pixels Feature Extraction Object tracking Object Classification 21
#2 Classify parasite type in blood smear images Images of blood smears with 3 different parasitic infections Babesiosis Plasmodium Trypanosomiasis Use computer vision techniques to identify features Bag of words, SURF, etc. Train and evaluate various classifier models SVM 22
Traditional Approach vs. Machine Learning Machine learning uses data and produces a program to perform a task Task: Image Category Recognition Hand Written Program boats mugs hats Computer Vision Machine Learning boats mugs hats If brightness > 0.5 then hat If edge_density < 4 and major_axis > 5 then boat model = < Machine Learning > (data, label) Algorithm 23
Machine Learning Workflow Using Images Training Data Feature Extraction Learning or Modelling Training Classifier / Model hat Input Image Feature Extraction Classification 24 Classification
Challenges: Machine Learning Workflow Using Images Training Data Feature Extraction Learning or Modelling 1 Challenge 1 Handling large sets of images Challenge 2 Challenge 3 2 3 How to extract discriminative information from images How to model data using machine learning Input Image Feature Extraction Classifier / Model Classification 25 hat
What is Feature Extraction? Bag of Words SURF HOG Image Pixels Feature Extraction Representations often invariant to changes in scale, rotation, illumination More compact than storing pixel data Feature selection based on nature of problem Sparse Dense 26
Many Options for Features and Machine Learning Feature Extraction Computer Vision System Toolbox BRISK,FREAK, SURF Histogram of Oriented Gradients (HoG) Using box filters(integral images ) Bag of visual words Color-based features Frequency-domain features Machine Learning Statistics and Machine Learning Toolbox SVM Decision trees AdaBoost Bagged trees k-nn Discriminant analysis Bayes classifiers Many permutations and combinations to fit the needs of your problem 27
Bag of Words Image Processing Toolbox Class / Label Perform image processing, analysis, and algorithm development Image Processing Toolbox provides a comprehensive set of reference-standard algorithms, functions, and apps for image processing, analysis, visualization, and algorithm development. You can perform image analysis, image segmentation, image enhancement, noise reduction, geometric transformations, and image registration. Many toolbox functions support multicore processors, GPUs, and C-code generation. Image Processing Toolbox supports a diverse set of image types, including high dynamic range, gigapixel resolution, embedded ICC profile, and tomographic. Visualization functions and apps let you explore images and videos, examine a region of pixels, adjust color and contrast, create contours or histograms, and manipulate regions of interest (ROIs). The toolbox supports workflows for processing, displaying, and navigating large images. Training Data Vocabulary / Bag of Words 28
Bag of Visual Words ( features) mugs Class / Label Training Data Vocabulary / Bag of Words 29
Image Classification with Bag of Words Training Data Bag = Visual Vocabulary Bag Classifier hat Input Image 30
#2 Classification of Parasite Type from blood smears >> demo 31
Challenges for Machine Learning with Images Solutions Challenge 1 Handling large sets of images Easy to handle large sets of images imageset imagedatastore (new in R2016a) Challenge 2 How to extract discriminative information from images Bag of words for feature extraction Many more available in Computer Vision System Toolbox Challenge 3 How to model data using machine learning Machine Learning App (R2015a) 32
Why MATLAB for Machine Learning? App driven workflow Spend more time solving problems, less time coding Easily manage large sets of images built in functions to handle large datasets Domain specific toolboxes Image processing, signal processing, financial data, RADAR etc. Advanced Classifiers Fuzzy Logic Toolbox, Neural Network Toolbox Extensive documentation and examples helps you get started quickly 33
Outline AI, machine learning and deep learning overview Essential tools for machine learning Deep learning for computer vision applications 36
Cat What is Deep Learning? Deep learning is a type of machine learning that learns tasks directly from data Dog Bird Learned Features Car Dog Cat Bird Car Data Task 37
Import the Latest Models Pretrained Models* AlexNet VGG-16 VGG-19 GoogLeNet InceptionV3 Resnet50 Resnet101 AlexNet PRETRAINED MODEL Caffe M O D E L S Import Models from Frameworks Caffe Model Importer TensorFlow/Keras Model Importer VGG-16 PRETRAINED MODEL GoogLeNet PRETRAINED MODEL ResNet PRETRAINED MODEL TensorFlow/Keras M O D E L S * single line of code to access model 38
Download and Install Pre-trained Networks 1. Go to the Home tab and click the Add-Ons icon. 2. Search pretrained models. Click AlexNet add-on to open its detailed information page. 39
Download and Install Pre-trained Networks 3. Install the add-on. 4. Import the model by typing net = alexnet in command window! 40
Two Approaches for Training New Model 1. Train a Deep Neural Network from Scratch 2. Fine-tune a pre-trained model (transfer learning) 41
Train a Deep Neural Network from Scratch Recommended only when: Training data 1000s to millions of labeled images Computation Compute intensive (requires GPU) Training Time Days to Weeks for real problems Model accuracy High (can over fit to small datasets) 42
MNIST: The Hello, World! of Computer Vision What? A set of handwritten digits from 0-9 Why? An easy task for machine learning beginners How many? 60,000 training images 10,000 test images Best results? 99.79% accuracy = 7 Sources: http://yann.lecun.com/exdb/mnist/ https://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results 43
Transfer Learning Workflow Load pretrained network Early layers that learned low-level features (edges, blobs, colors) Last layers that learned task specific features Replace final layers New layers to learn features specific to your data Train network Training images Training options Predict and assess network accuracy Test images Deploy results Probability Boat Plane Car Train Trained Network 1 million images 1000s classes Fewer classes Learn faster 100s images 10s classes 44
Why Perform Transfer Learning Requires less data and training time Reference models (like AlexNet, VGG-16, VGG-19) are great feature extractors Leverage best network types from top researchers AlexNet PRETRAINED MODEL Caffe M O D E L S VGG-16 PRETRAINED MODEL GoogLeNet PRETRAINED MODEL ResNet PRETRAINED MODEL TensorFlow/Keras M O D E L S 45
Detection and Localization Using Deep Learning Regions with Convolutional Neural Network Features (R-CNN) Semantic Segmentation using SegNet 46
Workflow to Train Detectors Images Label Ground Truth Ground Truth Train detector Object detector Machine Learning Deep Learning 47
MATLAB Tools to Train Detectors Images Label Ground Truth Ground Truth Train detector Object detector 48
MATLAB Tools to Train Detectors Images Label Ground Truth Ground Truth Train detector Object detector Use same ground truth to try different detectors. Machine Learning Cascade Object Detector Aggregate Channel Features Single Line of Code to Train Each Detector Deep Learning R-CNN Fast R-CNN Faster R-CNN E.g. traincascadeobjectdetector trainfasterrcnnobjectdetector 49