Bayesian Deep Learning for Integrated Intelligence: Bridging the Gap between Perception and Inference

Transcription

1 1 Bayesian Deep Learning for Integrated Intelligence: Bridging the Gap between Perception and Inference Hao Wang Department of Computer Science and Engineering Joint work with Naiyan Wang, Xingjian Shi, and Dit-Yan Yeung

2 2 Perception and Inference See (visual object recognition) Read (text understanding) Hear (speech recognition) Comprehensive AI Think (inference and reasoning)

3 Perception 3 Bayesian Deep Learning (BDL) Motivation: Our goal Perception & Inference/reasoning Deep Learning & Graphical Models Inference/reasoning Deep learning Graphical model Bayesian deep learning

4 4 Perception and Inference Perception component Content understanding Task-Specific component Target task Bayesian deep learning (BDL) Maximum a posteriori (MAP) Markov chain Monte Carlo (MCMC) Variational inference (VI)

5 5 Example: Medical Diagnosis Perception component Symptoms Task-Specific component Reasoning and inference Bayesian deep learning (BDL)

6 6 Example: Movie Recommender Systems Perception component Content understanding Task-Specific component Similarity, preferences Recommendation Bayesian deep learning (BDL)

7 7 A Principled Probabilistic Framework Perception Component Task-Specific Component Perception Variables Task Variables Hinge Variables [ Wang et al ]

8 8 BDL Models for Different Applications [ Wang et al ]

9 9 Bayesian Deep Learning: Under a Principled Framework Probabilistic Graphical Models

10 10 Collaborative Deep Learning [ Wang et al (KDD) ]

11 11 Recommender Systems Rating matrix: Matrix completion Observed preferences: To predict:

12 12 Recommender Systems with Content Content information: Plots, directors, actors, etc.

13 13 Modeling the Content Information Handcrafted features Automatically learn features Automatically learn features and adapt for ratings

14 14 Modeling the Content Information 1. Powerful features for content information Deep learning 2. Feedback from rating information Non-i.i.d. Collaborative deep learning

15 15 Deep Learning Stacked denoising autoencoders Convolutional neural networks Recurrent neural networks Typically for i.i.d. data

16 16 Modeling the Content Information 1. Powerful features for content information Deep learning 2. Feedback from rating information Non-i.i.d. Collaborative deep learning (CDL)

17 17 Contribution Collaborative deep learning: * deep learning for non-i.i.d. data * joint representation learning and collaborative filtering

18 18 Contribution Collaborative deep learning Complex target: * beyond targets like classification and regression * to complete a low-rank matrix

19 19 Contribution Collaborative deep learning Complex target First hierarchical Bayesian models for deep hybrid recommender system

20 20 Stacked Denoising Autoencoders (SDAE) Corrupted input Clean input [ Vincent et al ]

21 21 Probabilistic Matrix Factorization (PMF) Graphical model: Notation: latent vector of item j latent vector of user i rating of item j from user i Generative process: Objective function if using MAP: [ Salakhutdinov et al ]

22 22 Probabilistic SDAE Graphical model: Generative process: Generalized SDAE Notation: corrupted input clean input weights and biases

23 23 Collaborative Deep Learning (CDL) Graphical model: Collaborative deep learning SDAE Two-way interaction More powerful representation Infer missing ratings from content Infer missing content from ratings Notation: rating of item j from user i latent vector of item j latent vector of user i corrupted input clean input weights and biases content representation

24 24 A Principled Probabilistic Framework (Recap) Perception Component Task-Specific Component Perception Variables Task Variables Hinge Variables [ Wang et al ]

25 25 CDL with Two Components Graphical model: Collaborative deep learning SDAE Two-way interaction More powerful representation Infer missing ratings from content Infer missing content from ratings Notation: rating of item j from user i latent vector of item j latent vector of user i corrupted input clean input weights and biases content representation

26 26 Collaborative Deep Learning Neural network representation for degenerated CDL

27 27 Collaborative Deep Learning Information flows from ratings to content

28 28 Collaborative Deep Learning Information flows from content to ratings

29 29 Collaborative Deep Learning Representation learning <-> recommendation

30 30 Learning maximizing the posterior probability is equivalent to maximizing the joint log-likelihood

31 31 Learning Prior (regularization) for user latent vectors, weights, and biases

32 32 Learning Generating item latent vectors from content representation with Gaussian offset

33 33 Learning Generating clean input from the output of probabilistic SDAE with Gaussian offset

34 34 Learning Generating the input of Layer l from the output of Layer l-1 with Gaussian offset

35 35 Learning measures the error of predicted ratings

36 36 Learning If goes to infinity, the likelihood simplifies to

37 37 Update Rules For U and V, use block coordinate descent: For W and b, use a modified version of backpropagation:

38 38 Datasets Content information Titles and abstracts Titles and abstracts Movie plots [ Wang et al ] [ Wang et al ]

39 39 Evaluation Metrics Recall: Mean Average Precision (map): Higher recall and map indicate better recommendation performance

40 40 Comparing Methods Hybrid methods using BOW and ratings Loosely coupled; interaction is not two-way PMF+LDA

41 41 When the ratings are very sparse: citeulike-t, sparse setting Netflix, sparse setting When the ratings are dense: citeulike-t, dense setting Netflix, dense setting

42 42 Mean Average Precision (map) Exactly the same as Oord et al. 2013, we set the cutoff point at 500 for each user. A relative performance boost of about 50%

43 43 Number of Layers Sparse Setting Dense Setting The best performance is achieved when the number of layers is 2 or 3 (4 or 6 layers of generalized neural networks).

44 44 Example User Romance Movies Moonstruck True Romance Precision: 30% VS 20%

45 45 Example User Action & Drama Movies Johnny English American Beauty Precision: 50% VS 20%

46 46 Example User Precision: 90% VS 50%

47 47 Summary: Collaborative Deep Learning Non-i.i.d (collaborative) deep learning With a complex target First hierarchical Bayesian models for hybrid deep recommender system Significantly advance the state of the art

48 48 Marginalized CDL Transformation to latent factors CDL: Reconstruction error Transformation to latent factors Marginalized CDL: Reconstruction error [ Li et al., CIKM 2015 ]

49 49 Collaborative Deep Ranking [ Ying et al., PAKDD 2016 ]

50 Generative Process: Collaborative Deep Ranking 50

51 51 Symmetric CDL Both item content and user attributes User attributes: age, gender, occupation, country, city, geolacation, domain, etc [ Li et al., CIKM 2015 ]

52 52 Symmetric CDL Marginalized CDL: Item content Symmetric CDL: Item content User attributes

53 53 Other Extensions of CDL Word2vec, tf-idf Sampling-based, variational inference Tagging information, networks

54 54 Relational Stacked Denoising Autoencoders [ Wang et al (AAAI) ]

55 55 BDL for Topic Models and Relational Learning Topic hierarchy Topic generation Word generation Topic-word relation Inter-document relation BDL-Based Topic Models

56 56 Relational SDAE as Relational Topic Models Perception component Task-Specific component Topic hierarchy Inter-document relation BDL-Based Topic Models [ Wang et al (AAAI) ]

57 57 Relational SDAE: Motivation Unsupervised representation learning Enhance representation power with relational information

58 58 Probabilistic SDAE Graphical model: Generative process: Generalized SDAE Notation: corrupted input clean input weights and biases

59 59 Relational SDAE: Graphical Model Notation: corrupted input clean input adjacency matrix

60 60 Relational SDAE: Two Components Perception Component Task-Specific Component

61 Relational SDAE: Generative Process 61

62 Relational SDAE: Generative Process 62

63 63 Multi-Relational SDAE: Graphical Model Product of Q+1 Gaussians Multiple networks: citation networks co-author networks Notation: corrupted input clean input adjacency matrix

64 64 Relational SDAE: Objective Function Network A Relational Matrix S Relational Matrix S Middle-Layer Representations

65 Update Rules 65

66 From Representation to Tag Recommendation 66

67 Algorithm 67

68 Datasets 68

69 Sparse Setting, citeulike-a 69

70 Dense Setting, citeulike-a 70

71 Sparse Setting, movielens-plot 71

72 Dense Setting, movielens-plot 72

73 73 Case Study 1: Tagging Scientific Articles Precision: 10% VS 60%

74 74 Case Study 2: Tagging Movies (SDAE) Precision: 30% VS 60%

75 75 Case Study 2: Tagging Movies (RSDAE) Does not appear in the tag lists of movies linked to E.T. the Extra-Terrestrial Very difficult to discover this tag

76 76 Relational SDAE as Deep Relational Topic Models Perception component Task-Specific component Topic hierarchy Inter-document relation BDL-Based Topic Models Unified into a probabilistic relational model for relational deep learning [ Wang et al (AAAI) ]

77 77 Applications of Bayesian Deep Learning: Under a Principled Framework Relational SDAE Collaborative Deep Learning Probabilistic Graphical Models

78 78 Take-home Messages Probabilistic graphical models for formulating both representation learning and inference/reasoning components Learnable representation serving as a bridge Tight, two-way interaction is crucial

79 79 Future Goals General Framework: 1. Ability of understanding text, images, and videos 2. Ability of inference and planning under uncertainty 3. Close the gap between human intelligence and artificial intelligence

80 80 Thanks! Q&A