Distributed Representation of Sentences

Distributed Representation of Sentences LU Yangyang luyy11@sei.pku.edu.cn July 16,2014 @ KERE Seminar

Outline Distributed Representation of Sentences and Documents. ICML 14 Word Vector Paragraph Vector Experiments of NLP Tasks A Convolutional Neural Network for Modelling Sentences. ACL 14 DCNN: Convolutional Neural Networks Experiments of NLP Tasks Multilingual Models for Compositional Distributed Semantics. ACL 14 Composition Models Experiments Summary

Authors Distributed Representation of Sentences and Documents. ICML 14 1 Quoc Le, Tomas Mikolov Google Inc, Mountain View A Convolutional Neural Network for Modelling Sentences. ACL 14 2 Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom University of Oxford Multilingual Models for Compositional Distributed Semantics. ACL 14 Karl Moritz Hermann, Phil Blunsom University of Oxford 1 http://icml.cc/2014/index/article/15.htm 2 http://acl2014.org/acl2014/index.html

Outline Distributed Representation of Sentences and Documents. ICML 14 Word Vector Paragraph Vector Experiments of NLP Tasks A Convolutional Neural Network for Modelling Sentences. ACL 14 Multilingual Models for Compositional Distributed Semantics. ACL 14 Summary

Recall: Word Vector 3 Every word: A unique vector, represented by a column in a matrix W Given a sequence of training words w 1, w 2, w 3,..., w T : 3 Mikolov T, et al. Efficient estimation of word representations in vector space[c]. ICLR workshop, 2013

Recall: Word Vector 3 Every word: A unique vector, represented by a column in a matrix W Given a sequence of training words w 1, w 2, w 3,..., w T : Predicting a word given the other words in a context (CBOW) Predicting the surrounding words given a word (Skip-gram) 3 Mikolov T, et al. Efficient estimation of word representations in vector space[c]. ICLR workshop, 2013

Recall: Word Vector The Skip-gram Model 4 Predicting the surrounding words given a word in sentence The objective: maximize where 1 T T t=1 c j c,j 0 log p(w t+j w t ) c : the size of the training context 4 Mikolov T, et al. Distributed representations of words and phrases and their compositionality[j]. Advances in Neural Information Processing Systems, 2013

Recall: Word Vector Continuous Bag-of-Words Model(CBOW) 5 Predicting a word given the other words in a context The projection layer: shared for all words (not just the projection matrix) The objective: maximize T 1 k log p(w t w t k,..., w t+k ) T t=k 5 Mikolov T, et al. Efficient estimation of word representations in vector space[c]. ICLR workshop, 2013

Word Vector The objective: maximize 1 T T k t=k log p(w t w t k,..., w t+k ) The prediction task: via a multiple classifier (e.g. softmax 6 ) p(w t w t k,..., w t+k ) = eyw i eyi y = b + Uh(w t k,..., w t+k ; W ) where U, b : the softmax parameters h : a concatenation or average of word vectors extracted from W 6 GOTO 53

Paragraph Vector PV-DM: A Distributed Memory Model The paragraph vectors are asked to contribute to the prediction task of the next word given many contexts sampled from the paragraph. The paragraph acts as a memory that remembers what is missing from the current context or the topic of the paragraph.

PV-DM Every paragraph: a column in matrix D Shared across all contexts generated from the same paragraph but not across paragraphs Every word: a column in matrix W Shared across paragraphs Sampled from a fixed-length context over the paragraph Concatenate paragraph and word vectors The only change compared to the word vector model: y = b + Uh(w t k,..., w t+k, d; W, D) where h : constructed from W and D d : the vector of the paragraph from which the context is sampled

Paragraph Vector without word ordering PV-DBOW: Distributed Bag-Of-Words 7 Ignore the context words in the input Force the model to predict words randomly sampled from the paragraph in the output Sample a text window Sample a random word from the text window Form a classification task given the Paragraph Vector 7 Skip-gram Model: GOTO 7

Dataset: Sentiment Analysis Stanford Sentiment Treebank Dataset 8 11855 sentences taken from the movie review site Rotten Tomatoes train/test/development: 8544/2210/1101 sentences sentence/subphrase labels: 5-way fine-grained(+ + / + /0/ / ), binary coarse-grained(pos/neg) here only consider labeling the full sentences treat a sentence as a paragraph Experiment protocols: Paraphrase Vector: a concatenation of PV-DM and PV-DBOW PV-DM: 400 dimensions, PV-DBOW: 400 dimensions The optimal window size: 8 Predictor of the movie rating: a logistic regression 8 Socher, R. et al. Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP, 2013

Dataset: Sentiment Analysis IMDB Dataset 9 100, 000 movie reviews taken from IMDB each movie review: several sentences labeled train/unlabeled train/labeled test: 25, 000/50, 000/25, 000 labels: binary (pos/neg) Experimental protocols: PV-DM: 400 dimensions, PV-DBOW: 400 dimensions Learning word vectors and paragraph vectors: 25, 000 labeled + 50, 000 unlabeled The predictor: a neural network with one hidden layer with 50 units and a logistic classifier The optimal window size: 10 9 Maas, et al. Learning word vectors for sentiment analysis. ACL, 2011

Sentiment Analysis (cont.)

Information Retrieval with Paragraph Vector Dataset: 1, 000, 000 most popular queries top 10 results, by a search engine Constructing a triplet of paragraphs: 1 st, 2 nd : results of the same query 3 rd : randomly sampled from the rest collection (different query) Task: identify which of the triplet are the results of the same query

Outline Distributed Representation of Sentences and Documents. ICML 14 A Convolutional Neural Network for Modelling Sentences. ACL 14 DCNN: Convolutional Neural Networks Experiments of NLP Tasks Multilingual Models for Compositional Distributed Semantics. ACL 14 Summary

Recall: Max-TDNN Sentence Model 10 TDNNs: Time-Delay Neural Networks Modeling long-distance dependencies time refers to the idea that a sequence has a notion of order. A TDNN reads the sequence in an online fashion: at time t 1, one sees x t, the t-th word in the sentence. A classical TDNN layer: A convolution on a given sequence x( ) Outputting another sequence o( ) 10 Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML, 2008

DCNN: Overview Convolutional Neural Networks with Dynamic k-max Pooling

Wide Convolution Each word w i R d Sentence matrix s R d s Weight matrix for convolving m R d m Matrix after convolution c R d (s+m 1)

(Dynamic) k-max Pooling k-max Pooling: A generalisation of the max pooling over the time dimension 11 Different from the local max pooling operations 12 11 Max-TDNN: Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML, 2008 12

(Dynamic) k-max Pooling k-max Pooling: A generalisation of the max pooling over the time dimension 11 Different from the local max pooling operations 12 Given a value k and a sequence p R p (p k), k-max pooling selects the subsequence p k max of the k highest values of p. The order of the values in p k max max corresponds to their original order in p. Dynamic k-max Pooling: k l = max(k top, where L l L s) l : the number of the current convolutional layer to which the pooling is applied L : the total number of convolutional layers in the network k top : the fixed pooling parameter for the topmost convolutional layer 11 Max-TDNN: Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML, 2008 12

Non-linear Feature Function Apply the convolution + non-linear layers 13 each d-dimension column a in the matrix a: M = [diag(m :,1 ),..., diag(m :,m)] where m : the weights of the d filters of the wide convolution A wide convolution + a (dynamic) k-max pooling layer + a non-linear function + the input sentence matrix a first order feature map 13 Temporarily ignore the pooling layer

Multiple Feature Maps Repeating: wide convolution + (dynamic) k-max pooling + non-linear function feature maps of increasing order 14 F i j = n k=1 m i j,k * F i 1 k where F i j : the j-th feature map of the i-th order * : wide convolution m i j,k : convolving matrix( all the m i j,k form an order-4 tensor) 14 LeCun Y, Bengio Y. Convolutional networks for images, speech, and time series[j]. The handbook of brain theory and neural networks, 1995

Folding In the formulation of the network so far: Applying feature detectors to an individual row Creating complex dependencies across the same rows in multiple feature maps Feature detectors in different rows, however, are independent of each other until the top fully connected layer. Folding: For a map of d rows, folding returns a map of d/2 rows Halving the size of the representation With a folding layer, a feature detector of the i-th order depends now on two rows of feature values in the lower maps of order i?1

Training The top layer of the network has a fully connected layer followed by a softmax non-linearity The softmax layer: predicts the probability distribution over classes given the input sentence The objective: To minimise the cross-entropy of the predicted and true distributions Including an L 2 regularization term

Sentiment Prediction in Movie Reviews Stanford Sentiment Treebank Dataset

Question Type Classification Six different question types Train/test: 5452/500 TREC Dataset DCNN: word dimension d = 32, a single convolution layer with filters of size 8 and 5 feature maps

Twitter Sentiment Prediction with Distant Supervision A tweet is automatically labelled as positive or negative depending on the emotion that occurs in it Train/test: 1.6 million(emotion-based labels)/400 (hand-annotated labels) Preprocessing: a vocabulary of 76643 word types DCNN: word dimension d = 60, other parameters same as the binary sentiment prediction task of Stanford Sentiment Treebank

Visualising Feature Detectors A filter in the DCNN: associated with a feature detector or neuron that learns during training to be particularly active when presented with a specific sequence of input words The first layer: continuous n-grams Higher layers: multiple separate n-grams

Outline Distributed Representation of Sentences and Documents. ICML 14 A Convolutional Neural Network for Modelling Sentences. ACL 14 Multilingual Models for Compositional Distributed Semantics. ACL 14 Composition Models Experiments Summary

Multilingual Models for Compositional Distributed Semantics Representing meaning across languages in a shared multilingual semantic space Proposing a novel unsupervised technique leverages parallel corpora employs semantic transfer through compositional representations Experiments on two corpora: cross-lingual document classification on the Reuters RCV1/RCV2 corpora classification on a massively multilingual corpus which we derive from the TED corpus

Overview Word representation: a continuous vector in R d Semantic representations of sentence and document: Computed by a compositional vector model(cvm) A multilingual objective function: Using a noise-contrastive update between semantic representations of different languages to learn these word embeddings (a) The cat sat on the red mat. (b) 猫坐在红色的垫子上 (a) The cat sat on the red mat. (b) Die Katze saß auf der roten Matte.

Approach Given enough parallel data, a shared representation of two parallel sentences would be forced to capture the common elements between these two sentences. What parallel sentences share, of course, are their semantics. Define a bilingual energy: where E bi (a, b) = f(a) g(b) 2 C : a parallel corpus x, y : two different languages (a, b) C : two sentences of languages x, y f : X R d g : Y R d

Approach (cont.) The objective: minimize E bi for all semantically equivalent sentences in the corpus E hl (a, b, n) = [m + E bi (a, b) E bi (a, n)] + where [x] + = max(x, 0) (a, b) C : positive sample (a, n) C : negative(or noise) sample The final objective function: minimize J(θ) = k E hl (a, b, n i ) + λ 2 θ 2 (a,b) C i=1 where θ : all the parameters in the model

Composition Models: CVM Focus on composition functions that do not require any syntactic information ADD model: f(x) = n i x i A sentence is represented by the sum of its word vectors A distributed bag-of-words approach: ignore the sentence order

Document-level Semantics For a number of tasks, such as topic modelling, representations of objects beyond the sentence level are required. Extend model to document-level learning: recursively applying the composition and objective function

Dataset: The Europarl corpus v7(rcv) 15 Experiment settings used for the Cross-Lingual Document Classification(CLDC) task considered the English-German and English-French language pairs A massively multilingual corpus based on the TED corpus 16 for IWSLT 2013 training: 12, 078 parallel documents (12 languages) used for the topic classification task: 15 most frequent keywords as topics Experiment protocols: All model weights were randomly initialised using a Gaussian distribution (μ = 0, σ 2 = 0.1). The number of noise samples for each positive samples: {1, 10, 50} The dimension of all embeddings: d = 128 Iterations: 100 for RCV, 500 for TED, 5 for joint 15 http://www.statmt.org/europarl/ 16 https://wit3.fbk.eu/

RCV1/RCV2 Document Classification ADD: training on 500k sentence pairs of the English-German parallel section ADD+: using an additional 500k parallel sentences from the English-French corpus Training the document classifier: using varying sizes between 100 and 10, 000 documents

TED Corpus Experiments Using the training data of the corpus to learn distributed representations across 12 languages In the single mode:vectors are learnt from a single language pair (en-x) In the joint mode: vector learning is performed on all parallel sub-corpora simultaneously.

Linguistic Analysis

Summary Mikolov, ICML 14 An unsupervised learning of paragraph vector PV-DM PV-DBOW Learning to predict the surrounding words in contexts sampled from the paragraph Lossing the word order information NLP tasks: Sentiment prediction (Stanford, IMDB) Information retrieval (computing similarity between snippets)

Summary(cont.) Kalchbrenner,ACL 14 A dynamic convolutional neural network DCNN Wide convolution + folding + (dynamic) k-max pooling + non-linearity NLP tasks: Sentiment prediction (Stanford, Twitter) Question type classification Visualizing feature detectors

Hermann,ACL 14 Summary(cont.) A novel method for learning multilingual word embeddings Leveraging parallel data Defining a multilingual objective function Coupled with simple composition functions CVM & DocCVM: ADD, BI NLP tasks: Cross-lingual document classification (Reuter RCV1/RCV2) Topic classification (TED) ALL (Mikolov 14, Kalchbrenner 14, Hermann 14): Without requiring external features as provided by parsers or other resources

Related Neural Sentence Models Neural Bag-of-Words(NBoW) models Mikolov T. et al. Distributed Representations of Words and Phrases and their Compositionality. NIPS, 2013 Bengio Y. et al. A Neural Probabilistic Language Model. JMLR, 2006 Models that adopts a more general structure Socher R. et al. Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP, 2013 Socher R. et al. Grounded Compositional Semantics for Finding and Describing Images with Sentences. TACL, 2013 Jordan B. Pollack. Recursive distributed representations. Artificial Intelligence, 1990 Models based on convolution and TDNN architeture Kalchbrenner N. and Blunsom P. Recurrent Convolutional Neural Networks for Discourse Compositionality. ACL, 2013 Collobert R. and Weston J. A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML, 2008

Thank You for Listening! Q & A

A Neural Probabilistic Language Model 17 18 y = b + W x + U tanh(d + Hx) x = (C(w t 1 ), C(w t 2 ),..., C(w t n+1 )) 17 Bengio Y. et al. A Neural Probabilistic Language Model. JMLR, 2006 18 Word Vector: GOTO 9

Stanford Sentiment Treebank 19 19 Socher R. et al. Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP, 2013