End-to-End Memory Networks with Knowledge Carryover for Multi-Turn Spoken Language Understanding

Transcription

1 End-to-End Memory Networks wit Knowledge Carryover for Multi-Turn Spoken Language Understanding Yun-Nung Cen, Dilek Hakkani-Tür, Gokan Tur, Jianfeng Gao, and Li Deng National Taiwan University, Taipei, Taiwan Microsoft Researc, Redmond, A, USA {y.v.cen, dilek, {jfgao, Abstract Spoken language understanding (SLU) is a core component of a spoken dialogue system. In te traditional arcitecture of dialogue systems, te SLU component treats eac utterance independent of eac oter, and ten te following components aggregate te multi-turn information in te separate pases. However, tere are two callenges: 1) errors from previous turns may be propagated and ten degrade te performance of te current turn; 2) knowledge mentioned in te long istory may not be carried into te current turn. Tis paper addresses te above issues by proposing an arcitecture using end-to-end memory networks to model knowledge carryover in multi-turn conversations, were utterances encoded wit intents and slots can be stored as embeddings in te memory and te decoding pase applies an attention model to leverage previously stored semantics for intent prediction and slot tagging simultaneously. Te experiments on Microsoft Cortana conversational data sow tat te proposed memory network arcitecture can effectively extract salient semantics for modeling knowledge carryover in te multi-turn conversations and outperform te results using te state-of-te-art recurrent neural network framework (RNN) designed for single-turn SLU. Index Terms: spoken language understanding, end-to-end, memory network, embedding 1. Introduction In te past decades, goal-oriented spoken dialogue systems (SDS) are being incorporated in various devices and allow users to speak to systems in order to finis tasks more efficiently, for example, te virtual personal assistants Microsoft s Cortana and Apple s Siri. A key component of te understanding system is an spoken language understanding (SLU) module-it parses user utterances into semantic frames tat capture te core meaning, were tree main tasks of SLU are domain classification, intent determination, and slot filling [1]. A typical pipeline of SLU is to first decide te domain given te input utterance, and based on te domain, to predict te intent and to fill associated slots corresponding to a domain-specific semantic template. Te upper block of Figure 1 sows a communication-related user utterance, just send to bob about fising tis weekend and its semantic frame, send (contact name= bob, subject= fising tis weekend ) [2]. Traditionally, domain detection and intent prediction are framed as classification problems, were several classifiers suc as support vector macines and maximum entropy are employed [3, 4, 5]. Ten slot filling is framed as a sequence tagging task, were te IOB (in-outbegin) format is applied for representing slot tags as illustrated in Figure 1, and idden Markov models (HMM) or conditional D I U S communication send_ just sent to bob about fising tis weekend O O O O O B-contact_name B-subject I-subject I-subject send_ (contact_name= bob, subject= fising tis weekend ) U 1 S 1 send to bob B-contact_name send_ (contact_name= bob ) are we going to fis tis weekend U 2 S B-message I-message I-message I-message 2 I-message I-message I-message send_ (message= are we going to fis tis weekend ) Figure 1: Example utterances (U) annotated wit its domain (D) and intent (I) and semantic slots in te IOB format (S). Te upper block sows an example for a single-turn utterance, and te lower block sows similar utterances in te multi-turn scenario. random fields (CRF) ave been employed for tagging [6, 7, 8]. it te advances on deep learning, deep belief networks (DBNs) wit deep neural networks (DNNs) ave been applied to for domain and intent classification [9, 10, 11]. Recently, Ravuri et al. proposed an RNN arcitecture for intent determination [12]. For slot filling, deep learning as been viewed as a feature generator and te neural arcitecture can be merged wit CRFs [13]. Yao et al. and Mesnil et al. later employed RNNs for sequence labeling in order to perform slot filling [14, 15]. Recently, Hakkani-Tür proposed RNN-based joint semantic parsing for predicting intents and filling slots in te mean time [16]. However, above work focuses on SLU for single-turn interactions, were eac utterance is treated independently. Te contextual information as been sown useful for SLU modeling [17, 18, 19, 20]. For example, te lower block of Figure 1 sows two utterances, were te latter containing te message content in te , so it is more likely to estimate te semantic slot message wit te same intent send if te contextual knowledge is kept. Bargava et al. incorporated te information from previous intra-session utterances into te SLU tasks on a given utterance by applying SVM-HMMs to sequence tagging and obtained te improvement [17]. Also, contextual information as been incorporated into te recurrent neural network (RNN) for improved domain classification, in-

2 Contextual Sentence Encoder istory utterances {x i } RNN mem x 1 x 2 x i p i m i Knowledge Attention Distribution Memory Representation Sentence Inner Encoder Product RNN in x 1 x 2 x i c current utterance u eigted Sum Knowledge Encoding Representation Figure 2: Te illustration of te proposed end-to-end memory network model for multi-turn SLU. kg slot tagging sequence o y RNN Tagger tent prediction, and slot filling [18, 21]. However, most prior work exploited information only from te previous turn, ignoring te long-term contexts. Anoter constraint is tat te models require supervision at eac layer of te network, and tere is also no unified arcitecture tat can perform multi-turn SLU in an end-to-end framework. Recently tere as been a resurgence in computational models using explicit storage and a notion of attention [22, 23, 24, 25]; manipulating suc a storage allows multiple computational steps and can model long-term dependencies in sequential utterances. Basically, te storage is endowed wit a continuous representation modeled by neural networks, were te stored representations can be read and written to encode knowledge. Motivated by te idea, tis paper presents a recurrent neural network (RNN) arcitecture were te recurrence reads from a possibly large external memory before tagging te current utterance. Te model training does not require paired data for eac layer of te network; tat is, te proposed model can be trained end-to-end directly from input-output pairs. To te best of our knowledge, tis is te first attempt of employing an end-to-end neural network model to model long-term knowledge carryover for multi-turn SLU. 2. End-to-End Memory Networks For te SLU task, our model takes a discrete set of istory utterances {x i} tat are stored in te memory, a current utterance c = w 1,..., w T, and outputs corresponding semantic tags y = y 1,..., y T, were a semantic tag consists intent and slot information. Te proposed model is illustrated in Figure 2 and detailed below Arcitecture Te model embeds all utterances into a continuous space and stores all x s embedding to te memory. Te representation of te current utterance is ten compared wit memory representations to encode carried knowledge via an attention mecanism. Ten te encoded knowledge is taken togeter wit te word sequence for estimating te semantic tags. Four main procedures are described below. Memory Representation: To store te knowledge in te previous turns, we convert eac utterance from previous turns, x i, into a memory vectors m i wit dimension d by embedding te utterances in a continuous space troug an RNN. Te current utterance c is also embedded to a vector u wit te same dimension. m i = RNN mem(x i), (1) u = RNN in(c), (2) were RNN mem and RNN in are tied togeter for encoding contexts and te current utterance consistently. Terefore, te sequential information can be kept for better representations [16]. Knowledge Attention Distribution: In te embedding space, we compute te matc between te current utterance u and eac memory vector m i by taking te inner product followed by a softmax. p i = softmax(u T m i), (3) were softmax(z i) = e z i / j ez j and p i can be viewed as attention distribution for modeling knowledge carryover in order to understand te current utterance. Knowledge Encoding Representation: In order to encode te knowledge from istory, a istory vector is a sum over te memory embeddings weigted by te attention distribution. = i p im i. (4) Because te function from input to output is smoot, we can easily compute gradients and back propagate troug it. Ten te sum of te memory vector and te current input embedding u are ten passed troug a weigt matrix kg to generate an output knowledge encoding vector o, o = kg( + u), (5) were kg is a fully-connected neural network for encoding carried knowledge. Sequence Tagging: Different from te classification task most of work focused on [25], our memory arcitecture is to provide additional knowledge for improving tagging performance. Terefore, to estimate te tag sequence y corresponding to an input word sequence c, we use an RNN module for training a slot tagger, were te encoded knowledge o is fed into te input of te model in order to model knowledge carryover: y = RNN(o, c). (6)

3 y t-1 y t y t+1 V V V t-1 t t+1 M U M U M U o w t-1 w t w t+1 Figure 3: Te RNN model arcitecture for tagging. Te dotted red lines sow te encoded knowledge from istory in te multiturn interactions Recurrent Neural Network (RNN) Tagger Te goal of te SLU model is to assign a semantic tag for eac word in te current utterance. Tat is, given c = w 1,..., w n, te model is to predict y = y 1,..., y n were eac tag y i is aligned wit te word w i. e use te Elman RNN arcitecture, consisting of an input layer, a idden layer, and an output layer [26]. Te input, idden and output layers consist of a set of neurons representing te input, idden and output at eac time step t (w t, t, and y t) respectively. Te solid black part in te Figure 3 illustrates te arcitecture of te vanilla RNN model. t = φ( w t + U t 1), (7) ŷ t = softmax(v t), (8) were φ is a smoot bounded function suc as tan, and ŷ t is te probability distribution over of semantic tags given te current idden state t. Te sequence probability can be formulated as p(y c) = p(y w 1,..., w T ) = i p(y i w 1,..., w i). (9) Te model can be trained using backpropagation to maximize te conditional likeliood of te training set labels. To overcome te frequent gradient vanising issue wen modeling long-term dependencies, gated RNN was designed to use a more sopisticated activation function tan a usual activation function, consisting of affine transformation followed by a simple element-wise nonlinearity by using gating units [27], suc as long sort-term memory (LSTM) and gated recurrent unit (GRU) [28, 29]. RNNs employing eiter of tese recurrent units ave been sown to perform well in tasks tat require capturing long-term dependencies [15, 30, 31, 32]. In tis paper, we use RNN wit GRUs to allow eac recurrent unit to adaptively capture dependencies of different time scales Gated Recurrent Units (GRU) A GRU as two gates, a reset gate r, and an update gate z [29, 27]. Te reset gate determines te combination between te new input and te previous memory, and te update gate decides ow muc te unit updates its activation, or content. r = σ( r w t + U r t 1), (10) z = σ( z w t + U z t 1), (11) were σ is a logistic sigmoid function. Ten te final activation of te GRU at time t, t, is a linear interpolation between te previous activation t 1 and te candidate activation t: t = (1 z) t + z t 1, (12) t = φ( w t + U ( t 1 r))), (13) z r IN OUT Figure 4: Te illustration of a GRU cell as depicted in [27]. were is an element-wise multiplication. en te reset gate off, it effectively makes te unit act as if it is reading te first symbol of an input sequence, allowing it to forget te previously computed state. Figure 4 sows te gating mecanism of a GRU cell. RNN-GRU can yield comparable performance as RNN-LSTM wit need of fewer parameters and less data for generalization [27, 33]. Terefore, tis paper employs GRUs for all RNN models in te experiments Knowledge Carryover In order to model te encoded knowledge from previous turns, for eac time step t, te knowledge encoding vector o in (5) is fed into te RNN model togeter wit te word w t. Terefore, te idden layer in te RNN model can be formulated as t = φ(mo + w t + U t 1) (14) to replace (7) as illustrated in Figure 3, were te dotted red lines indicate te carried knowledge. RNN-GRU can incorporate te encoded knowledge in te similar way, were Mo can be added into (10), (11) and (12) for modeling contextual knowledge similarly Model Training To train te RNN-GRU wit knowledge carryover, te weigts of M,, and U for bot gates and te weigts of RNN mem, RNN in, and kg can be jointly updated via backpropagation from te RNN tagger Dataset 3. Experiments Te data is collected from Microsoft Cortana, were we extract multi-turn interactions (#turn 5) in te communication domain for experiments. Tere are 32 semantic tags (te concatenation of intents and slots). Te number of multi-turn utterances for training is 1,005, one for testing is 1,001, and one for development is 207. Tere are 13,779 single-turn utterances, wic are used to train te baseline SLU model for comparison Implementation Setting For training models, we use mini-batc stocastic gradient descent wit batc size 50 and te adam optimizer wit default parameters (a fixed learning rate 0.001, β 1 = 0.9, β 2 = 0.999, ɛ = 1e 08 ) [34]. Te number of iterations per batc is set to be 50 in te experiments. Te dimension of word and memory embeddings is set as 150 and te size of te idden layer in te RNN is set as 100. Te memory size is 20 to store carried knowledge from previous 20 turns. Te dropout rate is set to be 0.5 to avoid overfitting.

4 Table 1: Te performance of multi-turn SLU in terms of first turn only, oter turns, and overall results (%). First Turn Oter Turns Overall Model Training Knowledge Encoding P R F1 P R F1 P R F1 (a) single-turn RNN Tagger (b) multi-turn (c) multi-turn current utterance (c) Encoder-Tagger (d) multi-turn istory + current (x, c) (e) Memory Network multi-turn istory + current (x, c) Table 2: Te performance of multi-turn SLU in terms of intent and slot results (%). Model Intent Slot P R F1 P R F1 (a) (b) (c) (d) (e) Results In order to evaluate te proposed model for multi-turn SLU, we compare te performance of following model arcitectures in Table 1 and Table 2. RNN Tagger treats eac test utterance independently and performs sequence tagging via RNN-GRUs, were te training set comes from single-turn interactions (row (a)) or multi-turn interactions (row (b)). Encoder-Tagger encodes te knowledge before prediction using RNN-GRUs, and ten estimates eac tag by considering not only te current input word but also te encoded information via anoter RNN-GRUs, were we encode knowledge using te current utterance only (row (c)) or entire istory togeter wit te current utterance (row (d)). Note tat tere is no attention and memory mecanisms in tis model. Te entire istory is concatenated togeter for modeling te knowledge. Memory Network takes istory and current utterances into account for encoding knowledge wit attention and memory mecanisms in an end-to-end fasion, and ten performs sequence tagging using RNN-GRUs as described in Section 2 (row (e)). Te evaluation metrics are precision (P), recall (R) and F- measure (F1) for semantic tags, were eac tagging result is considered correct if te word-beginning and te word-inside and te word-outside predictions are all correct (including bot intents and slots). For te evaluation results, we sow te performance of te testing set in terms of (1) first turn only, (2) oter turns, and (3) all turns from a full dialogue in Table 1. Furtermore, we sow te performance of evaluating intents and slots separately in Table Comparing between Single-Turn and Multi-Turn Data For te rows (a) and (b) in Table 1, training on single-turn data may work well wen testing te first-turn utterances, acieving 60.6% on F1. However, for oter turns, its performance is muc worse due to lack of modeling contextual knowledge in te multi-turn interactions and mismatc between training and testing data. Treating eac utterance from multi-turn interactions independently performs 55.8% on F1 for te first-turn utterances, even toug te size of training data is smaller. Te reason is probably tat tere is no mismatc between training and testing Comparing between Encoded Knowledge For employing encoder-tagger models (rows (c) and (d)), additionally encoding te istory utterances improves te tagging results for bot first-turn and following turns. Also, te encoder-tagger significantly outperform te RNN tagger (47.4% to 62.5%), sowing tat encoder-tagger is able to capture clues from long-term dependencies Effectiveness of te Proposed Model From Table 1, we find tat te best overall performance comes from te proposed memory networks (row (e)), wic acieves 67.1% on te overall F1 score and sows te effectiveness of modeling long-term knowledge for SLU. Te proposed model works well wen tagging te turns wit previous contexts, were te F1 score of SLU is about 65.7%. Interestingly, te performance of first-turn utterances is also better tan all baselines, probably because te capability of modeling following turns can benefit te performance of first-turn utterances. Comparing wit te row (d), memory network is able to effectively capture salient knowledge for better tagging. Also, in terms of efficiency, te proposed model is more tan 10x faster tan te encoder-tagger model, because eac utterance is modeled separately and stored in te memory for later reuse Analysis of Intent and Slot Performance Table 2 presents intent-only and slot-only performance of te same set of models. For bot intent and slot performance, te model trained on single-turn utterances performs worst. For intent performance, encoder-tagger wit istory (row (d)) and memory network (row (e)) significantly outperform oters. Te difference between te rows (d) and (e) may not be significant. On te oter and, in terms of slot performance, te best result is from te memory network, demonstrating tat knowledge carryover modeled by te proposed model can benefit inference of semantic intents and slots in te multi-turn scenario. 4. Conclusions Tis paper proposes end-to-end memory networks to store contextual knowledge, wic can be exploited dynamically during testing for manipulating knowledge carryover in order to model long-term knowledge for multi-turn understanding. Te model embeds istory utterances into a continuous space and store tem in te memory. Te decoding pase applies an attention model to encode te carried knowledge and ten perform multiturn SLU. Te experiments sow te feasibility and robustness of modeling knowledge carryover troug memory networks.

5 5. References [1] G. Tur and R. De Mori, Spoken language understanding: Systems for extracting semantic information from speec. Jon iley & Sons, [2] Y.-N. Cen,. Y. ang, A. Gersman, and A. I. Rudnicky, Matrix factorization wit knowledge grap propagation for unsupervised spoken language understanding, Proceedings of ACL- IJCNLP, [3] P. Haffner, G. Tur, and J. H. rigt, Optimizing svms for complex call classification, in 2003 IEEE International Conference on Acoustics, Speec, and Signal Processing, Proceedings.(ICASSP), vol. 1. IEEE, 2003, pp. I 632. [4] C. Celba, M. Maajan, and A. Acero, Speec utterance classification, in 2003 IEEE International Conference on Acoustics, Speec, and Signal Processing, Proceedings.(ICASSP), vol. 1. IEEE, 2003, pp. I 280. [5] Y.-N. Cen, D. Hakkani-Tur, and G. Tur, Deriving local relational surface forms from dependency-based entity embeddings for unsupervised spoken language understanding, in 2014 IEEE Spoken Language Tecnology orksop (SLT). IEEE, 2014, pp [6] R. Pieraccini, E. Tzoukermann, Z. Gorelov, J.-L. Gauvain, E. Levin, C.-H. Lee, and J. G. ilpon, A speec understanding system based on statistical representation of semantics, in 1992 IEEE International Conference on Acoustics, Speec, and Signal Processing (ICASSP), vol. 1. IEEE, 1992, pp [7] Y.-Y. ang, L. Deng, and A. Acero, Spoken language understanding, IEEE Signal Processing Magazine, vol. 22, no. 5, pp , [8] C. Raymond and G. Riccardi, Generative and discriminative algoritms for spoken language understanding. in INTERSPEECH, 2007, pp [9] R. Sarikaya, G. E. Hinton, and B. Ramabadran, Deep belief nets for natural language call-routing, in 2011 IEEE International Conference on Acoustics, Speec and Signal Processing (ICASSP). IEEE, 2011, pp [10] G. Tur, L. Deng, D. Hakkani-Tür, and X. He, Towards deeper understanding: Deep convex networks for semantic utterance classification, in 2012 IEEE International Conference on Acoustics, Speec and Signal Processing (ICASSP). IEEE, 2012, pp [11] R. Sarikaya, G. E. Hinton, and A. Deoras, Application of deep belief networks for natural language understanding, IEEE/ACM Transactions on Audio, Speec, and Language Processing, vol. 22, no. 4, pp , [12] S. Ravuri and A. Stolcke, Recurrent neural network and lstm models for lexical utterance classification, in Sixteent Annual Conference of te International Speec Communication Association, [13] P. Xu and R. Sarikaya, Convolutional neural network based triangular CRF for joint intent detection and slot filling, in 2013 IEEE orksop on Automatic Speec Recognition and Understanding (ASRU). IEEE, 2013, pp [14] K. Yao, G. Zweig, M.-Y. Hwang, Y. Si, and D. Yu, Recurrent neural networks for language understanding. in INTERSPEECH, 2013, pp [15] G. Mesnil, Y. Daupin, K. Yao, Y. Bengio, L. Deng, D. Hakkani- Tur, X. He, L. Heck, G. Tur, D. Yu et al., Using recurrent neural networks for slot filling in spoken language understanding, IEEE/ACM Transactions on Audio, Speec, and Language Processing, vol. 23, no. 3, pp , [16] D. Hakkani-Tür, G. Tur, A. Celikyilmaz, Y.-N. Cen, J. Gao, D. Li, and Y.-Y. ang, Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM, in INTERSPEECH, [17] A. Bargava, A. Celikyilmaz, D. Hakkani-Tur, and R. Sarikaya, Easy contextual intent prediction and slot detection, in 2013 IEEE International Conference on Acoustics, Speec and Signal Processing (ICASSP). IEEE, 2013, pp [18] P. Xu and R. Sarikaya, Contextual domain classification in spoken language understanding systems using recurrent neural network, in 2014 IEEE International Conference on Acoustics, Speec and Signal Processing (ICASSP). IEEE, 2014, pp [19] Y.-N. Cen, M. Sun, A. I. Rudnicky, and A. Gersman, Leveraging beavioral patterns of mobile applications for personalized spoken language understanding, in Proceedings of 17t ACM International Conference on Multimodel Interaction (ICMI). ACM, 2015, pp [20] M. Sun, Y.-N. Cen, and A. I. Rudnicky, An intelligent assistant for ig-level task understanding, in Proceedings of Te 21st Annual Meeting of te Intelligent Interfaces Community (IUI), 2016, pp [21] Y. Si, K. Yao, H. Cen, Y.-C. Pan, M.-Y. Hwang, and B. Peng, Contextual spoken language understanding using recurrent neural networks, in 2015 IEEE International Conference on Acoustics, Speec and Signal Processing (ICASSP). IEEE, 2015, pp [22] D. Badanau, K. Co, and Y. Bengio, Neural macine translation by jointly learning to align and translate, in International Conference on Learning Representations (ICLR), [23] A. Graves, G. ayne, and I. Danielka, Neural turing macines, arxiv preprint arxiv: , [24] J. eston, S. Copra, and A. Bordesa, Memory networks, in International Conference on Learning Representations (ICLR), [25] S. Sukbaatar, J. eston, R. Fergus et al., End-to-end memory networks, in Advances in Neural Information Processing Systems, 2015, pp [26] J. L. Elman, Finding structure in time, Cognitive science, vol. 14, no. 2, pp , [27] J. Cung, C. Gulcere, K. Co, and Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, arxiv preprint arxiv: , [28] S. Hocreiter and J. Scmiduber, Long sort-term memory, Neural computation, vol. 9, no. 8, pp , [29] K. Co, B. van Merriënboer, D. Badanau, and Y. Bengio, On te properties of neural macine translation: Encoder-decoder approaces, arxiv preprint arxiv: , [30] K. Yao, B. Peng, Y. Zang, D. Yu, G. Zweig, and Y. Si, Spoken language understanding using long sort-term memory neural networks, in 2014 IEEE Spoken Language Tecnology orksop (SLT). IEEE, 2014, pp [31] A. Graves, A.-r. Moamed, and G. Hinton, Speec recognition wit deep recurrent neural networks, in 2013 IEEE International Conference on Acoustics, Speec and Signal Processing (ICASSP). IEEE, 2013, pp [32] I. Sutskever, O. Vinyals, and Q. V. Le, Sequence to sequence learning wit neural networks, in Advances in neural information processing systems, 2014, pp [33] R. Jozefowicz,. Zaremba, and I. Sutskever, An empirical exploration of recurrent network arcitectures, in Proceedings of te 32nd International Conference on Macine Learning (ICML- 15), 2015, pp [34] D. Kingma and J. Ba, Adam: A metod for stocastic optimization, arxiv preprint arxiv: , 2014.