Context Maintenance in Dialog

Transcription

1 Context Maintenance in Dialog Khyathi Raghavi Chandu, Aakanksha Naik, Aditya Chandrasekar Language Technologies Institute, Carnegie Mellon University Pittsburgh PA Abstract 1 Introduction Traditional Question-Answering systems provide relevant answers to questions by searching for and selecting the most relevant answer from a database, which may be structured (SQL Database) or unstructured (corpus of text documents). Maintaining context (information from previous questions) is an aspect that is missing in most traditional Question-Answering systems. In this project, we aim to develop a factoid question-answering dialog system that is capable of maintaining and utilizing context information. As a part of this work, we develop an open-domain factoid question answering dialog system and add some basic context-maintenance features to the system. we manage to perform some rudimentary context maintenance but find that effective context-maintenance in an open-domain question-answering dialog system is very challenging. Hence we restrict our domain to movie question-answering to further improve context maintenance. The movie question-answering dialog system that we develop aims to help users look for movies by asking questions about the movie, including attributes like names of the actors, the name of the director and the year of release. We also investigate the problem of asking the correct question at each turn in the dialog so as to extract the necessary information needed while leveraging the context gathered from previous questions and answers to quickly minimize the set of options to which the movie that the user is looking for may belong. The rest of this report is organized as follows: Section 2 describes our factoid question-answering system, section 3 describes the Context Maintenance strategies used in the Factoid Question Answering Dialog System and the problems associated with maintaining context in an open-domain, section 4 describes our movie question-answering dialog system and our approaches on how to ask the right set of questions about the movie at each turn and section 5 details our lessons learned and future work. 2 Factoid Question-Answering System We create a question-answering dialog system that is capable of maintaining context and answering factoid questions. The two main sub-modules of our system are: Question-Answering system and Context-Manager. At each dialog-turn, the user utterance is first directed to the Context Manager. The Context Manager embeds contextual information into the user utterance and passes this new utterance as a query to the Question-Answering system. The Question-Answering system predicts the Answer-Type for the query using an SVM classifier. Unigrams, bigrams and word-embeddings are used as features for Answer-Type prediction. The system also retrieves relevant documents for the query by performing a web search. POS Tagging, Chunking and Named Entity Recognition are then performed on these documents to retrieve candidate answers for the query in accordance with the answer-type identified by the SVM classifier. Finally, the candidate answers are ranked based on cosine similarity between the query and the retrieved sentences and the top candidate answer is

2 Figure 1: The architecture of the Factoid Question Answering Dialog System returned to the user. The Context Manager maintains context by resolving co-referent mentions and storing some important entities mentioned by the user in each turn. The next section provides more details on how context maintenance is achieved in our system. 3 Context Maintenance in Open-Domain Factoid Question-Answering In order to maintain context in open-domain factoid question answering, we first need to understand how users use context in actual conversations. Since no suitable datasets are available for this purpose, we construct a synthetic dataset of 60 sample conversations by simulating the interaction between users and a factoid question answering dialog system. We then observe that in many of these conversations, context can be maintained by simply resolving co-referent mentions. Consider the following simulated conversation: User: Who is the President of the United States? System: Donald J Trump User: In which state was he born? In this conversation, if we resolve that he refers to Donald J Trump, we can replace he by Donald J Trump in the third utterance, thereby changing the utterance to, In which state was Donald J Trump born?. This modified utterance can then be used to query our factoid question answering system, which will produce the appropriate response. This shows that coreference resolution is one of the important methods which can be used to maintain context in a factoid question answering dialog system. One basic method of incorporating coreference resolution is to run a state-of-the-art coreference system over the current user utterance as well as all utterances from previous turns and replace all co-referent mentions with the phrases they are linked to. We apply this technique to all 60 conversations from our synthetic dataset to study how well context is maintained by this method. We use the Stanford Deterministic Coreference resolution system for this task. We observe that the Stanford co-reference resolution system resolves third person singular pronouns and some noun phrases correctly, but does not do a good job at resolving that, this, there and split antecedents. We also notice that it resolves first person and second person pronouns, such as you and me, which would be detrimental for a dialog system. Based on our observations, we make some heuristic extensions to the system. 2

3 We ignore the resolution of split antecedents. We incorporate the following strategies on top of the Stanford system to aid in the resolution of some mentions which are not resolved by the system: If this and that occur in a sentence and if they do not occur as an adjective, then replace with it so that it can be resolved by the Stanford system. If there occurs in a sentence, then find the most recently mentioned location within the previous few utterances and substitute with that location. Do not resolve first and second person pronouns. Do not replace any phrases with pronouns. Store the person and location named-entities so that we can retrieve the most recent entity for resolving when and who questions. We can also use the most recent location entity for resolving there. The following are a few examples of conversations in which context maintenance by resolving coreference using the Stanford coreference system and our strategies works well: Original sentence: Who is the president of the United States? Modified sentence: Who is the president of the United States? Original sentence: Donald J. Trump Modified sentence: Donald J. Trump Original sentence: What is his chief of staff s name? Modified sentence: What is the president of the United States chief of staff s name? Original sentence: What is the color of the sky? Modified sentence: What is the color of the sky? Original sentence: Blue Modified sentence: Blue Original sentence: Isn t the ocean of the same color? Modified sentence: Isn t the ocean of the color of the sky However, we observe that coreference resolution does not completely solve the problem of context maintenance. For example, consider the following conversation: Original sentence: Alexa, Play Cheap Thrills by Sia Modified sentence: Alexa, Play Cheap Thrills by Sia Original sentence: Alexa, Play the entire album Modified sentence: Alexa, Play the entire album In this conversation, we can see clearly that play the entire album" should have been resolved to Play the entire album by Sia which contains the song Cheap Thrills". Coreference resolution, as we see, is not able to resolve and maintain this information. Using coreference resolution and storing the entities mentioned by the user, we can perform context maintenance in a completely domain-agnostic way. This type of context maintenance can maintain coherence to some extent, but it cannot talk/remember specific information about entities. For example, this system may remember that the user mentioned "France", but it will not store the user s opinion about France. Talking or remembering specific facts about entities requires the system to identify which aspects of entities are interesting, which is very difficult in an open domain. Therefore, in this work, we restrict our domain to movies so that we can maintain context more efficiently. The next section discusses about how we build a task-oriented dialog system for the movie domain and how we perform efficient domain-specific context maintenance. 4 The Movie Question-Answering Dialog System We focus on building a task-oriented dialog system whose goal is to search for a movie given some information about the movie. In order to do this, the dialog system has to have access to a knowledge base consisting of various attributes of the movie. We use the Wikipedia info boxes to construct this 3

4 Figure 2: Wikipedia infobox for the movie, "Shawshank Redemption". knowledge base for movies as they contain information about various attributes of a movie, like name, year released, director s name, cast information, etc. We assume that the most popular movies are the ones that are generally talked about, and therefore, we restrict our system database to the 250 most popular movies from IMDb. We extract and normalize the titles of these movies and then scrape the info boxes for these movies from the Wikipedia pages corresponding to the normalized titles. We also extract the text from the Wikipedia articles of these pages. Figure 1 shows the infobox on the Wikipedia page corresponding to the movie, "Shawshank Redemption". The data in the infoboxes provides information about various attributes of the movie and this data can be treated as slots and values for our dialog system. At each turn, we narrow our search based context accumulated from the previous turns which is stored in terms of these slots and values. From the infoboxes of these top 250 movies, we could collect 99 possible slots. After frequency based pruning, a set of 22 slots was selected, for which context is maintained. Information about each movie is stored in the following way: {MovieName: {slot1: [values], slot2: [values] } } The 22 generalized slots which we deal with are the following: [ fname, Starring, Running time, Directed by, Release date, Language, Country, Distributed by, Cinematography, Produced by, Edited by, Music by, Budget, Box office, Screenplay by, Productioncompany, Based on, Written by, Productioncompanies, Story by, Genre, Narrated by ] All the above slot names are self explanatory except for fname, which stores the name of the file containing the plot of the movie. The basic components of our dialog system are the Intent Detection Module, Context Maintenance Module and the Question Generator Module. The Intent Detection module predicts the intent behind each utterance of the user. We do not have annotated training data for intent classification for our dialog system. Hence, we experiment with two approaches for intent detection: semantic parsing and entity detection. In the semantic parsing approach, we create a set of seed words for each slot (for example, the seed set for the slot "directed by" consists of the following words: "director", "directed"). For each user utterance, we then detect the presence of these seed words and use the children of dependencies rooted at these seed words to populate slot values. The problem we faced with this approach is that dependencies between variations of slot names and values are not correctly identified (dependency parsing errors) sometimes due to which the system is unable to capture the information being provided by the user. Hence we adopted an entity detection based method where we check each user utterance for the presence of every value name from the list of all values for all slots of the movies and assign the corresponding slot from the dictionary to this value if it is found in the utterance. This hypothesis fails in the cases 4

5 where the same value is present in multiple slots. For example, Aamir Khan is a value that is present in both the slots directed by and starring. Moreover, this requires the user to specify information (such as actor/ director names) in the exact same way that it is specified in the Wikipedia Infobox. In spite of these issues, this technique performed better as compared to the semantic parsing based approach. Hence, the final system uses this approach. The Context Maintenance module maintains a list of attributes about which information has already been provided by the user. This is done so that the user is not queried for information about the same slot repeatedly. The user sometimes may not be able to answer the question being asked and hence, such slots should not be asked about again. Moreover, this module narrows down the list of possible movies that can fit the user s criteria, based on the stored attribute values. The Question Generation module takes care of asking the right question at each turn in the dialog with the final aim of finding the name of the movie that best fits the criteria of the user. This module adds interactive capability to the dialog system. The Question Generation module helps in narrowing down the set of possible movies by asking for information about specific attributes. The attribute which can provide largest reduction in the search space should be selected at each step by the Question Generation module. One way of selecting the attribute that gives largest reduction in the search space, would be to select the attribute which has the maximum number of values. The drawback of this method however is that the attribute with the maximum number of values at a particular turn may not be the most discriminative attribute. For example, consider the attribute, Starring. This attribute may have the maximum number of values (names of actors) at a particular turn, but may not be the most discriminative attribute at that turn, as a particular actor may have starred in all movies under consideration at that turn. In fact, the most discriminative attribute at a particular stage would be the one whose distribution over the set of possible values is closest to a uniform distribution. This motivates our second approach for generating questions by choosing the most discriminative attribute. We compute KL divergence values between the value distribution for each attribute and the uniform distribution. We then choose the attribute that has the minimum value of KL divergence as the most discriminative attribute about which the user will be queried next. Figure 2 illustrates this approach of selecting the most discriminative attribute at each turn using KL divergence. The KL divergence for two distributions P and Q is calculated using the following formula: KLdiv x (P Q) = i P (i)) ln( Q(i) P (i) ) = 1 ln(x) lnx i P (i)ln( P (i) Q(i) ) (1) In addition, to introduce more diversity into the questions being asked, the system randomly decides whether to ask the user about a specific attribute (using either of the two strategies discussed above) or to simply ask the user to provide more information. In this way, we provide the user the opportunity to directly give information that he is sure about rather than forcing him to provide replies to questions which he may not be able to answer. 5 Evaluation We trained the question classification system on 10,000 training questions from Li and Roth dataset. The answer type categories are abbreviation, entity, human, location and numeric. When tested on 5,500 questions, we got an accuracy of 76% for the task of question classification. To evaluate the gains from incorporating coreference, we manually analyzed a set of five conversations that real users had with our system. While none of the conversations contained any coreferent mentions, our strategies restricted first and second person pronouns from being replaced which was beneficial for our system. 6 Lessons Learned and Future Work we started out with an attempt to tackle the problem of domain-agnostic context maintenance. But we found that context maintenance in the open domain is a complicated problem for a dialog system with several passive and active hints that it would need to pick up on. Hence, we chose to restrict our domain to movies in the latter part of the project. Additionally, asking the user a question gives an 5

6 Figure 3: Approach for attribute selection using KL divergence impression that the bot is interested in the conversation. Hence we looked into the problem of making Alexa ask intelligent questions to improve interactivity as well as to obtain information necessary to reduce the search space. We face some issues in our Movie Question-Answering Dialog System because the Automatic Speech Recognition (ASR) technology is not particularly good with named entities. This creates a problem for our domain, especially when there are foreign names as values for slots. The choice of whether to ask the next best question or to request for more information can be shifted from a random decision (in our current system) to a statistical model based on confidence scores provided by a classifier. We again face the issue of lack of annotated data but we can devise strategies to create an annotated dataset for this step in the future. Moreover, we observed that the most discriminative slots are sometimes slots that people may not know directly. For example, budget is a slot that is generally highly discriminative for different movies but people may not be aware of exact figures to answer questions pertaining to these slots. Also, when asked about such slots, the user may answer with expressions such as low budget / expensive film etc. instead of providing exact answers. Hence the system needs to incorporate inference strategies to be able to draw such conclusions from the user utterance to perform better. Another example slot which may benefit from such strategies is running time, where people might be able to specify short or long film as opposed to the exact duration. Another interesting problem in this domain is delving deep into the unstructured text i.e the plot of the movie and helping users find movies they are looking for based on occurrences from the movie plot. We leave this for our future work. Acknowledgments We would like to thank Ran Zhao, Prof. Alex Rudnicky and Prof. Alan Black for their valuable suggestions and inputs throughout the duration of this project. 6