COMPONENT BASED SUMMARIZATION USING AUTOMATIC IDENTIFICATION OF CROSS-DOCUMENT STRUCTURAL RELATIONSHIP

Transcription

1 IADIS International Conference Applied Computing 2012 COMPONENT BASED SUMMARIZATION USING AUTOMATIC IDENTIFICATION OF CROSS-DOCUMENT STRUCTURAL RELATIONSHIP Yogan Jaya Kumar 1, Naomie Salim 2 and Albaraa Abuobieda 3 1 Faculty of Information and Communication Technology, Universiti Teknikal Malaysia Melaka Durian Tunggal, Melaka, Malaysia 2 Faculty of Computer Science and Information Systems, Universiti Teknologi Malaysia , Skudai, Johor, Malaysia 3 Faculty Faculty of Computer Studies, International University of Africa , Khartoum, Sudan ABSTRACT The world we live today witnesses a fast moving information age due to the ever increasing information available online. People are being exposed vast online documents, being retrieved from various sources. The need for automatic document summarization system has deemed necessary to alleviate information overload. In the context of online news documents, different news sources reporting on a particular event tend to contain common components that make up the main story of the news. Based on this conception, we propose component based summarization, i.e. taking into account the generic components of a news story to produce quality summaries. We focus particularly on news stories related to natural disaster events. Besides that, we also investigate the automatic identification of cross structural relationships (CST) between sentences using case base reasoning (CBR) approach. The identified CST relations will be used to extract highly relevant sentences to be included in the summary. As for the evaluation process, the performance of our proposed approach was evaluated using ROUGE: - a standard evaluation metric used in text summarization. KEYWORDS Multi document summarization, Cross-document structural relationship, Case base reasoning, Machine learning. 1. INTRODUCTION There have been many research works concerning text document summarization in academia (Gupta and Lehal, 2010). Research on text summarization can be of different nature ranging from single document summarization to multi document summarization. In this work we concentrate our attention on the problem of producing multi document summary for news articles, particularly news stories related to natural disaster events. Imagine that a user tries to find information about the news regarding the earthquake which occurred in Sendai, Japan. The user will probably receive dozens of articles, possibly related. Thus it is favorable to have a system which can generate a summary containing the most important information contained in those articles. Such systems are designed to take a cluster of related documents and produce a shorter or concise version of the original documents. Much of the work thus far has been in the context of generic summarization (Nenkova and McKeown, 2011). In generic summarization, the importance of information is determined only with respect to the input alone without relating to the goal for generating the summary. This approach is very imprecise and less knowledge-rich. Furthermore, such approach views all documents as homogenous texts regardless of the genre of its domain, i.e. generic summaries make no assumptions about the domain of its source information. This shortcoming has led to the development of summarization systems centered upon various domains of interest. For example in summarizing business articles, biomedical documents and etc. (Wu and Liu, 2003, Verma et al, 2007). As this study involves multi document, we will also investigate the studies related to multi document analysis. Text analysis in documents has nowadays become very prominent, especially when it involves multiple documents e.g. news articles. The idea of cross-document structural relationship is to investigate the existence of inter-document relationships. These relations are based on the CST model (Cross-document 59

2 ISBN: IADIS Structure Theory) (Radev, 2000). Documents which are related to the same topic usually contain semantically related textual units. A number of researchers have addressed the benefits of CST for summarization task (Zhang et al, 2002 and Jorge and Pardo, 2010). However the major limitation of their work is that the CST relationships need to be manually annotated by human expert; which is a drawback for an automatic summarization system. In this study, our proposed approach takes into account the generic components of a news story and performs sentence selection based on automatic identification of CST relationships. We believe that providing comprehensive contextual information coverage would be ideal for news summary creation, as it is close to the way how humans prepare a news related summary. The rest of the paper is organized as follows: Section 2 presents the overview of our approach. It covers the proposed framework together with the CST relationship identification process. Section 3 gives the evaluation results for both the CST relationship identification and the generated summaries. Finally we end with conclusion in Section OVERVIEW OF APPROACH If we look back at previous approaches concerning text summarization, we can observe that the approaches are mainly based on low knowledge representations without any attempt to understand the text. Moreover, until now, most text summarization models incorporate only bag of words as text representation and do not include much contextual information. For example, to provide coverage related to the locations, people and events particularly for a news story. We believe that providing comprehensive contextual information coverage would be ideal for summary creation. As far as news documents are concerned, different news sources reporting on a particular event tend to contain common components that make up the main story of the news. The most common components of a news article consist of WHO, WHEN, WHERE, WHAT and HOW. In the process of news story production, these are the core components which a journalist must collect, interpret, organize, and transmit (Neal and Brown, 1982). Furthermore, such components which are integrated in news articles are very close to how human perceive news information content. Figure 1. General components of news articles related to natural disaster events In the context of natural disaster events, the news components can be mapped to these events. That is including components such as the description of the disaster (HOW), information about affected locations (WHERE), persons involved (WHO), the damages to human and properties (WHAT), the relief efforts (WHAT) and the organizations involved (WHO) (see Figure 1). Such occurrence of component sentence with its information content description is what the readers usually search for while reading news stories related to natural disaster events. 60

3 IADIS International Conference Applied Computing Framework Overview Figure 2 shows the general design for our proposed component based summarization (CBS) framework. First it receives a cluster of news articles that need to be summarized. Using the GATE tool, all component sentences are extracted from these documents. The extracted component sentences are then further preprocessed and directed into CBR model to identify the CST relationships it holds. Based on the type and number of CST relationship, each component sentence will be scored and ranked in their predefined component clusters. Redundant sentences will be removed by using word overlap check. To generate the summary, high ranking sentences are selected according to each component cluster size until the desired summary length is met. Finally the selected summary sentences are sorted according to its original position in the document Component Sentence Extraction Figure 2. Component based summarization (CBS) framework To extract the component sentences from the news articles, we looked at some existing information extraction (IE) techniques. Over the years, a number of information extraction techniques have been developed. A comprehensive review and analysis of these techniques can be found in (Moens, 2003). We have employed the technique which uses gazetteer lists. Despite its simplicity, many IE systems have shown that this technique works well in various applications (Wimalasuriya and Dou, 2009). As opposed to IE technique which uses linguistic rules, this technique recognizes individual words or phrases instead of patterns. This approach is similar to the one used for named entity recognition (NER) task. First, the words or phrases to be identified for a particular category are stated in a list, known as a gazetteer list. In our work, these categories refer to the components in the news documents (refer to Figure 1). Once the text documents are annotated by the entities of each component, then by using Java Annotation Patterns Engine (JAPE) grammar, the component sentences are recognized and extracted. In this work, we used the General Architecture for Text Engineering (GATE) tool (Cunningham et al, 2002), which is a widely used NLP framework that provides the platform to employ this technique Identification of CST Relationships In this section, we will discuss all the steps to be considered for automatically identifying the CST relationships between sentences pairs. Cross-document relationships between sentences can indicate the sentences with high relevance in a particular document cluster. In our work, the sentence relevancy is evaluated with respect to its component cluster. We have considered four types CST relationship namely Identity, Subsumption, Description and Overlap. As most of the other CST relations are covered by these four relations, we consider them sufficient for our summarization task. Table 1 lists the details of the four CST relations that need to be identified. 61

4 ISBN: IADIS Table 1. CST relations used in this work Relationship Identity Subsumption Description Overlap (partial equivalence) Description The same text appears in more than one location S1 contains all information in S2, plus additional information not in S2 S1 describes an entity mentioned in S2 S1 provides facts X and Y while S2 provides facts X and Z; X, Y, and Z should all be non-trivial. At first, all the extracted component sentences will be preprocessed using stop word removal and word stemming. Then feature extraction is performed. We represent each of the sentence pairs using lexical and semantic features. Below we describe the features that were computed for each sentences pair: Cosine similarity cosine similarity is used to measure the similarity of two sentences (S). Here the sentences are represented as word vectors with tf-idf as its element (i) value: S1, i S2, i cos( S1, S2) (1) 2 2 ( S1, i) ( S2, i) Word overlap this feature represents the measure based on the number of overlapping words in the two sentences. This measure is not sensitive to the word order in the sentences (Zahri and Fukumoto, 2011): # commonwords( S1, S2) overlap( S1, S2) (2) # words( S1) # words( S2) Length ty pe of S 1 this feature gives the length type of the first sentence when the lengths of two sentences are compared: lengtype( S1) 1 if length( S1) length( S2), -1 if length( S1) length( S2), (3) 0 if length( S1) length( S2) NP simila rity this feature represents the noun phrase (NP) similarity between two sentences. The similarity between the NPs is calculated according to Jaccard coefficient as defined as in the following equation: NP( S1) NP( S2) NP( S1, S2) (4) NP( S1) NP( S2) VP simila rity this feature represents the verb phrase (VP) similarity between two sentences. The similarity between the VPs is calculated according to Jaccard coefficient as defined as in the following equation: VP( S1) VP( S2) VP( S1, S2) (5) VP( S1) VP( S2) We have proposed the case based reasoning (CBR) approach to perform the identification of CST relationship in this work. CBR is a supervised based learning algorithm which has four major phases i.e. Retrieve, Reuse, Revise and Adapt. Once we have extracted all the features from each sentence pair, we represent them as feature vectors (inputs). These inputs together with their respective outputs (CST relationship types) represent the cases in the casebase. Next to identify the relationship type of a new case (sentence pair), we will compare the input feature vector of the new case with existing cases in casebase. We use the cosine measure to retrieve the similar cases. If the similarity value is more than the predefined threshold value, the model will reuse the solution. Thus, the solution (relationship type) to the new case will then be the output of the most similar case retrieved from the casebase. If the similarity value is less than the threshold value, the model will revise the new case as No relation type and adapt the revised new case into the casebase. This process is depicted in Figure 3. 62

5 IADIS International Conference Applied Computing Sentence Scoring Figure 3. CBR process for CST relationship identification In text summarization, one of the key phases prior towards sentence selection is sentence scoring. This phase involves giving score to each sentence based on some scoring metrics and then rank those sentences based on the highest score. The high ranking sentences will then become the potential sentence to be included in the summary. In our work, the sentence scoring will be based on the type and number of CST relationships i.e. the final score is obtained by integrating all the CST relationship scores: 4 Score( S)= Score( Rk ) (6) k 1 where: a) Score( S ) = the score of the sentence S b) Score( R k ) = the score of the CST relation k, i.e. the total number of relation k owned by sentence S Sentence Selection After sentence scoring and ranking completes, the final phase will be the sentence selection phase. The phase completes a summary by adding the high ranking sentence in a text until the summary length is met. This is the most standard method for sentence selection. However in our proposed component based summarization approach, we choose the sentences based on the size of the component clusters. The size ratio of each component cluster is first computed. Then the highly ranked sentences in each component cluster are picked according to its size ratio. This process ends when it reaches the desired summary length. Finally the selected summary sentences are ordered according to its original position in the document. 3. EVALUATION As our proposed component based summarization approach uses automatically identified CST relations for summary generation, it is not fair to just evaluate the final summary alone. The performance of CBR classification should also be evaluated. This is essential because the performance of the classification has direct implication on the final results of the system. Thus in this section, we will show the evaluation results for both CBR classification model for CST relationship identification and the overall component based summarization (CBS) model for generating the summaries. 63

6 ISBN: IADIS 3.1 CBR Classification Performance It is important to compare different techniques on the same datasets to see if the performance of the technique being proposed is comparable or performs better than the other techniques. We have compared the performance of our proposed CBR model with Neural Network (NN) and Support Vector Machine (SVM), which are two popular machine learning techniques used for classification tasks. In conducting the experiment, we used the dataset taken from CSTBank a corpus consisting clusters of English news articles annotated with CST relationships (Radev and Otterbacher, 2003). Our training and testing set consist of sentence pairs with its corresponding CST type label. We selected 476 sentence pairs for training and 206 sentence pairs for testing. Figure 4 shows the comparison of F-measures between the three techniques. It can be observed that CBR performs better than SVM and NN in classifying three out of five relations. It was also found that overall our CBR model achieved highest classification accuracy, which is 80.58%. The ability of CBR to perform well in CST relationship type identification could be closely related to nature of its learning method itself, i.e. lazy learning. As opposed to eager learning methods which need to generalize the training data to classify new cases, lazy learning is a learning method which performs classification based on the similarity of that problem with already known problems. Concerning our problem domain, since texts data have high variability, the key advantage of lazy learning is that instead of estimating the target function once for the entire instance space, this method can estimate it locally for each new instance to be classified. Besides that, CBR can better fit our CST relationship identification problem since it is capable to adapt new cases into its casebase. This will not require retraining of data as opposed to SVM and NN. Figure 4. Comparison of F-measures between SVM, NN and CBR 3.2 Summary Evaluation Our system was evaluated using 44 news articles (related to natural disaster events) from different document sets of test data obtained from Document Understanding Conference (DUC) DUC 2002 is a standard corpus used in text summarization studies. As to evaluate the generated summary, we used ROUGE, a common tool used for this purpose. ROUGE (Recall-Oriented Understudy for Gisting Evaluation), proposed by Lin (2004) is package for automatic evaluation for summaries. This system measures the quality of a system generated summary by comparing it to a human created summary (gold standard). There are many variances in ROUGE evaluation measure; however it was found that ROUGE-1, ROUGE-2, ROUGE-S and ROUGE-SU worked well in multi document summarization tasks (Lin, 2004). Thus we employ these four measures in this work. We selected two model (human) summaries from DUC 2002, namely HI and H2. H1 is used as reference summary to measure the quality of the generated summaries for each method (the proposed CBS method, baseline and H2-H1). H2 is used as human with human benchmark (H2-H1). We also use the baseline (without using component) as our comparison model. 64

7 IADIS International Conference Applied Computing 2012 Table 2. Summarization results comparison between CBS and baseline (without using component) using ROUGE-1, ROUGE-2, ROUGE-S and ROUGE-SU Using Component Without Component Measure AVG-R AVG-P AVG-F AVG-R AVG-P AVG-F ROUGE ROUGE ROUGE-S ROUGE-SU The evaluation was based on the average recall, precision and F-measure of the ROUGE metrics. Table 2 shows the comparison between the proposed CBS method and the baseline method. The baseline method also uses CST relations to select the most relevant sentences for its summary generation. However it threats all sentences to be homogenous without integrating the components associated to it. The experimental result shows that adopting news components improves the quality of the summary. Also, as can be seen in Figure 5, the graphs indicate that the CBS method is close to human benchmark in terms of its performance. We could say that incorporating the structure or components of a news document does influence the nature of automatic summarization. Such way of utilizing news component s content knowledge will benefit the summarization process as it gives an intuitive thought on the kind of information that is essential to be included in the summary. (a) (b) (c) Figure 5. The CBS, baseline and H2-H1 comparison: Average precision, recall and f-measure using (a) ROUGE-1, (b) ROUGE-2, (c) ROUGE-S and (d) ROUGE-SU (d) 65

8 ISBN: IADIS 4. CONCLUSION In this paper, we have introduced a method based on the generic components of news for multi document summarization problem where our approach is focused on generating summaries for news articles related to natural disaster events. Our proposed component based summarization model (CBS) is integrated with crossdocument relationships (CST) to identify highly relevant sentences to be included in the summary. As opposed to previous works which highlighted the benefit of CST relations using manually annotated text, in this work we have attempted to automatically identify the CST relations. In order to achieve this task, we designed a case based reasoning (CBR) model and obtained good classification results. The overall performance of our proposed CBS model was assessed using the dataset from DUC 2002 whereby its performance was measured using ROUGE measures. The results showed that the proposed method is effective and came close to the human benchmark scores. This supports our hypothesis i.e. providing comprehensive contextual information coverage using generic components of news would be ideal for news summary creation, as it is close to the way how humans prepare a news related summary. Although we focus on natural disaster news stories, the concepts and techniques are applicable to other domains as well. As future work, we plan to improve the CST relationship identification by using feature weighting and also considering suitable learning algorithm to improve the sentence scoring phase to better rank the important sentences. ACKNOWLEDGEMENT This research is supported by the Ministry of Higher Education (MOHE), Universiti Teknikal Malaysia Melaka (UTeM) and Universiti Teknologi Malaysia (UTM). REFERENCES Gupta, V. and G.S. Lehal, A Survey of Text Summarization Extractive. J. Emerging Technologies in Web Intelligence, Vol. 2, No.3, pp Nenkova, A. and McKeown, K., Automatic Summarization. Foundations and Trends in Information Retrieval,Vol. 5, pp Wu, C. and Liu, C., Ontology-based Text Summarization for Business News Articles. Proceeding of the Computers and Their Applications, pp Verma, R., P. Chen and W. Lu, A Semantic free-text summarization system using ontology knowledge. Proceedings of the Document Understanding Conference, pp Radev, D.R., A Common Theory of Information Fusion from Multiple Text Sources Step One: Cross-Document Structure. Proceeding SIGDIAL, Vol. 10, pp Zhang, Z., Blair-Goldensohn, S., and Radev, D.R., Towards CST-Enhanced Summarization. In Proceedings of AAAI/IAAI, pp Jorge, M.L.C., Pardo, T.S., Experiments with CST-based Multidocument Summarization, Workshop on Graphbased Methods for Natural Language Processing, ACL. Uppsala, Sweden, pp Neal, James M. and Suzanne S. Brown Newswriting and Reporting. Surjeet Publications, Delhi. Moens, M., Information Extraction: Algorithms and Prospects in a Retrieval Context, The Information Retrieval Series, Springer-Verlag, Secaucus, NJ. Wimalasuriya D. C. and Dou, D., Ontology-Based Information Extraction: An Introduction and a Survey of Current Approaches. Journal of Information Science, pp Cunningham, H., Bontcheva, K., Tablan, V. and Maynard, D., GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics (ACL'02). Philadelphia. Zahri, N.A.H.B. and Fukumoto, F., Multi-document Summarization Using Link Analysis Based on Rhetorical Relations between Sentences. In Proceedings of CICLing, Vol. 2, pp Lin, C.Y., ROUGE: A Package for Automatic Evaluation of Summaries, In Proceedings of Workshop on Text Summarization of ACL, Spain. Radev, D.R., Otterbacher, J., CSTBank PhaseI. Available from: 66