UCSG Shallow Parsing: Optimum Chunk Sequence Selection

UCSG Shallow Parsing: Optimum Chunk Sequence Selection B Hanumantha Rao and Kavi Narayana Murthy Department of Computer and Information Sciences, University of Hyderabad, Hyderabad, India. hanu 499@yahoo.com,knmuh@yahoo.com Abstract This paper is about syntactic analysis of natural language sentences. The focus is on wide coverage partial parsing architectures. In this work we enhance and enrich the UCSG shallow parsing architecture being developed here over the last many years. UCSG architecture combines linguistic grammars in the form of Finite State Machines for recognising all potential chunks and HMMs to rate and rank the chunks so produced. Here we have explored Mutual Information statistics for rating and ranking chunks as also complete parses (chunk sequences) so as to place the best parses near the top. The main aim of this work is to identify the best word group (chunk) sequence or global parse for a given sentence using a information-theoretic measure called mutual information. This method is based on the hypothesis that the best chunk can be obtained by analysing the mutual information values of the chunk tag n-grams. In the initial version of UCSG, HMMs were local to chunks. Global information such as the probability of a chunk of a given type starting a sentence or the probability of a chunk of a particular type occurring next to a chunk of a given type are also useful. We try to capture this global information in the form of mutual information score and use it in improving the ranks of correct chunks. Combining the two methods, namely HMMs and Mutual Information, to get the full information of a chunk to improve the ranks further, is another important aspect of this work. Later, a Best First Search module uses these ranked chunks to find the final parse. We have also added 1000 sentence manually parsed corpus to the existing 4000 manually parsed data. Experiments on the existing and newly added text corpora are included. 1 Introduction Developing high performance, wide coverage computational grammars and parsers has proved to be a very difficult task. Hence the recent interest in shallow or partial parsing systems. Shallow or Partial Parsing means The task of recovering only a limited amount of syntactic information from natural language sentences. Shallow parsing does not attempt to resolve all semantically significant decisions. Most of the times it is restricted to identifying phrases in the sentence. Partial syntactical information can help to solve many natural language processing tasks, such as information extraction, text summarisation, machine translation and spoken language understanding. Purely linguistic approaches have not proved practicable for developing wide coverage parsers and purely machine learning approaches are also impracticable in many cases due to non-availability of large enough training corpora. Only a judicious combination of the two approaches can lead to wide coverage, robust parsing systems. In this work, we are using the UCSG shallow parser architecture being developed at University of Hyderabad, which combines linguistic constrains expressed in the form of finite state grammars with statistical rating using HMMs built from a large POS-tagged corpus and a Best First search for global optimisation for determining the best parser for the give sentence [3, 2]. The UCSG shallow parsing architecture is depicted in figure1. The input to the system is either a plain or a POS tagged sentence. The aim is to produce all possible parses in ranked order hoping to get the best parse to the top. In this work by parse we mean a sequence of chunks. Chunks are sequence of words. A chunk or a word group as we prefer to call it in UCSG, is a structural unit, a non-over lapping and non-recursive sequence of words, that can as a whole, play a role in some prediction [3, 2]. 1

UCSG is a three module architecture. The first module is a Finite State Grammar (FSM) Parser. The goal of this module is to accept all valid word groups but not necessarily the only word groups that are appropriate in context for a given sentence. We do not restrict or prune the various possibilities. Instead we use a separate module to rate and rank these word groups later. The second module of this architecture uses a set of Hidden Markov Models (HMM) for rating and ranking the word groups produced by the FSM module. We can of course limit the number of parses generated if required but the ability to produce all possible parses is fundamental to the architecture. Note that we do not produces all possible parses first and then rate and rank them. Instead we attempt to produce the parses in best first order. An example is given below: input sentence: Savitri is loved by sita. ##<NP0><Savitri>##<VBZ><is>##<VVN_VVD><loved> ##<PRN_AVP_PRP><by>##<NP0><Sita>## Chunks got after FSM : <ng><0-1><np0><savitri> <vg><1-2><vbz><is> <vg><1-3><vbz><is>##<vvn><loved> <vg><1-4><vbz><is>##<vvn><loved>##<avp><by> <vg><2-3><vvd><loved> <vgs><2-3><vvn><loved> <vg><2-4><vvd><loved>##<avp><by> <ajg><2-3><vvn><loved> <vgs><2-4><vvn><loved>##<avp><by> <ng><3-5><prp><by>##<np0><sita> <part><3-4><avp><by> <ng><4-5><np0><sita> Figure 1. UCSG Shallow Parsing Architecture The aim here is to assign the highest rank for the correct chunk and to push down other chunks. Since a final parse is a sequence of chunks that covers the given sentence with no overlaps or gaps, we evaluate the alternatives at each position in the sentence in a left-to-right manner. We evaluate the second module using Mean Score, which is the mean of the distribution of ranks of correct chunks produced for a given training corpus. Ideally, all the correct chunks would be at the top and hence the score would be 1. The aim is to get Mean Score as close to 1 as possible. We have used a manually parsed corpus of 4000 sentences developed by us to evaluate the Mean Score. The third module is for identifying the best chunk sequence or global parse for a given sentence. This module generate all possible parses hopefully in best first order. Words and chunks in a sentence are referred to in terms of the positions they occupy in the sentence. Positions are marked between words, starting from zero to the left of the first word. The very first word is between positions 0 and 1. A word group containing the third and fourth words in the sentence can be referred as W 2,4. The following steps describe how we map a given sentence and word groups present in the sentence into a graph. The positions in the sentence are treated as nodes of the resulting graph. If a sentence contains N words then the graph contains N + 1 nodes corresponding to the N + 1 positions in the sentence. Word group W i,j is represented as an edge form node i to node j. The probability of a word group W i,j given by HMM module and the transition probability from previous word group type to current word group type are combined to estimate the cost of an arc between the nodes i and j. We always start from the initial node 0. Length of the sentence N is the goal node. Now our parse selection problem of a sentence containing N words becomes the task of finding an optimal path from node 0 to node N.

Choosing the locally best chunks at each position in a given sentence does not necessarily give us the best parse (chunk sequence) in all cases. The HMMs used in the initial version of UCSG are local to chunks. They use only with-in-chunk information for calculating the score of a chunk, ignoring the global information such as the probability of a chunk of a given type starting a sentence or the probability of a chunk of a particular type occurring next to a chunk of a given type. In this work, we have used Mutual Information statistics to capture this global information. 2 Manually Parsed Corpus Development We have a manually parsed corpus of 4000 sentences already developed in our centre here. In order to add to this effort and to test the performance of the system on new data, we have developed an additional 1000 sentence manually parsed corpus, taking plain sentences from Reuters corpus. This corpus is thus very useful for evaluating the various modules of the parsing architecture and also for bootstrapping. This corpus was developed by parsing the sentences using the UCSG shallow parser for English and then manually checking the top parse and making corrections where required. We felt this was far easier than parsing the sentences entirely by hand. 3 Rating and ing Chunks using Mutual information 3.1 Mutual Information In his treatise on information theory Transmission of Information [4] Fano discusses mutual information as a measure of the interdependence of two signals in a message. In probability theory and information theory, mutual information between two random variables is a quantity that measures the mutual dependence of the two variables. This bigram mutual information is a function of the probabilities of the two events: MI(x, y) = log Pr(x, y) Pr(x) Pr(y) Consider these events not as signals but as a sequence of part-of-speech tags in a sentence. Then an estimate of mutual information of two categories xy is: #xyincorpus total#of bigramsincorpus M I(x, y) log ( ) ( ) #x #y corpussize corpussize We can thus take advantage of context in determining the rank of particular chunk using pairs of tokens or bigrams. 3.2 Rating and ing chunks In UCSG shallow parsing Architecture [1] the first module produces all possible chunks according to the given finite state grammar. Due to lexical ambiguities and inherent ambiguities in the grammar, we often produce many alternatives. By giving the top rank to the correct chunk and pushing down other chunks we can improve the performance of the parser. At each possible branching point or position in a sentence we find all the chunk types on either side of the point, calculate the bigram mutual information score between each pair of chunk types and associate this score with the right side chunk. In the final module, we start form left and select the chunk on the right side with a high score. This goes on till the end to complete one single parse. The following steps describe you how we calculate MI score of any chunk: At each position i in the sentence, find all the chunk groups ending with i and all the chunk groups beginning with i. Let W j1,i, W j2,i, W j3,i...w jn,i are all chunks ending with i and W i,k1, W i,k2, W i,k3...w i,km are all chunks beginning with i. MI(x,y) gives us the mutual information score between x and y. MI score of any chunk group which is beginning with i is calculated by combining all the MI scores of chunk groups which are ending with i. MI(W i,k1 ) = n p=1 MI(W jp,i, W i,k1 ) MI(W i,k2 ) = n p=1 MI(W jp,i, W i,k1 ) MI(W i,k3 ) = n p=1 MI(W jp,i, W i,k1 ) : : MI(W i,km ) = n p=1 MI(W jp,i, W i,km )

Initially we calculate the score of a chunk using bigram mutual information, which only checks the chunk types. This way, we are giving equal weight to the chunks of same type even though they were differ in terms of length. Thus, differentiating long and short chunks of same type becomes a difficult task. We have therefore extended the set of chunk types to include the chunk lengths. Now noun groups of 3 words, for example, will be treated as different from a noun group of 4 words. In this work, we have used the term LMI to denote MI applied to this new, refined set of chunk types. Note that the basic mutual information technique itself is not changed, we are only applying MI to a more fine grained set of chunk types. Our observations show that LMI is better than the general MI. LMI gives better Mean Score than general MI or HMM alone. However, it gives less recall compared to the HMM method. This result encourages us to combine the two techniques. By combining the two techniques we will hopefully get the full information of any single chunk - both the local as well as the global information. 3.3 Weighted Voting Technique We find that HMM is giving good recall while LMI improves the Mean score. The best combination of these techniques can be helpful in improving the whole system further. A voting system is a means of choosing between a number of options, based on their performance. Let W1, W2 be the two methods and let f1, f2 be the weights. We maximise W1*f1+W2*f2. 3.3.1 Results Since we need high Recall and a low (close to 1) Mean Score, we can search for optimum weights by defining a score that combines these two requirements: CombinedScore = M R 25 + Recall. Performance for the original version of UCSG which uses only HMMs is given first: Table 1. Performance of the HMM Module on the Manually Parsed Corpus of 4000 sentences - Plain Sentences as Input Cutoff Mean Recall Combined Score Score (%) [MR*-25+Recall*1] 1 1 47.25 22.25 2 1.35 72.81 39.03 3 1.60 85.95 45.88 4 1.76 92.20 48.06 5 1.87 95.30 48.53 6 1.94 96.97 48.42 7 1.99 97.85 48.17 8 2.01 98.36 47.89 9 2.04 98.67 47.66 10 2.06 98.93 47.40 We give below the performance of the system using an optimum weighting based on extensive experimentation. We find that it is useful to make the LMI weight a function of the length of the chunks. Best results have been obtained when the weight is half of the chunk length. Table 2. Performance of the Weighted Voting technique(hmm+lmi) Module on the Manually Parsed Corpus of 4000 sentences - Plain Sentences as Input Cutoff Mean Recall Combined Score Score (%) [MR*-25+Recall*1] 1 1 51.00 36.76 2 1.32 75.06 42.05 3 1.55 86.75 48.08 4 1.69 92.49 50.01 5 1.80 95.47 50.41 6 1.87 97.05 50.28 7 1.91 97.85 50.03 8 1.94 98.37 49.75 9 1.96 98.69 49.50 10 1.98 98.92 49.26 We see that both the Mean Score and the Recall have improved, as also the Combined Score. 4 Optimum Chunk sequence Selection The parse generation module has been evaluated on the manually parsed corpus in terms of rank of the fully correct parse and also in terms of percentage of correct chunks in

the top parse. Plain sentences and POS tagged sentences have been considered separately for input. The results are summarised in tables given below. Here, we have restricted the parsing time taken by the best first search algorithm to 10 epoch seconds for each sentence because the time and space complexity increases exponentially as branching factor (b) and length of the sentence (n) increases. 4.1 Experiments and Results Table 4. Performance of the Best First Search Module after using MI in UCSG - Test Data of 4000 Sentences Plain POS tagged Sentences Sentences 1 1077 1967 2 368 522 3 175 178 4 87 143 5 74 95 % of Correct parses 60.09 69.20 % of Correct chunks 79.77 84.29 Total Recall 60.00 91.25 In first experiment we have taken 4000 plain and POS tagged sentences from the manually parsed corpus developed by us here for evaluating the two systems. We give below the results. The performance of the original system using only HMMs is included for comparison. Table 3. Performance of Best First Search Module in initial version of UCSG - Test Data of 4000 Sentences Plain POS tagged Sentences Sentences 1 1194 2194 2 343 487 3 153 163 4 76 124 5 58 88 % of Correct parses 62.45 72.27 % of Correct chunks 84.033 88.31 Total Recall 55.15 88.42 All these experiments have been done using epoch time limit of 10 seconds. For training MI we have used 4000 manually parsed corpus. We observe that performance has not improved. MI improved Mean Scores. We observe that the ratings of good chunks are improved but the ratings for other chunks are also improved. As a result, the performance of the final chunk sequence generation module does not necessarily improve. We find that this is because of sparse data problem. More experimentation is required to find optimal solutions. In order to check for corpus effects, we have taken 1000 sentences from Reuters News Corpus for testing. For training we have used 4000 manually parsed corpus. The results are given below. Table 5. Performance of the Best First Search Module - Test Data of 1000 Sentences (Initial Version) (UCSG with MI) 1 227 162 2 75 88 3 52 50 4 26 24 5 13 19 % of Correct parses 63.68 60.54 % of Correct chunks 83.5 77.05 Total Recall 50.42 50.41

Again we see that we have not been able to obtain better performance in the final parse. It has been shown [2] that the POS tags of the top parse are largely correct. Here we explore the idea of using the UCSG shallow parser as a POS tagger. We run the parser twice, first time only for obtaining the POS tags for words as given the top parse. The results are given below: Table 6. Performance of the Best First Search Module using existed system as POS tagger- Test Data of 4000 Sentences Initial Version UCSG with MI 1 1184 1076 2 179 204 3 40 46 4 18 33 5 17 22 % of Correct parses 57.38 56.81 % of Correct chunks 83.45 79.01 Total Recall 55.15 60 Because of time limits, we are able to get only a small portion of the sentences parsed. Most of the POS tags in the top parse are correct. There will be a few errors and hence when we parse the same sentences a second time, the performance in terms of accuracy will remain more or less the same as original with the small reduction explained out by the POS tag errors. Of course the second pass with POS tagged sentences as input will be much faster. 5 Significant Observations The following observations relate to the two versions specified below: System A: Initial version of the UCSG parser which uses HMMs only for rating and ranking the chunks System B: New version of the UCSG parser which uses the optimal combination of LMI and HMM for rating and ranking the chunks. Case1: Correct chunks have been given bad ranks (other than first) in system A. This is the most common case. We give some examples here. All the phrases we show here are in the format: Each entry includes the chunk type, the starting and ending positions, the chunk itself with the POS tags of all the words, log probability given by HMM, rank, number of items in the set, and the serial number of the branching points. The correct word group is will have taken. Phrases got in System B: <vg><6-9><vm0><will>##<vhb><have>##<vvn><taken> <7.313257557><1><5><7> <vg><6-8><vm0><will>##<vhb><have> <5.477835005><2><5><7> <vg><6-7><vm0><will><0.848599753><3><5><7> <ng><6-7><np0><will><-4.9158645><4><5><7> <ng><6-7><nn1><will><-9.8589973><5><5><7> <vg><7-9><vhb><have>##<vvn><taken> <3.26479549255><1><2><8> <vg><7-8><vhb><have><1.3058264337><2><2><8> <ajg><8-9><vvn><taken><-8.0795313><1><2><9> <vgs><8-9><vvn><taken><-9.2045137><2><2><9> Phrases got in System A: <vg><6-7><vm0><will><-6.28724041706727><1><5><7> <vg><6-8><vm0><will>##<vhb><have> <-7.67284079772121><2><5><7> <ng><6-7><np0><will><-9.3417681><3><5><7> <vg><6-9><vm0><will>##<vhb><have>##<vvn><taken> <-11.9799212377966><4><5><7> <ng><6-7><nn1><will> <-14.2849009073139><5><5><7> <vg><7-8><vhb><have><-4.82147060423909><1><2><8> <vg><7-9><vhb><have>##<vvn><taken> <-9.12855104431446><2><2><8> <vgs><8-9><vvn><taken> <-10.0745943157187><1><2><9> <ajg><8-9><vvn><taken> <-16.4378927128617><2><2><9> of correct chunk: System B rank : 1 System A rank : 3 Final Parse : In both systems the top parse has the correct chunk, even though system A fails to give rank 1 to the correct chunk Reason: The combined score of will, have and taken is worse than the score for will have taken and hence the correct chunk will be selected even if it is not at the top. Case2: This is the reverse of case 1. Here system A gives the correct rank but not System B. For example, The correct chunk sequence is neither abdul nor karim For the first chunk of the above sequence neither

Phrases got in System B: <ng><0-4><dt0><neither>##<np0><abdul> ##<CJC><nor>##<NP0><karim><1.47619><1><4><1> <ng><0-2><dt0><neither>##<np0><abdul> <-0.478724618262946><2><4><1> <avg><0-1><av0><neither><-2.9116095><3><4><1> <ng><0-1><pnp><neither><-4.86802762><4><4><1> <ng><1-4><np0><abdul>##<cjc><nor>##<np0><karim> <6.01487797><1><2><2> Phrases got in System A: <avg><0-1><av0><neither><-6.9307659><1><4><1> <ng><0-1><pnp><neither><-10.6992615><2><4><1> <ng><0-2><dt0><neither>##<np0><abdul> <-13.174931478><3><4><1> <ng><0-4><dt0><neither>##<np0><abdul> ##<CJC><nor>##<NP0><karim><-23.9099><4><4><1> <ng><1-2><np0><abdul> <-3.4382><1><2><2> <ng><1-4><np0><abdul>##<cjc><nor>## <NP0><karim><-14.173283081881><2><2><2> System B rank : 3 System A rank : 1 Final Parse: In both systems the top parse has the correct chunk, even though System B fails to give rank 1 Reason: Similar to case 1 Case 3: This is the case Where all the correct chunks have been given 1st rank. Yet the top parse is not the correct parse The Correct Chunk Sequence is have you seen ganguli the artists drawings Phrases got in System B: <vg><0-1><vhb><have><-1.6157><1><1><1> <ng><1-2><pnc><you><3.6783><1><1><2> <ng><2-4><vvn><seen>##<np0><ganguli> <1.3770835165><1><3><3> <vgs><2-3><vvn><seen><-3.082><2><3><3> <ajg><2-3><vvn><seen><-10.007><3><3><3> <ng><3-4><np0><ganguli><2.363><1><1><4> <ng><4-7><at0><the>##<nps><artist s>## <NN2><drawings><3.3364937><1><1><5> Phrases got in System A: <vg><0-1><vhb><have><-4.8214706><1><1><1> <ng><1-2><pnc><you><-3.70164664><1><1><2> <vgs><2-3><vvn><seen><-10.02640><1><3><3> <ng><2-4><vvn><seen>##<np0><ganguli> <-11.4226004><2><3><3> <ajg><2-3><vvn><seen><-16.389707><3><3><3> <ng><3-4><np0><ganguli><-3.4382><1><1><4> <ng><4-7><at0><the>##<nps><artist s> ##<NN2><drawings> <-16.171527555><1><1><5> System B rank: Except one chunk (seen) all the other correct chunks are given rank1 System A rank: All the correct chunks have been given rank 1 Final Parse: Top parse is not the correct parse Reason: Sum of correct chunk (seenand ganguli) scores is less than the wrong chunk s (seen ganguli) score 6 Conclusions and Future work In this work we have explored the use of mutual information between chunk types for improving the UCSG shallow parsing architecture. Initial results are encouraging. More experimentation is required to get optimal results. We have also added 1000 more sentences to the manually parsed corpus. Using MI we can improve the ranks of correct chunks. We can use some threshold point to push the bad chunks further down. A variety of machine learning strategies as also sentence level linguistic constraints can be incorporated to further improve the chunk sequence selection. The partial parsing system is currently producing only chunk sequences as parse output. Thematic role assignment (such as subject and object) should be included. Further enhancements to handle clause structure would take it closer to a full syntactic parsing system. It is clear that if we reduce the sparse data effect we will get even better results. We can use linguists features with MI in selection of correct chunk. We can extend the bigram mutual information to generalised mutual information, where we take a window of chunk types in calculating MI between two chunks. We have worked with chunk type only. We can extend it to include the lexical information also. References [1] G. B. Kumar and K. N. Murthy. Ucsg shallow parser. Proceedings of CICLING 2006, LNCS, 3878:156 167, 2006. [2] D. M. Magerman and M. P. Marcus. Parsing natural language using mutual information statistics. pages 984 989, 1990. [3] K. N. Murthy. Universal Clause Structure Grammar. PhD Thesis, University of Hyderabad, 1995. [4] F. R. Transmission of information. 1961.