Extending Sparse Classification Knowledge via NLP Analysis of Classification Descriptions

Extending Sparse Classification Knowledge via NLP Analysis of Classification Descriptions Attila Ondi 1, Jacob Staples 1, and Tony Stirtzinger 1 1 Securboration, Inc. 1050 W. NASA Blvd, Melbourne, FL, USA Abstract - Supervised machine learning algorithms, particularly those operating on free text, depend upon the quality of their training datasets to correctly classify unlabeled text instances. In many cases where the classification task is nontrivial, it is difficult to obtain a large enough set of training data to achieve good classification accuracy. In this work we examine one such case in the context of a system designed to ground free text to an organizational hierarchy which is ontologically modeled. We explore the impact of utilizing information garnered from a highly customized Natural Language Processing (NLP) analysis of this ontology to augment a very sparse initial training dataset and compare this to a more labor intensive extraction of a small set of key words and phrases associated with each concept. We demonstrate an approach with significant improvement in classifier performance for concepts having little or no initial training data coverage. Keywords: Hierarchical document classification, Automatic document classification, Machine learning, NLP, Ontology 1 Introduction In this work we describe a software classification system which employs NLP analysis and machine learning algorithms to automatically determine whether a human produced text applies to one or more of a set of concepts. The machine learning portion of this system is essentially a supervised learner whose accuracy relies heavily upon the quality of the training instances it has encountered. Unfortunately the concept space over which this solution is deployed is sufficiently large that it was not feasible to obtain a large quantity of high-quality training instances. In fact, many concepts were observed to never be explicitly referenced by training instances. Although the classification portion of the system evolves dynamically to improve its classifications over time, the initial classifications produced by the system were of poor quality due to this deficiency of initial training data. We discuss several approaches taken to improve these initial classifications and construct knowledge in the face of limited or absent training data. We compare two mechanisms of capturing expert user knowledge in terms of their impact on classifier accuracy and recall and how well they interact with traditional training instances. 2 Related Work From a machine learning perspective, this work deals with the well-studied field of supervised learning algorithms [1]. There is a wide body of research related to applying such algorithms to human produced text, for an overview see [2]. Of particular relevance is the multi-topic document classification problem [3] in which a document is analyzed by a machine classifier and determined to be applicable or not applicable to a list of topics. The sparseness of labeled training instances is by no means unique to this problem domain. Significant research has been conducted in the area of semi-supervised learners which attempt to generalize a small amount of labeled training data to a larger amount of unlabeled data which can subsequently be used for training. For a thorough overview of semi supervised techniques see [4]. In this work we examine a somewhat different position than that in which most semi-supervised systems are deployed, however, because we assume access to a limited number of labeled training instances but do not assume the existence of unlabeled training instances. Instead we assume access to expert generated knowledge either in the form of an ontology [5] modeling the structure of concepts in the concept space or in the form of a key-word document, either of which we attempt to derive unlabeled instances from and whose instance labels are trivial to reverse engineer from the ontology structure. 3 Approach We explore two unique options to expand the initial classifier knowledge. The first is to exploit the knowledge of subject matter experts by hand-populating a database of key phrases they have determined likely to be associated with each concept in the concept space. The second approach is to utilize an expert-created ontology describing the nature and relationship between the concepts in the concept space to determine which lexical features are associated with what concepts by making the assumption that the definition of a

concept will contain language similar to the documents associated with that concept. Though their angles of attack on this problem are quite different, it is important to keep in mind that these approaches are attempting to do fundamentally the same thing derive new training instances which can be provided to the classifier. Before describing the mechanics behind generating these instances, we will give a brief overview of the classifier and its operation. 3.1 Classifier The minutiae of the supervised learning classification algorithm utilized in this work are not germane to this discussion and are described elsewhere [10]. For the purposes of this paper, it is sufficient to understand the grounding engine mechanics for two key operations the classifier performs: classification and training. Classification is an operation performed on a text document to determine which concepts are present in it. In order to classify a given document as exhibiting or not exhibiting a set of concepts, the document is first decomposed into features using Natural Language Processing (NLP) techniques described later. These features are provided as input to the grounding engine. The output of the grounding engine is a label for each concept indicating how confident the classifier is that the concept exists in the document whose features were provided. Note that the features passed to the classifier are either sense-ambiguous (for example a stemmed form of a word with some notion of its part of speech) or sense-specific (for example a WordNet synset identifier). In this work we explore the implications of each feature type. Training is an operation performed on a text document and a set of labeled concepts indicating which are correctly or incorrectly associated with the text. When updated with a training instance, the classifier internally adjusts its structure such that in the future the features provided will be more likely to be associated with the concepts labeled correct and less likely to be associated with the concepts labeled incorrect. For clarity, we now introduce a logically distinct type of training, called bootstrap training, which differs slightly from the training described above. Bootstrap training is unique in that it is always performed before any standard training instances are encountered and is used to construct the initial body of knowledge needed to very roughly associate features with concepts. Because the bootstrap training labels in this work were implicitly derived, we could not make any assumptions about false negative concepts using a bootstrapping instance. When a bootstrap training instance was encountered, the classifier therefore only considered the concept arguments labeled as correct for the given text. Bootstrapping is useful because, as mentioned earlier, for many of the concept types on which our classification system operated, there were a significant number of concepts having a trivial number of training instances (or none at all). Without some form of correction, these holes in the initial knowledge of the classifier meant that the classification algorithm would always assign some concepts zero confidence. Bootstrapping is less crude than the typical approach of assigning each concept a small virtual probability. 3.2 Feature extraction using Natural Language Processing The features used as input to the classifier were extracted from document text via a NLP pipeline using the UIMA [6] framework and OpenNlp [7] components. The results of the OpenNlp components were augmented by a heuristic lemmatizer based on an extended WordNet [8] dictionary. 3.2.1 Custom WordNet dictionary WordNet is a machine-usable dictionary of words organized into synonym sets (synsets). Each word in the dictionary is associated with a sense, which is captured by a synset; and each synset is associated with a textual description, along with the words belonging to this synset. In this sense the WordNet dictionary assumes the role of a thesaurus as well. We modified WordNet in two ways to better fit the needs of the effort described here. The first modification was at the content level the dictionary lookup logic was modified to support customized synsets stored in a file. The addition of this custom read logic enables us to fully modify the WordNet dictionary by removing, altering, or creating new synsets in the dictionary. Nearly 1200 field-specific jargon and acronym terms were added to the dictionary. These 1200 terms were selected because they were important words previously discarded by the lemmatizer using a standard WordNet dictionary. After implementing the changes to WordNet the lemmatizer was able to recognize these terms as words. The second modification was at the software level the custom dictionary read logic was modified such that it loads all dictionary entries into memory (requiring around 350MB in our implementation). This change resulted in significant speedup over the file-based MIT implementation packaged with the dictionary for random word lookups. We observed a speedup of roughly 11x for random synset retrievals and 1.5x

for repeated synset retrievals compared to MIT s implementation. The lower speedup for repeated lookups is due to the use of caching in the MIT implementation, which mitigates costly file access times for dictionary entries which are frequently accessed. 3.2.2 Lemmatizer Although the standard OpenNlp components provide reasonable accuracy for determining the correct part-ofspeech (PoS) tags for words appearing in the document, the process is not perfect. To attempt to correct the erroneous PoS tags, we developed a custom component that uses simple heuristic rules to guess the potential textual form of the lemma of the word along with the correct PoS tag. The lemma of a word is simply the base form of the word that forms the head of an entry in a dictionary (e.g. the lemmas of the words ate and mice are eat and mouse, respectively). Our heuristic for finding the actual PoS tag for a word observed in a document uses the OpenNlp PoS tag guess for the word and the PoS tag of the previous non-filler word. If the custom dictionary contains a lemma/pos pair matching the known lemma and OpenNlp PoS tag guess, the OpenNlp tag guess is assumed to be correct. Otherwise, we consult the dictionary using the following PoS tags (in the given order): verb, noun, adjective and adverb. The first matching PoS from this list when paired with the lemma is returned as correct. Note that the heuristic rules of calculating the lemma from the textual form of the word change based on the current PoS tag candidate. If no lemma/pos tag matches an entry in the custom dictionary then the OpenNlp tag guess is assumed to be correct and the lemma is assigned the textual form of the word. 3.2.3 Sense disambiguation The sense disambiguation scheme utilized is based on the Lesk algorithm described in [9]. The algorithm outlined below operates on sentences extracted from the document, performing the steps for each unprocessed word in the sentence. If, at any step in the algorithm, only one viable sense of the word/pos remains, that sense is selected and the following steps are not performed. First, exclude senses that are hyponyms of non-applicable senses (e.g. sport terms). Second, check if senses from the current and another word are part of the same synset hierarchy; if yes, the corresponding senses are selected for both words. Third, check if the current and the neighboring word are related to each other via lexical parallelism. Two words exhibit lexical parallelism if there exists a hypernym/hyponym (for nouns), similar-to (for adjectives), pertains-to or morphological similarity (for adverb) relationship between any of their possible senses. If lexical parallelism is observed, the senses in relation are selected for the corresponding words. Fourth, if the current word is a verb, check if it is collocated with any of the neighboring word lemmas. A verb lemma is collocated with another lemma if the verb has a lemma in any of its hypernym synsets that is morphologically similar to the other lemma. Fifth, perform the Lesk algorithm: check for maximum lemma overlap between the current sentence and the textual description of the possible senses. Sixth, extract example usage from the sysnset textual definition and check for patterns with the neighboring words. If all steps were successfully performed and there are still multiple candidate synsets associated with the word, all the remaining synsets are accepted as equally likely correct senses. 3.3 Bootstrapping mechanisms We explored two options for generating the bootstrapping training set. For both options, we attempted to use both sense-ambiguous (lemma/pos pair) and sensespecific (synset ID) features. In the sense-specific case we resolved the senses of the bootstrapping training set manually and employed the automatic sense disambiguation mechanism described above during training. 3.3.1 Bootstrapping based on keyword mapping The first bootstrapping approach we took was to utilize a keyword mapping document which describes the key words and phrases associated with each concept in the concept space. To generate the bootstrapping training set, we created a set of single-feature training documents derived from the keyword mapping document. The label for the training instance contained in each of these documents was simply a list of the concepts correctly associated with the keyword. This approach scales poorly because a domain expert must distill each concept into a list of the most important phrases and terms associated with that concept. Furthermore, it is important that these phrases and words not overlap to provide maximum differentiation between the concepts. Avoiding overlap becomes increasingly difficult as the size of the concept space grows. Additionally, it is not clear whether the few key words or phrases selected by the expert will have sufficient lexical overlap with the contents of an arbitrary document to produce meaningful classification results.

3.3.2 Bootstrapping based on ontology lexicalization The second bootstrapping approach we explored was to utilize an ontology created by a domain expert that described the concepts in the concept space. To generate the training set, we created a single virtual training document for each concept. The training document consisted of the definition, description and other descriptive textual information extracted from the ontology describing the associated concept. The classification label associated with that document was the label of the concept from which the information was extracted. This approach scales better than manually generating the key word mapping document because the ontology is required to contain textual information (e.g. description) associated with each concept. The downside is that the training instances generated in this fashion may contain misleading knowledge since nonkeywords may be included in the ontology text fields. As an example of this: the definition of a concept might always begin with the word Definition which will lead classified documents containing the word Definition to be erroneously grounded to all concepts. 4 Results To gauge the performance of the various proposed mechanisms for training the classifier we conducted 50 runs with randomly generated training (T) and validation (V) sets for each run. Each validation set consisted of V randomly selected classification instances from a pool of C correctly classified documents (C = 63). The remaining T classification instances were used for training the classifier. The training and validation sets were selected such that they were guaranteed to be disjoint if possible (i.e., no document was validated using a classification instance previously used for training in the same run unless C < T + V ). The training set was augmented with instances derived using various combinations of the following approaches: 1. No bootstrapping 2. Human-created concept keywords 3. Lexicalized ontology For each validation instance the top 10 scoring concepts after classification were selected and the remainder was rejected. These 10 concepts were compared to the correct concept labels and the recall, precision, and fscore (the harmonic mean of the former two) values were computed as arithmetic averages over all runs. To more clearly demonstrate the problem we faced with limited training data, we first examine the impact of decreasing the size of the training set with no training augmentation. Figure 1: Impact of training set size on classifier performance with no training augmentation As expected, decreasing the size of the training set had a deleterious effect on precision, recall and fscore. Note that with the 1T/10V configuration the classifier s performance was actually slightly worse than if 10 concepts had been selected from the roughly 600 in the hierarchy at random. This worse-than-random behavior was due primarily to incomplete coverage of the concept space by the single training instance. Because our classification system frequently encountered instances where there were few or no initial training instances as in the 1T/10V configuration, we clearly needed a mechanism for bootstrapping classifier knowledge. To understand how effective bootstrapping can be on its own and with standard training, we tested the following training configurations: B: bootstrap training only B+T: bootstrap training followed by standard training T: standard training only We also explored the impact of sense disambiguation during parsing using the following bootstrapping configurations: Dk: hand disambiguation of keyword senses, automatic disambiguation of training instances Dd: hand disambiguation of concept ontology descriptions, automatic disambiguation of training instances

Ak: ambiguous keywords (no disambiguation of training instances) Ad: ambiguous concept ontology descriptions (no disambiguation of training instances) Note that the Dk and Dd configurations above employed sense disambiguation. To maximize the benefit of sense disambiguation, we manually disambiguated the bootstrapping data. Figure 2 shows the performance of various training configurations in terms of their fscores normalized to the unbootstrapped 63T/10V case from Figure 1. In Figure 2, we observe that bootstrapping alone (B) performs poorly relative to the training alone (T) and both (B+T) configurations. This is expected because the quality of the bootstrapping training instances is significantly lower than the quality of real-world generated training instances. Figure 2 : Impact of training configuration on classifier performance (63T/10V) It is clear from Figure 2 that bootstrapping alone is a poor substitute for quality training data but does provide significantly better recall than simply guessing (which would produce a recall of 3.3% on average). On its own the best performing bootstrapping configuration was (Dk) which utilized the hand generated key word document. We also observe that sense disambiguation (used in the Dk and Dd cases) only provided a benefit over using ambiguous features when bootstrapping alone (B) was used. The benefit of sense disambiguation in this case is 46% greater for keyword (Dk) than with lexicalized ontology (Dd) bootstrapping. This significant improvement over lexicalized ontology bootstrapping is a direct result of the highly targeted and minimally overlapping language used in the keyword document whereas the lexicalized ontology bootstrap instances were observed to contain many overlapping terms and phrases between unique concepts. The next interesting observation from Figure 2 is that the bootstrapping and training configuration (B+T) performed better than training alone (T). This indicates that bootstrapping was beneficial even for concepts covered by labeled training instances. To quantify the interaction between bootstrapping and normal training, we next varied the training set size and examined the performance of the (Ad, B+T) case above. Note that no sense disambiguation was used during bootstrapping or training and that lexicalized ontology bootstrapping was used. These results are shown in Figure 3 as improvement factors for recall, precision and fscore relative to the corresponding values for the training only (T) case from Figure 1. Figure 3: Relative performance of lexicalized ontology bootstrapping + training compared to training alone For all of the tested training set sizes shown above, bootstrapping resulted in a significant improvement in fscore relative to the trained only case. Intuitively, the benefit of bootstrapping decreases as the training set increases in size and this was also observed. It is interesting to note that bootstrapping improved precision appreciably more than recall in all cases. This is because, during the post bootstrapping training process, exact concept matches were fed back to the system with higher weights than close concept matches.

It should be noted that while the relative improvements in recall, precision, and fscore for the 1T/10V case were large, classifier recall, precision, and fscore at that point were roughly 32%, 74% and 45%, respectively. Although still far from perfect, this result was significantly better than the results obtained with few or no initial training instances. 5 Conclusion and Future Work We have explored several approaches which can be used to generate initial training data using domain expert knowledge. The most effective of these was building a bootstrapping training set using a lexicalization of an expert generated ontology describing the concept space. Our lexicalized bootstrapping mechanism was able to achieve an fscore of 40% even with no initial training instances present. Additionally, we observed 15% improvement in fscore values using bootstrapping and training together even for the most trained configuration tested. One possible improvement to the techniques described in this work would be to experiment with more sophisticated sense disambiguation schemes. Our efforts to leverage WordNet to exploit synset relationships such as hypernyms and hyponyms were unsuccessful largely because we were unable to accurately map words in text to WordNet synsets automatically. In such a system it would be possible to derive a large number of similar words to those in the bootstrapping training set using the synset hypernym and hyponym relational links and we anticipate improved performance for bootstrapping using these approaches. [5] N. Guarino. Formal Ontology in Information Systems ; Proceedings of FOIS 98, Vol. 46, 3 15, June 1998. [6] D. F. Lally. Uima: An architectural approach to unstructured information processing in the corporate research environment ; Natural Language Engineering, Vol. 10, 327 348, 2004. [7] J. Hockenmaier, G. Bierner and J. Baldridge. Extending the Coverage of a CCG System ; Research in Language and Computation, Special Issue on Linguistic Theory and Grammar Implementation, Vol, 2, Issue 2, 165 208, 2004. [8] G. A. Miller. WordNet: a lexical database for English ; Communications of the ACM, Vol. 38, Issue 11, 39 41, 1995. [9] M. Lesk. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone ; Proceedings of the 5th annual international conference on systems documentation (SIGDOC 86), 24 26, 1986. [10] J. Staples, A. Ondi and T. Stirtzinger. Semiautonomous hierarchical document classification using an interactive grounding framework ; Proceedings of the ICAI 12, (to appear), 2012 6 Acknowledgement This work was supported in part by the Air Force Research Laboratory, Contract Nos. FA8750-09-D-0195-0004, FA8750-09-D-0195-0006 and FA8750-08-C-0109 7 References [1] T. M. Mitchell. Machine Learning. McGrawHill, 1997. [2] M. Berry. Survey of Text Mining: Clustering, Classification, and Retrieval, First Edition. Springer, 2008. [3] H. B. Bernick. Automatic document classification ; Journal of the ACM, Vol. 10, Issue 2, 151 162, 1963. [4] N. Chawla and G. Karakoulas. Learning from labeled and unlabeled data: an empirical study across techniques and domains ; Journal of Artificial Intelligence Research, Vol. 23, 331 366, 2005.