Feature Reduction Techniques for Arabic Text Categorization

Feature Reduction Techniques for Arabic Text Categorization Rehab Duwairi Department of Computer Information Systems, Jordan University of Science and Technology, Irbid, Jordan. E-mail: rehab@just.edu.jo Mohammad Nayef Al-Refai Department of Computer Science, Jordan University of Science and Technology, Irbid, Jordan. E-mail: refai2006moh@yahoo.com Natheer Khasawneh Department of Computer Engineering; Jordan University of Science and Technology, Irbid, Jordan. E-mail: natheer@just.edu.jo This paper presents and compares three feature reduction techniques that were applied to Arabic text. The techniques include stemming, light stemming, and word clusters. The effects of the aforementioned techniques were studied and analyzed on the K-nearest-neighbor classifier. Stemming reduces words to their stems. Light stemming, by comparison, removes common affixes from words without reducing them to their stems. Word clusters group synonymous words into clusters and each cluster is represented by a single word. The purpose of employing the previous methods is to reduce the size of document vectors without affecting the accuracy of the classifiers. The comparison metric includes size of document vectors, classification time, and accuracy (in terms of precision and recall). Several experiments were carried out using four different representations of the same corpus: the first version uses stem-vectors, the second uses light stem-vectors, the third uses word clusters, and the fourth uses the original words (without any transformation) as representatives of documents. The corpus consists of 15,000 documents that fall into three categories: sports, economics, and politics. In terms of vector sizes and classification time, the stemmed vectors consumed the smallest size and the least time necessary to classify a testing dataset that consists of 6,000 documents. The light stemmed vectors superseded the other three representations in terms of classification accuracy. Introduction The exponential growth in the availability of online information and in Internet usage has created an urgent demand Received September 26, 2008; revised June 2, 2009; accepted June 3, 2009 2009 ASIS&T Published online 13 July 2009 in Wiley InterScience (www.interscience.wiley.com)..21173 for fast and useful access to information (Correa & Ludermir, 2002; Ker & Chen, 2000; Pierre, 2000). People need help in finding, filtering, and managing resources. Furthermore, today s large repositories of information present the problem of how to analyze the information and how to facilitate navigation to the information. This mass of information must be organized to make it comprehensible to people, and the most successful paradigm is to categorize different documents according to their topics. Text categorization, or text classification, is one of many information management tasks. It is a way of assigning documents to predefined categories based on document contents. Categorization is generally done to organize information automatically. The need of automated classification arises basically because of the paced growth and change of the Web, where manual organization becomes almost impossible without expending massive time and effort (Pierre, 2000). Information retrieval, text routing, filtering, and understanding are some examples of wide range applications for text categorization (Dumais, Platt, Heckerman, & Sahami, 1998). Many categorization algorithms have been applied to text categorization, for example, the Naïve Bayes probabilistic classifiers (Eyheramendy, Lewis, & Madiagn, 2003), Decision Tree classifiers (Bednar, 2006), Neural Networks (Basu, Walters, & Shepherd, 2003), K-nearest-neighbor classifiers (KNN) (Gongde, Hui, David, Yaxin, & Kieran, 2004) and Support Vector Machines (Sebastiani, 2005). With the increasing size of datasets used in text classification, the number and quality of features provided to describe the data has become a relevant and challenging problem. There is a need for effective feature reduction strategies (Yan et al., 2005; Yang & Pedersen, 1997). Some standard feature JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 60(11):2347 2352, 2009

reduction processes can initially be used to reduce the high number of features such as eliminating stopwords, stemming, and removing very frequent/infrequent words. Feature selection strategies discover a subset of features that are relevant to the task to be learned and that causes a reduction in the training and testing data sizes (Seo, Ankolekar, & Sycara, 2004). The classifier built with only the relevant subset of features would have better predictive accuracy than the classifier built from the entire set of features (Mejia-Lavalle & Morales, 2006). If we keep working on a high dimension dataset space, two main problems may arise: computational complexity and overfitting. In this paper the researchers present and compare three heuristic feature selection measures for Arabic text: stemming, light stemming, and word clusters. Stemming reduces words to their stems (roots). Light stemming, on the other hand, removes common affixes from words without reducing them to their roots. Word clusters partition the words that appear in a document into clusters based on the synonymy relation. Afterwards each cluster is represented by a single word (called cluster representative). The effects of the above three techniques, in addition to the case of using the original words of a document, such as feature selection techniques, were assessed for text categorization. The assessment framework includes comparing the document vector sizes, preprocessing and classifications times, and classifier accuracy. The KNN classifier was applied to an Arabic dataset. The dataset consists of 15,000 Arabic documents; the documents are collected, filtered, and classified manually into three categories: Sports, Economics, and Politics. The smallest vector sizes and the smallest time were achieved in the case of stemming, while the highest classifier accuracy in terms of precision and recall was obtained in the case of light stemming. This paper is organized as follows: The first section is the introduction; the second section describes the proposed framework, which consists of feature selection techniques and the classification subsystem. The third section presents and analyzes the results of this paper. Finally, the last section summarizes the conclusions and briefly highlights future work. System Framework The Proposed Feature Selection Measures Stemming algorithms are needed in many applications such as natural language processing, compression of data, and information retrieval systems. Very little work in the literature utilizes stemming algorithms for Arabic text categorization, such as the work of Sawaf, Zaplo, and Ney (2001), and the work of Elkourdi, Bensaid, and Rachidi (2004), and Duwairi (2006). Applying stemming algorithms as a feature selection method reduces the number of features because lexical forms (of words) are derived from basic building blocks and, hence, many features that are generated from the same stem are represented as one feature (their stem). This technique reduces the size of document vectors and increases the speed of learning and categorization phases for many classifiers, especially for classifiers that scan the whole training dataset for each test document. The stemming algorithm of Al-Shalabi, Kanaan, and Al-Serhan (2003) was followed here as a feature selection method. Arabic words roots consist of three letters. Very few words have four, five, or six letters. The algorithm reported in Al-Shalabi et al. (2003) finds the three-letter roots for Arabic words without depending on any root or pattern files. For example, using Al-Shalabi et al. s algorithm would reduce the Arabic words which mean the library, the writer, and the book, respectively, to one stem, which means write. The main idea for using light stemming is that many word variants do not have similar meanings or semantics. However, these word variants are generated from the same root. Thus, root extraction algorithms affect the meanings of words. Light stemming, by comparison, aims to enhance the categorization performance while retaining the words meanings. It removes some defined prefixes and suffixes from the word instead of extracting the original root (Aljlayl & Frieder, 2002). For example, the word means the book and the word means the writers ; they are extracted from the same root write, but they have different meanings. Thus, the stemming approach reduces their semantics. The light stemming approach, on the other hand, maps the word which means the book to which means book, and stems the word which means the writers to which means writer. Light stemming keeps the words meanings unaffected. We applied the light stemming approach of Aljlayl and Frieder (2002) as a feature selection method. The basis of their light stemming algorithm consists of several rounds that attempt to locate and remove the most frequent prefixes and suffixes from the word. Word clustering aggregates synonymous words, which have various syntactical forms but have similar meanings, into clusters. For example, the verbs and have similar meaning, which is run, and therefore would be aggregated into a single cluster even though they have different roots. Classifiers cannot deal with such words as correlated words that provide similar semantic interpretations. A word cluster vector is created for every document by partitioning the words that appear in that document into groups based on their synonymy relation. Every cluster is then represented by a single word the one that is commonly used in that context. To alleviate minor syntactical variations among words in the same cluster, the words were light stemmed. Using this approach, a document vector would consist of cluster representatives only and their frequencies. This fundamentally reduces the size of document vectors. The distribution of words into clusters is performed by carrying out a thesaurus lookup for every word that appears in a document. If that word matches a cluster (list) in the thesaurus then that word is replaced by that list s representative. Since the dataset consists of three categories, sports, politics, and economics, we built a thesaurus for Arabic terms that are related to these topics. Many resources were utilized 2348 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2009

FIG. 1. The main modules in the system. to build this thesaurus such as the following Arabic dictionaries:,,,, and other electronic resources such as, Microsoft Office 2003 thesaurus, and many Internet sites that provide information about the Arabic terms of politics, sports, and economics, or their synonymous terms. The Classification Subsystem The motivation of our research is to assess the effects of feature reduction methods on the KNN classifier. Therefore, the documents in the dataset were processed and represented in four versions and then the classifier was run on each version. Each document in the dataset was represented using the following four vectors: A stem vector: where words that appear in the document were reduced to their stems. A light stem vector: where words are processed by removing common affixes using the algorithm described previously (Aljlayl & Frieder, 2002). A word clusters vector where synonymous words that appear in the document are represented by a single word that is their cluster representative. A word vector where words of a document are used as is, without any transformation. Figure 1 shows the main modules in the system. The following paragraphs describe each of these modules. The preprocessor: preprocessing aims to represent documents in a format that is understandable to the classifier. Common functions for preprocessors include document conversion, stopword removal, and term weighting. The stemmer and light stemmer modules apply stemming and light stemming to the keywords of a document, respectively. The word cluster module groups synonymous words. After each transformation the keywords are weighted using term frequency (TF). The KNN classifier takes as input a test document (represented using the four vectors described above) and assigns a label to that document by calculating its similarity to every document in the training dataset. The training dataset was also represented using the four representational vectors used for test documents. The label of the test document is determined based on the labels of the closest K neighbors to that document. The best value of K was 10 (for this dataset) and it was determined experimentally. Experimentation and Results Analysis Dataset Description and Vector Sizes The dataset consists of 15,000 Arabic text documents. These documents were prepared manually by collecting them from online resources. The dataset was filtered and classified manually into three categories: politics, sports, and economics. Each category consists of 5,000 documents. The dataset was divided into two parts: training and testing. The testing data consist of 6,000 documents, 2,000 documents per category. The training data, on the other hand, consist of 9,000 documents, 3,000 documents per category. Every document was represented by four vectors depending on the feature reduction technique employed. In particular, word, word clusters, stemmed, and light stemmed vectors. Table 1 describes the characteristics of the four versions of document vectors. The purpose of this table is to show that feature reduction methods reduce the dataset size, and hence minimize the required memory space to handle the dataset. As can be seen from the table, the stemmed vectors consumed the least space (35 MB) and the smallest number of features (5,341,696). This is expected, as stemming reduces several words to one stem. The largest vectors in terms of size and JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2009 2349

TABLE 1. The properties of the training dataset in terms of size and number of features. Training dataset Size in Total number of keywords version megabytes for 9,000 documents Word vectors 86 MB 8,987,278 Stemmed vectors 35 MB 5,341,696 Light-stemmed vectors 59.5 MB 7,092,884 Word-cluster vectors 57 MB 6,845,538 TABLE 2. The elapsed classification time (in seconds) for 6,000 test documents. Preprocessing Classification Total Experiment time time Time Word vectors 1,778 12,013 13,791 Stemmed vectors 1,252 9,773 11,025 Light-stemmed vectors 1,433 10,886 12,319 Word-cluster vectors 1,881 10,155 12,036 number of features were the word vectors (where no feature reduction technique was employed). Again, this is expected, as words with minor syntactical variations would each be represented as a distinct keyword on its own. Finally, the light-stemmed vectors consumed (59.5 MB) with (7,092,884) features. This is slightly higher than the stemmed vectors as only certain prefixes and suffixes are removed from the words before storing them in their corresponding vectors. Wordcluster vectors fall between the stemmed and light stemmed versions. Preprocessing and Classification Times The experiments were performed by categorizing the test documents (6,000 documents) in four cases based on the feature reduction method utilized. In every case the KNN classifier was used. The experiments were carried out on a Pentium IV personal computer (PC), with a RAM of size 256 MB. Table 2 shows the elapsed preprocessing and classification times for all test documents. Preprocessing time depends on the activities performed during this process. In our work, preprocessing includes the removal of punctuation marks, tags, and stopwords, which is common to all experiments. Preprocessing also includes term weighting, which is common to all experiments but also depends on the feature reduction algorithm. Terms weighting time is proportional to the number of terms in a given document: the more terms the higher the preprocessing time. Table 2 shows that the lowest preprocessing time was achieved in the case of stemming. This is because the size of the vectors is smaller when compared with the other three vector types. Also, the stemming algorithm utilized is efficient in the sense that it needs to scan a given word only once to deduce its stem (Al-Shalabi et al., 2003). The next-best preprocessing time was achieved in the case of light stemming. To a certain extent, stemming and light stemming are similar in the sense that they both need to process every word in a document either by running the stemming or light stemming algorithms. The worst preprocessing time was in the case of word clusters, as this requires accessing the thesaurus in addition to the document to create a document vector. Classification time indicates the time necessary to classify the 6,000 test documents using the KNN classifier. As can be seen from the table, classifying documents using the stemmed vectors needed the least time, as document vector sizes are rather small. Classifying documents using the light-stemmed and word-cluster vectors consumed 10,886 and 10,155 seconds, respectively. The two values slightly vary because vector sizes of the two methods are similar (see Table 1). Word vectors needed the largest time to classify the collection of test documents; again, this is due to the fact that word vectors are the largest. To sum up, classification time using the KNN classifier is proportional to document vector sizes: the smaller the vector sizes the smaller the classification time. The last column in Table 2 shows the total time. Total time is the sum of the preprocessing time and classification time. Stemmed vectors achieve the lowest total time. The largest time was achieved in the case of word vectors. Classifier Accuracy Versus Feature Reduction Techniques This section investigates the effects of feature reduction techniques on classifier accuracy. The accuracy of the classifier is judged by the standard precision and recall values widely used by the research community. These were originally used to evaluate information retrieval systems and are currently used to assess the accuracy of classifiers. Assume that we have a binary classification problem. This means we have only one category to predict (say, C). The sample of documents consists of both documents that belong to this category and documents that do not belong to the category. Let TP (true positives) be the number of documents that were classified by the classifier to be members of C and they are true members of C (human agrees with classifier). Let FN (false negatives) be the number of documents that truly belong to C but the classifier failed to single them out. Let FP (false positives) be the number of documents that were misclassified by the classifier to belong to C. Finally, let TN (true negatives) be the number of documents that were truly classified not to belong to C. Recall is defined as TP/(TP + FN) and Precision is given by TP/(TP + FP). Table 3 shows the precision values for the politics, economics, and sports documents that were fed to the KNN classifier. Every group of test documents was fed to the classifier four times: once in the form of word vectors, the second in the form of stemmed vectors, then as light-stemmed vectors, and finally as word-clusters vectors. As can be seen from the table the highest value of precision was achieved for politics documents, in the case of word cluster vectors. The precision for the light stemming case was slightly less than the word clusters case. The worst precision 2350 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2009

TABLE 3. Classifier accuracy (using precision) for word vectors, stemmed-vectors, light-stemmed vectors and word-clusters vectors. Word Stemmed Light-stemmed Word-clusters vectors vectors vectors vectors Politics 0.722 0.7448 0.83187 0.8404 Economics 0.7642 0.92433 0.9334 0.90938 Sports 0.95813 0.9873 0.9806 0.9763 TABLE 4. Classifier accuracy (using recall) for word vectors, stemmed-vectors, light-stemmed vectors, and word-clusters vectors. Word Stemmed Light-stemmed Word clusters vectors vectors vectors vectors Politics 0.9435 0.956 0.9475 0.9295 Economics 0.6565 0.7085 0.82 0.838 Sports 0.801 0.938 0.9635 0.9495 FIG. 2. Average precision of the classifier. FIG. 3. Average recall of the classifier. value for the politics documents was in the case of the word vectors. The highest precision value for the economics documents, by comparison, was achieved in the case of light-stemmed vectors; the lowest value was achieved in the case of the word vectors. Finally, the highest precision value for the sports documents was achieved in the case of light-stemmed vectors and the worst was in the case of word vectors. The conclusion is that using words without applying any stemming, light stemming, or word cluster algorithms result in the classifier being too sensitive to the minor syntactical variations of the same word and therefore these words are considered not correlated, which consequently adversely affects the precision of the classifier. Figure 2 depicts the average precision for all test documents (politics, economics, and sports); the best average value was achieved in the case of light stemming and the worst average value resulted in the case of word vectors. Table 4 shows the recall values for the three categories. The highest recall for politics documents was achieved in the case where the documents represented as stemmed vectors. The highest recall value for economics documents, by comparison, was achieved when the documents were represented as word clusters. Finally, the highest recall for sports documents was achieved in the case where the documents were represented as light-stemmed vectors. Figure 3 shows the average recall values for all test documents against the four employed feature reduction techniques. The two best values were achieved in the cases of light stemming and word clusters, respectively. Conclusions and Future Work In this study we applied three feature selection methods for Arabic text categorization. The Arabic dataset was collected manually from Internet sites such as Arabic journals. The dataset was filtered and classified manually into three categories: politics, sports, and economics. Each category consists of 5,000 documents. The dataset was divided into two parts: training and testing. The testing data consist of 6,000 documents, 2,000 documents for each category. The training data consist of 9,000 documents, 3,000 documents per category. The techniques used for feature selection are stemming (Al-Shalabi et al., 2003), light stemming (Aljlayl & Frieder, 2002), and word clusters. Stemming finds the three-letter roots for Arabic words without depending on any root or pattern files. Light stemming, on the other hand, removes the common suffixes and prefixes from the words. Word clusters group synonymous light stems using a prepared thesaurus and then chooses a light-stemmed word to represent the cluster. The KNN classifier was used to classify the test documents. The experiments were done in the following manner, the KNN classifier was run four times on four versions of the JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2009 2351

dataset. In the first version, a document is represented as a vector that includes all the words that appear in that document. In the second version, words of a given document are reduced to their stems (roots) then the corresponding document vector is created. In the third version the words that constitute a given document were light stemmed and then the corresponding vector is generated. In the final version, the words of a given document were grouped into clusters based on the synonymy relation and each cluster was represented by a single word (cluster representative). Afterwards, that document vector was formed by using cluster representatives only. Our experiments have shown that stemming reduces vector sizes, and therefore improves the classification time. However, it adversely affects the accuracy of the classifier in terms of precision and recall. The precision and recall reached their highest values when using the light stemming approach. These results are of interest to anyone working in Arabic information retrieval, text filtering, or text categorization. In the future we plan to extend this framework to include statistical feature selection techniques such as χ 2, information gain, and the Gini index (Shang et al., 2007; Shankar & Karypis, 2000). Finally, the thesaurus, which we used in this work, was built manually. We plan to improve this thesaurus by utilizing automatic algorithms and then screening the synonymy lists by language experts. References Aljlayl, M., & Frieder, O. (2002). On Arabic search: Improving the retrieval effectiveness via a light stemming approach. In Proceedings of the ACM 11th Conference on Information and Knowledge Management (pp. 340 347). New York: ACM Press. Al-Shalabi, R., Kanaan, G., & Al-Serhan, H. (2003, December). A new approach for extracting Arabic roots. Paper presented at the International Arab Conference on Information Technology (ACIT), Alexandra, Egypt. Basu, A., Walters, C., & Shepherd, M. (2003). Support vector machines for text categorization. In Proceedings of the 36th Annual Hawaii International Conference on System Sciences (pp. 103 109). Los Alamitos, California: IEEE Press. Retrieved July 2, 2009, from http://ieeexplore. ieee.org/stamp/stamp.jsp?tp=&arnumber=1174243&isnumber=26341 Bednar, P. (2006, January). Active learning of SVM and decision tree classifiers for text categorization. Paper presented at the Fourth Slovakian- Hungarian Joint Symposium on Applied Machine Intelligence, Herlany, Slovakia. Correa, R.F., & Ludermir, T.B. (2002, November). Automatic text categorization: Case study. Paper presented at the VII Brazilian Symposium on Neural Networks, Pernambuco, Brazil. Dumais, S., Platt, J., Heckerman, D., & Sahami, M. (1998). Inductive learning algorithms and representations for text categorization. In Proceedings of the Seventh International Conference on Information and Knowledge Management (pp. 148 155). New York: ACM Press. Duwairi, R.M. (2006). Machine learning forarabic text categorization. Journal of the American Society for Information Science and Technology, 57(8), 1005 1010. Elkourdi, M., Bensaid, A., & Rachidi, T. (2004). Automatic Arabic document categorization based on the naïve Bayes algorithm. In Proceedings of COLING 20th Workshop on Computational Approaches to Arabic Script-based Languages (pp. 51 58). Retrieved July 2, 2009, from http://www.arabicscript.org/w5/pdf/proceedings.pdf Eyheramendy, S., Lewis, D., & Madiagn, D. (2003). On the naïve Bayes model for text categorization. Paper presented at the Ninth International Conference on Artificial Intelligence and Statistics, Key West, FL. Gongde, G., Hui, W., David, A.B., Yaxin, B., & Kieran, G. (2004). An knn model-based approach and its application in text categorization. In A. Gelbukh (Ed.), Proceedings of the Fifth International Conference on Intelligent Text Processing and Computational Linguistics (CICLing) (pp. 559 570). Lecture Notes in Computer Science, Vol. 2945. Berlin/Heidelberg, Germany: Springer. Ker, S., & Chen, J. (2000). A text categorization based on summarization technique. In J. Klavans & J. Gonzalo (Eds.), Proceedings of the ACL-2000 Workshop on Recent Advances in Natural Language Processing and Information Retrieval, held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics (pp. 79 83). New Brunswick, NJ: The Association for Computational Linguistics. Mejia-Lavalle, M., & Morales, E. (2006). Feature selection considering attribute inter-dependencies. In International Workshop on Feature Selection for Data Mining: Interfacing Machine Learning and Statistics (pp. 50 58). Providence, RI: American Mathematical Society. Pierre, J. (2000, September). Practical issues for automated categorization of web pages. Paper presented at the 2000 Workshop on the Semantic Web, Lisbon, Portugal. Retrieved May 29, 2009, from http://citeseer.ist. psu.edu/pierre00practical.html Sawaf, H., Zaplo, J., & Ney, H. (2001, July). Statistical classification methods for Arabic news articles. Paper presented at the Arabic Natural Language Processing Workshop. Toulonse, France. Sebastiani, F. (2005). Text categorization. In A. Zanasi (Ed.). Text mining and its applications to intelligence, CRM and knowledge management (pp. 109 129). Southampton, UK: WIT Press. Seo, Y., Ankolekar, A., & Sycara, K. (2004). Feature selection for extracting semantically rich words. Technical Report CMU-RI-TR-04-18, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA. Shang, W., Huoang, H., Zhu, H., Lin,Y., Qu,Y., & Wang, Z. (2007). A novel feature selection algorithm for text categorization. Expert Systems with Applications, 33(1), 1 5. Shankar, S., & Karypis, G. (2000). A feature weight adjustment algorithm for document categorization. In Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM Press. Yan, J., Liu N., Zhang B., Yan S., Chen Z., Cheng Q., Fan W., & Ma W. (2005). OCFS: Optimal orthogonal centroid feature selection for text categorization. In Proceedings of the 28th Annual International ACM SIGIR Conference (SIGIR 2005) (pp. 122 129). New York: ACM Press. Yang, Y., & Pedersen, J. (1997). A comparative study on feature selection in text categorization. In J.D.H. Fisher (Ed.). The Fourteenth International Conference on Machine Learning (ICML 97) (pp. 412 420). San Francisco: Morgan Kaufmann. 2352 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY November 2009