Marathi POS Tagger Prof. Pushpak Bhattacharyya Veena Dixit Sachin Burange Sushant Devlekar IIT Bombay
About Marathi Language Marathi is the state language of Maharashtra, a province in the western part of India. Marathi is spoken by about 16 million people. It is 15th in the world in the population wise ranking list. It belongs to the Indo Aryan language family with many influences from Dravidian languages. fahd IIT Bombay
Part of Speech Tagger Basic aim of Part of Speech Tagging is to identify correct word cartogry(such as Noun,Verb etc) from a given sentence. Part of Speech (POS) tagging is a crucial step in any language processing system. Parsing, Machine Translation, Information Extraction- all these tasks have to employ POS tagging in the initial stages. fahd POS tagging has its own challenges, some of which are POS ambiguity, unknown words and Proper nouns. IIT Bombay
Part of Speech Tagger POS Tagging techniques overview : In reality this picture is more complicated IIT Bombay
IIT Bombay Modules completed in Marathi POS Tagger Verb Computation. (example बसत, ख त त,ख श ल) Conjunct Computation. (example आ ण,पण,पर त ) Interjection Computation.(अर व, ब प र,etc ) Noun Computation (Currently doing) (Example र हल, ब डक ल,ब ईल,ख ऊ )
IIT Bombay Marathi POS Tagger Modules. Tokenization. (Common Module) Stemmer Morphological anyalser. Tag Generator (Common Module)
IIT Bombay Marathi POS Tagger Verb Basic Prathama taakhyaata थम त य त Dwitiya taakhyaata त य त य त Laakhyaata ल य त Vaakhyaata व य त I: aakhyaata ई -आ य त U: aakhyaata ऊ-आ य त I:laa khyaata ईल य त Chaakhyaata/Aayachaakhyaata च य त
POS Tagger : Indian Languages As indian languages has morphologocally rich so importance of linguistic in Indian language is increased. For example र म ल, here in this example र म+आ+ल this breaking never be ignored as each cluster giving important information
Way towards Marathi POS We started with verb as, it is most important category of the words. Implementation of Aakhyatya(आ य त) Theory is been implemented. Aakhyatya(आ य त [7]) Theory refers to the group of suffixes which gives information like (G_N_P_T_A_M)
Pratham Takhyata प ष Gen प ल ग Masc एकवचन ल ग Fem नप सक ल ग Neut अन कवचन Plural थम First त य Sec त त य Third त, त य, त, त (त ) त, त त स य स, त स, त स (त स) त, त त य, त, त त, त त (तS) त त
Dvitiya takhyata प ष प ल ग ल ग नप सक ल ग ए. अ. ए. अ. ए. अ. थम त, त त, त य, त, त त, त (त ) (त ) त य त स त, त त, त त त त स त, य त, त त (त स) (त त त) त त य त त त य त, त (तS) त, त
Lakhyata प ष प ल ग ल ग नप सक ल ग ए. अ. ए. अ. ए. अ. थम ल, ल ल, ल य, ल ल, ल ल, ल (ल ) (ल ) त य ल स ल, ल त, ल त ल त ल स ल, ल त, य, त, ल त (ल स) (ल, ल त) त त य ल ल ल य ल, ल ल (लS) ल, ल
Vakhyata/Avakhyata प ष प ल ग ल ग नप सक ल ग ए. अ. ए. अ. ए. अ. थम आव आव आव आ य (आव ) (आव ) त य आव स आव, आव त, आव (आवS) आव स आ य आ य त (आव स) (आव आव त) त त य आव आव आव त आव आ य आ य त आव आव (आवS) आव, आव त आव आव त
I-akhyata प ष एकवचन अन कवचन थम ईए ऊ, ओ, ऊ, ओ त य स आ, आ त त य ईए त
Ilakhyata प ष एकवचन अन कवचन थम ईन एन ऊ, ऊ त य श ल आल त त य ईल एल त ल
* Computation * Tokenization is a process of separating the different Tokens. for example बसत, ;, ( Tokenization can be done in various ways * StringTokenizer Class. * Regex Expression * java.text.* package
* Computation * Stemming is important in the system and we show the process with an example. Suppose the input word is बसत (to sit)". In बसत ", two suffixes matched is of the category verb, -to" The stems formed after removal of this suffixs is बस (sit) Searching this stems in the lexicon shows that बस" is present in the lexicon Applications : Multilingual search engines, POS Tagger etc.
* Computation * Verb Module RF1 RF2 RF3 VP ST P1 P2 P3 AT Engine IP Engine TO DR
* Computation * RF 1 RF 2 RF 3 IP :Marathi text in UNICODE Engine ST :It identifies the longest suffix and the stem for the input word. VP : It modifies the suffix wherever it is necessary. DR : It is a dictionary of all types of root words (It counts more than 2000 verbs) RF1: It consists of rules relating irregular stems to their root forms. RF2: It consists of rules relating suffixes and the corresponding features.(त ) RF3: It consists of rules regarding the most frequent and the most deviated verb forms. (ह ण ) AT : It generates tags based on the results returned by the engine. TO : The tagged verb form is returned as output. V P IP S T P 1 P2 Engine DR P 3 A T T O
Rule Files RF1: Rule File 1: It consists of rules relating irregular stems to their root forms. We have analyzed around 25 irregular verbs. In total 65 rules, relate irregular stems to the corresponding root forms. 1. <w>कर_kar<s> क _karne_to do<r>
Rule Files continued RF1: Rule File 2: It consists of rules relating suffixes and the corresponding features. We have implemented over 1700 rules. The syntax of the rules is as follows. <r>ण _Ne<c>त _to<s>present<t>habitual<a>indicative<m>m<g>s<n>1<p>. The rule states that if suffix त _to is separated, then add changing part ण _Ne to the regular stem and search the root in the dictionary DR. If the root is found then respective TAM GNP information will be extracted.
Rule Files continued.. RF1: Rule File 3: It consists of rules regarding the most frequent and the most deviated verb forms. The root verbs,ह ण _hone_to become, न ह ण _na hone_not. The syntax of the rules is the same as the rules in RF1, which relates the deviated form of the stems and the root. (The format of RF3 is same as RF2 and RF1).
Other Modules Engine: It processes the suffixes according to their categories. We have implemented processes P1, P2 and P3 for the corresponding categories. Suffix without space Regular verb (P1) (example त,त त etc) Irregular verb (P2) (example क ल etc.) Suffix with space (P3) ( ल _आह etc.) Assign Tags (AT): This module generates the tags based on the results returned by the engine. We have defined the number of tags as listed in the table 6 in Appendix A. Tags are attached to respective word. Tagged Output (TO): The verb forms detected from the text is displayed along with the respective tag.
Evaluation Total number of tagged verb forms 2176 Total number of correctly detected tagged verb forms 2166 Undetected verb forms: 97 Total number of verb forms present in the corpus : 2263 Following precision and recall values are with ambiguity. Precision 0.9995 Recall 0.97
Conjunct Computation * Modules. * Tokenization. (Common Module) * Sorting the Conjunct List. * Searching Word using Binary Search. * Tag Generator (Common Module) (Tag is Conj )
Interject Computation * Modules. * Tokenization. (Common Module) * Sorting the interject List. * Searching Word using Binary Search. * Tag Generator (Common Module) (Tag is Intej )
Noun Commutation CM PP Case 0 0 Direct (र म,र न) 0 1 Direct (र मल ) 1 0 *Direct/Oblique 1 1 Oblique ( र म ल )
10 Case घ ड ( N_M_S_D ) Ex : घ ड पळ ल. घ ड ( N_M_P_D ) Ex : घ ड पळ ल. घ य ( N_M_S_Voc) Ex : घ य इकड य.