TOWARDS A KNOWLEDGE-BASED FREE-TEXT RESPONSE ASSESSMENT SYSTEM

IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2008) TOWARDS A KNOWLEDGE-BASED FREE-TEXT RESPONSE ASSESSMENT SYSTEM Panagiotis Blitsas Interdisciplinary Program of Graduate Studies in Basic & Applied Cognitive Science National & Kapodistrian University of Athens Panepistimioupolis, Ilissia, Athens 15784, Greece Maria Grigoriadou Department of Informatics & Telecommunications National & Kapodistrian University of Athens Panepistimioupolis, Ilissia, Athens 15784, Greece ABSTRACT In the present work, the architecture of a knowledge-based free-text response assessment system is presented, which can assess free-text responses on open-ended questions based on text comprehension theories. Its main advantages are the facts of extending the knowledge base and assessing responses on questions of different types. It is constituted from three basic modules: The first module is the Normalization Module (NoM), which is responsible for converting free-text responses into normalized responses, as well as, converting technical text into functional system. The second module is the Functional System Module (FSM), which depicts all entities of a technical text and the relations among them, and is presented, by the representation of microstructure and macrostructure of a Computer Networks domain technical text, according to Denhiere-Baudet Text Comprehension Model. The third module is the Assessment Module (AM), responsible for assessing the normalized responses. It is based on data, obtained by experimental studies on microstructure and macrostructure, constructed by secondary school students, during reading technical texts and responding to questions. KEYWORDS Assessment, Functional system, Microstructure, Macrostructure. 1. INTRODUCTION Nowadays, much effort is put in creating models to simulate psycholinguistic theories with regard to human processes of text comprehension, as the Construction-Integration Model (Kintsch, 2001; Kintsch, 1992), the Theory of Latent Semantic Analysis of Knowledge Representation (Landauer & Dumais 1997), and Denhiere-Baudet Text Comprehension Model (Baudet & Denhiere 1992). Text comprehension models use cognitive representations, which focus in deeper levels of understanding, such as conceptual conclusions based on knowledge, inductive reductions and world knowledge, combined with the surface levels of understanding, which involve lexical processing, syntactic analysis and text interpretation. Our research is focused on Denhiere-Baudet Text Comprehension Model, because it is the most appropriate for elaborating technical text, after analyzing the cognitive categories that are involved in such a kind of texts. According to this model, the individual, while reading a technical text, manufactures, progressively, its representation, namely atoms, states, events and actions of the world that is described in the text, as well as, temporal and causal relations, which connect these cognitive categories. The term 'atom' is used for the entities that participate in the knowledge representation. The term 'state' is static and describes a situation where no change happens during a period of time. The term 'event' describes an action that causes changes, not evoked by a person, but by a non-human action e.g. a machine. An 'action' causes changes and is evoked by a person. 37

ISBN: 978-972-8924-69-0 2008 IADIS A technical text depicts a technical system, which contains a set of associated atoms, fixed by hierarchical relations of all/part type. In the case of a technical system, atoms refer to system units. A system unit is a set of entities with a specific function. A technical system, organized as a tree of goal/subgoals, is called a Functional System. According to this model, a technical text should support the construction of two representations, Microstructure and Macrostructure. Microstructure includes two basic structures: (1) Relational structure, which is an ontology of entities and units, which are characterized by the values of their attributes, and one or more static relations among them and (2) Transformational structure, which is the description of the progressive modification of the states of the system units, namely a description of sequence of events executed on these units. Furthermore, in the same structure a description of temporal or/and causal relations among these events is included. Macrostructure includes a teleological structure of goal and subgoals of various operations of the system units and the whole system itself, which, otherwise, are called macroevents. The initial system state changes in order to reach a final state and satisfy a predetermined system goal. In turn, every unit, embodied in the whole system, has been developed, in order to satisfy its own predetermined subgoal. Furthermore, Macrostructure includes Microstructure. So, transformational structure of every unit and, in general, of the whole system is the mean, through which the previous subgoals are being achieved. In the present work, we present the architecture of a free-text response assessment system based on expert's knowledge representation, according to Denhiere-Baudet Text Comprehension Model. This system refers to assessing responses on questions upon the content of technical text, and especially texts about Informatics and Communications. Such a kind of an automated free-text response assessment system on open-ended questions can support students achieving constructivistic learning, while interacting with this. In section 2, the architecture of the Free Text Response Assessment System is presented. In section 3, 4, and 5 the Normalization Module, the Functional System Module and the Assessment Module are presented, respectively. In section 6, there is a discussion about our research and, in section 7 our future plans are presented. 2. ARCHITECTURE OF THE FREE-TEXT RESPONSE ASSESSMENT SYSTEM Figure 1. Architecture of the Free-Text Response Assessment System Figure 1 displays the architecture of the Free-Text Response Assessment System, based on Denhiere-Baudet Text Comprehension Model. 38

IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2008) In detail, the system modules are the following: Normalization Module (NoM): This module is responsible for converting student's free-text response into normalized response through Natural Language Processing (NLP) Functional System Module (FSM): This module is the ontology of the basic structures of the expert's knowledge representation, namely relational and transformational structure of microstructure, and teleological structure of macrostructure, depicted in the technical text. Assessment Module (AM): This module is responsible for rating the normalized response, coming from NoM. The description of these three basic modules and their submodules is following in the next sections. 3. NORMALIZATION MODULE This module has a double role. Its first goal is converting a free-text response into normalized response, and the second one is converting the content of a technical text into ontologies, describing the microstructure and macrostructure, enriching, this way, the content of FSM, which is being described in section 4. When we use the term 'normalized response', we mean the response, which includes the entities, meaning the units, the events and the macroevents, declared explicitly or implicitly in the respective free-text response. In order to achieve the previous goals, the Normalization Module has the following three submodules: Cleaning Module (CleM): It is responsible for cleaning up the student's free-text response from unnecessary words. Collaborating with the FSM, can decide, which word will be cleaned up and which one will be kept, as necessary. It means that when CleM meets a word that constitute entity or relation, appeared in FSM, it will keep this word, and additionally, it will build the representation of student's response. This representation is, in fact, the 'normalized' response. The output of CleM is led, as an input, to NoRM. During this processing stage, there is a need for CleM to collaborate with computational synonym dictionaries, such as WordNet, in order to decide if a word is a synonym of an entity or relation of FSM, so that CleM keeps the word as a necessary one and does not consider it as irrelevant. Conversion Module (CoM): It is responsible for distinguishing the technical text entities, into units, events and macroevents, in order to send them to FSM where microstructure and macrostructure ontologies will be constructed. Normalized Response Module (NoRM): It functions as a buffer of the student's normalized response. Its output is led to AM for estimating the errors, appeared in student's normalized response representation. So far, we have created a tool, in Greek, for implementing the basic submodules of Normalization Module. This tool includes a computational conceptual dictionary and supports input of new concepts and relations among them. Furthermore, it has the capability of accepting as an input a short text, and producing as an output a tree-like ontology of the concepts, which are appeared in the free text, and all the relations among them, belonging to the knowledge base of this tool. Additionally, by using CmapTools software (http://cmap.ihmc.us/) an ontology has been manually created. CmapTools gives the opportunity of handling ontological representations and can extract these representations, as entity triads, in form of entity - relation entity. These triads can be inputs to our tool and enrich the knowledge base, which constitutes FSM. 4. FUNCTIONAL SYSTEM MODULE The Functional System Module includes three submodules, which depict the Relational Structure (ReM), the Transformational Structure (TraM) and Teleological Structure (TeM) of expert's knowledge representation. This module takes, as an input, the output of CoM, and constructs the different structures. As an example for representing these three structures of expert's knowledge, we used a technical text, taken from the book 'Computer Science: An overview' (Brookshear, G., 2006). It refers to the combination of bus topology networks toward forming wide area computer networks: The repeater is little more than a device that connects two buses to form a single long bus. The repeater simply passes signals back and forth between the 39

ISBN: 978-972-8924-69-0 2008 IADIS two original buses (usually with some form of amplification) without considering the meaning of the signals. A bridge is similar to, but more complex than, a repeater. Like a repeater, it connects two buses, but it does not necessarily pass all messages across the connection. Instead, it looks at the destination address that accompanies each message and forwards a message across the connection only when that message is destined for a computer on the other side. The expert's microstructure and macrostructure representations are following. 4.1 Expert's Microstructure Representation At the microstructure level, there is a description of units, which constitute the system, as well as, the causal relations, which connect them. First stage of realizing microstructure representation is constructing relational structure of the text content. Relational structure has two basic ontologies: taxonomy and partonomy. Taxonomy is a tree-like structure, 'is a' type, which analyzes the taxonomical relations among the entities appeared in the technical text. Partonomy is a tree-like structure too, 'part of' or 'has' type, which expresses all/part-of relations among the text entities. These ontologies have to represent the knowledge, explicitly referred in the text, as well as, the implicit knowledge, which is activated during the reader's comprehending. Examples of text taxonomy and partonomy are following, in form of Prolog facts: is_a(repeater,device). is_a(bridge,device). is_a(bus,lan). is_a(lan,network). part_of(node,bus). part_of(lan,wan). part_of(content,signal). has(bus,csma_cd_protocol). Second stage of realizing expert's microstructure is representing transformational structure. An example of that is shown, schematically, in figure 2. Figure 2. Expert's Transformational Structure 4.2 Expert's Macrostructure Representation Expert's macrostructure includes teleological structure and microstructure. Teleological structure, describing in section 1, is depicted in TeM, and an example, referred to the text, is show schematically, in figure 3. Figure 3. Expert's Macrostructure 40

IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2008) 5. ASSESSMENT MODULE The Assessment Module can assess three kinds of questions: a. Ontological questions, through ReM, such as 'what is' b. Operation questions, through TraM, such as 'how does it work' c. Teleological questions, through TeM, such as 'why' or 'which one' In order to develop the Assessment Module, we had to obtain data of errors students make, during reading technical text and answering questions on microstructure and macrostructure they construct. An experiment on secondary level students took place for estimating the misconceptions students have during comprehending the technical text. Two question types were given to the participants: (i) inference questions, examining the transformational structure students formed during reading and (ii) bridging questions, examining the macrostructure they formed and the recall they achieved from different points of the text. Forms for completing events (for the first question type), and choosing the right device for a specific goal (for the second question type) were given. So, the demanded responses had a normalized form. One example question of each type is following in the next two subsections. 5.1 Response Assessment on Microstructure Questions An example of inference question, referred to transformational structure, which belongs to microstructure, is the following: Describe the operation of a Router, step by step, from the first state, in which the message is located in the source node of the source network to the state, in which the message is located in the destination node of the destination network. Three error types were mentioned on students' responses: missing events in the events sequence events replaced by goals causal errors among events The right sequence of events of Router operation (Router transformational structure) is: (destination network) protocol detection protocol control protocol comparison protocol conversion. For example, in a student's response, the next error types were appeared: (i) Causal error: instead of 'protocol control' precedes 'protocol conversion', in the response 'protocol conversion' precedes 'protocol control', (ii) Event replaced by a subgoal: in place of 'data detection', 'data transfer' appears. Figure 4 displays the analysis of student's transformational structure assessment process, and figure 5 displays an example of automated transformational structure question assessment, implemented in Prolog. In figure 5, AErrors are events replaced by subgoal errors, and BErrors are the causal errors. Finally, MEvents are the missing events in the response. Figure 4. Student's transformational structure assessment process 41

ISBN: 978-972-8924-69-0 2008 IADIS Figure 5. Automated transformational structure question assessment example 5.2 Response Assessment on Macrostructure Questions A bridging question demands a response after combining information from more than one point of the given text. An example of bridging question follows: In the case of connecting a Bus with a Ring topology network, which is the device, you have to use? Why the other devices are rejected? Justify your position. This question demands a position and justification for the choice of the right device and for the rejection of the others. The response is based on the macrostructure (including the microstructure of 'repeater', 'bridge', and 'router' devices). Router is not presented in the example text but is appeared in a bigger text, where the example text belongs. The following incomplete sentences were given to the students: The right device is...(1) because...(2) Device... (3) can't be used because... (4) Neither device... (5) can be used because... (6) The right response is: (1) Router, (2) protocol conversion, (3) bridge or repeater, (4) not protocol conversion, (5) repeater or bridge & (6) not protocol conversion. In students' responses, the following error types were appeared: 1. Wrong position (1) or/and justification (2) for the choice of the right device. 2. Wrong positions (3), (5) or/and justification (4), (6) for the rejection of the other devices. 3. Blanks in some of (2), (3), (4), (5) and (6). So, we have three different degrees of assessing. 1. Naïve response, when there is wrong position and justification for the choice of the right device. 2. Incomplete response, when there is right position for the choice of the right device and there is wrong response on (2), (3), (4), (5) or (6), and 3. Full response, when student completed the full right response. Figure 6 displays the analysis of student's macrostructure assessment process, and figure 7 displays an example of automated macrostructure question assessment, implemented in prolog. Figure 6. Student's macrostructure assessment process 42

IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2008) Figure 7. Automated macrostructure question assessment 6. DISCUSSION In this paper, we presented the architecture of a knowledge-based free-text response assessment system. The main purpose of this work was to illustrate the knowledge representation upon a technical text based on the model of Denhière and Baudet for technical text comprehension and the proposition of an automated assessment system, which could assess normalized responses. The fact that the microstructure is not sufficient for the description of a technical system, but the macrostructure is necessary too, is revealed. The complete analysis in microstructure and macrostructure level can support automated reasoning, which in turn, leads to automated comprehension in form of knowledge extraction through these two levels. So far, the most of the systems for computer-assisted assessment of free-text responses are based on the Latent Semantic Analysis Theory (Landauer & Dumais 1997). The main disadvantage of these systems is the fact that they are not incremental, because they are based on vector-oriented analysis. The present system could give a boost to a knowledge-based assessment, with ability of extending the knowledge base by adding new conceptual rules, expressing more causal, spatial and temporal relations, as well as, more goals and subgoals, which can describe new operations of a functional system, which in turn describes the content of a new technical text body. Furthermore, responses on different kinds of questions could be assessed, as bridging, inference and ontological questions. Such a system of automated comprehension and technical text assessment could give a lot of prospects to the way students of technical subjects could interact with computer and through processes of their automated responses assessment can achieve constructivistic learning. The process of an active response assessment can 43

ISBN: 978-972-8924-69-0 2008 IADIS help in their meta-knowledge, after giving them the possibility of estimating their errors and error types, and providing the possibility of correction. 7. FUTURE PLANS Our short-term plans include a further study on the misconceptions of students in Computer Science domain, which could provide more explanatory data about the error types they make, while comprehending technical texts, based on their prior knowledge in the domain (Kanidis, E. and Grigoriadou, M., 2007). These studies can help to enrich the Assessment Module with new assessing rules. Furthermore, one of our main goals is the implementation of the system. First, we elaborate the connection of the Normalization Module with computational dictionaries, such as WordNet, Visdic e.t.c. Second, we elaborate the automated enrichment of the Functional System Module knowledge base, in terms of providing an incremental way of inserting new technical text content. ACKNOWLEDGEMENT The work, described in this paper, was partially supported by a grant from the project Language Engineering Tools in Learning Environment: Application, Research, Innovation (Let s LEARN), which is funded by The General Secretariat for Research and Technology (Contract GSRT 092-g). REFERENCES Baudet, S. and Denhiere, G., 1992: Lecture, comprehension de texte et science cognitive. Presses Universitaires de France, Paris. Brookshear, G., 2006. Computer Science: An Overview, Pearson Addison Wesley, Ninth Edition Graesser, A. and Tipping, P., 1999: Understanding texts, In: Bechtel, W., Graham, G. (eds.) A Companion to Cognitive Science, Malden MA: Blackwell, Chapter 24. Gruber, T., 1993: Toward Principles for the Design of Ontologies Used for Knowledge Sharing, Technical Report KSL 93-04, Stanford University, Knowledge Systems Laboratory, Revision. Kintsch, W., 2001: Predication. Cognitive Science Vol. 25, pp 173-202. Kintsch, W., 1992: A cognitive architecture for comprehension. In: Pick, H.L., van den Broek, P., Knill, D.C. (eds.) The study of cognition: Conceptual and methodological issues Washington, DC, American Psychological Association, pp. 143-164. Landauer, T. K. and Dumais, S. T., 1997: A solution to Plato's problem: the Latent Semantic Analysis theory of acquisition, induction and representation of knowledge. Psychological Review, Vol. 104, No. 2, pp 211-240. Kanidis, E. and Grigoriadou, M., 2007: Reading about Computer Cache Memory: The Effects of Text Structure in Science Learning, Proceedings of the 2th European Conference on Cognitive Science. Lemaire, B, Denhiere, G., et al, 2006: A computational model for simulating text comprehension, Behavior Research methods, Vol. 38 No. 4, pp 628-637. Rinaldi, F., Dowdall, J., Hess M., Molla, D., Schwitter, R., 2002: Towards Response Extraction: An Application to Technical Domains. In: F. van Harmelen (eds.), Proceeding of the 15 th European Conference on Artificial Intelligence, the Netherlands, Amsterdam IOS Press, pp. 460-464. 44