The Reflection Assistant: Investigating the Effects of Reflective Activities in Problem Solving Environments Claudia Gama School of Cognitive and Computing Science University of Sussex Brighton UK BN1 9QH claudiag@cogs.susx.ac.uk Abstract: This paper presents theoretical issues about the role of metacognition in learning, as well as the architecture and design of a learning system dedicated to monitoring and helping students to use their metacognitive skills. This system is called the Reflection Assistant and is supposed to be used together with problem solving or case based learning systems. Thus, through interactions with the Reflection Assistant during problem solving activities, students can also improve their metacognitive skills (such as planning, self monitoring and assessing), as well as their knowledge in the specific domain. The purpose of this research is to investigate better means to increase metacognitive knowledge, including awareness on the learning process and capacity of transferring skills to new learning situations. The motivation for this research is that it is necessary to focus as much on the metacognitive processes of problem solving as on the cognitive ones to develop good problem solvers. 1. Introduction: Theoretical Issues 1.1 Metacognition and Learning The argument that drives this research is that metacognition plays a major role in cognitive development. Thus, enabling students to develop a conscious, explicit model of their metacognitive skills by means of reflective activities facilitates the improvement of both cognitive and metacognitive expertise. Research in the area of cognitive psychology (Flavell, 1976, 1979, Schoenfeld, 1987, Derry, 1992, amongst others) shows that an important type of knowledge underlying the construction of complex representations and solutions strategies is metacognitive knowledge. Metacognition refers to a cognitive system s intelligence about itself and its ability to regulate and control its own operation. So, knowing about one s own cognition forms the basis for most metacognitive abilities. These abilities include knowledge about one s perceptions, memories, decisions, and actions (Shimamura, 1994, p. 253). Metacognitive knowledge can be divided into three related but distinct categories of intellectual behavior (Schoenfeld, 1987): Knowledge about thought process This is related to people s ability to accurately assess their knowledge and knowledge acquisition process. In a learning situation one of the final goals is to help students develop good study skills. Those skills depend, in part, on students ability to make realistic assessments of what they can learn. Hence, it is important to know how likely it is that students will reflect on their thinking and how accurate those reflections will be. Control and Self regulation This aspect of metacognition is related to the monitoring of cognitive activities. One way to look at this aspect of metacognition is to think of it as a management issue, i.e., how well one manages her time and effort, as she is involved in cognitive tasks. In the context of learning self monitoring or self regulation refers to an individual s ability to conduct on line self checks of the problem solving process, for example; a skill that is particularly important when the student deals with an ambiguous or confusing problem. Thus, this self regulation could provide the student with information to guide her problem solving actions.
Beliefs and Intuitions These are related to the ideas about a learning topic that the student brings to her work, and how it shapes the way the student faces the topic. Another important aspect of learning is transfer. Educators hope that students will transfer learning from one problem to another within a course, from one year in school to another, between school and home, and from school to workplace. Transfer can be improved by helping students become more aware of themselves as learners who actively monitor their learning strategies and resources and assess their readiness for particular tests and performance. Metacognitive approaches to instruction have been shown to increase the degree to which students transfer to new situations without the need of explicit prompting (Bransford et al., 1999). 1.2 Problem Solving, Metacognition and Reflection According to Schoenfeld (1987), problem solving activities can be divided into three conceptual stages, named: Preactive (activities performed before starting to solve the problem), Interactive (activities performed as the student works on the problem) and Postactive (all those activities performed after finding a possible solution to the problem). Considering these conceptual stages, three important metacognitive processes can play a major role: planning (mainly in the preactive and also in the interactive stage), self monitoring (in the interactive stage) and assessing the problem solving efforts (in the postactive stage). Metacognitive processes are important contributors to problem solving performance across a wide range of domains (Davidson et al., 1994). In problem solving situations there are always metacognitive processes involved, such as (1) identifying and defining the problem, (2) mentally representing the problem, (3) planning how to proceed, and (4) evaluating what you know about your performance. The successful application of these metacognitive processes depends on characteristics of the problem (if it is a well structured or an ill structured problem), the problem solver, and the context in which the problem is presented. To the extent that what is desired is to identify and develop good problem solvers, what is needed is to focus as much on the metacognitive processes of problem solving as on the cognitive ones (Davidson et al., 1994). Hence, it is important that students can recognize and explore their own metacognitive processes. Reflective activities encourage students to analyse their performance, contrast their actions to those of others, abstract the actions they used in similar situations, and compare their actions to those of novices and experts. The reflection on the metacognitive skills can bring awareness of their thought process and increase their capacity of transferring these skills to new learning situations. Fig. 1 summarizes the interrelationship between the problem solving conceptual stages, the metacognitive processes, and the role of reflection in this context. Figure 1 Metacognition, Reflection and Problem Solving 2. The Design of the Reflection Assistant The Reflection Assistant is be a specialized system that monitors, analyses and improves student s metacognitive skills in problem solving situations. It is still in the design phase (it has not been implemented yet). It is designed to be used in combination with different problem solving systems (e.g.: mathematics problem solving systems, case based medical system, geometry system, etc.).
Indeed, the Reflection Assistant is disjoint of the specific domain learning system, and it has to be configured accordingly to the specific domain that it will be applied. The two major goals of the Reflection Assistant are: (1) Create learning environments that afford and encourage metacognitive processes. (2) Improve metacognitive skills development in general and thereby enable the transference of these skills to future learning situations. Through graphical tools and specific activities the Reflection Assistant makes perceptible to the student certain metacognitive skills used in distinct moments of instructional activities. Thus, initially students can use these mechanisms to monitor their learning process and become aware of the metacognitive strategies they are using when engaged in problem solving activities. After that, in a subsequent phase they can use appropriate mechanisms to help them to improve these metacognitive strategies. The important points this research aims to investigate are: (a) The appropriate timing for providing metacognitive scaffolding and reflection on the problem solving process. (b) The effective balance between reflection focused on the learning task (domain specific) and reflection on general metacognitive skills (domain independent) for a particular problem solving situation. 2.2 Description of the Reflection Assistant The Reflection Assistant contains four main components. The general architecture is shown in Fig. 2 and the function and characteristics of each component are explained. Figure 2 Architecture of the Reflection Assistant Metacognitive Knowledge Base This component has detailed information about metacognitive skills used in problem solving situations. Each metacognitive skill is described in the Reflection Assistant in terms of variables, which may have their values modified according to the importance that the specific skill plays in the specific domain. Student Metacognitive Model This is responsible for recording the student s responses and keeping track of the student s performance. The Student Metacognitive Model uses information from the Student Log and from the Metacognitive Knowledge Base to generate a structured model of the student s metacognitive capabilities and progress. This model informs the kind of assistance that should be provided to a particular student and also serves as an inspectable model for the student (in other words, the student can also view her use of strategies). Interaction Generator This component combines information provided by the Student Metacognitive Model and Metacognitive Knowledge Base to elaborate the content and format of the metacognitive activity proposed. Translator This is the communication mechanism (or protocol) that will make possible the integration of the Reflection Assistant with the learning system. There is a two way communication between the two
systems: (1) the learning system provides information about the actions of the student during her interaction (in the format of a log file); then, the Translator interprets this information and a metacognitive log is created in the Reflection Assistant. (2) The reflection assistant sends suggestions of possible new actions to the learning system, so adapting the teaching strategy according to the student metacognitive model. Despite being decoupled from the learning system, the Reflection Assistant is not completely independent of it. It keeps an intermediate level of interaction with the system through a communication channel. This interaction is reflected in the architecture of the system, where its main components need to contain several parameters that are modified depending on the circumstances under which the assistant is used (i.e., the specific learning system to which the Reflection Assistant will be associated with or the kind of reflective activities relevant to a specific domain or for certain problems). 2.3 The Different Types of Reflective Assistance Following the idea of dividing the problem solving activities into three conceptual stages (as explained in section 1.2) the Reflection Assistant can be used in different configurations. The reflection activities can happen either before starting a new problem/case; or during the interaction with the problem/case; or even after finishing the problem/case. A specific visualization method will be selected depending on both the timing of the reflection and the particular metacognitive skill emphasized (e.g.: planning, monitoring, etc.). Indeed, the combination Metacognitive skills Timing Visualization method (or approach) is determined by the characteristics of the domain and task and, also the student s preferences. Besides that, the degree of specificity of the behavior of the assistant depends on the level of integration between the Reflection Assistant and the Learning Environment. This integration is generated by the amount of information provided by the problem solving system to the Reflection Assistant s Translator. To organize these different arrangements the system provides three basic types of assistance, corresponding to a particular moment of the interaction. Before solving a new problem (or case) the Preparation to Reasoning Assistant helps the student to recall previous knowledge or studied cases. While the student is trying to solve the problem he can interact with the Support to problem solving Assistant. And, finally, after finding a solution, the student could use the Reflective Follow up Assistant that provides means to reflect on the performance in the task and the general progress. These assistants aim at providing structured support to the student, establishing check points to reflect over the learning process. The kinds of assistants that comprise the Reflection Assistant and which metacognitive skills (or processes) they are mainly related to are detailed below. Preparation to Reasoning Assistant Questions like What prior knowledge can help me with this particular task? What do I need to know before I can successfully deal with this task? What should I do and in which order? How much time do I have to complete the task? drive the activities proposed by this assistant. The idea is to anchor new concepts into the learner s existing cognitive knowledge to make them retrievable. The aim of this assistant is to prepare the student for the problem solving activity, promoting reflection upon the necessary skills to solve the problem, more specifically, the capacity to recall previous knowledge and make generalizations. Sometimes the reflection process does not occur, in part because students are not really motivated to perform reflection after solving the problem and in part because the current problem solving medium (paper and pencil) does not really lend it self to this activity (Collins, A. & Brown, J. S., 1988). Because the Reasoning Assistant is going to be used before the learning activity takes place, it may contribute to bypass problems of low motivation. Support to Problem Solving Assistant This assistant offers a means for monitoring the student s plan of action. It helps the student during the cognitive activity with mechanisms to reflect on the necessary skills to solve a problem, making them think about, among other things: How am I doing? Am I on the right track? How should I proceed? Should I move in another direction? What do I need to do if I don t understand? Also an important point is to use the Reflection Assistant as a means to aid the student when she feels stuck. Thus, the Support to problem solving Assistant should provide support for students to seek for clarification and to reexamine their problem solving strategy. Reflective Follow Up Assistant After concluding the cognitive task the student can use this assistant to verify her performance and compare it with previous interactions. Here, generalizations are again used to help the student to
organize the case or problem studied into a general knowledge framework. Examples of common points to reflect on in this moment can be: Did my particular course of thinking produce more or less than I had expected? What could I have done differently? How might I apply this line of thinking to other problems? 2.4 The Design of the Reflective Follow up Assistant: First Prototype There are several points to take into account in the development of such a learning system. The abstract nature of the metacognitive processes is indeed a major concern. Also, the graphical tools used to represent these processes have to be adequately developed. Moreover, these tools and metacognitive processes probably vary from one student to the other, depending of their own characteristics. So, in order to collect information about all these factors that influence the whole system, some initial studies are being performed. As part of this initial phase a first version of the Reflective Follow up Assistant is being designed and will be used with the PATSy system. PATSy (Patient Assessment Training SYstem) is a case based interactive learning environment that keeps a database of medical cases in the domain of aphasiology (Lum & Cox, 1998, Cox, 1999). PATSy provides a context in which students may practice diagnosis via deductive reasoning. Thus, the Reflection Assistant will help students to monitor their reasoning process. PATSy keeps an electronic log of all actions performed by the student (sequence of clinical tests chosen, cases studied, etc.). The first version of the Reflective Follow Up Assistant will provide two types of information: quantitative assessment and qualitative assessment of the metacognitive skills used during interaction with the PATSy system. Fig. 3 (a,b and c) show mock ups of this prototype. Quantitative Assessment After each session with the PATSy system students will see a graph showing the time spent in cognitive and metacognitive activities. For example, Fig. 3a shows a sample graph of a student. Using that, the student can see graphically the amount of time spent analysing the problem, planning, studying previous cases or solutions and attempting to solve the problem. Other information provided are the moments where the student reflected on her studying process. PATSy already provides some questions that should help students to step back and ask what they want to do next. Further information generated is the summary of the total amount of time spent in the task. Figure 3a Quantitative Assessment (sample 1) Figure 3b Quantitative Assessm. (sample 2) Another graph from the quantitative assessment is shown in Fig. 3b. It is a graphical comparison between the performance in the present case and in previous cases. Another possible comparison could be between the performance of one student with those of other students. Qualitative Assessment This kind of assessment aims at providing more detailed explanations to the student about how to improve her metacognitive skills. It is achieved by asking questions about the student s performance. For example, Fig. 3c shows a qualitative assessment of the planning skills of one student. The student is invited to explain why she failed in using the planning strategies she proposed to use at the beginning of the task. The idea is to make the student to reflect on the importance of a good plan of action that has
to be followed during the problem solving. In addition, the system will increase the planning abilities of that student. to provide scaffolding to 3. Conclusions This research investigates better means to make students learn important self monitoring and other metacognitive skills in computer based problem solving environments. The final goal is the design of a new learning environment that extend the possibilities of interactive learning systems, giving students opportunities for reflection, revision and feedback over their problem solving efforts. This paper has presented the design of the Reflective Follow up Assistant Module to be used in the postactive stage of problem solving. It focuses on both quantitative and qualitative assessments of students performance. More work has to be done in the design of the other modules of the Reflection Assistant: the Preparation to Reasoning Assistant and the Support to Problem Solving Assistant. A distinguished characteristic of the Reflection Assistant is that it is both a domain independent system and also maintains an intermediate level of interaction with the learning system that uses it. The granularity of interaction is an interesting point of investigation. This relates to the tension between teaching metacognitive skills within a particular context and ensuring their transferability to other contexts. Some initial studies have been done and the first prototype is under development. Figure 3c Sample of Qualitative Assessment References Bransford, J. D., Brown A. L. & Cocking, R. R. (Eds.) (1999). How People Learn: Brain, Mind, Experience, and School. National Academy Press. Washington, DC. Collins, A. & Brown, J. S. (1988). The Computer as a Tool for Learning Through Reflection. In Mandl, H. & Lesgold, A. (Eds.), Learning Issues for Intelligent Tutoring Systems. Springer Verlag. Cox, R. (1999). Modelling the Diagnostic Reasoning Skills of Expert Clinicians and Students in the Domain of Aphasiology. Proceedings of C LEMMAS: Roles Communicative Interaction in Learning to Model in Mathematics and Science. Brna, P.; Baker, M. & Stenning, K. (Eds.). Davidson, J. E. et al. (1994). The Role of Metacognition in Problem Solving. In Metcalfe, J. & Shimamura, A. P. (Eds.), Metacognition: Knowing about Knowing (pp. 207 226). The MIT Press. Derry, S. J. (1992). Adaptative Learning Environments Foundations and Frontiers. In Jones, Marlene & Winne, Philip H. (Eds.), Metacognitive Models of Learning and Instructional Systems Design (pp.257 286). Springer Verlag Berlin Heidelberg: Nato Asi Series Books. Flavell, John H. (1976). Metacognitive Aspects of Problem Solving. In Resnick, Lauren B. (Ed.), The Nature of Intelligence. Lawrence Erlbaum Associates. Flavell, John H. (1979). Metacognition and Cognitive Monitoring. American Psychologist Journal, 34 (10), 906 911.
Lum, C. & Cox, R. (1998). Patsy: A Case Based, Distributed Multimedia Approach to Patient Assessment Skills Training. WWW document: http://www.cogsci.ed.ac.uk/~rcox/patsy_paper/patsy.html Schoenfeld, A. H. (1987). What s All the Fuss About Metacognition? In Schoenfeld, Alan H. (Ed.), Cognitive Science and Mathematics Education. Lawrence Erlbaum Associates. Shimamura, A. P. (1994) The Neuropsychology of Metacognition. In Metcalfe, J. & Shimamura, A. P. (Eds.), Metacogntion: Knowing about Knowing (pp. 253 276). The MIT Press. Acknoledgements This research is supported by a grant from CNPq Brazil no. 200275 98.4. The author would like to thank Prof. Benedict du Boulay for his comments and suggestions on this paper.