Student-created Narrative-based Assessment Olaf Hallan Graven Buskerud University College, Norway Olaf.Hallan.Graven@hibu.no Prof Lachlan M MacKinnon Buskerud University College, Norway Lachlan.Mackinnon@hibu.no Introduction Abstract: The student created narrative based assessment is based on a student demonstrating their understanding of a topic by creating a narrative. That narrative usually takes the form of the exposition of a series of facts and the relationship between them, as a means of describing the overall conceptual view of the topic under consideration. The basis for this is how the student is constructing his learning. The use of student created narrative based assessment is assisting the student in obtaining learning and developing understanding. The proposed assessment system, is intended to be implemented as part of a learning environment with focus on the utilisation of a game based model and its inherent support for constructivist learning in a higher education environment. The assessment of a narrative is a time consuming task and students immersed into a games environment requires fast feedback. The proposed system design is utilising plagiarism detection tools as basis for the automated assessment. The student created narrative based assessment is based on a student demonstrating their understanding of a topic by creating a narrative. The narrative usually takes the form of the exposition of a series of facts and the relationship between them, as a means of describing the overall conceptual view of the topic under consideration. The basis for this is how the student is constructing their learning. The use of student created narrative based assessment is assessing how the student is obtaining learning and developing understanding. Each piece of learning that the student is undertaking is part of the overall picture. The students obtain elements through a task based mechanism. We then ask them to create a construct which describes the conceptual relationship between these elements overall within the context in which they are studying. The construction of the narrative is the student describing this relationship, between the elements. The resulting narrative is not used as a demonstration of the knowledge the student already processes, but the narrative is the means by which they develop the understanding of how those concepts interrelate. The creation of the narrative is assisting the students in building the understanding. After the created narrative is complete it will be assessed in term of its contextual description and relevance, it will demonstrate if the learning has taken place. The use of narrative assessment is a good way of demonstrating understanding. The mechanism that we seek to use, the essay or discursive text, to describe our understanding, has its foundations in the Greek and Roman traditions. A modern description, argument and history of writing as assessment is given by Hamp-Lyons.(Hamp-Lyons 2002) Any help given to the students during the creation of the narrative will be in the form of scaffolding. This scaffolding(boyle 1997) will aid the students in the learning process as they are creating the narrative, which is an important part of the learning process and will gradually be fading as the students own understanding increases. The initial scaffolding put into the learning material to help in the initial phases of narrative construction should guide the students around any major pitfalls they are likely to make in creating their early narratives, and save them from making to many erroneous connections in an early phase. It is important to note that the basis for the student created narrative based assessment are not about the students understanding or memorising the narrative written by others, they are to genuinely construct their own narrative. The authors of the learning material must be aware of this when offering scaffolding to aid in the students creation of their own narrative.
It is the authors firm belief that the preferred model for teaching is the constructivist model. The design basis for our virtual game based environment and the inbuilt assessment is to offer support for this. The constructivist model is based on the notion that students actively construct their own understanding, meaning and knowledge from their experiences. In this model the instructor has the role of guiding the students to create their own knowledge through learning activities. Our view is also that technology can be used to support constructivism, but technology as such is neither inherently constructivist nor necessary to support it. The assessment of a narrative is a time consuming task and students immersed into an environment, such as a proposed game based environment require fast feedback. To accommodate this we need to include mechanisms for automated assessment into the virtual learning. Various forms of automatic assessment have been carried out with varying success over years.(foxley, Higgins et al. 2001). Assessments using multiple choice questions, point and click interface, diagrams or mathematical expressions are examples of automated assessment that have been successful. Automated assessment of discursive or essays solutions however are not as straight forward. Research into involving computers in assessment started in the early 1960s. More recent research, have addressed the automatic grading of essays (Larkey 1998; Foltz, Laham et al. 1999; Landauer, Laham et al. 2000; Shermis, Mzumara et al. 2001; Kanejiya, Kumary et al. 2003; Lonsdale and Strong-Krause 2003; Duwairi 2006) Some of the methods used previously have in more recent times been used as the basis for some plagiarism detection tools, this led us to investigate a possible new use of these tools. Our proposed basis for developing automated assessment and fast feedback is the utilisation of grading of student work using tools developed for plagiarism detection. Game based models for presenting learning material The proposed assessment system, student created narrative based assessment, is designed as part of a learning environment with focus on the utilisation of a game based model and its inherent support for constructivist learning in a higher education environment. The gaming format also lends itself to support the known advantages of narrative from oral traditions and fits with the younger generation s interest in current trends in the entertainment industry. The authors have previously(graven and MacKinnon 2006) published a design for a game based virtual learning environment. This virtual learning environment utilises different starting points, abilities and other gaming elements to personalise the environment for individual students. In the proposed design for a game based learning environment the subject area is divided in topics and subtopics, which are then modelled as levels in the game. The different levels will contain multiple quests each representing some areas of material that the student should learn. The aim of the game is to complete all the levels. The progression in the game is controlled via the assessment part of the system which is the focus of this paper. The quests are only there to aid in the process of learning, and will supply the students with concepts that they will later be assess in. It is possible to model access to higher levels in the game without necessarily completing the current levels. The decision on what levels should be accessible at any one time and the overall control of the progress of the student is left up to the designers of the level Our model of supporting the students though the learning process is based on scaffolding and fading techniques. Initially the students will be offered much support, but as they progress through the levels this support will be gradually removed. Within the model we are describing, the scaffolding is to provide early support and structure for the students to aid them in their thinking. The supplied scaffolding will then be fading to allow the students to gain recognition to the processes they themselves use to understand, and achieve meta cognition. The use of scaffolding to guide and direct students is the authors suggested model to strike a balance between control and freedom. In the game based learning environment quests will be the major way to presenting learning material to the students. The quest form is borrowed from computer games(aarseth 2002), the quest format is built on the simple principle of setting some task for the player and then rewarding them on the successful completion of that task. If the player does not succeed the option is to go back and try the same or another task again. The quests will expose students to learning material and through the completion of a quest students will acquire an understanding of the key concepts modelled in that quest or learning material. Constructivism promotes social and communication skills by creating a learning environment that emphasizes collaboration and exchange of ideas. This social community is naturally present in lectures and tutorials.
The authors view is not that the only learning that can occur is face to face, but rather that social activities are an important and significant part of the process. As Mayes (Mayes October 10th 1997) puts it: Lectures are occasions where the individual is confirmed as a member of a learning community. One of the challenges in online learning is to create this feeling of belonging to a community. To encourage and facilitate this important social aspect of learning and encourage students to assist each other, the players helping others in the game are to be rewarded according to rules built into the game. During a quest the player can be given any amount of help, as long as this help does not violate the basic principle of the learning experience. The aim and focus for completing the quests are to learn. The results from quests do not influence the judgement of the outcome of the assessment process, but it will give access to the assessment, this to allow the student to receive any help necessary from the game environment, fellow students or instructors during game-play. When the student have completed the required and desired number of quests and collected a number of items/concepts they will move on to a special part of the game, the assessment that will decide if the progression to the next level is permitted or blocked. The assessment requires the student to combine the items/concepts collected through game-play i.e. exploring the learning material. The outcome of the assessment will be one of: Everything is fine, go on; Mostly fine, you may progress but are recommended to go back and revisit some quests; Progression blocked, the student has not shown sufficient understanding and the required learning objectives have not been met and the student is required to return to the same level as previous and redo some or many identified quests before being allowed to attempt the assessment again. The basis of narrative based assessment The student created narrative based assessment is based on the student demonstrating their understanding of a topic by creating a narrative. That narrative usually takes the form of the exposition of a series of facts and the relationship between them, as a means of describing the overall conceptual view of the topic under consideration. The basis for this is how the student is constructing their learning. The use of student created narrative based assessment is assessing how the student is obtaining learning and developing understating. Educators generally have been consistent in arguing that higher education should be about encouraging students to take a deep approach to their studies, i.e. they should learn the material and not just memorise it so they can retell in exams. One of the major challenges of this is that even if the teaching of the students does take a deep approach it does not necessarily lead to the desired learning outcomes. Assessment, as so clearly shown by (Boud 1990; Biggs 2003), drives the learning process and over-rides practically every other aspect of curriculum design, students will be guided by the assessment and not by what they are told. The general understanding is in order to get the best learning and most proper assessment, the assessment should be designed into the system and be an integral part of the learning process. The importance of having the assessment as an integral part and not something added just at the end is well documented.(boud 1995; Biggs 2003) Assessment has two goals, covered by formative and summative assessment. Formative assessment is designed to aid the learning process, it is a mechanism to encourage student reflection by the means of synthesis of their existing understanding and then reflection on the feedback they receive on that understanding. The summative assessment evaluates the students knowledge in the area. It is important that summative assessment is included as part of the course, not solely at the end. From definition by The American Association for Higher Education, November 1995 Assessment is an on-going process aimed at understanding and improving student learning. The evaluation of courses must contain both formative and summative assessment. Traditional assessment is based on students producing essays on demand. Producing a well written text will for most people require several revisions and consequently the time to make them. In addition the added stress of and exam situation have an adverse effect on the students ability to produce a well written text. The outcome of this is that the student will fall back on knowledge-telling strategies and we will not demonstrate the full extent of their knowledge. Cho(Cho 2003) in his paper Assessing writing: Are we bound by only one method? asks the question How realistic and fair is it, then, to expect somebody to draft a well-organized essay in less than an hour on a topic that the writer may or may not have thought about before?.
According to Scardamalia and Bereiter(Bereiter and Scardamalia 1987), the most influential model of the cognitive processes in writing is the one proposed by Hayes and Flower in 1980 this model describes three cognitive processes-planning, translating, and reviewing, their main contribution of this model was showing that these cognitive processes are not necessarily strictly sequential; rather, these processes may interact recursively with one another. In addition to this Scardamalia and Bereiter divides the method selected by individual writers into two major categories: knowledge telling and knowledge transformation. The knowledge telling process is when the memory is searched for content relevant to the topic and whatever comes to mind is written down. Knowledge transforming process is a more complex process in which goals are incorporated into the planning process and only content relevant to both the topic and the goals is written down, this process does also include multiple revisions of the produced text. A more detailed description can be found in Assessment of Planning, Translating, and Revising in Junior High Writers (Berninger, Whitaker et al. 1996) Research by Powers and Fowler(Powers and Fowles 1996) have shown that even small increases of time at written tests can give significant improvements in test score. The same results have been reported by others, in addition to the results time constraint will also influence the writing style, in our case we will give the students ample time to prepare and their writing style should not be influenced by time at all. This is important as we do not want the students resorting to knowledge-telling strategies as described by Bereiter and Scardamalias(Bereiter and Scardamalia 1987) model, we want the students anchored in the constructivist model, transforming the knowledge and making it their own. The strategy of knowledge-telling is a low level strategy that is triggered by imminent deadlines and other stressful factors, the problem of students falling back on this strategy is a problem that out game based environment should be able to deal with as we are tracking the students though the learning process, and should be able to spot this. The distinction between knowledge telling strategies and knowledge transformation strategies are similar to the distinction in learning between deep and shallow learning, described by Marton and Saljo (Marton and Saljo 1976). We are proposing a novel solution to the problem of, producing essays on demand, by assisting the students in creating a coherent narrative during the learning to demonstrate that the learning have taken place. There are distinct differences between student-created narrative based assessment and traditional essay style questions, though they both is based on the notion of students creating essays to demonstrate that learning have taken place. Traditional essay style assessment is based on the students to construct and develop their learning over a period of time and they are then placed into a situation where they are required to demonstrate that this learning have taken place over a short period of time. The assessment is performed after the learning has taken place and is not integrated into the learning process. In the proposed student created narrative based assessment the creation of the narrative by the student is seen as an integral part of the learning process. One of the key concepts for us is that, we find that during the creation and development of a narrative the students are developing their understanding, in addition the resulting narrative can be used to demonstrate understanding of the topic area and thereby prove that learning have taken place. Our assessment model does not pose any time limit on the students, neither is it imposing any limit on number of attempts. Students are allowed to use the feedback from the assessment of their constructed narrative in a formative fashion if that is their desired learning strategy. We are describing something more than the traditional essays questions, and our addition is not in technology or environment e.g. essay style questions online etc. What we are concerned with is that in creating a narrative, a student has to be able to understand their topic and they have to be able to express this understanding. The creation of this narrative is a supported constructivist process, with scaffolding that gradually fade as the student progress up the levels and narrative that are adapted to the concepts we are assessing. Narrative as we understand it, a mathematical expression, a storyboard, a diagram or an essay, are a excellent way to demonstrate conceptual understanding. Narrative based assessment in a games environment The authors have previously(graven and MacKinnon 2006) published a design for a game based virtual learning environment. This virtual learning environment utilises different starting points, abilities and other gaming elements to personalise the environment for individual students. A game based system will provide a testing environment for the student created narrative based assessment. Exploration and the freedom to direct one s own learning is fundamental to constructivist methods.
The aim is for the student coming through the game gain concepts and understanding in a constructivist and exploratory model, the authors argues that this is inherent in MMORPG games, in that they allow just such exploration and freedom. A MMORPG typically offers the player a number of options like available paths and environments to explore and different quests to attempt. The students are the faced with a construction of a narrative in a constructivist fashion, with scaffolding that gradually fade as the student progress up the levels and narrative that are adapted to the concepts we are assessing. Showing students the entrance and then letting them get on with their learning is not an efficient or even sensible strategy for education, so new students in the system will need to be guided and led, while maintaining enough freedom to allow them to explore and direct their own learning. The student will collect a number of items or concepts during the learning phase. When they have collected the required or desired number of elements/concepts, and have acquired an understanding of the context of these elements and how they fit together, they will progress and attempt to complete the assessment. The students are free to explore their environment and the elements/concepts that the student are requires to collect and understand, are not required to be collected in any specific order. The context in where and how the collected items and concepts are found and are the basis for how these are to be understood, and later used to demonstrate this understanding in the constructed narrative. At the point were they have completed a predetermined number of quests the student will be allowed to progress to the assessment part. The reason why the student will be required to perform a certain number of quests is to ensure that they have done work. At the point of allowing the student through the assessment part, we know from the requirements that the students have completed some quests requiring them to undertake some work on the material: Due to the systems ability to track them we also know what quests they have attempted and how they performed can be utilised in later feedback, as recommendations after completed assessment attempts. Spending some time working with the learning material will also give them a clear understanding of what is expected of them and a opportunity to get involved in the social dimension of learning and possibly help others. The assessment will be a task set for them where they are required to combine the collected elements/concepts into a narrative. The created narrative may require more elements than they have collected, particularly in scenarios where the students are allowed entry into the assessment without the requirement to complete all quests. In addition it will present them with challenge of combining the elements they have collected into a coherent narrative. The assessment is divided into two parts. The first part is to help the student to construct a narrative, we are here supporting the student. The student is supported via scaffolding, this means that they are given controlled help and possibly a structure where they can fit the previously collected concepts together. Depending on which level the student have progressed to within the game, the amount of support provided are going to vary. In the early stages/levels support will be plentiful, as the student progress the support will fade and at the last stages may be completely removed. In the early levels the focus will be to teach the student how to construct a narrative, and it is therefore a necessity to keep the requirements of the created narrative simple in this phase. The help provided to the student will be to identify the important concepts and how to construct an argument, how the information interrelates and how to then sequence the information into the constructed narrative. All this support information can be modelled as standard learning material, i.e. in a quest form, inside the assessment part of the environment. After the student has explored this supporting learning material on how to create a narrative and is ready to complete the assessment exercise proper, they will step into the final part where the assessment will be done. Into this final stage only the items/concept collected during the quests will be allowed. The task set to the student will be to use the elements that they bring with them and arrange them into whatever the task specifies: a model, an equation, an essay or whatever the assessment requires. The previous given support on how to create a narrative in this particular assessment will guide the student to the appropriate form of narrative. In the cases where the students have completed only a number of quests and not collected all available concepts and items available to them in the game, the designer of the task will have to make a decision on how this is to be handled. Assessing the constructed narrative The assessment of a narrative is a time consuming task and students immersed into an environment, such as a games environment requires fast feedback to maintain a state of flow(csikszentmihalyi 1990), have I understood
this, passed the test and able to continue or not? To accommodate this we need to build in mechanisms for automated assessment into the virtual learning environment. Automated verification of solutions created by the students can be hard or reasonably straight forward depending on the type of solution expected. Various forms of automatic assessment have been carried out with success over years.( Foxley, Higgins et al. 2001). Assessments using multiple choice questions, point and click interface, diagrams or mathematical expressions have been successful. It must be noted that all these assessment have been constructed in a fashion similar to multiple choice questions or fill in the blank and not as free drawn diagrams or equations. Automated assessment of discursive or essays solutions are not straight forward. (Larkey 1998; Foltz, Laham et al. 1999; Landauer, Laham et al. 2000; Lancaster and Culwin 2001; Shermis, Mzumara et al. 2001; Kanejiya, Kumary et al. 2003; Lonsdale and Strong-Krause 2003; Duwairi 2006) An assessment in essay format contains a variety of relationships between concepts and even how these concepts are expressed will influence the correctness of the answer. Some of these solutions can be assessed through pattern based solutions and others may be assed on the interrelationship between the objects expressed, we can here look at object/concept proximity. Some of the methods used previously have in more recent times been used as the basis for some plagiarism detection tools. This led us to investigate a possible new use of these tools. Our proposed basis for developing automated assessment and fast feedback is the utilisation of grading of student work using tools developed for plagiarism detection. There will be instances were we will expect the student to present a narrative contain items/concepts in a specific context and in a certain order, because the concepts have a relationship with one another. Some of the concepts we expect the student to present will have a fixed relationship and the order are important, others will have a more free form and must be allowed in a variety of orders. The context of concepts presented to the students in the earlier quest/ learning material will influence how they incorporate them into their constructed narrative, but the assessment system must not require this to be the case. Unless the contexts in which these concepts are presented are the only valid, other contexts must be allowed. The authors will utilise algorithms tested and published in open-source and or/free tools as a basis for our system, an overview of available tools have been produced by Lancaster and Culwin(Lancaster and Culwin 2004; Lancaster and Culwin 2005). There are several clear advantages with working with free tools, the first is that in the proposed environment we will utilise these plagiarism detection algorithms and tools in a new and not by the original designers intended fashion. Free tools do publish the criteria selection for the metrics they utilise for assessing the written essays, this provides us with the confidence needed for the automated assessment. To be able to produce a result with a high degree of certainty, the system will need to use for than one set of plagiarism algorithms. One set of algorithms used will need to be based on structural metrics, where the documents first are tokenised. The process of tokenisation takes the form of removing as much of the unnecessary information and fill text as possible from the students submission. The processing of the text will involve identifying a set of words, expressions and concepts and substituting these with tokens. The next step in the process will involve the comparison of chains of these tokens with previously prepared targets. The number and length of token chains found to be equal to the target token chains will give a measure of how close the student created narrative are to the target. One variation of the token algorithms described above would be to perform the same comparison described on the chains of tokens on a untokenised text, in this case the comparison would be finding runs of similar words. To create a better chance of detecting similarities the text would need to be pre-prepared to remove spelling errors, grammatical errors and other factors that have not influenced of the degree of understating demonstrated by the student. One other set of algorithms used to grade the student created narrative can be standard grammatical analysers similar to one found in word processors. These algorithms can verify that the text produced by the students is a full text and not just listings of words and concepts encountered though the learning process. Another source of tools to verify that the narrative produces meets with a minimum set of requirements to be considered a text, is general essay assessment tools(shermis, Mzumara et al. 2001; Hamp-Lyons 2002; Shermis and Burstein 2003).
There are multiple tools to detect plagiarism the authors have been investigating some in particular: Visualisation and analysis of similarity tool (VAST) and Sherlock: Visualisation and analysis of similarity tool (VAST) (Lancaster and Culwin 2001; Lancaster and Culwin 2004) been developed by the Centre for Interactive Systems Engineering, based primarily at London South Bank University. VAST is primarily used to investigate two documents and trace out the similarities within them. The tool works with words both sequenced and unsequenced, and produce metrics on both. The visualisation part of the toll will not be utilised in the games environment but the metrics will be of great value in determining the degree of understanding demonstrated by the student. Sherlock is developed at School of Information Technologies at University of Sydney, Australia. Rob Pike created the original version. Sherlock is a tools designed to find similarities between text documents by utilising digital signatures, formed by turning several words in the input into a series of bits and joining those bits into a number. Conclusion The assessment approach described in this paper differs from traditional narrative based assessment. In that it provides a support mechanism for the student in their skills in narrative construction, and formatively feedback information to aid the students to discover the conceptual linkages between the elements of information that they have gained within the context of the topic they are studying. The summative assessment that then follows can also be used formatively by the student to continue to develop their understanding until they reach a point when they achieve a required threshold, from both the system and their view. The student is in control of the process and may progress when the required level set by the system is reached or postpone the progression until they are satisfied with their own level of understanding. The mechanisms to provide this fast feedback within the system are based on automated assessment tools utilising plagiarism detection software, this is also novel. We have here shown the basis for the assessment system and a outline design for the complete system and are now moving on to implementing actual working prototypes. References Bereiter, C. and M. Scardamalia (1987). The psychology of written composition. Hillsdale, N.J., L. Erlbaum Associates. Berninger, V., D. Whitaker, et al. (1996). "Assessment of Planning, Translating, and Revising in Junior High Writers." Journal of School Psychology 34(1): 23-52. Biggs, J. B. (2003). Teaching for quality learning at university : what the student does. Phildelphia, Pa., Society for Research into Higher Education : Open University Press. Boud, D. (1990). "Assessment and the Promotion of Academic Values." Studies in Higher Education 15(1): 101-111. Boud, D. (1995). "Assessment and Learning: contradictory or complementary?" ssessment for Learning in Higher education: 35-48. Boyle, T. (1997). Design for multimedia learning. London, Prentice Hall. Cho, Y. (2003). "Assessing writing: Are we bound by only one method." Assessing writing 8: 165-191. Csikszentmihalyi, M. (1990). Flow : the psychology of optimal experience. New York, HarperPerennial.
Duwairi, R. M. (2006). "A framework for the computerized assessment of university student essays." Computers in Human Behavior 22(3): 381-388. Foltz, P. W., D. Laham, et al. (1999). Automated essay scoring: Applications to educational technology. EdMedia'99. Foxley, E., C. Higgins, et al. (2001). The CourseMaster Automated Assessment System a next generation Ceilidh. Computer Assisted Assessment Workshop. Graven, O. H. and L. M. MacKinnon (2006). Exploitation of games and virtual environments for e-learning. ITHET. Sydney, Australia, IEEE Press. Hamp-Lyons, L. (2002). "The scope of writing assessment." Assessing writing 8: 5-16. Hext, J. B. and J. W. Winngs (1969). "An automatic grading scheme for simple programming exercises." Communication of the ACM. Kanejiya, D., A. Kumary, et al. (2003). Automatic evaluation of students answers using syntactically Enhanced LSA. HLT-NAACL03. Lancaster, T. and F. Culwin (2001). Towards an error free plagiarism detection process. ITiCSE 2001. Lancaster, T. and F. Culwin (2004). Using freely available tools to produce a partially automated plagiarism detection process. ASCILITE, Perth. Lancaster, T. and F. Culwin (2005). "Classification of plagiarism detection engines." Italics e-journal 4(2). Landauer, T. K., D. Laham, et al. (2000). "The intelligent essay assessor." Ieee Intelligent Systems & Their Applications 15(5): 27-31. Larkey, L. S. (1998). Automatic essay grading using text categorization techniques. International Conference of Research and Development in Information Retrieval. Lonsdale, D. and D. Strong-Krause (2003). Automated Rating of ESL Essays. HLTNAACL-03 workshop on building natural language processing,. Marton, F. and R. Saljo (1976). "On qualitative differences in learning I. Outcome and process." British Journal of Educational Psychology 46. Mayes, J. T. (October 10th 1997). Dialogue with a dumb terminal. Times Higher Education Supplement. Powers, D. E. and M. E. Fowles (1996). "Effects of applying different time limits to a proposed GRE writing test." Journal of Educational Measurement 33(4): 433-452. Shermis, M. and J. C. Burstein (2003). Automated Essay Scoring, Lawrence Erlbaum Associates. Shermis, M. D., H. R. Mzumara, et al. (2001). "On-line Grading of Student Essays: PEG goes on the World Wide Web." Assessment and Evaluation in Higher Education,. Aarseth, E. (2002). Quest Games as Post-Narrative Discourse. Narrative across Media, The Johns Hopkins University Press.