MULTIMODAL REFORMULATION DURING SHARED SYNCHRONOUS NOTE-TAKING AND ITS POTENTIAL PEDAGOGICAL CONSEQUENCES FOR TEACHERS AND STUDENTS

MULTIMODAL REFORMULATION DURING SHARED SYNCHRONOUS NOTE-TAKING AND ITS POTENTIAL PEDAGOGICAL CONSEQUENCES FOR TEACHERS AND STUDENTS Kristine Lund*, Gregory Dyke, Jean-Jacques Girardot Kristine.Lund@univ-lyon2.fr, Gregory.Dyke@emse.fr, Jean- Jacques.Girardot@emse.fr *CNRS, Université de Lyon, ICAR, École des Mines de St. Etienne, RIM Abstract In this paper, we show how analyses of teacher talk and student note-taking performed by dyads within a computer-mediated shared text writing interface can illustrate the ways in which talk is reformulated into writing and restructured, paving the way for deeper analyses concerning student knowledge construction. Note-taking is viewed as an epistemic activity to be cultivated within pedagogical situations. We also show how student coordination in the note-taking space can influence possibilities for collaborative knowledge structuring and we illustrate the opportunities that viewing students note-taking gives teachers. Key Words Multimodal reformulation, note-taking, computer-mediated shared synchronous text writing, knowledge construction Introduction Although there has been much research conducted on how students take notes (e.g. Sutherland, Badger & White, 2002) the techniques often involve analyzing the form or characteristics of written notes according to pre-established normative categories and interviewing students to find out why and how they took notes. Suritsky & Hughes (1991) define note-taking as a series of activities involving listening, understanding and assimilation, making a choice of what to record and even include revising. One of the major currents of research looks for patterns in note-taking and attempts to measure their efficiency in relation

2 to students pedagogical results. For example, the value of detailed notes rises as time passes between when the notes were taken and the evaluation of students knowledge (Williams & Eggert, 2002). In other words, if students take detailed notes, they obtain higher scores when tested later than earlier in relation to when notes were taken. In a similar vein, Nye, et al (1984) found a strong correlation between the quantity of notes taken and students performance during exams. In this study, we prefer to adopt the viewpoint of Castello & Monereo, (2005), who have worked to turn note-taking into a tool for conceptual change and knowledge construction (called epistemic note-taking) rather than see it be used as a kind of data-collection system incorporating an external memory function with the sole goal of storing information for the student to later memorize and partially reproduce for a future exam. Firstly, these authors taught students different note-taking procedures (e.g. those focusing on information repetition, vs. enhancing comprehension and restructuring content) that were adapted to particular note-taking contexts. Secondly, they turned students notes into a formative assessment tool through a process of teacher guidance and regulation with the goal being that students use note-taking for self-regulation of their own learning. Finally, these authors established note-taking teams in the classroom with the objective of promoting collaborative learning through the use of activities which emphasize peer-assessment. In our own work, we have not undertaken such an ambitious and comprehensive project at this stage. However, we did focus on studying the process of note-taking using a synchronously shared text editor and thus put together an empirical study that allows us to record and replay this process over the duration of many sessions. Our goals were firstly, to study on a micro-analytical level, the ways in which pairs of students chose to reformulate teacher talk into notes (both temporally and qualitatively) when the students write in the same shared interface. Such a microanalytical understanding could help both students and teachers become aware of their respective note-taking and teaching strategies and is a first step towards improving the pedagogical situation that was the basis of our empirical study. Secondly, we wondered in what ways cooperative face-to-face note-taking by students could make itself felt in the tutor s pedagogical activity. In what follows, we further motivate our approach and we present our empirical study, our method and our preliminary results. Finally, we end with a discussion and perspectives for future work. Multimodal Reformulation The authors Kress & Van Leeuwen (2001) define modes as the abstract non-material resources that help us to understand our surroundings (e.g. the sense of touch, hearing, smell, taste, but also ways of expression like gesture and speech). They define media as specific material forms in which modes can be expressed (e.g. the human body, a mechanical body, a computer). A mode such as writing can be carried out within at least four different media (firstly, a human hand with pen and paper, a typewriter or a computer keyboard and secondly a robot s hand manipulating some kind of writing device). In our empirical situation described in the next section there are essentially two different modes used by participants: talk and writing. A teacher and two students speak together face-to-face and also type with a personal keyboard in a shared note-taking space. Such a situation changes the definition of note-taking given above (Suritsky & Hughes, op. cit.) in that two additional factors must now be considered: 1) does one student take into account the other student s note-taking and if so, how? And 2) does the teacher react to students note-taking and again if so, how?

3 Apothéloz (2001) gives us a definition of reformulation useful for our empirical situation e.g. continually showing every step of the way the manner in which what has been formulated is articulated with what has already been formulated. In our case, the reformulation can be multimodal (from talk to writing or writing to talk) and also multi-actor (teacher to student, student to student or student to teacher). Our third research question takes into account the multimodal nature of the interaction we study how can we describe the multimodal reformulations in our corpus? And finally, how can reformulations bear witness to students knowledge appropriation? Empirical Study Within the LEAD 1 European research project, we observed nine dyads over the course of three to five meetings with their teacher for an introductory-level computer-programming project at the École des Mines in St. Etienne, a total of 33 meetings varying from 30 minutes to one hour in duration. These meetings took place face-to-face with the assistance of a chat and a real-time shared text editor, both of which the two students and the teacher had access to, on their laptops (cf. Figure 1). Figure 1. The empirical situation The tutorial trilogues were not designed to follow a specific pedagogical scenario. Instead a rough parallel could be drawn between them and industrial project review meetings where the teacher could correspond to either the person who ordered the program, a consultantpedagogue or a programming expert and the students to the design and development team. Aspects discussed during the meeting included task understanding and choices about program design (e.g. functional specifications, implementation). Variously, during the meeting, students took notes, defined tasks to be accomplished before the next meeting, and prepared the layout of their final report and presentation. The data collected for each session includes multi-track audio (one track for each participant), video, the interaction log-file produced by the shared text editor and the chat (implemented in-house on the DREW 2 collaborative software platform Corbel, Girardot & Jaillon, 2002) as well as field notes taken by an observing researcher. Conditions varied among dyads: Two supervising tutors were involved, supervising three different projects. 1 The European project LEAD (Technology-enhanced learning and problem-solving discussions: Networked learning environments in the classroom) is funded by the 6th framework Information Society Technology LEAD IST-028027. http://www.lead2learning.org/ 2 The Dialogical Reasoning Educational Web tool (DREW) was designed and developed within the SCALE project (Internet-based intelligent tool to Support Collaborative Argumentation-based LEarning in secondary schools, funded by the 5th framework Information Societies Technology SCALE IST-1999-10664.

4 Method In order to answer our research questions, it is essential to be able to view (replay) the dialogue transcript/video in combination with the interaction log files (chat, text-editor) and to understand how the text was written in the shared text editor. Currently, the dialogue of ten out of the 33 sessions have been transcribed with Elan (http://www.lat-mpi.eu/tools/elan/) in order to timestamp the beginning and end of dialogue turns, important for synchronizing with typed note-taking. These files were imported into Tatiana (Trace Analysis Tool for Interaction ANAlysts), developed in part within the LEAD project (Dyke, Lund & Giradot, 2009; Dyke, Girardot, Lund & Corbel, 2007; Corbel, Girardot & Lund, 2006). Figure 2 shows a view of the Tatiana interface, containing a set of replayables, defined as artifacts retaining some notion of ordering of events and interactions in time. All replayables can be replayed in synchronization. Figure 2. Various replayables in Tatiana: traces of a shared text editor (top left), transcription of talk (middle left), writing units obtained from text trace (top center), visualization of reformulation (bottom left), synchronized with external tools, DREW replayer (top right), video player (middle right), remote control (bottom right). Exploratory analyses were carried out by one of the authors by categorizing both tutor dialogue and student note-taking in terms of content as defined by keyword or phrase. Figure 3 shows the transcription and the writing units taken from the shared note-taking interface being categorized with the same schema. Content expressed for the first time in one of the modes was assigned a color and a keyword or group of words and if it was judged that this content gave rise to reformulation then the reformulation was categorized in the same way (e.g. the teacher talks about graphical interfaces and the student writes graphical interface ).

5 Figure 3. A screen shot of Tatiana showing categorization of talk and writing. Decisions on what was considered as reformulation were subsequently checked and agreed upon by the other authors. Such labeling made possible the automatic generation of the temporal visualization above at the bottom of Figure 2 where the transcription of teacher talk is represented by the middle line (almost continuous talk) and whereas the lines above and below are student A and student B s note-taking. The timescale is the same between dialogue and note-taking. Figure 4 shows how the researcher can manually make links between events occurring in different parts of the corpus in order to further analyses. For example, one can select an element in either the transcription of teacher-talk (top) or in the student s writing units (bottom left) and then in the bottom right part of Figure 4 it becomes possible to link these two elements as well as qualify the nature of the link. Here, an arrow is drawn, thus creating a link signifying a reformulation, but the arrow is not named. In future analyses, we expect to characterize the nature of the reformulation. Figure 4. A screen shot of Tatiana showing how links can be made between events in a corpus (e.g. between talk and writing).

6 In the next section, we will show how the arrows that researchers create between elements of talk and writing in this corpus can be used to visualize moments of multimodal reformulations in a temporal manner. Preliminary Results Here we present a series of moments of multimodal reformulations discovered during the analysis of two sessions (the same teacher with two different pairs of students). Figure 5 and Figure 6 show two instances where the teacher noticed what the students were writing as notes and reacted to them. In the first he explains what a Turing machine is when the students question the concept in the notes and in the second, he encourages them to go further in planning the different steps they must take in organizing their computer-science programming project when he sees that they remain quite general in their note-taking. Note that the first instance is a written incitement to reformulate talk in a more explicit way while the second is an instance of inciting by talk to reformulate existing text in a more explicit way. Figure 5. The teacher (session 1) answers a question he sees written in the shared interface. Figure 6. The teacher (session 2) encourages the students to be explicit in stating the steps in a process (talk and written text in original French). In Figure 7 the teacher is talking about a program called the game of life and is explaining the constraints that need to be respected during the programming that the students will be doing. We can see from the figure that the student writes down the conditions of a cell being born before the teacher talks about it so this seems to be an example of a student having some notions about the material the teacher is presenting. This example contains reformulation from talk to writing, coupled with new content added to the reformulation, content that appears immediately afterward in teacher-talk.

7 Figure 7. Student B seems to anticipate in his notes what the tutor will say. Figure 8 shows how the two students emphasize different parts of the teacher dialogue in their notes, but also how they build on each other s contributions. The textboard colors show which student types what text (red for student A and blue for student B). Student A adds implementations possibles: after student B writes au depart, NxP cases, thus structuring the notes within a larger context.

8 Figure 8. Student A and B s textboard with their notes at the end of the sessions in the original French (above) and a visualization showing Student A and student B choosing different aspects of teacher-talk to include in their notes (translated into English). Our next example (Figure 9) also illustrates how two students may have coordinated their note-taking. Student A begins to write about the description of musical notes and opens a parenthesis potentially to list the elements that make up that description. But then he erases that parenthesis. It may be because just before he erases it, he sees that student B lists the elements that describe a musical note (its frequency, tempo, etc.). This would imply that student B considers the note-taking as a more collaborative enterprise where the goal is for both students to contribute, but without redundancies. Figure 9. Written co-elaborated student-student reformulation of teacher talk. Figure 10 and Figure 11 show how a student can take notes on content in a different order than that presented in teacher-talk. In the first example, only student A takes notes (although B is aware of the note-taking). The teacher is again explaining how to program the Game of Life; student A chooses to organize chronologically the information he hears in a different way in his notes. In the second example student B is taking notes and also changes the chronological order of the information. In order to ascertain to what extent the students understand the teacher-talk as evidenced by the restructuring of their notes, we must go back to the original data in context. Tatiana allows for such zooming in and back to primary data from a phenomenon that has been pinpointed. In order to fully comprehend the dynamics of the situation and ascertain the level of student comprehension, one must replay the interaction in order to see how note-taking occurred in relation to teacher talk. These analyses are under way.

9 Figure 10. Student A takes notes on a later part of tutor dialogue first and then takes notes on an earlier part of dialogue. Figure 11. Student B shows similar behavior to student A in Figure 10. Figure 12 illustrates a number of instances where both student A and student B type their notes while placing content in a chronological order that is different than what the teacher says. Our next step is to study further this content in order to describe the mechanisms of restructuring and reformulation (as has been begun for Figure 8 and Figure 10). It is also shown in this figure that in general, students type notes somewhat later than when the teacher-talk occurs. Despite the obviousness of this second result, such a visualization could make salient the places during a course where notes are taken at a much later than usual time-span.

10 Figure 12. Chronological disorder between note-taking and teacher-talk (The figure is in French the cloud on top points to the instances where writing notes begins after speaking and the cloud on the bottom to the instances where the most recent concepts are written down first, followed by concepts mentioned earlier in teacher-talk. Our next example (Figure 13) is a closer look at how teacher-talk is restructured differently by students in their notes. The figure is in French, but the colors give an initial idea of what each student chose to include in his notes as opposed to what the teacher said. On the right side of the figure, we see that student B was able to structure the teacher-talk into a coherent list, whereas student A began the list with what was to be done secondly, then follows it with what should be done first, showing less of an ability to restructure information on the fly. Figure 13. Differences in the structure of reformulation; phrases having the same color occur both in teacher-talk and in student notes, although sometimes reformulated. Our final example (Figure 14) is not a reformulation per se, but rather an event that takes place that has en effect on note-taking for the rest of the students time together. In the beginning of the interaction, student A and student B both began writing in the same area of the note-taking space which resulted in their typing being mixed together. In order to avoid that in the future one of the students named the space above in the area student A and the space below in the area student B. From thereafter, each student typed in his space, thus eliminating the possibility to build on each other s sentences. Figure 14. A conflict during synchronous writing provokes a spatial organization.

11 Conclusion and perspectives In this paper we have done a preliminary micro-analytic study of face-to-face multimodal interaction (talk and writing) between a teacher and two students. The main activity of the students was to take notes in a shared interface (shared also with the teacher) while the teacher explained computer-science programming concepts in the framework of a project the students were working on together. Participating teachers were used to meeting with students during such projects, but the shared interface was new for both teachers and students. Tatiana software (Trace Analysis Tool for Interaction Analysis) was used to synchronize corpus elements that were recorded and post-treated (e.g. teacher-talk transcription) or traced on the computer (e.g. student note-taking). These elements were analyzed in order to answer three main research questions in relation to collaborative note-taking. Firstly and in general, how can we describe the multimodal reformulations in our corpus? More specifically, how do students reformulate teacher-talk and how does a teacher react to note-taking? Can we pinpoint student or teacher strategies? We propose to discuss our results in part, within the framework developed by Castello & Monereo (op. cit) defining four key points illustrating quality of note-taking: personal organization do students organize notes in a way differing from the teacher's presentation (while maintaining proper relations between concepts)?, content amplification do students make use of other information sources and integrate what they hear into notes that combine both teacher-talk and their previous knowledge? level of reflection do students write down their doubts, questions or other reflective comments? quality of note synthesis do students write notes that are comprehensive, incomplete or selective in relation to the thematic units developed in class, and do they just copy down the teacher's talk or do they personalize their note-taking? Some of our examples showed how students restructured teacher-talk as they reformulated it (personal organization). In some cases, students did such restructuring in their own corner of the shared writing space, in others they took into account their partner s note-taking and built upon it (personal organization stimulated by awareness of another's organization). We also saw how an early conflict in writing could lead to separating the shared writing space into two distinct spaces, which led to separate note-taking and no collaboration. Such a result shows how characteristics of collaborative spaces may limit cooperative behavior. We found two specific cases of a teacher reacting to notes he saw being written in the interface: in one, the teacher explained a concept when the student wrote a question mark next to a word in the note-taking space (student's level of reflection) and in another, the teacher encouraged the students to develop their project planning when he saw that their notes were not very specific. Such a reaction can be construed as an opportunistic teacher strategy. We have not yet found any instances of content amplification nor have we performed analyses on the quality of note synthesis aside from noting particular moments of personalization of note-taking. Our current goal is to analyze in more detail how the reformulations characterized in the above four key points may shed light on students understanding of the subject matter. Some authors go so far as to compare note-taking activity with the egocentric speech as studied by Vygotsky (1981), that is, teacher-talk (public external speech) on the one hand and private and internal speech of thought of the student on the other hand (Castello & Monereo, op. cit). In this case, note-taking may be considered as a way of appropriating meaning. The nature of our corpus and the synchronization of the different elements of the corpus within Tatiana software renders possible the analyses it takes to answer this question.

12 Bibliography APOTHÉLOZ, D., 2001, «Les formulations collaboratives du texte dans une rédaction conversationnelle : modes d expansion syntaxique, techniques métalangagières, grandeurs discursives manipulées, etc.» dans Le processus rédactionnel. Ecrire à plusieurs voix Gaulmyn, M.-M. de, Bouchard, R. et Rabatel, A. (dir.), L Harmattan, Paris, pp. 49-66. CASTELLO, M. & MONEREO, C., 2005, Student s note-taking as a knowledge construction tool. L1-Educational Studies in Language and Literature, 5 (3); 265-285. CORBEL, A., GIRARDOT, J.J., & JAILLON, P., 2002, DREW: A Dialogical Reasoning Web Tool, ICTE2002, International Conference on ICT's in Education. Badajoz, Spain, pp. 516-521. CORBEL, A, GIRARDOT, J.J., & LUND K., 2006, A method for capitalizing upon and synthesizing analyses of human interactions, In (eds) W. van Diggelen & V. Scarano, Workshop proceedings Exploring the potentials of networked-computing support for face-to-face collaborative learning. EC-TEL 2006 First European Conference on Technology Enhanced Learning, Crete, pp. 38-4 DYKE, G. & LUND, K., 2007, «Implications d un modèle de coopération pour la conception d outils collaboratifs» dans Actes de la conference internationale EPAL ("Echanger pour apprendre en ligne: outils, tâches, interactions, multimodalité, corpus"), Grenoble 7-9 juin 2007, University of Stendhal. Publié en ligne sur le site : http://w3.u-grenoble3.fr/epal/dossier/06_act/actes.htm DYKE, G. LUND, K. & GIRARDOT, J.-J. (2009) «Tatiana: an environment to support the CSCL analysis process» dans (O Malley, C., Suthers, D., Reimann, P., and Dimitracopoulou, A., eds), Computer Supported Collaborative Learning Practices: CSCL2009 Conference Proceedings, pages 58-67, Rhodes, Greece. DYKE, G., GIRARDOT, J.-J., LUND, K., & CORBEL, A., 2007, «Analysing Face to Face Computer-mediated Interactions» dans Actes de European Association of Research in Learning and Instruction (EARLI) 07. Budapest, Hungary. IGO, L.B,,KIEWRA, K.A & BRUNING, R., 2008, Individual Differences and Intervention Flaws : A Sequential Explanatory Study of College Students Copy-and-Paste Note Taking. Journal of Mixed Methods Research 2:2, pp. 149-168. KRESS, G. & VAN LEEUWEN, T. 2001, Multi-modal Discourse. Arnold, London. LUND, K., VAN DIGGELEN, W., DYKE, G., OVERDIJK, M., GIRARDOT, J.J., & CORBEL, A., 2008, «A researcher perspective on the analysis and presentation of interaction log files from CSCL situations within the LEAD project» dans les Actes de International Conference of the Learning Sciences (ICLS) 2008, Utrecht, The Netherlands. NYE, P., CROOK, T.J, POWLEY, M. & TRIPP, G. 1984, Student note-taking related to university examination performance. Higher Education 13, pp. 85-97. SURITSKY, S.K. & HUGHES, C.A. 1991, Benefits of notetaking: Implications for secondary and postsecondary students with learning disabilities. Learning Disability Quarterly, 14:4, pp 7-18. SUTHERLAND P., BADGER R. & WHITE G., 2002, How New Students Take Notes at Lectures Journal of Further and Higher Education, 26:4, 1 pp. 377-388(12). WILLIAMS, R.L. & EGGERT, A.C. 2002, Notetaking in college classes: student patterns and instructional strategies. The Journal of General Education. 51:3, pp. 173-199.

13 Biographical notice Kristine Lund is a CNRS Research Engineer in the Human and Social Sciences at the University of Lyon, France where she is the vice-director of the Language Sciences laboratory Interactions, Corpus, Learning and Representations. Dr. Lund's research interests include the multimodal co-construction of complex knowledge in goal-oriented computer-mediated human interaction.