An Evaluation of Computer Mediated Communication to Support Group Discussion in Continuing Education

Transcription

1 Educational Technology & Society 3(3) 2000 ISSN An Evaluation of Computer Mediated Communication to Support Group Discussion in Continuing Education Dr. Rachel Pilkington University Research Fellow Computer Based learning Unit, University of Leeds Leeds, LS2 9JT United Kingdom Tel: Fax: R.M.Pilkington@cbl.leeds.ac.uk Dr. Catherine Bennett Project Officer Department of Psychology, University of York Heslington, York, Y10 5DD United Kingdom C.Bennett@psych.york.ac.uk Dr. Sarah Vaughan Senior CAL Co-ordinator King's College London Computing & IT Services, Strand London WC2R 2LS United Kingdom sarah.vaughan@kcl.ac.uk Abstract An evaluation of a taught module within a Post-Graduate Master of Education course is reported. Participants on the course were mainly teachers learning for continuing professional development. The evaluation studies the use of ICT to provide flexible learning support. A Virtual Classroom environment was developed using WebCT and within this environment CHAT seminars were used to discuss issues arising from students' pre-reading of set papers. Participation in seminars was evaluated in relation to the quality of both written assignments and collaborative group-work. The evaluation raised the issue of how to increase the participation of some students in on-line CHAT and group-work. Above average participation in the CHAT was accompanied by above average performance in group-work. However, there seemed to be no such relationship between speaking in the CHAT and performing well on the written assignment. From students questionnaires the main reasons for non-participation in CHAT seminars was difficulty in accessing the CHAT from home at the scheduled time. Issues arising from these findings are discussed in relation to structuring courses to meet the needs of full-time, part-time and distant students. Keywords Computer Mediated Communication, Flexible Learning, Networked Learning, Continuing Education, Discourse Analysis Introduction This study evaluates the use of Information and Communication Technology (ICT) to provide flexible learning including Distance Learning support for a Post-Graduate Master of Education course. To meet students needs a Virtual Classroom environment was developed using WebCT computer based tools. This environment is illustrated in Figure 1. Use of this environment was evaluated in relation to the quality of work produced by students and the nature of their collaboration and communication through the tools. Some preliminary results and emerging issues will be discussed. Why Flexible Learning? The student base is changing - those wanting to take post-graduate courses are no longer only recent graduates. Increasingly, students need to balance work and family commitments with the demands of learning new skills (McCombs, 1991). As a consequence Educational Institutions must respond rapidly and flexibly to meet students' needs. 349

2 Britain towards 2010 published by the Department of Trade and Industry says students no longer need to be resident at universities...the document dismissed objections about face to face interaction between student and teacher - for most students it does not happen under traditional methods - and lifestyle". (Donald MacLeod reporting in the Guardian HE, 25th January 2000). The argument is that, using ICT, flexibility can be provided for students living and working at locations distant from the University. However, to be effective ICT must meet requirements of quality. How do we meet these requirements given the financial constraints that inhibit face to face teaching? This is the problem for the design of the module critically evaluated here. Students enrolling on the course may be full-time, part-time or fully distant, International or Home students. For these students we not only need a course that delivers informative content but one that builds on existing employability skills, promotes critical thinking and stimulates students to become autonomous learners whilst, at the same time delivering practical ICT skills. Increasingly, society needs workers skilled in the retrieval, manipulation and dissemination of information (particularly but not exclusively through ICT). As a consequence of these shifting trends our Educational Institutions must themselves respond rapidly and flexibly to meet the need, not only for training in ICT, but training in the skills required for autonomous, life-long learning (Cowham, 1998). Designing for Flexible Learning - Our Approach Theories of learning suggest that deep conceptual understanding and the development of intellectual skills require active, constructive tasks (Piaget, 1970), that provide scope for verbal interaction (Vygotskii, 1978). Collaborative Learning exposes students to new information and alternative perspectives encouraging them to reflect on and revise conceptions (Blaye, Light, Joiner & Sheldon 1991; Littleton & Light, 1998). Articulation through Exploratory talk (inquiring, explaining and justifying) has been shown to be particularly beneficial for these purposes (Mercer and Wegerif, 1999). From the above discussion we have developed an approach (Barker & Pilkington, 2000) to guide course construction in a way that is grounded in theories of teaching and learning. This attempts to marry approaches based on setting learning objectives (Gagne, 1972; Bloom, 1976; Anderson & Sosniak, 1994) with a more learner-centred and constructive approach to setting learning tasks. This approach suggests that the course needs to provide opportunities for:! Acquisition - new knowledge needs to be acquired or constructed through exposure to new information.! Articulation - students need to test and elaborate this understanding further through self-explanation and discussion.! Application - to develop skills, there must be opportunity to apply knowledge by completing authentic tasks.! Assessment - constructive feedback from peers or tutor helps students to set and re-set learning goals. These stages may overlap and take place at different levels during the same activity. Acquisition could in principle take place in a variety of different ways, through self-paced individual reading, serendipitously in discussion, or by active 'discovery' (e.g. by using a simulation). However, exposure to (human or material) that can help students to grasp key concepts is required. To ensure meaningful learning rather than rote learning there needs to be reflective activity both during and following acquisition. This enables students to test and elaborate their understanding. Ideally there is a reflective cycle (Kolb, 1984) in which both learner and teacher are engaged. Thus, constructive feedback is used to refine understanding, pointing to gaps or inconsistencies in knowledge and prompting new goals to acquire new knowledge. Articulation of understanding in discussion can prompt such feedback from others but assessment addresses the need to present considered pieces of work for more explicit peer or tutor review. Feedback in the form of both student evaluation and student performance provides a similar role for the tutor in refining the setting of appropriate learning tasks. In applying this approach to the design of Networked Learning we have designed a Virtual Learning Environment using WebCT s integrated Computer Mediated Communication (CMC) and World Wide Web (WWW) Course Management Tools. Using these tools, flexibility can be enhanced for students living and working at a distance as follows:! Acquisition - where prior knowledge is variable students can learn at their own pace through access to online notes. These provide a more flexible alternative to the lecture enabling more articulation in face to face time. 350

3 ! Articulation - Computer Mediated Communication (CMC) CHAT and Bulletin Board tools can supplement face to face sessions (and for distant students replace face to face tutorials). Through these tools students can increase their interaction with peers and tutor giving opportunity for self-explanation, argument and constructive feedback.! Application - using CMC tools, and Web-Based Group-Space(s) students can apply knowledge by collaborating together to construct a resource (an authentic design task).! Assessment - formative and summative feedback can be given via CMC at various levels through a mixture of tutor set tasks/quizzes and peer or tutor comments via private CHAT Bulletin Board. The Module and Course The Module evaluated here "Learning and the New Technologies" is a core module in "MEd in Information Technology, Multimedia and Education" and "MEd in Learning Systems Design" courses. The module aims to develop students' understanding of learning theory and stimulate creative reflection on implications for the design of computer-based learning. A Virtual Learning Environment (VLE) using WebCT s CMC and World Wide Web (WWW) Tools was used to support the module. Figure 1 shows the home page with icons through which CHAT, Bulletin Board and Course Content were accessed. Figure 1. The Home Page The module ran for 12 weeks. It consisted of 11 two-hour face to face classroom sessions and 11 one-hour remote on-line CHAT seminars conducted through WebCT. One tutor and 2 demonstrators taught the module and there were 20 registered students (class of 1998). Half the students were part-time and half full-time students. There were no full Distance Learners enrolled for 1998 (data reported here). Thus, all students attended face to face sessions. Most (15/20) were teachers with an existing role in teaching ICT and seeking continuing professional development. Full-time students were mainly International Students for whom English was a second language but who were resident in Leeds to study. In contrast all part-time students were Home students, but many travelled considerable distances to Leeds. 351

4 Course Materials Lecture notes and Slides were pre-prepared, edited and uploaded into the WebCT environment. Lecture Notes were plain text with hyper-links to glossary definitions. Slides were downloadable as Power Point mixed media presentations. Having the course material on-line enabled acquisition to take place at a distance, as a self-paced and individual activity. The emphasis was on giving maximum opportunity for articulation and application in face to face time. The expectation was that students would preview each week's slide presentation before attending the face to face session for that week. Associated information on schedule, assignments, exercises and group-work was also available on-line. Figure 2. Course Slides On-Line CHAT Seminars The CHAT seminars were held at a regular time prior to the face to face session in order to discuss issues arising out of students reading of set papers. These evening seminars were tutor-led and content oriented. The aim was to provide a reflective and critical forum for discussion in which all students were encouraged to participate. The CHAT provided a useful means for the tutor to check on the understanding of key concepts to be reviewed in the face to face session the following day. A reconstruction of the CHAT Room in use is illustrated in Figure 3. The names of students participating have been changed but the dialogue is reconstructed verbatim. In this example students 'discover for themselves' the issue of whether self-explanation is truly self-explanation if it is prompted by the tutor's inquiries! Face to Face Sessions In the expectation that on-line course material was pre-read before attending face to face sessions there was no need for the tutor to give a lecture during face to face sessions. However sessions typically began with up to twenty minutes of orientation, reviewing issues arising out of the notes, CHAT and Bulletin Board discussions. This was followed by either a collaborative practical exercise or (from weeks 5-10) collaborative small-group project work. This structure enabled maximum time to be given to students to articulate their understanding and to apply it to constructive and interactive tasks - activities which are generally preferred in a face to face context. 352

5 Figure 3. CHAT Room in Use The Small-Group Project Work Design issues were made authentic for students by involving them in collaboratively producing their own mini project. From weeks 5 to 10, students organised their time, working in a group of 5 or 6, to produce a web-based resource (set of web pages). Students were free to choose the members of their group and the topic of their group s resource. Although the assessment of this mini-project did not count directly towards their grade, students were encouraged to produce a resource that would help them complete their written essay assignments. Figure 5 shows an example resource. Managing Collaborating Together Within the WebCT Virtual Classroom, private forums (private Bulletin Boards) were also set up to support small-group project work. This was aimed at helping students to manage their collaboration outside contact hours. Private and CHAT rooms were also available through WebCT. In week 10 students presented their pages and made their resource public for the rest of the class to view on the Web. They then peer-reviewed each other s work using the Bulletin Board and a supplied evaluation framework (see appendix). The tutor reviewed the presentation and content of each resource, and the balance and management of the group s collaboration to give feedback to each group. The Evaluation of the Module The evaluation of the module examines the use of the Virtual Classroom both to access course materials and to participate in on-line CHAT seminars on course reading. The quality and the quantity of the discourse produced 353

6 by students and tutor in the on-line CHAT was also evaluated using dialogue analysis techniques (Pilkington, 1999). In particular the evaluation examined:! Student responses on course feedback questionnaires.! Student participation in on-line CHAT seminars on course reading.! The quality and the quantity of the discourse produced by students and tutor in the on-line CHAT (See Pilkington, Treasure-Jones & Kneser, 1999 for details of the analysis method).! Student group-work scores.! Student written assignment scores. These were final essay assignment grades. Figure 4. Example of Student' Constructed Small-Group Resource (class of '99) Overall Student Impressions Fifteen out of twenty students returned an evaluation questionnaire. Overall, 80% of those giving a response rated the course quality as good to very good. Some students were supportive of the collaborative group-work and wished that assessment had been based on this, others saw problems in working at a distance. Part-timers did not value the on-line CHAT seminars as much as full-time (mainly) International Students. Patterns of Use From students questionnaires the main reasons for non-participation in the CHAT seminars was difficulty in accessing the CHAT from home due to other commitments at the scheduled time, or not having access to a Java enabled browser. From Table 1 it can be seen that six students did not participate in the on-line CHAT at all and five of these were part-time Home students. Most Home students showed little use of the CHAT. However, two Home students proved the exception to this rule producing 89% of all Home student s turns. No International student (all full-timers) failed to participate at all. International Students showed a higher level of participation than Home students (though turn length was shorter). Access difficulties also affected part-time students' approach to group-work. There was evidence from group-work evaluations that these students adopted a more co-operative than collaborative approach to constructing the on-line resource and from questionnaires, of a 354

7 frustration in not having sufficient common time to manage collaboration outside the face to face sessions. Table 1 is sorted by ranking students according to the number of weeks in which they spoke in the CHAT. It can be seen that this indicates that a low participation in the CHAT is likely to be accompanied by under average performance on the Group-work score. Group-work scores showed a significant positive correlation with number of weeks speaking in the CHAT (Spearman rank correlation r s = 0.74, N = 20, p < 0.01). This result needs to be treated with caution since each individual in the group is allocated the same grade based on rating the group resource produced. However, it supports the impression (from talking to students) that part-time Home students, frustrated by not having time to manage group-work, adopted a more independent approach to group-work which proved both less satisfying and less productive (see appendix for criteria in marking Group-work). Student Home or International Part Time or Full Time Number of weeks Spoke in the CHAT (average = 2.5) + = above average groupwork score Groupwork Score (average = 20.18) Number of Course Content Pages HIT (average = 409) + = above average Hits S17 H FT S02 H PT S06 H PT S19 H PT S09 H PT S20 H PT S15 I FT S07 H PT S16 H PT S11 I FT S12 I FT S14 H PT S18 H PT S03 I FT S13 I FT S05 I FT S04 I FT S08 I FT S10 H FT S01 H PT Table 1. Participation in CHAT Hits on Course Content Pages and Group-work Scores Participation in CHAT Seminars Table 2 divides students into Home students (resident and studying in the UK) and International (studying away from home in the UK). For the reasons above, most Home and International students showed different patterns of participation in the on-line CHAT. Many (60%) of Home students (who were mainly part-time students) did not take part in the CHAT at all whereas none of the International students failed to participate. Two Home students proved the exception to the rule, between them producing 89% of Home students' turns. These students both began the course as full-time students, although one later converted to part-time status. From a more detailed dialogue analysis of eight CHAT sessions the International students produced a similar number of words and turns as Home students, but these were more evenly distributed amongst them. Thus, although one International student was responsible for about 45% of all International students' words and turns, no International student logged in fewer than two times and no International student produced fewer than 2 turns or 23 words. Many International students also said on their questionnaires that they particularly valued reviewing the transcripts of the CHATs on the following day. Further dialogue analysis reported elsewhere (Pilkington, Treasure-Jones & Kneser, 1999) showed that International speakers had a slightly higher rate of initiating than Home students - indicating that they were not more reticent to speak than Home students were (though turn length was generally short). The proportion of quality content-related talk of an exploratory (critical and inquiring style) was a constant 45% across International and Home students alike, although the two dominant Home students produced a higher ratio of this type of talk than other students did. 355

8 Student Home or International Part Time or Full Time Number of words in CHAT Number of Turns in CHAT Number of exploratory style moves in CHAT Groupwork Score S01 H PT S06 H PT S07 H PT S09 H PT S10 H FT S14 H PT S16 H PT S17 H FT S18 H PT S19 H PT S20 H PT S02 H PT S03 I FT S04 I FT S05 I FT S08 I FT S11 I FT S12 I FT S13 I FT S15 I FT Table 2. Home and International Student Participation in CHAT Written Assignment Score Inclusiveness of Dialogue The on-line CHAT was a strongly tutor-led discussion. From a total of 9831 words and 664 turns over eight analysed seminars the tutor produced 4487 words and 248 turns. This represents 46% of words and 37% of turns. One full-time International Student and two Home students accounted for another 3638 words and 284 turns between them - leaving just 17% of the word total (15% of turns) to the remaining students. Looking at the quality of the participation, the proportion of on-topic Exploratory Dialogue (see Pilkington, Treasure-Jones & Kneser op cit.) was a constant 45% for International and Home students alike. The two dominant Home students produced a higher ratio of this type of talk than other students did (25% of the total number of such moves). However, the tutor accounted for 54% of such moves. This raises the issue of how the CHAT could be made more inclusive and whether students would have benefited from this. From an analysis of a private CHAT session between two students in a one-to-one situation, students were more active and inquiring privately than in a tutor-led CHAT on the same topic held later the same day. This may suggest that the tutor should take a more hands-off approach in seminars and encourage students to express themselves more. However, it also indicates that when space for one-to-one peer-to-peer interaction is provided it is spontaneously and constructively used by some students (see Pilkington, Treasure-Jones & Kneser, 1999). The Role of Assessment Table 3 is sorted according to mark achieved in the final written assignment. Two students failed the module as a result of not submitting a written assignment. The average group mark is calculated without including a mark for these two students listed at the bottom of table 3. It was noted earlier that there seemed to be a reasonably strong correspondence between participation in the CHAT and a good mark in the group-work assessment. However, students who made no use of the on-line CHAT were not less likely to produce a well-written assignment. Since it is the written assignment (and not the group-work mark) that contributes to the final mark, there was no assessment-based incentive to participate in CHAT or group-work. 356

9 Student Home or International Part Time or Full Time Number of words in CHAT (average = 255) + = above average number of words + = above average written assignment S17 H FT S20 H PT S06 H PT S09 H PT S10 H FT S18 H PT S11 I FT S02 H PT S12 I FT S13 I FT S01 H PT S05 I FT S14 H PT S19 H PT S08 I FT S15 I FT S04 I FT S16 H PT S07 H PT S03 I FT Table 3. CHAT use in relation to Group-work and Essay Assignment Marks Written Assignment Score (average = 58.5) Conclusions and Issues Raised Differences in participation in the CHAT seminars between Home (mainly part-time) and International (all fulltime) students emerged from the evaluation presented here. In particular International students showed a more even distribution of participation with more International students participating. The reasons for nonparticipation in CHAT seminars were mainly pragmatic. Some students (mainly part-timers) had problems attending the evening CHAT seminars. The p.m. time-slot for the CHAT was a compromise between fulltimers staying late and part-timers logging in from home. This proved ideal for few. This year, with 6 students registered as Distance Learners, the solution was to run two seminar slots, a lunchtime slot and an evening slot. This has doubled tutor contact time but has also doubled part-timers' participation in CHAT seminars on set reading. Difficulties in scheduling common time to CHAT also affected part-time students' approach to group-work. Difficulties in meeting face to face tended to lead towards more distributed and individual patterns of working, in turn limiting the score obtained for group-work. These difficulties continue to affect part-time students' approach to group-work this year. This year, 2 International full Distance Students (not attending any face to face sessions) were able to schedule lunchtime CHAT and Bulletin Board discussions with full-time campusbased students. This enabled them to collaborate in group-work, suggesting that (at least for part-time students) pressure of time and not the functionality of the tools is a more limiting factor affecting the quality of groupwork. That two students failed to submit a written assignment and several students required extensions due to work or family problems further suggests that time to study and common time to meet (on-line or face to face) are more critical factors for part-time students. However, in piloting the VLE with a small number of full Distance Students this year, first impressions are that CHAT and Bulletin Board tools have not fully met these students need for a sense of presence. One student suggested video-conferencing so that they could really meet their fellow students. Kahl & Cropley (1986) suggest Distance Students feel more 'isolated' than face-to-face students and experience lower levels of self-confidence as a result. This can lead them to drop out. Students who commented on this, rated face-to-face contact above CHAT and CHAT above Bulletin Board in giving them a sense of presence. The Bulletin Board was sparsely used by full-time, part-time or Distance students, despite being seeded by the tutor on each of the topics discussed in the CHAT seminars; despite the difficulties involved the symmetric CHAT and not the asymmetric Bulletin Board was the preferred discussion tool. The speed of response was too low for the Bulletin Board to meet this need. 357

10 It was noted that there was a significant correlation between participation in the CHAT and a good mark in the group-work assessment. However, there was no such relationship between speaking in the CHAT and the written assignment mark. Students who made no use of the on-line CHAT (mainly Home part-timers) were not less likely to produce a well-written assignment. However, this is probably true for students well able to frame an argument on the topic. Participating in the CHAT may have assisted students starting from a lower level of knowledge or inexperienced in academic argument in English. From the questionnaires, full-time International students tended to value the CHAT more (even those who were silent) and were also more likely to read CHAT transcripts posted the next day. It was difficult for some International students to keep up (in real time) with discussion in a second language. The transcripts provided an opportunity for students to check their understanding. As already noted, part-time students were under considerable pressure to produce good quality work in the time available to them, indeed, two students failed to submit a written assignment. Since it is the written assignment that contributes to the final course grade and not the group-work score there was no assessment-based incentive for part-time students to participate in the CHAT or the group-work. This raises the issue of whether skills such as oral debate, computer-based textual and graphical design and team-working skills (the skills acquired and applied in the CHAT seminars and group-work), should be formally assessed. For this to be effective a component of the assessment should probably be based on student logs of the collaborative process and involve a private peer-rated as well as a tutor rated component. The on-line CHAT seminars on set reading were strongly tutor-led. Based on previous research, CMC tools are poorly used unless the tutor actively facilitates discussion (Rohfeld & Himestra, 1995, p.91). From the dialogue analysis (Pilkington, Treasure-Jones & Kneser, 1999) the tutor interacting with two or three active students was effective in modelling Exploratory Dialogue. However, the analysis revealed a need to encourage more inclusive participation. The value of differing roles for the tutor (e.g. explaining, inquiring or encouraging) and the ideal balance between these is now the subject of further work. All students benefited from some aspects of the Virtual Learning Environment. Part-time and distant students benefited from open and flexible access to course material and those who chose to participate in CHAT seminars benefited from not having to travel to meetings. Full-time International students benefited from flexible access to written transcripts of tutorial discussion as well as text-based lectures. When the VLE is seen as an add-on support to an existing course (a course with problems in meeting the different needs of part-time, full-time, Home and International Students), then the advantages are clear. This additional support was not achieved without costs both in terms of additional development time and additional student contact hours. Whether these benefits will prove sufficient to enable full Distance Learners to achieve comparable results remains to be seen. References Anderson, L., W. & Sosniak, L. A. (1994.). Bloom's Taxonomy: a Forty Year Retrospective, Chicago, IL: National Society for the Study of Education. Barker T. & Pilkington, R. (2000). An Investigation and Design of Networked Learning in Inner-City Leeds. Paper presented at the Networked Learning 2000, April, Lancaster. Blaye, A., Light, P., Joiner, R. & Sheldon, S. (1991). Collaboration as a facilitator of planning and problemsolving on a computer-based task. British Journal of Developmental Psychology, 9, Bloom, B. S. (1976). Human Characteristics and School Learning, New York: McGraw-Hill. Cowham, T. (1998). Challenges for change - information and communication technology and lifelong learning. Paper presented at the International Conference Networked Lifelong Learning, April, The University of Sheffield, Sheffield. Gagné, R. M. (1972). Domains of Learning. Interchange, 3 (1), 1-8. Kahl, T. N. & Cropley, A. J. (1986). Face-to-face versus distance learning: psychological consequences and practical implications. Distance education, 7 (1), Kolb, D. A. (1984). Experiental Learning: Experience as the Source of Learning and Development, Englewood Cliffs, NJ: Prentice-Hall. 358

11 Littleton, K. & Light, P. (1998). Learning with Computers: Analysing productive interactions, London: Routledge. Mercer, N. & Wegerif, R. (1999). Children's talk and the development of reasoning in the classroom. British Educational Research Journal, 25 (1), McCombs, B., L. (1991). Motivation in Lifelong Learning. Educational Psychologist, 26 (2), Piaget, J. (1970). Science of education and the psychology of the child (Derek Coltman, Trans.), New York: Orion Press. Pilkington, R. M. (1999). Analysing Educational Discourse: The DISCOUNT Scheme (CBLU Technical Report 99/2), Leeds: Computer Based Learning Unit. Pilkington, R. M., Treasure-Jones, T. & Kneser, C. (1999). Educational CHAT: using Exchange Structure Analysis to investigate communicative roles in CMC seminars. In P. Brna, M. Baker, & K. Stenning (Eds.) Roles of Communicative Interaction in Learning to Model in Mathematics and Science: Proceedings of C-LEMMAS TMR Conference, Computer Based Learning Unit, University of Leeds. Rofeld, R. W. & Hiemestra, R. (1995). Moderating discussions in the electronic classroom. In Z. Berge & M. Collins (Eds.) Computer Mediated Communication and the Online Classroom, Vol. 3, Distance Learning. Creskill, New Jersey: Hampton Press, WebCT (1999). World Wide Web Course Tools. WebCT Educational Technologies Corporation, Vygotskii, L. (1978). Mind in Society, Cambridge, MA: Harvard University Press. 359

12 Appendix Evaluation criteria for group-work. Group-work was marked independently by tutors and demonstrators who rated the work from 1 to 3 on each of the columns in the table below (where three represents very good, 2 good and 1 average an additional discretionary 1/2 mark could also be awarded). The marks from each of the three markers were summed and averaged to give the final overall score. Groups Collaboration Content Presentation Best Feature Students Group Management Relevance Interface Navigation S11 S12 S15 S17 A manager. Co-operative loosely synthesised Content contribution by member Manager too much? Not quite seamless Objectives & Target group Implicit only Many useful collected. Some original work to tie them together Organisation (structure and linkage) Loose topics not linked well across participants Inconsistent some poor choices of colour scheme Good but many links didn't work overall S2 S6 S19 Managed but co-operative and loose synthesis Not balanced or seamless - but logistical problems Implicit only Some good found Not linked well across participants Good but inconsistent Some links didn't work overall S7 S9 S14 S16 S18 S20 Well managed Co-operative And synthesis at structural level Largely balanced But not seamless Objectives clear but implicit targets Excellent and coherent set of Very good organisation - structure less coherent from content point of view Good consistency achieved Good - most links worked overall S3 S5 S8 S13 S1 S4 S10 Well managed Seamless and well synthesised Strong coordination but cooperative? Well balanced / seamless Stated objectives though not target group Very good and an excellent range of Very good organisation - best structure Very good consistency achieved though some poor choice of colour Most links worked Well balanced/ seamless Stated Excellent and coherent Creative idea with original work Very good organisation - structure coherent from content point of view as well Good consistency achieved Good - most links worked A good original piece of writing plus many Good quality of books and journal articles linked in - useful resource Coherent and top page Original text to address a specific question with appropriate well selected embedded Coherent and top page overall