UM 2007 WORKSHOP 2 Corfu, Greece, June, 2007

Transcription

1 UM 2007 WORKSHOP 2 Corfu, Greece, June, 2007 Personalisation in e-learning environments at individual and group level Peter Brusilovsky Maria Grigoriadou Kyparissia Papanikolaou 11 th International Conference on User Modeling, Corfu, Greece

2 Preface In recent years, renewed interest in both individual and group-centred learning has yielded adaptive instruction and collaborative learning to provide flexible and powerful alternatives to the design of e-learning environments. Adaptive instruction argues that, learners will be able to achieve their goals more efficiently when pedagogical procedures accommodate their individual differences. Collaborative learning is supposed to lead to a deeper level of learning, critical thinking, and shared understanding, providing opportunities for developing social and communication skills. Accordingly, adaptive e-learning environments focus on the development of ways of supporting learners to undertake control over their own learning, whilst collaborative e- learning environments focus on effective collaboration. A number of important contributions concerned with designing personalisation in the area of adaptive educational systems have been made to date, mainly focusing on individuals. Considering that a group consists of individuals and also has its own identity, personalisation at group level may target both to the members of a group and to the whole group. Exploring the ways that personalisation may affect and increase different types of interactions that trigger learning mechanisms at individual and group level, is a challenging goal. The focus of this workshop is on the design of personalisation at individual and group level. The ultimate objective is to enrich our knowledge of how to design personalised e-learning environments based on individual and group models. By bringing together researchers on adaptive educational systems, collaborative learning environments, interaction analysis, learner modelling, group modelling, this workshop aims to create a forum to share ideas about the relationship among individual and group interaction and development, and discuss different approaches about how personalisation could enhance particular forms of interaction that trigger learning mechanisms. Challenging issues to which this workshop aims to contribute are: Modelling the learner/group: What types of characteristics should be included in the learner model representing a learner (a) as an individual (b) as a member of a group (e.g. individual differences, preferences/needs, interaction behaviour, collaboration/communication behaviour)? What is the group identity? How do the individual s characteristics affect the group interaction? Interaction data and analysis: What data should be gathered, how and in what forms, in order to have a sufficient, workable and interoperable set of interaction data reflecting the group dynamic? How analysing different types of interactions such as learner-system interactions, learner-teacher interactions, peer interactions, group interaction, could stimulate reflection on the learning process? What techniques can effectively be used in gathering and analysing interaction data? Group formation: Which are the critical issues in grouping learners, e.g. context, task to perform, learners characteristics, group heterogeneity / homogeneity? How to group learners based on their individual differences? Which algorithms support effective clustering of homogeneous and heterogeneous groups of learners? Assessment & Feedback: How the assessment process could feed the learner or group model and the personalisation process? How to design personalised forms and types of feedback based on the learner and/or group model? Personalization at group level: What kind of innovative personalization techniques can be applied on group level? How modelling the whole group (instead of a single learner) may enhance system abilities to assist the learner and the teacher? Representation of learner and group model: What representation forms could be used for the externalisation of the learner and/or group modelling process? The workshop had sixteen submissions. Each of these was rigorously reviewed by three members of our expert international panel of reviewers on the Programme Committee. We have selected the six as full papers, four as short papers and three as posters in these proceedings. 2

3 We thank the members of the Programme Committee for their significant contributions to the workshop, providing helpful feedback on submissions and careful reviews. Kyparissia Papanikolaou Maria Grigoriadou Peter Brusilovsky Programme Chairs Peter Brusilovsky, University of Pittsburgh, USA Maria Grigoriadou, University of Athens, Greece Kyparissia Papanikolaou, School of Pedagogical and Technological Education, Athens, Greece Programme Committee Susan Bull, University of Birmingham, UK Paul De Bra, Eindhoven University of Technology, The Netherlands Judy Kay, University of Sydney, Australia Rose Luckin, Institute of Education, UK George Magoulas, University of London, UK Jon Dron, University of Brighton, UK Angelique Dimitracopoulou, University of the Aegean, Greece Nikolaos Avouris, University of Patras, Greece Grammatikh Tsaganou, University of Athens, Greece Ricardo Conejo Muñoz, Universidad de Málaga, Spain 3

4 Table of Contents Pages Adapting to Groups of Learners Judith Masthoff 5-6 Computer based Interaction Analysis Supporting Self-regulation: an Emerging Research Field 7 Angelique Dimitracopoulou FULL PAPERS Design Services-Enabled Personalised Support for Planning of Lifelong Learning Based on Individual and Group Characteristics 8-15 Hassan Baajour, George D. Magoulas, and Alexandra Poulovassilis Collecting and Analyzing Interaction Data in Computer-Based Group Learning Discussions: An overview Tharrenos Bratitsis and Angelique Dimitracopoulou Group Interaction Prompted by a Simple Assessed Open Learner Model that can be Optionally Released to Peers Susan Bull and Mark Britland Forming Homogeneous, Heterogeneous and Mixed Groups of Learners Agoritsa Gogoulou, Evangelia Gouli, George Boas, Evgenia Liakou, and Maria Grigoriadou Investigating Individual-to-Group and Group-to-Individual Influences Kyparissia Papanikolaou, Evangelia Gouli, and Maria Grigoriadou Adaptation of Feedback in e-learning System at Individual and Group Level Ekaterina Vasilyeva, Mykola Pechenizkiy, and Paul De Bra SHORT PAPERS Investigation of Group Formation using Low Complexity Algorithms Christos E. Christodoulopoulos and Kyparissia Papanikolaou Evidential Multiple Choice Questions Javier Diaz, Maria Rifqi, and Bernadette Bouchon-Meunier How to Adapt the Visualization of Programs? Andrés Moreno, Roman Bednarik, and Michael Yudelson Semantic Modeling for Group Formation Asma Ounnas, Hugh C Davis, and David E Millard POSTERS Interpretative e-learning personalization: methodology, formal aspects and generic scenarios of individual/group dynamics. A case of a course in art history Ioannis Kanellos, Thomas Le Bras, Ioana Suciu, and sister Daniilia An Affective Model for Personalized Learning in Adaptive Educational Hypermedia Systems Makis Leontidis and Constantin Halatsis The Squeaky Wheel Algorithm: Automatic Grouping of Students for Collaborative Projects Steven L. Tanimoto

5 Adapting to Groups of Learners Judith Masthoff University of Aberdeen, Aberdeen, Scotland, UK Almost all work on adaptive systems to date focuses on adapting to individual learners. However, there are many situations when it would be good if we could adapt to a group of learners rather than to an individual. For instance, learners may have to share a device, as is often the case in interactive Television (which tends to be viewed in groups), and ambient intelligent environments. A teacher may also want a group of learners to share their learning experience, as some pedagogical theories emphasize the role of learning together. For instance, Lave and Wenger's concept of "situated learning" promotes the notion that learning takes place from the process of engagement in a community of practice [1], which may be a classroom community. A related approach is Vicarious Learning [2], which asserts that people can benefit from observing and modelling the behaviours, attitudes and emotional reactions of other learners. Also, socialconstructivism emphasises the benefits of learning in engagement with others. This raises questions on how to select activities that will benefit the group as a whole. In this presentation, we will draw from our experiences in group recommender systems, and attempt to show how these can be applied to adaptive e-learning. Adapting to groups is even more complicated than adapting to individuals. In this presentation, we will discuss how group adaptation works, what its problems are, and what advances have been made. Interestingly, we will show that group adaptation techniques have many uses as well when adapting to individual learners. The main problem group adaptation needs to solve is how to adapt to the group as a whole based on information about what is good for individual users. For instance, suppose the group contains three people, Peter, Jane and Mary. Suppose a system is aware that these three individuals are present and knows how good each of a set of items is for them (e.g. lessons, activities). Table 1 gives example ratings on a scale of 1 (really bad) to 10 (really good). Which items should the system select, given time for four items? Table 1. Example of individual ratings for ten items (A to J) A B C D E F G H I J Peter Jane Mary Many different strategies exist for aggregating ratings of individuals into a rating of a group (e.g. used in elections, like when selecting the leader of a political party). Eleven of these (inspired by Social Choice Theory) are discussed in [3]. For instance, one could average the ratings of the individuals to obtain a group rating (making E and F the most preferred items by the group): the Average Strategy. One could take the minimum of the ratings, assuming that a group is as happy as its least happy member: the Least Misery Strategy. One could use a combination of the Average and Least Misery strategy, taking the average of ratings but only for those items whose ratings are all above a threshold: The Average Without Misery Strategy. We conducted a series of experiments in the context of a group recommender system to investigate which strategy is best (see [3] for details). In Experiment 1, we investigated how people would solve this problem, so given ratings for individuals (as in Table 1), which items they thought should be presented to the group, if there was time for say six items. We compared our subjects decisions (and rationale) with those of the aggregation strategies. We found that humans care about fairness, and about preventing misery and starvation ( this one is for Mary, as she has had nothing she liked so far ). Subjects behaviour reflected that of several of the strategies (e.g. Average, Least Misery, and Average Without Misery were used), while other strategies were clearly not used. 5

6 In Experiment 2, we presented subjects with item sequences chosen by the aggregation strategies. Subjects rated how satisfied they thought the group members would be with those sequences, and explained their ratings. We found that the Multiplicative Strategy (which multiplies the individual ratings) performed best, in the sense that all subjects thought its sequence would keep all members of the group satisfied. Several strategies could be discarded as they clearly were judged to result in misery for group members. We also compared the subjects judgements with predictions by simple satisfaction modelling functions. Amongst other, we found that more accurate predictions resulted from using quadratic ratings, which e.g. makes the difference between a rating of 9 and 10 bigger than that between a rating of 5 and 6. We also investigated how to deal with the order of a sequence, but this clearly needs more work in a learning context. Some findings are likely to be applicable here as well. For instance, it is likely that the motivation and confidence produced by a previous activity will affect a learner s judgement of the next activity. Also, the performance of the learners will need to be taken into account when selecting the next activity. When adapting to a group of learners, you cannot give everybody what is good for them all of the time. However, you do not want anybody to get too dissatisfied. For instance, in a class it would be bad if a learner were to leave and never come back, because they got really demotivated. An ideal teacher would adapt to the learners in such a way that they get activities that are really good for them most of the time. To achieve this, it is unavoidable that learners will occasionally get activities that are less appropriate, but this should happen at a moment when they can cope with it (e.g. when being in a good mood because they loved the previous activities). Therefore, it is important to monitor continuously the affective state of each learner. Rather than using self-rating (which can be tedious for learners) or measuring (sensors are intrusive and not that good yet), it would be nice if we could predict it. In [4], we investigated different ways to model affective state. We compared the predictions of these models with the predictions of users. We also performed an experiment in an educational domain to compare the predictions with the real feelings of users. Group adaptation techniques can also be used when adapting to an individual learner. Multiple criteria play a role when deciding on the next activity for a learner. For instance, its difficulty level, how suited it is with respect to learning style, how engaging it is, etc. Selecting an activity then may well resemble the situation described in Table 1, except that now Peter, Jane and Mary will be replaced by criteria. We have done some research on this problem (in a news domain), and found that the techniques are applicable, though weights need to be added to model the relative importance of criteria [5]. Group adaptation can also help to overcome the cold-start problem [6]: when you do not know much yet about a new learner, you can select activities for them that would keep the group of existing learners happy. References 1. Lave, J., & Wenger, E., Situated Learning: Legitimate Peripheral Participation, Cambridge: Cambridge University Press. 2. Bandura, A., Social Learning Theory. New York: General Learning Press 3. Masthoff, J. (2004). Group Modeling: Selecting a Sequence of Television Items to Suit a Group of Viewers. UMUAI, 14, Masthoff, J. and A.Gatt (2006). In pursuit of satisfaction and the prevention of embarrassment: Affective state in group recommender systems. UMUAI, 16, Masthoff, J. (2004). Selecting news to suit a group of criteria: An exploration. 4th Workshop on Personalization in Future TV - Methods, Technologies, Applications for Personalized TV, Eindhoven, Netherlands. 6. Masthoff, J. (2003). Modeling the multiple people that are me. In: P. Brusilovsky, A.Corbett, and F. de Rosis (eds.) Proceedings of the 2003 User Modeling Conference, Johnstown, PA, Berlin: Springer Verlag,

7 Computer based Interaction Analysis Supporting Self-regulation: an Emerging Research Field Angelique Dimitracopoulou LTEE Laboratory, University of the Aegean, 1 Democratias Ave, Rhodes, Greece adimitr@rhodes.aegean.gr Up to the present, computer based interaction analysis was mainly applied and taken into account by a learning environment, in order to either adapt and personalize itself meeting the users needs and preferences, or even provide appropriate guiding messages for the users. In these cases the locus of control is on the system. The field of Computer based Interaction Analysis for the support of the participants self-regulation in technology based learning activities (for individuals or groups) is a new direction of research that explores the potentiality of attributing the locus of control to the users themselves [3]. Its purpose is to offer an enriched and powerful interface (integrating functionalities deriving from the interaction analysis outputs) and primarily a cognitive and metacognitive support to learning environment participants (e.g. students, moderators, teachers) as well as to observers of those activities (e.g. teachers, researchers), which need to analyze and understand the complex cognitive and social phenomena that may occur. The core aim is to offer directly the means to the human actors (usually via visualized representations of appropriate interaction analysis indicators) so as to be aware of and regulate their behaviour, either as individuals or as cognitive groups. In fact, the corresponding interaction analysis tools support the users in three major levels: awareness, metacognition and evaluation. The objective is the optimization of the activity through: a) refined participation and learning outcome for the students through reflection, self-assessment and self-regulation, b) better activity design, regulation, coordination and evaluation by the moderator. The Computer based Interaction Analysis tools can be embedded or linked to various kinds of learning environments [1], [2], [4], [5] addressed to individuals (e.g. simulation, modelling environments, etc) or groups (e.g. forums, chats, etc). The Interaction Analysis outputs are presented to the participants or the observers in an appropriate format (graphical, numerical or literal), in a singular or a combined form, via significant interaction analysis indicators. Usually, different kinds of indicators or combinations of them are appropriate for different interaction analysis tools users, respecting their predefined or emerging roles in the learning activity timeline. During the workshop, we will have the opportunity to reflect on the potentialities, the limits and the perspectives of this field. For that matter, we will present the general features of the interaction analysis tools and their general functioning modalities. Then, we will present a theoretical framework, allowing the examination of the explicit and implicit features of interaction analysis indicators, so as to outline the corresponding state of the art. Finally, we will discuss the expectations on these indicators and derive the research perspectives that will allow the exploration of the Interaction Analysis tools effects on the users, in an essential way. The ultimate goal will be to reflect upon the potential of designing adaptive systems complementarily to systems adopting the interaction analysis for supporting humans self-regulation. References 1. Biuk-Aghai R., Simoff S.: An integrated framework for Knowledge extraction in Collaborative Virtual Environments. In ACM-2001, Group 01, Sept.30-Oct.3, 2001, Boulder, Colorado, USA 2. Cheng R., Vassileva J.: Adaptive rewarding mechanism for sustainable online learning community. Artificial Intelligence in Education. C.-K. Looi et al. (Eds). (2005). IOS Press 3. Dimitracopoulou, A et al.: State of the art of interaction analysis for Metacognitive Support & Diagnosis. IA JEIRP Deliverable D Kaleidoscope NoE, Available online at: 4. Jermann P.: Computer Support for Interaction Regulation in Collaborative Problem Solving, PhD Thesis, University of Geneva (2004) 5. Reimann P.: How to support groups in learning: More than problem solving. (keynote talk) in Aleven et al. (ed.), Artificial Intelligence in Education (2003). Supplement Proceedings University of Sydney,

8 Designing Services-Enabled Personalisation for Planning of Lifelong Learning Based on Individual and Group Characteristics Hassan Baajour 1, George D. Magoulas 1,2 and Alexandra Poulovassilis 1,2 1 London Knowledge Lab, Emerald Street, WC1N 3QS London, UK 2 Department of Computer Science and Information Systems, Birkbeck College University of London, Malet Street, WC1E 7HY London, UK {hassan, gmagoulas, ap}@dcs.bbk.ac.uk Abstract. Lifelong Learning is a complex environment where strategic planning, coordination and collaboration between partners are needed. Lifelong learners form a rather diverse student population with a variety of backgrounds, evolving needs and varying accessibility requirements. Making lifelong learning happen in practice requires learners becoming more aware of their own studying and thinking processes, and tools and guidance that support the planning of learning throughout life. To this end, technology should be used effectively to assist lifelong learners to access, compose and manage their learning under varying circumstances and settings, such as institutional, informal and work-based. Models also need to be developed and frameworks need to be extended that would allow local, regional, national and international systems to work together to provide lifelong learners coherent access to e-infrastructures within and across institutions. This paper explores a lifelong learning dimension in the context of the MyPlan project 1, which aims to create a personal space where learners are supported in planning their learning throughout life. It investigates the role user modelling can play in this context to support personalisation when searching for new learning opportunities and receiving recommendations, and to facilite reflection both on learner s individual experiences and experiences of others. 1 Introduction Lifelong learning requires continuous engagement in acquiring and applying new knowledge and skills in order to meet the evolving needs of individuals and organisations. It has created new challenges with respect to understanding, exploring and supporting new learning dimensions, such as self-directed learning, reflection, collaborative and organisational learning, and learning on demand [8]. The emergence of Lifelong Learning Networks (LLNs) and University Federations, [9], [20], which are envisaged as groups of institutions that come together across a city, area or region to offer new progression opportunities for lifelong learners, creates new drivers for change to accommodate the needs of learners who will move between various groups and institutions, and who may work in different countries. Much of this has been made possible by the new possibilities offered by the Internet and the new approaches in systems design [6], [9]. However, the audience of these systems is considerably diverse, as lifelong learners may belong to radically different groups; they come from different backgrounds, and are very much concerned with the accessibility of the various systems within LLNs or federations, the transferring of their personal information for cross-institutional e-learning, and the flexibility of the curricula offered. Thus, new approaches are needed for managing and exchanging performance or competency data between different e- portfolio systems [10], [11], 12], assembling or manipulating fragmented user data that were stored in different distributed systems [6], [20]. The issue of allowing users some level of control over viewing, editing, managing and realising their attributes has been investigated in the context of open learner modelling. A number of 1 The MyPlan-Personal Planning for Learning Throught Life project ( is funded by the e-learning Capital Programme of the Joint Information System Committee, UK ( 8

9 approaches have been developed to maintain and compare learners own and a system s beliefs about aspects of their user model, such as their knowledge, collaborative peer assessment in discussions, and interactive dialogues [1], [4], [16]. However, in lifelong learning a more holistic approached is needed as is evidenced by the growing adoption of e-portfolios and personal learning planning systems. In this direction, our previous work [2], [3], [19] has investigated how lifelong learners could be better supported in their choices by developing a system which allows users to search for information about cross-institutional learning opportunities, and to create, maintain and share individual timelines of their past and future learning, work and life experiences. We also investigated (see [18]) the exchange of information between such learning planning systems and e-portfolio systems as learners move between institutions. The MyPlan project is extending this work by aiming to create a personalised space that more effectively addresses the needs of the diverse population of lifelong learners, allowing planning of and reflection on learning and providing personalised recommendation of relevant future learning routes based on previous learning and future learning goals. Building on the earlier L4All system of [2], [3], [19], it will allow users to create open information-rich user models that will not only represent their interests, background, qualifications, goals and objectives but will also provide a holistic view of their learning, work and life experiences in the form of a timeline. This approach provides continuity between learning and work experiences and differs considerably from other open learner models in that the learning pathway also integrates social factors, providing support throughout lifelong learning rather than compartmentalising learning into one stage or period. The paper is organised as follows. Section 2 introduces the learner model, which is defined as an ontology describing the different characteristics of a learner and the relationships between the different concepts. Section 3 presents personalisation services specific to supporting independent lifelong learners in planning their learning and use cases in the context of the MyPlan project. The last section presents concluding remarks. 2 Lifelong Learner Modelling The main driver behind the standardisation of user modelling approaches is to enable interoperability and reusability of user models in order to allow personalisation of web-based systems by integrating the available user models. In this vein, several researchers have argued in favour of generic learner models, as the adoption of a universal set of user model elements and attributes following well-defined rules would facilitate communication and integration between different kinds of user models embedded in different personalised systems. In [13] a user modelling language, UserML, was introduced as a platform for exchanging data in a ubiquitous environment. Later, in [14], the same authors proposed the General User Model Ontology which uses UserML to represent user models in a way that would enable different systems to represent and exchange user models. Other work [6] has focused on the use of open standards and services to enable other personalised systems to plug into a server-based learner modelling component which gathers data from several sources and use it or contribute data. In [5] a learner profile based on standards and ontologies was described that uses concepts including preference, performance and portfolio for exchanging data, such as competency records, between e-assessment and e-portfolio systems. Ontology-based learner modelling has also been investigated in [16] and [20] with the content of the ontology dependent on the application. These approaches could also be useful in the context of lifelong learning. Particularly in LLNs and other forms of cross-institutional e-learning, users typically interact with several different distributed applications, all working with their own learner profiles. Representing and exchanging user data is a desirable property in this context and an ontology-based model would potentially facilitate this process. Below, we introduce an ontology-based user model that has been developed for MyPlan. The ontology, the MyPlan Learner Ontology (MyPlanLO), conceptualises the different characteristics of lifelong learners identified in our previous studies [2], [18]. It defines lifelong learners in terms of concepts, their structure and relationships. It has been developed using Protégé, and a partial view on Protégé OWL-Viz is represented in Fig. 1. 9

10 Fig. 1. A partial graph-based view of the MyPlan learner ontology. The MyPlanLO captures metadata comprising different user characteristics. It is structured according to Instructional Management Systems-Learner Information Package (IMS-LIP) [15] but incorporates also MyPlan-specific elements and is implemented using RDF/OWL. A sample of elements, their attributes and associated MyPlan functionalities are shown in Table 1. In particular, the element timeline consists of episodes of learning, work and life. It is used to provide a holistic view of lifelong learning, allowing learners to understand and reflect on the social as well as the educational factors that may influence not only how they learn but also how external factors influence their career decisions and educational choices. Our user studies (see [18]) have shown that, despite privacy concerns, learners are willing to share their experiences with their peers by setting their timeline access privilege to public. 10

11 Table 2. Sample of MyPlan learner model elements, attributes and related system functionalities. Element Attributes Functionalities Identification Names, Postal address Access into the system , Phone, photo Search and contact others Accessibility Preferences, language information, disabilities, learning style Goals Learning type, date, description Organise local communities Customise the user interface Categorise users according to preferred language and accessibility requirements Search for lifelong learning opportunities Search for people like me Organise communities of similar users QCL Qualifications, certificates, Filter search results licences, awards, prizes Organise communities according to their QCL Activities type, level, date Recommend lifelong learning opportunities Competencies Title, description Filter content Personalise the content Organise communities according to their Titles Interests Hobbies and recreational Capturing user interests could be used to organise activities communities with similar interests. Personalise the layout Filter the content Recommendations Evaluations Type, level Rate lifelong learning activities and timelines Affiliations Organisations, educational institutions, employers, government departments Timeline Owner, keywords, title, privilege, episode Search for courses Organise communities of users Filter content Personalise the interface Search for lifelong learning opportunities or other users Search timelines that match keywords/title or contain specific episodes Identify people like me. 3 Services-enabled Personalisation for Planning Lifelong Learning MyPlan builds on the L4All system [3], [19], adopting a service-based architecture following the E-Learning Framework (ELF) specification ( The ELF is an initiative of the UK s JISC and Australia s Department of Education, Science and Training; in June 2006, New Zealand s Ministry of Education and SURF in Netherlands also became partners. L4All integrates tools, common access management services and web services, and also interacts with services provided by external suppliers such as LearnDirect ( Our design approach recognises the central importance of community involvement, and models the process of planning of lifelong learning using services and processes that describe sequences of steps and the services and data involved in each step [2], [3]. Personalisation in this context emerges through aggregation of services that implement system functionalities. It can be materialised by creating, managing and storing user data (e.g. qualifications, interests, goals, learning style), usage data (e.g. ratings, learning opportunities selections from search results, types and levels of courses attended to date), usage regularities (e.g. types of episodes in the timeline and frequency of their occurrence) and relationships between user behaviours from a diverse set of applications running on a LLN using the MyPlanLO (e.g. competencies stored in an e-portfolio system). Services have been identified through the ELF and a methodology that used a series of workshops, focus groups and interviews with lifelong learners and experts to identify user requirements and evaluate system features and functionalities [2] (see also [18]). These include services for the management of individual learner models and policies for updating, registering user-related data, authorisation and authentication. Table 2 presents a set of personalisation services that are relevant for planning of lifelong learning. Their aim is to enhance individual learners engagement with the lifelong learning process by offering control over designing and 11

12 reflecting on their learning pathways. Personalisation services can also support building communities of learners with similar interests, and can facilitate information sharing with other members of the lifelong learning community and with peers. Table 3. Sample of MyPlan personalisation services. Personalisation Service Search Ranking Filtering Advertisement Customisation Recommendation Similarity Validation/ eligibility Reflective log Rating Automatic profile update Notification Occupation profile matching Communication Classification User support Functionality Searching for content, pathways or peers in a personalised manner by narrowing down or broadening the search depending on the learner model, e.g. preferences and learners goals. Adaptive link ordering based on self-rating and other users ratings. Filtering of links about relevant learning opportunities and pathways. Deliver personalised advertisements about learning opportunities based on user s activity with the system and user preferences. Allow users to update their profile and preferences, activate/deactivate features of the system that depend on user preferences, career and learning goals. Personalise the delivery and presentation of content depending on learner background, preferences and characteristics; receive personalised messages/information, or a calendar that is configured to show only the user s relevant events. Provide recommendations about other users (people like me ), i.e. users with similar profiles and/or timelines. Recommendations about courses can be also provided (e.g. courses I should do) by matching properties of the courses with user profile characteristics and learning history. Evaluate similarity measures in order to provide recommendations. Validate acceptance into next stage of learning by checking whether the learner can access specific modules, or generating warnings regarding irrelevant plans and contradictions in the lifelong learning plan. Allows learners to record their reflections about parts of their learning plan, e.g. how modules/courses helped them achieve a personal goal. The system can utilise reflections of some users and display them automatically to a particular user with similar characteristics, who shows an interest in the same subject. Support for the use of secondary metadata (user ratings and text annotations) for resources. Record rating of search results, lifelong learning opportunities, pathways, implicitly or explicitly. Support the mapping of learner activities in the system against specific competencies, which updates services with user preferences information. Monitor learners favourite timelines, detect changes or modified reflections and initialise services that inform the interested learner about these. Notify users about communication spaces that relate to their interests. Match user profile attributes with expert-supplied occupation profiles using metadata for courses and professions. Create communication spaces for users with similar interests, e.g. developing weblogs with many users/authors that belong to the same community. Access to communication spaces can be personalised to meet the needs, interests and preferences of particular user groups. Categorise users into groups for use by other personalisation services. This can be achieved by grouping according to user stereotypes, by grouping individual user models that share similar attributes, or both. Provides support regarding the use of various system features and functionalities. The ontology approach is eminently suitable for categorising and associating a user with a specific stereotype dynamically, as assumptions about a learner s characteristics can be inferred automatically whenever any changes occur to a particular concept, property or relationship representing learner-related information. For example, personalised searching and filtering of results or recommendations can be generated based on specific stereotypes (particularly when the system is used within the context of a LLN). Moreover, maintaining the ontology as new groups of users are formulated through clustering based on the similarity of their goals, timelines, competencies and other relevant user attributes can facilitate generating recommendations based on group characteristics. For example, when a user rates or comments on a timeline or a course, 12

13 the system could notify other interested members of the same group automatically. Servicesenabled personalisation requires user and usage data, usage regularities and user behaviours coming from the wider networked environment to operate. While some of these data may provoke users privacy concerns (e.g. transferring personally identifiable information, validating qualifications, and importing reflections from other systems), we have found that other data, such as timeline episodes and goals, are willingly shared among peers [18]. There are a number of approaches for privacy-preserving personalisation [17]. The approach explored here combines users participation in setting privacy preferences (e.g. about their timelines) with Shibboleth 2 federated access management [20] adopting the eduperson standard [7]. 3.1 Personalisation Scenario Based on Individual Characteristics Fig. 2 illustrates a Search for courses scenario as an example of individual level personalisation. Kim completed her secondary school. She registered on the system and starts looking for courses on IT. There are thousands of them, and she is a little dispirited. She refines her search by specifying that she wants a short course satisfying a set of search criteria e.g. subject, location (by postcode), level. This customises her search and narrows down the list of courses, but there is still a wide range of options available to her. She is still not quite sure what she should do, so she selects an option to filter the search results according to her learner model. 1. Registration Register Registration Data user-name; password; fullname; age; address; qualifications; preferences; interests 2. Keyword-based searching 3. Personalised search based on additional usersupplied data Search Courses Search Courses Presentation List of relevant results Customisation subject; postcode; course level; institution 4. Personalised search based on individual characteristics Presentation Narrow down results Filtered results Presentation Search Courses Filtering criteria User attributes: qualification; competencies; preferences; interests; goal Fig. 2 Personalised searching for courses based on individual characteristics and preferences Fig. 3. Searching for courses and receiving recommendations. User s timeline appears at the bottom. Users can drag and drop icons to add episodes to their timelines. Fig. 3 shows a screenshot of the system. The user s timeline is shown at the bottom of the screen. Among other options, the user can search for courses/learning opportunities. The my courses list presents a customised view of the search results, while the my recommendations lists contains courses relevant to this user which have been identified by matching with predefined learning pathways designed by educational experts and stored in the system. 3.2 Personalisation Scenario Based on Group Characteristics Fig. 4 presents an example personalisation that exploits characteristics which are similar among learners. Jack is interested in planning his lifelong learning. In order to create the timeline representing his lifelong learning pathway, he first creates his profile. The system categorises Jack 2 Access management protocol designed to facilitate communication among UK higher education institutions. 13

14 automatically into one of the static stereotype groups already existing in MyPlan. Based on that, Jack will be recommended a list of existing timelines for users that belong to this stereotype group. This will help Jack in designing his own timeline and will allow him to communicate with people like him. Alternatively, a more effective recommendation can take place whereby similarities between Jack and other users are identified by matching Jack s and other learners characteristics. This can be achieved using a clustering technique that takes into account learner model items such as current occupation, interests and learning goal. The interface of Fig. 5 allows users to specify criteria that can be used to identify similar users. Age; Occupation; Qualifications; Preferences; Interests; Goal; Ratings User Data Grouping Classification Similarity measures Clustering Categorise user Stereotype Categorisation Occupational profiles; FE student; HE student; Independent learner; Career advisor Create Profile Profile Creation Present pathways of people like me Recommend Timelines Present pathways designed by experts Identify people like me based on affiliation, occupation, goal, ratings, qualifications, preferences, interests, timeline episodes Recommendation based on group characteristics Recommend Career Pathways Stereotype-based pathway recommendations Fig. 4. Generating recommendations based on group characteristics and users similarities. Fig. 5. Identifying people like me and relevant timeline. 4 Concluding Remarks This paper has discussed an ontology-based approach for modelling the user and ways to personalise users interaction in the context of MyPlan, a system that aims to support the independent lifelong learner. MyPlan operates using web services, and uses standards-based technologies and open specifications. In a lifelong learning environment, an ontology-based approach is a good candidate for developing interoperable systems and learner models that can serve various e-learning systems for various purposes. A standards-based architectural approach can facilitate collaborative cooperation between different systems, enriching systems with userrelated information in order to afford a suitable level of personalisation. The paper has proposed a list of personalisation services for planning lifelong learning and has illustrated application scenarios of personalisation services that use various related learner model items operating at the individual and the group level. References 1. Bull, S. & Nghiem, T.: Helping learners to understand themselves with a learner model open to students, peers and instructors. In P. Brna & V. Dimitrova (eds), Proc. of Workshop on Individual and Group Modelling Methods that Help Learners Understand Themselves, International Conference on Intelligent Tutoring Systems (2002) de Freitas S., Harrison I., Magoulas G., Mee A., Mohamad F., Oliver M., Papamarkos G., Poulovassilis A.: The development of a system for supporting the lifelong learner, British Journal of Educational Technology, 37(6) (2006) de Freitas S., Magoulas G., Oliver M., Papamarkos G., Poulovassilis A., Harrison I., Mee A.: L4All- a web-service based system for lifelong learners. In Proc. of echallenges'2006 (Workshop on Next 14

15 Generation in Technology Enhanced Learning), October 2006, Barcelona. IOS Press (2006) Dimitrova, V.: STyLE-OLM interactive open learner modelling. International Journal of Artificial Intelligence in Education, 13 (2002) Dolog, P., & Schaefer, M.: Learner modeling on the Semantic Web. In Proc. of PerSWeb-2005 Workshop: Personalization on the Semantic Web, 10th International Conference on User Modeling, July, 2005, Edinburgh, UK, Springer Verlag. Also available online at: 6. Dolog, P. & Schaefer, M.: A framework for browsing, manipulating and maintaining interoperable learner profiles. In Proc. of 10th International Conference on User Modeling, July 2005, Edinburgh, UK. Springer Verlag. Also available online at 7. EduPerson Specification (2003). Internet2 Middleware Architecture Committee for Education, Directory Working Group (MACE-Dir). Available online at 8. Fischer, G., Sugimoto, M.: Supporting self-directed learners and learning communities with sociotechnical environments, Journal on Research and Practice in Technology Enhanced Learning, 1(1) (2006) Gaeta M., Ritrovato P., Salerno S.: Making e-learning a service oriented utility: the European Learning Grid Infrastructure Project. In Towards the Learning Grid: Advances in Human Learning Services, vol. 127, Frontiers in Artificial Intelligence and Applications P. Ritrovato, C.Allison, S.A. Cerri, T. Dimitrakos, M. Gaeta and S. Salerno (eds) (2005) Grant, S.: Clear e-portfolio definitions: a prerequisite for effective interoperability. In Proc. of eportfolio 2005 Cambridge, UK. October Grant, S.: Frameworks of competence: common or specific? In Proc. of TEN Competence Workshop, Sofia, Bulgaria, March Grant S.: Towards competence-related interoperability. In Proc. of the TENCompetence Workshop on Service Oriented Approaches and Lifelong Competence Development Infrastructures, Manchester, England (2007) 13. Heckmann, D., & Krueger, A: A User Modeling Markup Language (UserML) for Ubiquitous Computing, 9th International Conference on User Modeling, Johnstown, USA (2003) Also available online at: Heckmann, D., Schwartz, T., Brandherm, B., Schmitz, M., von Wilamowitz-Moellendorff, M.: GUMO, the general user model ontology. In Proc. International Conference on User Modeling, Edinburgh, UK Also available online at: IMS-LIP (2005). IMS Learner Information Package Specification, v1.0.1, IMS Global Consortium. Available online at: Kay, J. and Lum, A.: Ontology-based user modelling for the semantic web. In Proc. Workshop: Personalisation for the Semantic Web, 10th International Conference on User Modeling, Edinburgh, UK, (2005) Kobsa, A. and J. Schreck: Privacy through Pseudonymity in User-Adaptive Systems. ACM Transactions on Internet Technology 3(2) (2003) L4All (2006). Project deliverables. Available online at Magoulas G., Papamarkos G., Poulovassilis A.: A services-enabled environment for personalising lifelong learning pathways (2006). In Proc. of the Workshop on Applying Service Oriented Architectures to Adaptive Information Systems, International Conference on Adaptive Hypermedia, Dublin, June, (Lecture Notes in Learning and Teaching, Weibelzahl, S. and Cristea, A. (eds), pp ) 20. Niu, X., McCalla, G., Vassileva, J.: Purpose-based user modeling in a multi-agent portfolio management system. In Proc. of 9th International Conference on User Modeling, Johnstown, PA, June 2003, Springer, Lecture Notes Artifical Intelligence vol (2003) 21. UKF (2006). Recommendations for use of personal data, UK Access Management Federation for Education and Research. Available online at 15

16 Collecting and Analyzing Interaction Data in Computer-Based Group Learning Discussions: An overview Tharrenos Bratitsis and Angelique Dimitracopoulou LTEE Laboratory, University of the Aegean, 1 Democratias Ave, Rhodes, Greece bratitsis@aegean.gr, adimitr@rhodes.aegean.gr Abstract. This paper provides an overview of the dynamics underlying the process of collecting and analyzing raw data from collaborative learning activities. It is based on the research background which the implementation of DIAS relied upon; a system aiming at supporting asynchronous discussions participants on an awareness and metacognition level via visualized interaction analysis indicators. Open questions from the CSCW field are used as a guideline for selecting the appropriate data to collect. The Information Visualization field provides ideas for successfully presenting information obtained by applying Interaction Analysis methods on these data. Through simple examples, we try to demonstrate how such attempts may lead to powerful analysis tools, with cognitive and metacognitive extensions. 1 Introduction The past few years we witness an increased mobility in research concerning tools for analyzing and supporting learning activities, by distance in particular [27]. Recent developments in learning theory have emphasized the importance of context and social interaction [27]. In this vein, Computer Mediated Communication (CMC) tools and in particular asynchronous discussion forae are widely used in formal or informal educational contexts, applying principles of constructivism, emphasizing in social interaction during learning activities [14]. The past five years research focuses in finding methods for evolving and supporting critical thinking through interactions, taking place within asynchronous discussions, in order to achieve high quality learning [27]. Such a goal requires tools, frameworks and methods for the facilitation of monitoring, and/or selfreflection and therefore selfregulation that could be supported by the automated analysis of the complex interactions that occur. Computer based Interaction Analysis is an emerging field of research within the academic community, focusing in analyzing interactions among users [8]. 2 Background In computer supported learning, under the scope of contemporary learning theories, such as constructivism and sociocultural theory, interaction among participants and the need to support and enhance it is highlighted [4],[5],[14]. Towards this direction, supporting mechanisms in the form of adaptive tools addressed directly to the users should be researched [8],[11],[18], [27]. We distinguish three main contributing fields for the corresponding research: Computer Supported Cooperative Work (CSCW), Information Visualization (InVis) and Interaction Analysis (IA). Awareness is a very important research issue for the CSCW field the last 25 years related, to the need of users collaborating via computer means, to be aware of the context of the common work. This corresponds to a vast number of parameters, resulting to the use of the term awareness along with an attributive adjective (collaboration-, peripheral-, background-, passive-, workspace-). The plethora of adjectives indicates that awareness is an ambiguous term which needs to be further defined, when used [25]. According to Gutwin & Greenberg [15], awareness: a) is the knowledge related to the state of the (common) working area for a limited portion of time and space, b) provides knowledge about changes occurring in this working space, c) is maintained by interactions between the collaborators and the working environment (as new emerging action data, alter its status), and d) is a part of an activity (completing a task, working on something etc). Three are the most important categories for the CSCW domain: Informal, Workspace and Group Awareness. The former relates to people presence and social activity, such as knowing who 16

17 is online or who is communicating with whom, revealing collaborators availability, accessibility, approximation and readiness [6]. Such information can improve communication, coordination, of actions or even division of labor [25], leading to activity alignment within a group of people [16]. Workspace Awareness relates to users interactions with the working environment and can be defined as the up-to-the-moment understanding of another person s actions within the shared workspace [15]. This type of awareness enhances coordination, information and knowledge exchange, by revealing the each person s contribution. Finally, Group Awareness relates to the focus of attention [23] and includes information collected passively (unintentionally) [10], providing context to one s actions. Awareness is a very important factor when studying any kind of collaboration among people. Corresponding research questions include selecting appropriate data to record, analyze and to which degree of abstraction or aggregation. Which are the facts, upon which a user relies for monitoring his collaborators activity? How is awareness information exploited by the users? What other points of view exist, while analyzing recorded activity data? Visualization is the process of transforming data, information and knowledge into visual form, exploiting the capability of the human brain for processing significantly larger amount of visual information [12]. Information Visualization is the use of interactive visual representations via computers, aiming at amplifying cognition [7]. Visual representations are considered as means of external cognition, when internal cognitive representations are offloaded into this external mean in order to relieve cognitive burden [24]. InVis is mostly used for presenting abstract information which cannot be intuitionally depicted or automatically mapped to the physical world. Its central objective is to transform information to visual representations without altering its fundamental meaning, while providing a new perspective [17]. There is no specific norm in order to achieve such a goal. Usually it depends on the nature of the information, the intended use of the visualization and the inventiveness of the designer. Tufte [28] introduced the design principles of a successful visualization, which should: a) show the data, b) avoid distorting what the data have to say, c) present data condensed, d) make large data sets coherently, e) encourage inferential processes (e.g. comparing data), f) provide different perspectives on the data, from broad overview to the fine structure, and g) have a clear objective. Several techniques have been proposed for selecting the appropriate visualization and visual metaphors have been used to add perspective to data, especially in collaborative environments. Some of them are presented in section 3. Computer based Interaction Analysis is a recently emerged research field, mainly due to the complexity of interactions occurring within collaborative systems. The IA results are presented directly to technology based activities participants in an appropriate format (graphical, numerical, literal), interpretable by them. They provide an insight of their own current or previous activity, allowing them to reflect on a cognitive or metacognitive level in order to self-regulate their actions. Information is also provided to the activity observers, in order to analyze the complex cognitive and social phenomena that may occur [8]. This approach can produce flexible IA tools, which in an educational context support directly the learning activities participants (e.g. students, teachers) or even observers (e.g. administrators, researchers), in three major levels: awareness, metacognition and evaluation [18]. The expected outcome is the optimization of the activity through: a) better design, regulation, coordination and evaluation by the moderator, and b) refined participation and learning outcome for the students through reflection, self-assessment and self-regulation. The Interaction Analysis process consists in recording, filtering and processing data regarding system s usage and user activity variables, in order to produce the analysis indicators. These indicators may concern: a) the mode, the process or the quality of the considered cognitive system learning activity; b) the features or the quality of the interaction product; or c) the mode, the process or the quality of the collaboration, when acting in the frame of a social context forming via the technology based learning environment [8]. A distinction between low level and high level indicators can be made. The low level indicators do not usually have an autonomous interpretative value and are inferred directly from the interaction data. They mostly contain information similar to that described by the workspace awareness concept. High level indicators usually have an inherent interpretative value and are inferred by complex process from the interaction data. They are often produced by calculations involving several low level indicators and provide information related to the group awareness and other complex concepts. Many of the IA indicators, produced in visual form come under the InVis filed, presenting abstract information, providing insights and new perspectives of interactions among participants. Additionally, new meanings can be derived by combining indicators information, thus producing Interpretative Schemas [4] (guidelines for interpreting and combining information from different indicators). 17

18 3 Research Approach Our analysis focuses on asynchronous discussion learning activities. The main research objective is to implement adaptable supporting tools for all the participants (students, teachers, researchers) of such learning activities, regardless of their role (moderator, coordinator, observer, active participant). The adaptability of these tools regards providing the appropriate set of tools for each user role and profile and for every different activity settings or arising circumstances. For that we examined three fundamental research directions. In the CSCW field evidence can be found of providing awareness-like information as a feedback to users seems to improve collaboration, by enhancing coordination and activity alignment. On the other hand, the abstract nature of awareness information leads us to study the Information Visualization field in order to investigate the most appropriate methods of presenting it to the users, while providing new perspectives at the same time. Furthermore, the Interaction Analysis field suggests that such techniques, exploiting users interactions as raw data for analysis may lead to enhancement of their cognitive and metacognitive structures and finally to users selfregulating their actions in favor of the overall learning activity. The concluding research objective becomes clear. How can we implement adaptable participants supporting tools, by providing visualized feedback information with users interactions as raw data of analysis? In an initial research stage, the core questions are: a) What data is available?, b) How should they be analyzed?, c) How should the results of the analysis be visualized for users to obtain them as feedback information?, and d) What are the effects of this approach to the overall learning activity; in what ways are the users supported by such tools?. In latter steps of research, the correspondence of each tool based on this approach to the various user profiles and roles, as well as the discourse activities learning strategies should be examined. The initial step of our research was to examine all the available data for analysis within an asynchronous discussion platform. Being single activity environments, discussion forae include rather limited amount of raw data. Possible user actions are connecting to the tool, reading and writing messages, with time as a common attribute. For the connecting action, the working space is the subject, with attributes such as number of messages contained, number of corresponding authors, number of messages added of read between user sessions, duration of user sessions etc. On the other hand, messages are the artifacts of the working space which are manipulated by the participants. The attributes of a message include author, time of writing, how many times was it read, when and by whom, how is it positioned within the discussions structure (which thread, is it a reply, are there any replies to it etc). Additionally, depending on the design of the discussion other attributes may exist, such as type of message (question, answer, argument etc). All the aforementioned quantitative data constitute workspace awareness information and can be used to implement corresponding tools for the involved users (participants, researchers). Many presentation methods can be selected, including literal, tabular and graphical format. Although awareness information is considered to be very important during collaborative activities, it seems inadequate for the case of discussion based activities. According to the Information Visualization field, alternative methods should be examined in order to present such simple quantitative information more efficiently. Furthermore, ways of producing new perspective on the existing activity data, focusing on more qualitative aspects of analysis (cognitive and metacognitive), should be researched. The definition of high level indicators and Interpretative Schemas in the Interaction Analysis field seems to meet these goals. Thus the research should be focusing towards the direction of finding new ways of raw data analysis and visualization, in order to provide more insightful examination of the collaboration, in favor of the involved users and the overall activity. In a latter phase, the exploitation of these methods should be thoroughly examined. 4 Related Work While examining Forum and Forum Type software, we found that commercial or open source products, such as WebCT, WebWiz and PhpBB provide minimum analysis information. Most of them implement simple usage indicators, such as activity information (number of messages posted and read), statistical indicators (most and least busy day, etc), online users, number of unread messages, etc. We consider this as minimal information, which supports forum usage only as a subsidiary tool of a Learning System [2]. Several promising approaches implementing graphical representations of asynchronous discussions features and parameters can be found in recent 18

19 literature. For example, the i-bee system (Fig. 1a) visualizes relationships between users and keywords in online messages. It also provides snapshots of past discussions and animations, with keywords appearing as flowers and users as bees. The distance between flowers and bees, their status (e.g. flying/sleeping bee, blossomed/closed flower) and their orientation depend on discussion parameters, such as keyword usage frequency and recent user activity [20]. Another example of the using powerful visualization metaphors is the i-tree system (Fig. 1b) that visualizes the discussion status on mobile phones using a tree representation. The tree corresponds to a single user, whose activities designate the tree s appearance. Thereby the tree s log and branches are relevant to the number of messages, the leaves range and color are relevant to message reading, the fruits are relevant to the answers the user has received and the appearance of the sky is designated by the whole discussion status [21]. Mailgroup (Fig. 1c) is a Forum Type tool with integrated Social Network Analysis tools. It implements an alternative method of representing the message sequence in an asynchronous discussion, taking into account both chronological and logical constituents [22]. Other approaches also exist, integrating Fuzzy Logic techniques in order to assess and evaluate the collaboration level in a discussion based on several parameters (Degree system) [1] or providing a variety of visualized statistical information (add-on for the AulaNet platform Fig. 1d) in order to help the teacher coordinate discussions and obviate undesirable situations or progress of the discussion activity [11]. Fig. 1. i-bee and i-tree systems visualizations These approaches constitute a representative specimen of asynchronous discussion software, used for learning purposes. All of them provide tools and functionalities for facilitating and supporting user activity in various levels. Nevertheless a closer examination leads us to the conclusion that they can only be used under specific usage settings, as described further in [2]. 5 Interaction Analysis indicators An example from the DIAS system The DIAS system (Discussion Interaction Analysis System) has been developed by the LTEE laboratory of the University of the Aegean. It is a fully functional discussion forum platform, with an underlying database system for data recording and several implemented functionalities in order to facilitate user participation, as well as the moderators alternative discussion strategy planning. Additionally about sixty five visualized indicators (including all variations) are produced, varying from simple statistical awareness information to complex cognitive and metacognitive indicators. Different sets are addressed to the teacher or moderator and the students - users, along with the corresponding interpretation schemas for different discussion strategies or usage scenarios. Our main goal is to offer direct assistance to users, supporting them in the level of awareness of their actions, as well as their collaborators, in order to activate their metacognitive processes, thus allowing them to self-regulate their activities. In parallel, we aim at supporting the discussion moderators (e.g. teachers) in order to identify undesired situations and difficulties that require regulative interventions. The design of the system is based on three core design principles [2]: 1. Take into account the totality of the users involved in a learning activity, as well as the cognitive systems they may form, students as individuals (in various roles), as members of one or more groups or even communities, teachers in different roles according the category of learning activity, etc. Some indicators should represent individual activity and others group or community activity and its context. 2. Provide a rich range of Interaction Analysis indicators for the various user profiles and points of view of the activity process, its quality, as well as its product. Different indicators may be more appropriate during different time periods of the learning, for different learning task, as well as for different profiles of forum participants. 19

20 3. Create an independent, flexible, customizable and interoperable system. Forae are tools that can be used in a variety of contexts and activity categories. Furthermore forum participants take various roles and have different needs according to their discussion subjects, the available time etc. Thus, customization and flexibility are crucial characteristics. This lead us to the selection of open source web based technology, making it easy to share with the academic community. More information about the system s architecture and functionality can be reviewed in [2]. At this point, a set of two high level Interaction Analysis indicators produced by the DIAS system will be presented, attempting to demonstrate how the analysis and visualization method can provide new perspectives on raw data, positively influencing the collaboration (as discussed in section 6). Both are addressed to the teacher moderator and the researcher observer. After studying the methods and dimensions of a discussion s qualitative analysis, proposed by the literature, we distinguished five quantitative dimensions, used as indicators of quality in a discussion forum thread: a) Number of users, b) Number of messages, c) Thread depth, d) Thread width, and e) Mean number of words. In our approach we tried to create a high level indicator which would combine all five, while producing a numerical value, indicating the more qualitative threads in a discussion [3]. Specifically for the width, parameter, we extended the definition of message width [26], introducing the concept of Thread Width, defined as the largest amount of messages in one level, within a discussion thread. Two (2) different indicators were constructed: a) Thread Propagation Indicator (T P ), and b) Thread Propagation Word Indicator (T PW ). The T P indicator provides a measurement of the propagation of a discussion thread. During the calculation of the indicator s value, the total width, the depth and the distributed width of the thread are taken into account. Higher values indicate more complex threads, thus distinguishing the most interesting (contributing more to the discussion quality) ones, worth revising. We make the assumption that wider, longer and more complex evolved threads are the more interesting to revise, as they imply increased interaction among the participants. The T PW indicator provides a connection of the T P indicator s value with the number of words in the messages of each thread. A B C D E F G Fig. 2. Visualization of Indicators T P and T PW The mathematical formula for calculating the indicators and detailed information regarding the visualization of their values can be found in [3]. In Fig. 2, an example extracted from two different discussion activities is presented. For every thread, two bar charts are produced. The left contains five bars, corresponding to the aforementioned important parameters. The right contains two bars corresponding to the T P (grey) and the T PW (pink) indicator. The structure of each thread is presented on the left side of Fig. 2, in order to assist the results presentation. No spectacular visualization method is used; simple bar charts have been selected. So, one could wonder, where is the innovation? According to the InVis field, our aim should be to present coherent large data sets, encouraging comparison and providing different perspectives of abstract information, while having a clear objective. In this case, multidimensional data, deriving from user interactions are presented in one diagram, trying to visualize the differences in the evolvement of the various discussion threads. Important aspects of their structure, widely used as individual quality indicators throughout literature [3] are combined, producing a coherent, larger data set. The main objective is to distinguish the important threads within a discussion, based strictly on quantitative data, helping a user lacking time to catch up with the discussion evolvement. An underlying theoretical background clearly justifies this objective [3]. These aspects alone could rationalize the characterization of the proposed indicators and visualizations successful. 20

21 The described approach can be applied in all kinds of discourse activities. Depending on the teaching strategy deeper analysis could be applied, thus adding new perspective on the presented information, resulting to higher level of qualitative results. Let s examine for example the diagrams shown in Fig. 2, which depict part of the discussions of two groups of undergraduate students. The learning activity consisted in presenting a situation (the floor-planning of a computer laboratory in Primary School) with two possible solutions. Each student selected the solution which met his/her personal criteria more, thus forming two separate groups. Each group had to collaborate through a discussion forum in order to jointly formulate arguments, supportive of their selection. Additionally, they were asked to enumerate the advantages and disadvantages of each proposal, while trying to improve the one they supported, if possible, providing a progress report at the end of this stage. At a later stage, the two groups joined in a common debate-like discussion, trying to reach a common decision. This discourse activity aimed at motivating the students in examining all the aspects of designing a computer laboratory, taking into account the potential population (children aged 6-12 years old) and the rest of the curriculum. In Fig. 2, threads A through D correspond to Group A which could access Interaction Analysis information throughout the activity and threads E to F correspond to Group B which had no access to such information at all. While trying to find a connection between the indicators values and the quality of the discussions, we examined the content of each thread accordingly. The results were very interesting and similar for both groups. The value of T PW is almost equal or slightly higher than the value of T P for threads A, B and E (Fig. 2). We examined the number of arguments discussed in these threads, which were included in the final report. Three arguments were negotiated in thread A, one in thread B and three in thread E (table 1). The value of T PW is significantly less than the value of T P in threads C, D and F. In these threads many social messages appeared or arguments were repeated by several students, in their attempt to appear active even if they had nothing significant or new to add. Finally, in thread G where the first draft of the report produced by Group B was posted for coordination purposes, T PW is significantly higher than T P. These results, which were verified by the examination of additional threads, indicate that the ratio of the two indicators corresponds to the significance of each thread s content. Table 1. Correspondence of indicators values and thread content Thread T P T PW T PW / T P Number of Arguments contained A B C D E F G Report 6 Discussion This paper focuses on the implementation of tools, supportive of collaborative learning activities participants, from the perspective of providing visualized feedback information. The presented information derives from applying Interaction Analysis techniques on activity raw data in order to construct meaning, which is represented taking into account Information Visualization findings. The center of interest is asynchronous discussion platforms. The initial step is to select the appropriate raw data, depending on the nature of the collaborative activity and the analysis objective. As this data is limited, analysis methods and visualization techniques should be carefully examined. According to the Information Visualization field, visual metaphors is one way of adding meaning to abstract data, such as the one used in the i-tree system [21]. A growing tree metaphor is used in order to map a person s activity to something more familiar for the students, from the real world. Thus the extension of the branches, the healthy color of the leaves, the fruit and the blue sky are used to indicate excellent performance. The i-bee system [20] uses a similar metaphor, after applying some statistical analysis in order to detect keyword usage frequencies. The picture of a bee trying to approach a flower depicts a desired state. In the MailGroup system [22], an innovative alteration of the message sequence representation within discussion threads enhances users perception of turn-taking and participation in general. Furthermore, sociograms 21

22 are used to analyze and present qualitative aspects of the overall collaboration. Finally, a set of simple charts provides a new perspective on users participation, for the teacher who is trying to coordinate and supervise a learning activity, in the AulaNet platform [11]. A simple reorganization of the data structure reveals new meaning and interpretation opportunities for the teachers. In all the presented systems, the raw data are always the same; messages, users, and their corresponding attributes. An alteration of the visualization method or the point of view, while examining the data, is enough in order to construct new analysis pattern. Furthermore, the example from the DIAS system clearly shows that such approaches can lead to the construction of tools which transform data in a way that represents cognitive data. Initially, the purpose of designing the T P and T PW indicators was to combine all the quantitative information used as discussion quality criteria, throughout the corresponding literature, in one indicator, thus providing a more insightful and complete representation. The objective was to be able to reach more concrete conclusions on that matter, by distinguishing important threads, with more qualitative content [3]. Of course, with the described teaching strategy the indicators proved to be useful in an additional manner, providing a very interesting assessment tool. This does not occur under other circumstances, such as the cases of discussion forae used in a less structured way, for information exchange. During our research with the DIAS system in real settings we examined the Interaction Analysis indicators influence on the discussion activity evolvement, focusing on students (users) behavior. Our findings reveal that the indicators presence tends to improve the students collaboration, in various ways: a) by acting as a strong, additional participation motive, b) by assisting students understand and assess the discussions goals, c) by promoting self-confidence, and d) by assisting the students reflect upon the overall activity, thus selfregulating their actions [3], [4], [5]. Additionally we investigated the evaluative and assessing potentiality of the indicators and their combinations, confirming the power of visualizing information. A number of new questions, for deeper research emerged in the process, related to the correspondence of the representations to specific teaching and learning settings (activity design, group forming), to user profiles and roles, to age, sex and cognitive background of the users and many more aspects. What this paper has tried to emerge is the fact that simple ideas deriving from the initial point of view when collecting and trying to analyze simple data may lead to very interesting research questions and findings. The field can be vast [9] and foster innovative ideas, such as the ones reviewed in this paper. The power of visualized indicators has been outlined in the literature [19]. Especially the research area of implementing supporting, evaluating and assessing tools for CSCL systems in order to enhance the process and the learning outcome has a lot to gain by such approaches. A clear focus of analysis points of view, solid questions formulation, imagination in designing and many case studies is all that is needed. An additional positive outcome of such approaches could be the analysis of students behavior, under different teaching settings. In the example presented in this paper, indicators proportions have been correlated with the discussions content, revealing a pattern within the users contributions; the development of collaborative constructed arguments through dialogic negotiation along with a corresponding metric. Thus models of the expected user behavior may be constructed. Other indicators of the DIAS system can be used towards this direction, if used in a reverse engineering approach, likewise. Using them to study behavior, user modeling may occur, thus designating expected action patterns in future teaching activities. One of the research objectives with the DIAS system has been to designate the proper indicator set for the different needs of the users, depending on the cognitive structures they form, their various roles and the various teaching settings [4], [5]. Having accomplished that (as part of the future designed work), the same reverse engineering approach for user modeling can be followed for constructing models, which can be used to select the appropriate teaching approach in the future and thus lead to adaptable tools, automating the supporting process. References 1. Barros, B., Verdejo, F.: Analysing student interaction processes in order to improve collaboration. The DEGREE approach. IJ AIED, 11, (2000) Bratitsis, T., Dimitracopoulou, A.: Data Recording and Usage Interaction Analysis in Asynchronous Discussions: The D.I.A.S. System. Workshop on Usage Analysis in Learning Systems, 12th Int. Conference AIED 2005, Amsterdam (2005) 3. Bratitsis, T., Dimitracopoulou, A.: Indicators for Measuring Quality in Asynchronous Discussion Forae. Int. Conference CELDA 2006, Barcelona, Spain, IADIS (2006) 22

23 4. Bratitsis, T., Dimitracopoulou, A.: Interaction Analysis in Asynchronous Discussions: Lessons learned on the learners perspective, using the DIAS system, Int. conference CSCL 2007, NJ, USA (accepted) 5. Bratitsis, T., Dimitracopoulou, A.: Interpretation Issues in Monitoring and Analysing Group Interactions in Asynchronous Discussions. Int Journal of e-collaboration (2007 in press) 6. Cadiz, J.J., Venolia, G.D., Jancke, G. & Gupta, A.: Sideshow: Providing Peripheral Awareness of Important Information. Microsoft Research Technical Report MSR-TR (2001) 7. Card, K. S., Mackinlay, J. D., & Shneiderman, B.: Readings in Information Visualization, using vision to think. Morgan Kaufmann, Cal. USA (1999) 8. Dimitracopoulou, A., Kollias, V, Harrer, A., Martinez, A., Petrou, A. Dimitriadis, Y. Bollen, L. Marcos, J.A. & Wichmann, A.: State of the art of interaction analysis for Metacognitive Support & Diagnosis. IA JEIRP Deliverable D Kaleidoscope NoE, Available online at: 9. Dimitracopoulou A. & Bruillard E. Interfaces de Forum enrichies par la visualization d analyses automatiques des interactions et du contenu. (Guest Editors) E. Bruillard & G.-L. Baron. Special Issue on Forum des Discussions Asynchrones, Sciences et Techniques Educatives (in press 2007) 10. Dourish, P.: Extending awareness beyond synchronous collaboration. In T. Brinck & S.E. McDaniel (eds): CHI 97 Workshop on awareness in Collaborative Systems, Atlanta, Georgia, March Gerosa, M.A. Pimentel, M, Fuks, H. & de Lucena, C.J..: No need to read messages right now: helping mediators to steer educational forums using statistical and visual information. In: Koschmann, T., Chan, T., Suthers, D. (eds): Proceedings of CSCL 2005, LEA, USA (2005) Gershon, N., Eick, S. & Card, S. Information Visualization. Interactions, 5(2) (1998) Greenberg, S, Gutwin, C. & Cockburn, A.: Awareness through Fisheye Views in Relaxed-WYSIWIS Groupware. Proceedings of Graphics Interface, Toronto, Canada, May (1996) Gunawardena, C., Lowe, C. & Anderson, T.: Analysis of global online debate and development of interaction analysis model for examining social construction of knowledge in computer conferencing. Educational Computing Research, 17(4) (1997) Gutwin, C. & Greenberg, S.: A Framework of Awareness for Small Groups in Shared Workspace Groupware. Report 99-1, Department of Computer Science, University of Saskatchewan, Canada, (1999) 16. Gutwin, C. & Greenberg, S.: A Descriptive Framework of Workspace Awareness for Real-Time Groupware. Computer Supported Cooperative Work. The Journal of Collaborative Computing, vol 11. Kluwer Academic Publishers, Netherlands, (2002) Hearst, M.: Information visualization: Principles, promise, and pragmatics. Handouts of the tutorial at CHI 2003 Conference on Human Factors in Computing Systems (2003) 18. Jerman P., Soller A. & Muhlenbrock M.: From Mirroring to Guiding: A Review of State of the Art Technology for Supporting Collaborative Learning. In: P. Dillenbourg, A. Eurelings, & K. Hakkarainen (eds): Proceedings of EuroCSCL, Maastricht, (2001) Mazza, R. & Milani C.: Exploring usage analysis in learning systems: Gaining insights from visualizations. In Choquet, C., Luengo, V., Yacef, K. (organizers), Workshop on usage analysis in learning systems, The 12 th international conference AIED, 2005, Amsterdam, (2005) 20. Mochizuki, T., Kato, H., Hisamatsu, S., Yaegashi, K., Fujitani, S., Nagata, T., Nakahara, J., Nishimori, T. & Suzuki, M.: Promotion of self-assessment for learners in online discussion using the visualization software. In: Koschmann, T., Chan, T., Suthers, D. (eds): Proceedings of CSCL 2005, LEA, USA (2005) Nakahara, J., Kazaru, Y., Shinichi, H., Yamauchi, Y.: itree: Does the mobile phone encourage learners to be more involved in collaborative learning? In: Koschmann, T., Chan, T., Suthers, D. (eds): Proceedings of CSCL 2005, LEA editions, USA, (2005) Reyes, P., & Tchounikine, P.: Mining learning groups' activities in Forum-type tools. In: Koschmann, T., Chan, T., Suthers, D. (eds): CSCL 2005, LEA, USA, (2005) Roden, T.: Populating the application: A model of awareness for cooperative applications. In M.S. Ackerman (ed): CSCW 96: Proceedings of the Conference on Computer Supported Cooperative Work, Boston, Mass., November 16-20, New York: ACM press, (1996) Scaife, M. & Rogers, Y.: External cognition: How do graphical representations work? International Journal of Human-Computer Studies 45, (1996) Schmidt, K.: The problem with awareness. Introductory remarks on Awareness in CSCW. Computer Supported Cooperative Work. The Journal of Collaborative Computing, vol 11 (2002) Simoff, S. J. & Maher, M. L.: Analysing Participation in Collaborative Design Environments. Design Studies, 21 (2000) Stahl, G.: Group Cognition: Computer Support for Building Collaborative Knowledge. Acting with Technology Series, MIT Press, (2006) 28. Tufte, E.R.: The Visual Display of Quantitative Information. Graphics Press, Cheshire, USA, (1983) 23

24 Group Interaction Prompted by a Simple Assessed Open Learner Model that can be Optionally Released to Peers Susan Bull and Mark Britland Electronic, Electrical and Computer Engineering, University of Birmingham, Edgbaston, Birmingham, B15 2TT, U.K. s.bull@bham.ac.uk Abstract. This paper describes a study of a simple open learner model that can be optionally released to peers in named or anonymous form, to facilitate collaborative and competitive learning in a situation where the learner model contents contributed to students' assessment. Results suggest that, under the right conditions, students may start working sooner; discussion amongst students can be increased; many will find the individual learner models of others useful; and most will open their model to other users even if they do not find peer models helpful themselves - thus ensuring the availability of peer models for those who wish to use them. We consider the extent to which these findings may be transferable to other settings. 1 Introduction Open learner models (OLM) are learner models that are accessible to users. Reasons for opening the model to the learner include promoting reflection; and encouraging learners to take greater responsibility for organising their learning, facilitated by the information about their current knowledge state as represented in their learner model [1]. It has also been suggested that students working together on a task may benefit from being able to access a group model [2], and that access to the individual models of peers working on a task may prompt spontaneous peer tutoring [3]. Less attention has been directed at allowing students to access the learner models of each other in a more open context alongside a course, where students can: use peer models to help them find collaborative partners or helpers; work independently, competitively using peer models to help them identify whether they are ahead of other learners; or identify cases where they thought they were doing well but, in fact, are struggling in comparison to others. Access to the models of peers in this type of learning context may motivate students to work harder, and students can also be reassured by seeing that they are not the only person experiencing a particular problem at the time. This paper introduces OLMlets, a simple OLM to help users identify their learning needs. Based on UMPTEEN [4], OLMlets permits learners to grant others named or anonymous access to their model. We describe a study of OLMlets in a 3 rd year university course on Interactive Learning Environments, where the model formed 10% of the course mark. Thus students had extrinsic motivation to work with OLMlets as this contributed to the assessment and, given the nature of the course, they also had an understanding of educational technology (with a particular focus on intelligent tutoring systems), so were able to appreciate the benefits of OLMs. The aim in this paper is to provide an indication of what can be achieved with a simple OLM that can be released to others, and to consider which aspects might transfer to contexts where students have no specific interest in educational technology, and where the instructor may not wish to assess the learner models. 2 The OLMlets Open Learner Model OLMlets is domain-independent, designed for a range of course types, so the learner modelling is quite simple in order that it may be easily deployed with minimal information required from instructors. Relationships between concepts or prerequisite requirements, etc., are not modelled; OLMlets simply uses responses to multiple choice questions input by the instructor to infer knowledge level and misconceptions - where misconceptions are defined by the instructor (see 24

25 [5]). In this Interactive Learning Environments course, questions were designed to elicit problems most commonly found in coursework, e.g. confusing a domain model with a simple (e.g. text) description of a subject; confusing a learner model with a percentage based on all answers (regardless of their relevance to current understanding); the belief that matching an interaction to learning style will necessarily lead to improved learning; the belief that educational evaluation is about whether students like a system (i.e. not considering learning gains or other learning issues). Modelling occurs over the last 5 attempts at questions on a topic, with heavier weighting on the more recent of these responses to maintain the focus on current knowledge - the weighting increases by 0.3 for each successive answer of the 5 contributing to the learner model of a topic. This is represented in the model by a figure between 0 and 1, with 0 indicating no knowledge and 1 full understanding. Fig. 1. Open learner model: left - skill meter (showing own model and peer comparison); top right - graph; bottom right - text, table, boxes (combined image) The most common presentation method for simple OLMs is the skill meter [4,6,7,8,9]. The size of the colours in the OLMlets skill meter indicates knowledge level, areas of difficulty, and misconceptions (see Fig. 1). OLMlets offers 4 further presentation formats: graph (with positive and negative information on different sides of an axis); text (short statements of knowledge); table (topics ranked according to knowledge); boxes (with colour representing knowledge). Previous work found that skill meters were the most commonly used, but some students did prefer other representations, or used the skill meters in combination with one or more other presentations [10]. Where the existence of misconceptions is shown, students can obtain a description of the applicable misconception(s) by clicking on the link. This is then displayed as defined by the instructor in the form of a sentence prefixed by 'You may believe that ', which remains visible in the OLM unless the student hides it. An example for this course is: 'You may believe that a system does not have to understand the learner model'. Students can also access a comparison of their knowledge to the instructor's expectations of knowledge for each stage of the course (illustrated with the skill meters in Fig. 2); and information about the distribution of the group's knowledge for each topic, with a star indicating the learner's own position for ease of comparison. The aim is for students to use OLMlets to help them identify areas of difficulty, and then undertake the necessary reading (using links within OLMlets to course notes or external online information accessed from the 'M' (materials) icon in Fig. 1, or finding relevant information independently); or discuss problems with peers. Thus, OLMlets does not itself tutor or contain learning tasks: the intention is that students retain responsibility for organising their learning. The above features have been found useful in 5 university courses at different levels [5]. New to this version of OLMlets is the ability to access individual peer models (the student's own model is at the top, with others appearing underneath - see Fig. 1); and the ability to selectively open one's model to other users. In this paper we therefore concentrate on the use of peer models. Fig. 2 shows the manner in which students release their model. The top part of the table lists the course topics in the first column. Students are listed in the lower part of the second column (names have been hidden here), and instructors are listed in the lower part of the third column. Students can choose which parts of their model to release to each group - all, selected or none (upper cells), also allowing students to select which topics to release; and whether to release their model named or anonymously (middle cells). Students may create new groups to release their model in different ways to different users (e.g. named to friends and anonymously to others). These groups are populated by dragging and 25

26 dropping a user's name into the new column (group) they have created. Students can opt to view all or selected models available to them (or view no peer models), in any of the five formats (Fig. 1). Fig. 2. Comparisons: top left - instructor expectations comparison; bottom left - group distribution; right - releasing the learner model to other users 3 Viewing Peer Models and Opening the Learner Model to Peers This section presents results of the use of the OLMlets peer models. 3.1 Participants, Materials and Methods Participants were 29 students in their 3 rd year of a 3 year BEng or 4 year MEng degree in Computer Interactive Systems, Multimedia Systems or Computer Systems Engineering. Students used OLMlets alongside a course on Interactive Learning Environments. They therefore had a theoretical knowledge of learning issues relevant to educational technology, including an introduction to the potential educational benefits of OLMs. The OLMlets learner models contributed 10% to the course mark, the learner models being recorded for assessment at the end of the course. The course lasted for one term, assessed additionally by an introspective analysis of the student's own learning (10%) and an educational technology design/evaluation report (80%). OLMlets was introduced in a 1 hour lab. Subsequent use was at times and locations of students' choosing. Results were obtained from questionnaires, system logs and the learner models. Questionnaire statements required responses on a 5 point scale (strongly agree - strongly disagree). We here combine responses of strongly agree and agree (labelled agree), and strongly disagree and disagree (labelled disagree). 28 questionnaires were returned, 3 of which did not permit their use for research. Thus the questionnaire results presented are from 25 students (86% of users, 89% of questionnaires). Additional information about group interaction was obtained by retrospective reflection of a course member. 3.2 Results Table 1 shows the number of questions answered, and the final learner model representation achieved. The learner model figure is an average over the 6 topics, each of which is represented in the underlying model by a value between 0 and 1. All students answered over 100 questions (range ; mean 261; median 210). Nearly all achieved an expert model (1.0) by the end of the course. Only 3 achieved lower than 1.0 (0.4, 0.94, 0.96), with only 1 of these unacceptably low. Fig. 3 shows questionnaire responses about one's own and peer models. All students found their own model useful, and nearly 90% found comparing their model to the knowledge expected by that time (defined by the instructor), to be helpful. The group comparison was preferred by fewer people, with around half finding it useful. Individual peer models were found least useful when they were anonymous - but were still considered helpful by around 30% of users. In contrast, 56% were interested in the named models of peers, finding these helpful for their own learning. 26

27 Table 1. Usage levels and final learner model state Mean Median Range Questions attempted Final learner model Table 2. Releasing the learner model to peers Named Anonymous Named/Anon Hidden Week 3 5; 1 (some p) 4 1 (part, some p) 18 Week 5 13; 2 (some p) Week 7 16; 2 (some p) Week 9 18; 1 (some p) Table 3. Number of logins and accessing peer models Peer model access: 0-1 accesses 2-5 accesses 6+ accesses Mean qs Median qs 3-7 logins (n=9) * * * * * * * * * logins (n=10) * * * * * * * * * * logins (n=10) * * * * * * * * * * ow n individual model instructor expectations group comparison anonymous peer models named peer models agree undecided disagree 0% 20% 40% 60% 80% 100% Fig. 3. Utility of the individual and peer models view: stronger students view: similar students view: weaker students view: friends open: stronger students open: similar students open: weaker students open: friends agree undecided disagree 0% 20% 40% 60% 80% 100% Fig. 4. Viewing and opening the learner model to peers according to ability and friendship competitive use discussed OLM find collaborators agree undecided disagree 0% 20% 40% 60% 80% 100% Fig. 5. Collaborative and competitive use of peer models Taken from the system logs at fortnightly intervals, Table 2 shows the points in the course at which students chose to release their learner model to others, and details of the way in which they released it. In week 3 students had been using OLMlets for 1 week. At this stage, 11 of the 29 students opened their model to peers. 1 of these opened part of their model (the stronger topics) to selected peers only, to some named and some anonymously. 1 other student also opened their model to only some peers, but released to each of these a full model in named form. The remainder releasing their model opened a full model to all peers either named (5), or anonymously (4). By week 5, 20 of the 29 students had released their model, all releasing a full model. 9 kept their model hidden from peers. Most who opened their model to peers, opened a named model (15), though 2 of these opened such a model to selected peers only. Of the remainder, 4 opened an anonymous model to all peers; 1 opened their model to all peers, but to some named and others anonymously. The pattern in week 7 was similar to week 5, but with 3 models that were closed to others previously, now opened fully to all in named form. The final row shows how students had 27

28 released their model by week 9, the point of assessment. All 23 releasing their model, released a complete model. The majority (19) opened it to their peers with their personal identifying details, but with 1 of these releasing their model to selected peers only. 2 opened their model to all anonymously, and 2 opened it in different ways to different people. 6 still kept their model hidden. The 11 users releasing their model at the start of the course had varied models - some strong for the stage of the course (5), some quite weak (4), and 2 on track. The strength of a model was not a strong predictor of whether it would be released named or anonymously: there were 2 named and 3 anonymous strong models, 3 named and 1 anonymous weak models, and 1 named and 1 named/anonymous 'on track' models. However, those releasing their models later, appeared to release them when they felt they were adequate - i.e. newly released models were rarely weak. Despite the instructor expectations comparison, students had different approaches to completing their work with OLMlets before the assessment deadline. 1 achieved their final model state in week 3; 9 in week 5; 7 in week 7; 12 in week 9 (recall that only 1 student did not achieve a (near) expert model). Some preferred to answer many questions in a few sittings, reading and revising material at the time (some early in the course, some more evenly throughout). Others preferred to use OLMlets more frequently, for shorter periods. In this case a similar pattern was observed, where some completed their model relatively early, and some worked more evenly through the term. The mean and median questions attempted (Table 3) suggest that the third logging in the fewest number of times required fewer questions to demonstrate good knowledge; but the middle third and third logging in most frequently did not differ much in the number of questions required. The number of logins was not related to the week by which students completed their model. Once peer models are enabled, they remain visible until the student hides them or logs out. Thus the figures in Table 3 for enabling viewing of the models of peers (where each asterisk represents a student), apply to a complete session - this is logged only once, although there may have been multiple viewings of peer models within a session. It is assumed that if peer models were never enabled, students had no interest in them. If they were enabled once only, this may have been out of curiosity or in order to determine their potential utility. However, the lack of subsequent viewing in later interactions also indicates a lack of interest in peer models. (All students initially logged in at the same time, and all had some peer models available on their first access.) There was a tendency for those who logged in more frequently, to more frequently view peer models. This applied especially to those whose interactions were spread most evenly through the term. Fig. 4 gives questionnaire results for viewing peer models and releasing one's own model according to relative ability and friendship. Around 60% were viewing the models available to them, of those they believed to have stronger knowledge than themselves, and those they thought were at a similar level. Only 16% were interested in the models of users they considered weaker than themselves. Their friends' models, regardless of ability, were accessed as frequently as those of the stronger and similar students. While many opened their model to all peers, the questionnaire sought to determine the extent to which users specifically considered those to release their model to, even if they eventually opened to all. A similar, but less extreme pattern was found for releasing the model, with 72-80% considering it useful to open their model to friends and those they believed to be at a higher or similar level to themselves; and 52% to those they considered weaker. Fig. 5 shows the way in which students used peer models in collaborative and competitive learning. Around half stated they were using the peer models available to them competitively - i.e. to try to outperform other students. Almost as many used them collaboratively, discussing their models with each other. 5 (20%) used the peer models to actively seek collaborators or helpers. The following excerpts are from student descriptions of the utility of the peer models. Releasing the learner model I felt it would be useful, from the beginning, to allow everyone anonymous access to my model, in order to provide comparison models for other users. I felt this was fair, as I was expecting other people to do the same so I could compare myself against theirs. The reason I decided I was going to show my personal details is to hopefully encourage more communication within the group. I opened my learner model to my peers anonymously as I didn't want to be asked for help. I decided to open my learner model to all my peers and instructors to get myself recognized. I decided against opening my model to peers as I did on occasion discuss my current level with my peers out of lectures as a gauge for how far they had got to judge how far along I should be. I felt if I restricted my model from certain people it may be assumed I had issues or grudges with them. 28

29 Individual use of peer models Allowing me to see if I was progressing at the same, further or behind my peers allowed me to think about my work patterns, and was I doing enough? I also used it to examine what misconceptions were held by other people to avoid holding those misconceptions myself. I was continually comparing my learner model to my peers to see if they were experiencing any similar misconceptions or problematic areas or whether they were individual to me. Collaborative use of peer models By having this knowledge of other students it allowed me to ask for peer help, in trying to overcome my misconceptions and help those with different misconceptions that I knew about. I do not think studying should necessarily be a private thing. Several peers approached me to exchange ideas and useful papers seeing that I had been frequenting OLMlets! Another important aspect of opening a named model was that I was receiving responses from my peers. They would come up to me and point out that they were having difficulties in the same topics as me and one student even helped me to resolve these issues. Competitive use of peer models I saw my knowledge increase I became more competitive. I no longer just wanted to meet the weekly target. I wanted to race ahead of the weekly target and also my peers. In a way it was almost like competing for the top position. Non-use of peer models Whether other people have achieved some specific level does not affect my own target at all. Sometimes I may have different schedules from others, in other words, I may pay different amounts of attention at different times to different modules. I largely chose not to observe other people's learner models. From an early point I felt I wished to keep the momentum in my learning more self-motivated rather than peer-motivated. 3.3 Group Interaction The following description provides an insight from the students' perspective, of the kinds of interaction amongst the group that were prompted by peer models. This is based on the experiences of the second author, a student on the course who was therefore in a good position to summarise what was happening amongst the students, to supplement information from the questionnaires and logs. When OLMlets was first used in the set lab session there was not much discussion. Initially progress was measured against the expected level of knowledge, with the general focus of any conversation being whether a student was ahead or behind the expected levels. After it was announced in a lecture that the instructor had seen the first completed model from those available to her, there was an immediate increase in the level of discussion and competitiveness. Regular discussions involved people who had mentioned that they had a complete model, who were willing to explain points to others. As more people completed their learner models, the general feeling of the remaining students was to complete their own model, as it was an assessed item that carried a 10% weighting, and was 'a silly 10% not to gain'. As the weeks progressed, there was also a lot more quiet competition. This was partly a race against time, but it was also comparative to other users. This occurred not only amongst students who had not yet completed as much; it also arose amongst those who were ahead, comparing themselves against fellow students in a 'bout of superiority' (knowng they had completed part of the assessment, and had finished ahead of others). At the start some students kept to the expected levels as they thought that exceeding them may be disapproved of, and may jeopardise their OLMlets mark. As the course progressed and more people completed their model ahead of time, more of those who originally aimed to stick to the schedule also completed their models. However, there were still a few who were careful to finish the topics in line with expectations. For some this was because they were also working on the main assignment and following the guidelines as advice for the rate to proceed; but for others it seemed to be a fear of being penalised in the assessment for not interacting with the system 'correctly'. When several people were using OLMlets at the same time, most notably in one of the small computer rooms, there came to be almost a community feel. Students were comparing their model against those of people in the room, and discussions were occurring spontaneously all the time.the 29

30 main difference between this course and others was in the amount of interaction amongst students. This was not only about the extent of completion of the learner model, but also about the course content as students were aware of their relative strengths and weaknesses, and were keen to help each other understand problematic points. The aim was often to improve the learner model, but students had to learn to accomplish this, and they recognised that they were learning. Discussions were taking place even amongst students who do not usually talk to each other about work. 3.4 Discussion It is difficult to be sure of the extent to which high usage levels (measured by the number of questions attempted) is due to students' interest in educational technology, and to what extent it results from the fact that the learner model formed part of the assessment. An advantage of formally assessing the model is that all students interact with the system, and so more may use it also for formative assessment. However, previous work found that, when OLMlets is recommended as preparation for an assessment in courses where students had no particular interest in user modelling or educational technology, usage levels can also increase [5]. It appears likely, therefore, that if students are made aware of the potential benefits and how it relates to assessment (even if the learner models themselves are not assessed), the benefits of prompting reflection on learning either individually or collaboratively, may be achieved. The fact that OLMlets is an open learner model means that students can judge when to stop working, and nearly all students in this study strove to achieve a perfect model. The excellent results were not only relevant to the OLMlets assessment, but crucially also ensured students had a good understanding of key issues for their coursework, increasing the likelihood that they would achieve the intended learning outcomes of the course. While all students found their own model useful, and most found the expectations comparison helpful, students had different reactions to the group and individual peer models. It is not expected that everyone should want to work collaboratively or competitively with OLMs - individual use of one's own model may better suit some. The ability to view peer models is therefore included for those who find it helpful. As over half claimed to find it useful, we recommend inclusion of this feature. Students found named models more useful than anonymous ones. Further work will help identify whether this was a feature of this student group, or whether high usage related to assessment, or the effect of use alongside a complete course, led to greater interest in knowing the owner of a model (for example, to help identify suitable learning partners; to set personal targets according to what similar students are achieving; or to avoid models considered unlikely to be useful). By the end of the course, 23 of the 29 students had released their model, most named, and all but one of these opening to all peers. In all cases all parts of the model were released. At the start of the course, 11 students opened their model to peers immediately, with all but one of these opening the full model, and all but two opening to all peers. This included a mixture of strong and weak models (according to expectations for the stage of the course), indicating that some were happy to release a weak model. This may help reassure students who are having difficulties, that others are in a similar position (see [4]). Models were released both named and anonymously, this choice not necessarily related to the strength of the model. During the course more students released their models, with the proportion of named models growing. Often, models released later were released once they were strong. Thus, at the start users saw a mix of models, and as the number of available models grew, so did the proportion of strong models. If these patterns turn out to be consistent over a range of courses, when using OLMlets in several courses students may be able to interpret the progress of others by the release rate of models. However, if usage patterns differ across courses, more work will need to be directed towards determining how students can appreciate the knowledge of their peers in particular if they are choosing not to use the group distribution view. Those who used OLMlets in fewer sessions, were quite evenly split in terms of whether they chose to view peer models. Those logging in more frequently tended to view peer models quite frequently throughout, suggesting that interest in peer models (for those who chose to use them), was maintained over time. It is interesting that the group logging in less frequently had a higher proportion of users who were not interested in peer models. Further work could investigate whether these students were working less frequently for extended periods in order to complete the work in as few sittings as possible and, perhaps for this reason, did not wish to 'waste' time viewing the models of peers; or whether the fact that these students required fewer 30

31 questions to achieve their final learner model state also meant that there was less need for comparison to peer models. Students were generally more interested in the learner models of those they considered stronger or of a similar ability to themselves, suggesting that they were focussing on improvement and comparing themselves to people they expected to be doing at least as well as themselves. They were perhaps also more concerned with seeking help from more advanced students, than offering unsolicited help. Students were also interested in the learner models of their friends, probably because they would normally be more likely to discuss their learning with friends, and also out of general interest for their friends. Students were interested in both discussing their model with others, and in using peer models competitively, as a measure against which to aim. Furthermore, despite existing groupings of friends amongst the students, 20% still used the peer models to seek out collaborators or helpers. Thus the availability of peer models can be recommended for each of these purposes, assuming these are appropriate for the context of system use. As stated above, it is not expected that all students should be willing to open their model to others, or that they should all want to view peer models. The important issue is that sufficient students were happy to release their learner model to other users, for the approach to be successful for those who found it helpful. The excerpts from student explanations of how they used peer models are illustrative of the main comments received. The results have already shown that most students chose to open their learner model to their peers. A sense of 'fairness' was sometimes expressed to explain this. Some students hoped to encourage collaboration amongst the group by releasing a named learner model. Others who were not interested in working with peers did nevertheless open their model to peers, though sometimes anonymously to ensure that they did not receive communications from others. However, a few students preferred not to release their model - though some of these did still discuss their model with peers. Common reasons for using peer models include the ability to see whether one is on track, and to try to avoid specific problems that others might have. Some students were more interested in using the models competitively, to try to outperform their peers, or more overtly to gain recognition. In contrast, some found that the models of others did not affect their own learning, as they followed their personal schedules. In one case a student was concerned that withholding their learner model could be seen as reflecting a (negative) perception of others, and so felt some pressure to release it. Clearly it is a problem if a system designed to allow optional release of the learner model to suit the individual's preference, results in a student nevertheless feeling some kind of obligation to release their model. There was only one such case, and many more positive experiences reported but this is an issue that should be further considered. Overall, most students were happy to open their learner model to others, even if they did not use peer models themselves; and around half found peer models useful, suggesting the value of providing this facility. According to comments from several students, a major effect of the widespread availability of peer models appears to be a generally increased level of communication about the course content amongst the group, which applied also to some learners who chose not to release their own model; an increase in competition amongst competitive learners which, in turn, also spurred many others on to work earlier in the course than they might otherwise have done. The instructor expectations comparison appeared to be the most common method of measuring progress at the start of the course. However, as the course progressed and students noted that some were working ahead of schedule, as stated above, this appeared to motivate many others to do likewise. Thus, later in the course, many students may have been using the expectations comparison in conjunction with the peer models as a target that they wished to exceed, rather than as the intended guideline for progression. A benefit of this was that students were more knowledgeable about the course content at each stage of the course, than in previous years. However, a question has been raised about a few students believing that they should perhaps not exceed the weekly targets as this may negatively affect their assessment. It is unclear how this belief may have taken hold, but highlighting to students that the targets can be exceeded, may be sufficient to address this. Spontaneous collaboration occurred particularly on occasions when students were working on OLMlets at the same time in the same lab, against a background of a 'community feel'. Such synchronised working was likely to have occurred because students had similar timetables, and so were at the same time filling in gaps between lectures, with lab activities. If students are following different timetables it may not be so easy to obtain the levels of peer interaction observed here, and in such cases it may be helpful to schedule dedicated lab sessions if this is possible. It appears that there were greater levels of discussion of this course than typically occurred in other courses, prompted in large part by the high availability of peer models. Although students were often 31

32 focussing on completing their learner model in order to complete their assessment, this necessarily involved further learning, and students did appear to be aware of this development of their knowledge. The key question arising from this study is whether the willingness to open one's learner model to peers will extend readily to other contexts, in particular in courses where students do not have a specific interest in educational technology, and in courses where the learner model does not form part of the assessment. Advising students of the relevance of a system as assessment preparation may be helpful (see [5]); and simply mentioning student progress in lectures can have a strong effect on prompting system use. Both these measures can be employed easily in most courses. Similarly, stating that targets may be exceeded if students wish to work ahead of the expectations comparison should remove the problem of students believing that to do this may constitute 'incorrect' use of the system. Providing time during a lab session to ensure that students have the confidence to use the system and are aware of the facility to view peer models and release their own model to others (and that they may remain anonymous in doing this if they prefer), may be important in later achieving the level of interaction amongst course members that occurred in this course. Instructor enthusiasm may help to prompt such exploration of peer models initially, though this level of encouragement appears less important to ensure general use (students used the previous version of OLMlets (without peer models) in courses in which the URL only was given, simply with the recommendation that it would be useful preparation for an upcoming assessment [5]). In conclusion, it appears that achieving high usage levels (which could be facilitated as suggested above), may have two main benefits arising from peer models: (i) increasing discussion amongst students; (ii) increasing competition amongst competitive learners, which then spreads to others, resulting in students generally working earlier in a course than they might otherwise do. 4 Summary This paper has described the use of learner models that can be opened to peers in a course where the learner model was assessed. Most students chose to release their model to others, with the majority releasing it in named form. While many students found peer models useful, we do not expect that all will prefer to use peer models. Nevertheless, most models were made available to others so that those who found them helpful, could use them. A result of the availability of peer models was an increased level of competition and communication amongst students. We therefore propose that, in contexts where collaboration and/or competitive learning is appropriate, students are provided the facility to release their learner model to others, and encouraged to explore this. References 1. Bull, S. & Kay, J. (in press). Student Models that Invite the Learner In: The SMILI Open Learner Modelling Framework, International Journal of Artificial Intelligence in Education 2. Tongchai, N. & Brna, P.: Enhancing Metacognitive Skills through the Use of a Group Open Learner Model Based on the Zone of Proximal Development, Proceedings of Workshop on Learner Modelling for Reflection, International Conference on Artificial Intelligence in Education 2005, (2005) Bull, S. & Broady, E.: Spontaneous Peer Tutoring from Sharing Student Models, in B. du Boulay & R. Mizoguchi (eds), Artificial Intelligence in Education, IOS Press, Amsterdam, (1997) Bull, S., Mabbott, A. & Abu-Issa, A.S. (in press). UMPTEEN: Named and Anonymous Learner Model Access for Instructors and Peers, International Journal of Artificial Intelligence in Education 5. Bull, S., Quigley, S. & Mabbott, A.: Computer-Based Formative Assessment to Promote Reflection and Learner Autonomy, Engineering Education 1(1) (2006) Corbett, A.T. & Anderson, J.: Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge, User Modeling and User-Adapted Interaction 4 (1995) Mitrovic, A. (in press). Evaluating the Effect of Open Student Models on Self-Assessment, International Journal of Artificial Intelligence in Education 8. Papanikolaou, K.A., Grigoriadou, M., Kornilakis, H. & Magoulas, G.D.: Personalizing the Interaction in a Web-Based Educational Hypermedia System: The Case of INSPIRE, User Modeling and User- Adapted Interaction 13(3) (2003) Weber, G. & Brusilovsky, P.: ELM-ART: An Adaptive Versatile System for Web-Based Instruction, International Journal of Artificial Intelligence in Education 12(4) (2001) Bull, S. & Mabbott, A.: Inspections of a Domain-Independent Open Learner Model with Individual and Comparison Views, in M. Ikeda, K. Ashley & T-W. Chan (eds), Intelligent Tutoring Systems: 8th International Conference, Springer-Verlag, Berlin Heidelberg, (2006)

33 Forming Homogeneous, Heterogeneous and Mixed Groups of Learners Agoritsa Gogoulou, Evangelia Gouli, George Boas, Evgenia Liakou, and Maria Grigoriadou Department of Informatics & Telecommunications, University of Athens, Panepistimiopolis, GR Athens, Greece Abstract. The formation of groups based on learners personality and performance attributes is a challending goal in the area of collaborative learning environments. In this paper, a tool for group formation, referred to as OmadoGenesis, is presented in terms of the algorithms implemented and its functionality. The OmadoGenesis tool accommodates learners characteristics in the formation of pure homogenous, pure heterogeneous or mixed groups, that is groups that satisfy heterogeneity for a specific learners characteristic and homogeneity for another characteristic. 1 Introduction Collaborative learning describes a situation in which particular forms of interaction among people are expected to take place, which would trigger learning mechanisms. However, there is no guarantee that the expected interactions will actually occur [3]. Hence, a general concern is to develop ways to increase the probability that some types of interaction occur. One way to do this is to set up initial conditions related, among others, to the group size and the selection of group members [3]. To this end, group formation, that is the identification of those learners belonging to one specific group, is considered very important [2]. Various practices may be used for the assignment of learners into groups. Random assignment helps mix up learners but do not directly address the problems caused by social dynamics. Learner-formed groups almost guarantee that a person will be comfortable with his/her group, but such groups are often based on friendships. Other practices are based on the ability or performance level of each learner; usually instructors form groups taking into account learners performance to a pre-test. Researchers in the area [1], [8] emphasize the importance of personality attributes (personal and social characteristics) in group composition. They suggest that in addition to performance level, attributes such as gender, ethnic background, motivations, attitudes, interests, and personality (argumentative, extrovert, introvert, etc.), should be given due attention in the process of forming groups. It is also observed that although homogeneous groups are better at achieving specific aims, when learners with different abilities, experience, interests and personalities are combined (heterogeneous groups), they out-perform homogeneous groups in a broader range of tasks [8], [9]. In a manual environment (with paper-and-pencil), a great deal of time and effort may be needed in the formation, especially of heterogeneous, groups. This is because, the numbers and combinations of performance level and values of personality attributes to be considered may be too many to handle and manage. Also, the problem may be more difficult, when the interest focuses on the formation of mixed groups, i.e. groups that satisfy heterogeneity for a specific learners characteristic and homogeneity for another characteristic. Research efforts attempt to develop computer-based tools that support the automatic forming of groups based on learners characteristics. The system developed by Yang et al. [10] is an attempt to group similar learners according to their preferences and learning behaviours. The system uses a multi-agent mechanism to manage and organize learners and learner groups. Inaba et al. [7] incorporated the grouping and constructed a collaborative learning support system that detects appropriate situations for a learner to join in a learning group. Graf and Bekele [6] propose a mathematical model that addresses the group formation problem through the mapping of both performance and personality attributes into a learner vector space. Their tool supports the formation of heterogeneous groups and uses an Ant Colony Optimization algorithm for maximizing the heterogeneity of the groups. 33

34 Our work attempts to contribute in the field by proposing a tool, referred to as OmadoGenesis that can be used for the formation of homogeneous, heterogeneous and mixed groups based on learners characteristics. The tool implements three algorithms: one for pure homogeneous groups (Homo-A), one for pure heterogeneous groups (Hete-A) and a third one based on the concept of genetic algorithms for homogeneous, heterogeneous and mixed groups. Also, the tool enables the random assignment of learners into groups and the assignment by the instructor on the basis of his/her preferences or learners demands. Moreover, the tool provides a number of facilities to the instructor such as the selection of the desired learners characteristics, the definition of the group size (i.e. the desired number of learners in a group), the refinement of the grouping results by rearranging the learners, the setting of the algorithm s parameters, and the specification of the criterion for the determination of the moderator of each group. The paper is structured as follows. In Section 2, we give a brief description of the conceptual framework and the definitions of various terms used. Following, in Section 3, a presentation of the algorithms is given in terms of their functionality. In Section 4, we present the tool from the instructor s point of view. Finally, in Section 5, we discuss the preliminary results of the application of the algorithms with real data and conclude with our future plans. 2 Conceptual Framework and Definitions The Learner Space Model. Each learner is represented in a multidimensional space by a vector; each dimension corresponds to a learner s attribute A n (i.e. learner s personality and performance attributes such as competence level, learning style, indicator for collaborative behaviour, indicator for acting as evaluator in peer-assessment). Each attribute A n has a value X n which is represented for the learner i as X n (L i ). The vector is made up of the values X n of all attributes. That is, learner i in a n th -dimensional space is represented as L i (X 1,X 2,,X n ). The values of the attributes are mapped to numerical values. Each of the n attributes has five possible values, i.e. X n =1, 2, 3, 4, 5, where 1 corresponds to the qualitative characterization Insufficient, 2 corresponds to Rather Insufficient, 3 corresponds to Average, 4 corresponds to Rather Sufficient, and 5 corresponds to Sufficient. For example, in a 3-dimensional space, the representation L 10 (2,3,5) means that learner with i=10 has values X 1 =2, X 2 =3 and X 3 =5 for the attributes A 1, A 2, A 3 respectively. Group. A group i of k learners is represented as G i =G(L 1,L 2, L k ). For example, G 1 = G(L 1,L 2,L 3,L 4 ) is the first group composed of the four (k=4) learners L 1, L 2, L 3 and L 4. Homogeneity. In a group G i with k learners, homogeneity with respect to an attribute A n exists when learners have similar values of the attribute considered, that is X n (L 1 )= X n (L 2 )= = X n (L k ). Difference. Difference (D n (L i, L j )) is defined as the distance between the values X n of the attribute A n of two learners (L i and L j ), that is D n (L i, L j )=abs{x n (L i )-X n (L j )}, e.g. if L 1 (1,3,5) and L 2 (2,4,1), then D 2 (L 2, L 1 )=abs{x 2 (L 2 )-X 2 (L 1 )}=abs(4-3)=1. Heterogeneity. A group of k learners G(L 1,L 2,,L k ), is heterogeneous with respect to the attribute A n, when D n (L j, L j-1 ) >= T, where (i) 2<=j<=k, (ii) the values of attribute A n are sorted such as X n (L 1 )<X n (L 2 )< <X n (L k ) and (iii) T is the threshold, which is defined as the lowest desirable possible value of the difference between the values X n of the attribute A n. Matrix-Hete used for the construction of heterogeneous groups in Hete-A algorithm. A Matrix-Hete of n attributes (where n=1, 2, 3) is defined as a n-dimensional array Matrix- Hete[5x5x5], where each element Matrix-Hete[i, j, z] (i, j, z =1,2,3,4,5) represents the number of learners that X 1 (L i )=i, X 2 (L i )=j, X 3 (L i )=z and contains a reference to an one-dimensional array that holds the identities (ids) of all learners L i that X 1 (L i )=i, X 2 (L i )=j and X 3 (L i )=z. In the Matrix-Hete, the parameter ideal distance is defined, which refers to the distance between the cells of the array when there are more than one cells having the same (highest) value. Criteria for group formation. The group formation process complies with the following criteria: each learner belongs only to one group, only one member is specified as the moderator of the group, if there are N learners to be segregated in q groups of k learners, then q=n/k if (N mod k=0 or N mod k=1) or q=n/k+1 if (N mod k>1). 34

35 Group Quality. The formation of a group may take into account more than one attributes and may follow for each attribute either homogeneity or heterogeneity. Therefore, the quality of group G i, consisting of k learners with respect to attribute A n, that is QG i (A n ), where 0 <= QG i (A n ) <= 4 (the range of possible values of the attributes is 1 to 5; thus, the maximum quality value 4 results from the difference between the highest and the lowest attribute value), is defined as follows: In case of homogeneity, QG i (A n )= 4 (max{x n (L 1 ), X n (L 2 ), X n (L k )} min{x n (L 1 ), X n (L 2 ), X n (L k )}) In case of heterogeneity, The quality of group G i is defined as k QG i (A n ) = = j 2 min{x n (L j) - X n(lj - 1),1} QG i = QG i( An), where n is the number of attributes considered for the formation of group G i n Solution Quality. The total quality QS of a solution is the sum of the qualities of all groups in the solution, i.e. q QS = QGi, where q is the number of all groups in the solution and 0 <=QS<= 4*n*q (4 is i= 1 the maximum quality value of group G i and n is the number of attributes) The Group Quality and the Solution Quality have been defined and used in the context of the genetic algorithm. However, they are also used in the context of Homo-A and Hete-A algorithm in order to have an indication of the quality of the produced solution. Moreover, they can be used as a quality measure for comparing the solutions produced by the available algorithms. 3 Developing the Algorithms Initially, our efforts focused on the construction of pure homogeneous or heterogeneous groups, therefore we developed two different algorithms (Homo-A and Hete-A), each one devoted to each case respectively. Following, having as an objective to cover also the case of mixed groups, we turned our efforts to the genetic algorithms as this category of algorithms are very flexible and allow the instructor to modify each time the definition of quality (see Group Quality in Section 2) for composing different groups of learners. The Genetic Algorithm (GA), presented in the following, is based on the principles of genetic algorithms but was adapted in order to be applied in all three cases (homogeneity, heterogeneity and mixed). Homogeneous Algorithm (Homo-A). The Homogeneous algorithm (Homo-A) is proposed by the tool only if homogeneity has been set for the selected learners attributes (up to three attributes can be selected). Homo-A uses clusters in order to create groups of learners and is based on the functional principles of k-means algorithm. The algorithm works as follows: consider that q groups of k learners will be created. A learner L i (X 1,X 2, X n ) corresponds to a point (X 1,X 2, X n ), where 1<=X 1 <=5,, 1<=X n <=5. At first, the q centers of the q clusters are chosen at random. In other words, q points (X 1,X 2,,X n ) are considered to be the centers of the q groups. Then, for each center the k closest points (learners) are chosen and moved into the group that the center belongs to. The proximity between points is calculated using Euclidean distance. In the next step, after the creation of the groups, a new center is calculated. Generally, for k learners per group with n attributes L 1 (X 1,X 2,,X n ),., L k (X 1,X 2,,X n ), the center is calculated as follows : k k X 1 ( Li) Xn( Li) i=1 i=1 Center = (,, ) n n The previous steps are repeated until there are no changes in the centers of the groups. 35

36 Heterogeneous Algorithm (Hete-A). The Heterogeneous algorithm (Hete-A) is proposed by the tool only if heterogeneity has been set for all the selected learners attributes (up to three attributes can be selected). Hete-A is based on the Matrix-Hete. Fig. 1 presents a two-dimension Matrix- Hete for 90 learners; one dimension corresponds to attribute A 1 and the second one to attribute A 2. Each cell C[i,j] of the matrix where i, j =1,2,3,4,5, has a value denoting the number of learners having the values i and j for A 1 and A 2 attributes respectively, e.g. the cell C[3,5] with value 4 denotes that there are 4 learners (L 1, L 25, L 33 and L 82 ) having the value of 3 for attribute A 1 and the value of 5 for attribute A 2. A L 1 L 25 L 33 L A1 Fig. 1. Matrix-Hete for two learners attributes A1 and A2 The algorithm works as follows: consider that q groups of k learners will be created. In the first step, the cell with the highest value is chosen. One learner from this cell is randomly chosen and put into a group. When the learner is chosen then the value of the cell is decreased by one and the learner is subtracted from the array. The line and the column that this value belongs to are excluded. This procedure is repeated till k learners are put into the group. The whole process is repeated from the beginning (using each time the updated cell values of Matrix-Hete) till all learners are run out. If there are more than one cells that have the same highest value (e.g. the cells having the value 8 in Fig. 1) then the parameter ideal distance is used to choose the right cell. More specifically, the Euclidean distance is calculated between the cell having the highest value in the previous step and the cells having the same highest value specified in the current step. The cell that has distance (from the previous specified cell) closer to the ideal distance is chosen. It may happen that the current group cannot be completed although there are still free learners because a learner cannot be chosen as all lines and columns have been excluded. In this case all lines and columns are recovered with their updated values and the process is repeated till the group is complete or there are no more free learners. In case that the Matrix-Hete is one-dimensional then the only difference is that only a column is excluded. In the example of Fig. 1, assume that groups of three members have to be formed. The cell having the value of 22 is the first one chosen and one of the 22 learners is randomly selected. The corresponding line and column are excluded and the cell with the next highest value is chosen, that is the cell having the value of 9. One of the 9 learners is randomly selected and put into the group. Afterwards, as there are three cells with the same highest value (value of 8), the distance of these cells from the last specified cell (having the value of 9) is calculated. The cell that has distance closer to the ideal distance (ideal distance=2) is chosen, that is cell C[4,2]. One of these 8 learners is randomly selected and put as the third member in the group. Since the first group has been formed, the Matrix-Hete is updated with the new cell values and the whole process starts from the beginning in order to form the remaining 29 groups. Genetic Algorithm (GA). Genetic algorithms are inspired by Darwin's theory about evolution. The evolution starts from a population (generation) of randomly selected solutions. Solutions from one population are taken and modified through the genetic operators of crossover and mutation to form a new population. The new population is expected to be better than the old one. The selection of some solutions (parents) to form new solutions (offspring) is based on their quality (fitness), which is calculated by a fitness function. The more suitable the solutions are the more chances 36

37 they have to reproduce new solutions. This process is repeated until some condition (e.g. number of generations or improvement of the best solution) is satisfied. The Genetic Algorithm (GA), implemented in the OmadoGenesis tool, can be applied for the construction of homogeneous, heterogeneous or mixed groups. The GA is defined by the following set of parameters: (i) Number of Generations: number of times that the population evolves, (ii) Population: number of solutions in one generation, (iii) Transport to New Generation: number of best solutions of one generation that pass to the next generation without the genetic operation of crossover and mutation (elitism), (iv) Mutation Possibility: (random) number which specifies if parts of the solutions of the worst groups will change, after the genetic operation of crossover, and (v) Avoidance of Marginal Values: it concerns only heterogeneity and means that D n (L i, L j ) < 4, e.g. group G(L i,l j ) is not a desirable one, where X n (L i )=1 and X n (L j )=5, as D n (L i, L j ) = 4. The procedure of GA is as follows: - STEP 1. The first generation is created by composing random groups of k learners. A solution is the set of groups generated. The quality of each group G i (QG i ) and the quality of each solution (QS) are calculated. - STEP 2. In order to create the next generations (parameter Number of Generations), a number (parameter Transport to New Generation) of the best solutions is transferred to the new generation. Then, two solutions are chosen with the roulette wheel selection method (an imaginary roulette wheel is used so that each candidate solution represents a pocket on the wheel). The better the quality of a solution (the bigger the pocket on the wheel), the bigger the possibility of the solution to be chosen for reproduction. From the two solutions, two offspring are created using crossover and mutation operations and the one with the best quality is chosen and is passed to the next generation. - STEP 3. The procedure of STEP 2 is repeated until the required number of solutions in a generation is created, that is until the number of solutions in a generation is equal to Population Transport to New Generation. In the context of the OmadoGenesis tool, a graphical representation of the GA is offered and the instructor has the possibility to terminate the GA at any time s/he wishes and the best solution (with the highest QS) that has been found at this time is provided as the final solution. In the current implementation, the fitness function supports groups consisting of up to four members (as we are interesting in such size of groups) but it can be easily adjusted to support any group size. The mutation and crossover are the most important operations of the GA. The aim of mutation is to prevent solutions in population falling into a local optimum of the problem. More specifically, if N is the number of all learners then, the following are repeated N times for all the solutions: A random number p is produced. If p is smaller than the parameter mutation possibility then two more random integer numbers are produced which take values from 0 to N. These two random numbers are used to select two learners of the solution. If these two learners belong to groups with poorer quality than the quality of the best group of the solution then the exchange is done. This means that the first learner is moved to the group of the second learner and the second learner moves to the group of the first learner. Otherwise, there is no exchange. Regarding crossover operation, there are many crossover techniques used in genetic algorithms such as one point crossover, two point crossover etc. Instead of using these techniques, in our development, we created a new technique in which the measure of the quality of each group (QG i ) is used in order to create the offspring. More specifically, the procedure of crossover follows the steps described below: - STEP 1. The groups in each solution are classified in an ascending order, according to the quality of each group (QG i ). As a result, there are two solutions in which the first group of each solution has the worst quality and the last group has the best quality. - STEP 2. The second step starts from the best group of the second solution and continues for each group of this solution. For each learner of the candidate target group (i.e. the group that is examined in order to be placed in the offspring), the group in which this learner belongs to the first solution is found and the quality of this group is examined. If all learners of the candidate target group (in the second solution) belong to groups in the first solution with worse quality than the quality of the candidate target group, then the candidate target group is added to the offspring. - STEP 3. All groups from the first solution that were not influenced in the previous step (i.e. consisting of learners that were not added in the offspring) are added in the offspring. 37

38 - STEP 4. All the remained learners from the first solution belonging to groups that were influenced in the second step are added in the offspring according to their order in the first solution. These learners are assigned to groups following their sequential order. - STEP 5. The second offspring is created in the same way as the first one with the difference that the second step starts from the first solution and ends to the second solution and STEPS 3 and 4 refer to the second solution. For example, let us assume that homogeneity has been selected for attribute A 1 and the groups may consist of three learners. The two solutions A and B, presented in Table 1, are consisted of 5 groups (Column G i ). The column QG i represents the quality of each group. The crossover operation works as follows: The groups in each solution are classified in an ascending order according to their quality. The first candidate target group is the group of the second solution with the best quality, that is group G 5 =(L 10,L 6,L 12 ) of solution B. Learners with ids 10, 6, 12 are searched in the groups of solution A. Every learner of G 5 of solution B belongs to groups in solution A with worse quality than QG 5, so G 5 of B is placed in the offspring. Then, the candidate target group is G 4 =(L 11,L 4,L 9 ) of B. Learners L 4 and L 9 belong to group G 5 in solution A with better quality than G 4 of B, so G 4 of B is not placed in the offspring. Then, the candidate target group is G 3 =(L 15,L 8,L 5 ) of B. All learners of G 3, belong to groups in A with worse quality than QG 3, so G 3 of solution B is placed in the offspring. Then, the candidate target group is G 2 =(L 13,L 7,L 1 ) of B. All learners of G 2 belong to groups in A with worse quality than QG 2 of B, so G 2 of B is placed in the offspring. The last candidate target group is G 1 =(L 2, L 14,L 3 ) of B. Learner L 3 belongs to group G 5 in A which has better quality than QG 1 of B, so group G 1 of B is not placed in the offspring. So far, the offspring consists only of groups from solution B. The G 5 in solution A was not influenced in the previous procedure, so it is added to the offspring. The remained learners L 11, L 2, and L 14 belong to groups in A that were influenced in the previous procedure, so they are added in the offspring according to their order in solution A and form the 5 th group of the offspring. The first offspring produced after the crossover operation is depicted in Table 2. Table 1. Example of crossover operation. Solution A Solution B G i L i A 1 QG i G i L i A 1 QG i Table 2. The first offspring produced after the crossover operation. G i L i A The OmadoGenesis Tool The development of the OmadoGenesis Tool was inspired by our research work in the context of the SCALE and the PECASSE environments [4],[5] in order to support the group formation process. The characteristics kept in the learner model of these environments (i.e. the learner id and the values X 1, X 2, X n of the attributes A 1, A 2, A n ) constitute the main source of the OmadoGenesis tool. The instructor can select the learners s/he wishes as well as the attributes to be taken into account. In the following, the instructor can assign to each of the selected attributes whether s/he prefers homogeneity or heterogeneity to be applied. In addition, the instructor can define the condition to be hold for the determination of the moderator in each group (e.g. the 38

39 moderator should have value X i > 4 in attribute A i ). Finally, s/he sets the number of the members per group. The tool, taking into account the parameters set by the instructor, proposes the most suitable algorithm to be used. That is, for the creation of pure homogeneous or heterogeneous groups proposes the Homo-A or Hete-A algorithm respectively while in case of mixed groups the tool proposes the GA. However, in the cases of homogeneous or heterogeneous groups the instructor may also select the GA. Moreover, if the instructor wishes may ignore the available algorithms and select the random construction of the groups. Upon the setting of the above attributes, the instructor may proceed to the setting of the algorithm s attributes (e.g. Number of Generations and Population for GA, ideal distance for Hete-A). After the execution of the selected algorithm, the results of the group formation are presented and the groups are denoted in alternating colors helping the instructor to identify easily the members of each group. The instructor may intervene in the results and proceed to any re-arrangements in case s/he believes that a better result can be achieved or to avoid any problems during the collaboration. Fig. 2. A screen shot of the OmadoGenesis Tool Fig. 2 presents a screen shot of the tool. The instructor has selected to form groups consisting of three learners and two attributes to be used for heterogeneity. The tool proposed the Hete-A algorithm, and the screen shot presents the results after the execution of this algorithm. The moderator of each group has also been specified. 5 Preliminary Results & Future Work The application of the algorithms with real data reveals that good solutions can be produced, that is the quality of the solutions approximates the highest value of the quality. For example, in case that 52 learners have to be grouped in groups of 4 members and two attributes A 1 and A 2 are used for the group formation. The application of the three algorithms gives the results presented in Table 3. Considering that the highest value of quality is 101 (QS= (max quality for Attribute A 1 + max quality for Attribute A 2 ) * num_of_groups= (4+4)*13=101), the produced solutions can be considered good enough. It is worthwhile mentioning that in the formation of pure homogeneous groups, the GA and the Homo-A seem to have almost the same performance while in case of heterogeneity, the Hete-A seems to produce a better solution than the GA. Also, the application of the GA for the formation of mixed groups gives a quite good solution with high quality. 39

40 Table 3. The quality of the solutions produced by the three algorithms Homogeneity in both attributes GA Homo-A Quality (QS) Heterogeneity in both attributes GA Hete-A Quality (QS) Homogeneity in A1 and Heterogeneity in A2 GA Quality (QS) 96 Despite the preliminary positive results, further experiments need to be carried out and examine the quality solutions with respect to the variation of the values of the attributes taken into account for the group formation. Moreover, the support of an instructor s profile facility is in our plans; the parameters set by the instructor will be kept in his/her own profile and retrieved and made available each time s/he uses the tool. Also, in the direction of helping instructors to have the most qualitative results, we plan to investigate whether results about the effectiveness of each algorithm with respect to the data used could be drawn, so that the tool proposes the most suitable algorithm and result in a group formation process with minimum or no manual intervention. Finally, experiments with real data and the participation of instructors are considered valuable in order to elicit instructors point of view regarding the usability of the tool, the effectiveness (i.e. quality) of the produced results and the degree of easiness in intervening in the results and making the desired re-arrangements. The results of this research work may help instructors and researchers in the field of collaborative education. The experimentation with various personal and social characteristics in forming groups may give an insight to factors affecting students interaction as well as students performance in different learning situations. References 1. Bradley, J. H., Herbert, F., J.: The effect of personality type on team performance. Journal of Management Development 16 (1997) Daradoumis T., Guitert M., Giménez, F., Marquès, J., Lloret, T.: Supporting the Composition of Effective Virtual Groups for Collaborative Learning. In Proceedings of the International Conference on Computers in Education (ICCE 2002). IEEE Computer Society Press (2002) Dillenbourg, P.: What do you mean by collaborative learning?. In: Dillenbourg P. (eds): Collaborativelearning: Cognitive and Computational Approaches. Oxford: Elsevier (1999) Gogoulou, A., Gouli, E., Grigoriadou, M., Samarakou, M., Chinou, D.: A web-based educational setting supporting individualized learning, collaborative learning and assessment. Educational Technology & Society Journal (2007) (to appear) 5. Gouli, E., Gogoulou, A., Grigoriadou, M.: Supporting Self-, Peer- and Collaborative-Assessment in E- Learning: the case of the PECASSE environment. Journal of Interactive Learning Research (2007) (to appear) 6. Graf, S., Bekele, R.: Forming Heterogeneous Groups for Intelligent Collaborative Learning Systems with Ant Colony Optimization. In Proceedings of the 8 th International Conference on Intelligent Tutoring Systems (ITS 2006), Lecture Notes in Computer Science, Volume 4053/2006. Springer Berlin / Heidelberg (2006) Inaba, A., Supnithi, T., Ikeda, M., Mizoguchi, R., Toyoda, J.: How Can We Form Effective Collaborative Learning Groups?. In Proceedings of the 5 th International Conference on Intelligent Tutoring Systems (ITS 2000), Lecture Notes in Computer Science, Volume Springer-Verlag London (2000) Martin, E., Paredes, P.: Using learning styles for dynamic group formation in adaptive collaborative hypermedia systems. In Proceedings of the First International Workshop on Adaptive Hypermedia and Collaborative Web-based Systems (AHCW 2004) (2004) available at ~rcarro/ahcw04/martinparedes.pdf 9. Nijstad, B. A., De Dreu, C.: Creativity and Group innovation. Applied Psychology 51(3) (2002) Yang, F., Han, P., Shen, R., Kraemer, B., Fan, X.: Cooperative Learning in Self-Organizing E-Learner Communities Based on a Multi-Agents Mechanism. In AI 2003: Advances in Artificial Intelligence, Lecture Notes in Computer Science, Volume 2903/2003. Springer Berlin / Heidelberg (2003)

41 Investigating Individual-to-Group and Group-to-Individual Influences Kyparissia Papanikolaou 1, Evangelia Gouli 2, and Maria Grigoriadou 2 1 General Department of Education, School of Pedagogical and Technological Education, Greece 2 Department of Informatics & Telecommunications, University of Athens, Greece {spap@di.uoa.gr, lilag@di.uoa.gr, gregor@di.uoa.gr} Abstract. In this paper, we aim to investigate the impact a group may have on its members. Especially, we investigate factors that account for Group-to-Individual and Individual-to- Group influences and how these factors may be exploited to support learner and group modeling. To this end we conducted an empirical study where students worked on concept mapping tasks, individually and in groups. Based on the results of this study, we propose a set of factors that reflect the influence of (a) individuals as members of a group on group achievement and (b) the group, in terms of group characteristics and interaction, on individual achievement. These factors involve several individual and group characteristics. The way these characteristics may be used to inform the learner and group models with quantitative and qualitative data reflecting students work at individual and group level is discussed. 1 Introduction Personalization recognizes and builds upon individuals diverse strengths, interests, abilities and needs in order to foster engaged and independent learners able to reach their full potential. In this context, information about how a learner interacts with peers as a member of a group (influences and influenced by the group process) is a valuable resource involving several factors related to personal and social characteristics of the learner. Identifying such factors and their relationships is a challenging goal for personalization both at individual and group level as they affect individual and group interaction and achievement. In particular, information about learner s interaction with peers could be exploited to extend the learner and group model in order to further enhance reflection, promote learning and collaboration, support the provision of adaptive feedback at individual and/or group level. Although much research has been conducted on group effectiveness, fewer studies have investigated the impacts of groups on their members [1], [5]. In this work, we intend to extend our understanding about how the collaboration process is shaped by individuals and vice-versa. Especially, we are interesting in identifying factors that account for Group-to-Individual (G-to-I) and Individual-to-Group (I-to-G) influences. We also investigate how the factors involved may be exploited to support learner and group modeling. To this end, we conducted an empirical study where students worked on concept mapping (CM) tasks, individually and in groups. CM was adopted for two particular reasons: the expressive power of concept maps (CMaps) in externalizing the cognitive structure of learners and their appropriateness as a collaborative learning task. In particular, CM models and visualizes the way the human mind organizes knowledge [8], facilitating the assessment of specific outcomes, which is a critical issue in our research. It uses a simple formal convention: nodes, links and labels on the links, organized in a structure (hierarchical or non-hierarchical) to reflect the central concept of the map; nodes represent concepts, and links, annotated with labels, represent relationships between concepts [4]. The triple Concept-Relationship-Concept constitutes a proposition, which is the fundamental unit of the map. Furthermore, CM is considered an effective technique for investigating group interaction and influences between individual and group work, as it facilitates group negotiation of meaning and promotes a deeper mutual understanding between the members of a group [8]. 41

42 2 G-to-I and I-to-G Influence in CM Tasks: An Empirical Study In this study, we aim to identify factors that reflect the influences of G-to-I and I-to-G through a sequence of individual and collaborative CM tasks. Also, we aim to investigate the individual and group characteristics related to these factors, which may inform/extend learner and group models respectively. The following research questions guide the study: Which factors may reflect the Individual to Group and the Group to Individual influence? and Which individual and group characteristics are related to these factors and may inform the learner and group models?. Subjects. Thirty-four 4 th year undergraduate students (12 female and 22 male) enrolled in a semester-long course entitled Didactics of Informatics at the Department of Informatics and Telecommunications, University of Athens, participated in this study in order to earn extra credit toward their course grades. Students were asked to complete the 80-item Honey and Mumford Learning Styles Questionnaire (LSQ) [3] and the 44-item Index of Learning Styles (ILS) questionnaire [7] outside of class. In particular, the LSQ determines learning preferences following the Honey and Mumford (H&M) style categorisation: Activist, Reflector, Pragmatist, Theorist. Students were characterised based on their stronger preferences, i.e. a student may be characterized as activist & reflector in case s/he depicts strong preference on both styles. The ILS questionnaire was used to assess the students style according to the visual/verbal dimension of the Felder-Silverman model. In particular, we acknowledged levels of this dimension, i.e. a student may be characterized as high/average/low visualiser or verbaliser. Students were grouped in twelve groups based on their styles (ten groups of three students and two groups of two students). Procedure. The students work was organised in three phases: Phase A - Students work on their own developing their own products (individual pre products). During this phase, students were given an individual CM assignment as an in-class activity; students were quite familiar with the CM technique, as they had worked out several CM tasks during the semester. Students were asked to act as tutors preparing a Cmap (pre-map) with the COMPASS environment [2] about the central concept Computer Storage Units (a list of 22 concepts and 20 links was provided whilst students were allowed to add up to 5 new concepts) for a high school class a topic quite familiar to 4 th year undergraduate students. This map would be used as a didactical tool during the corresponding course. Phase B - Students work in groups developing a group product. Then, students organized in groups in order to jointly construct a CMap for the same central concept. In this case, students were asked to act as tutors and collaborate with their colleagues for the same target. For the group map construction, students used COMPASS and consulted their individual CMaps constructed previously. They had at their disposal the same list of concepts given in Phase A and they were allowed to add up to 5 new concepts in the group map. Phase C - Students work on their own developing their own products (individual post products). Lastly, students worked individually to construct a post-map on the same topic (using the same list of concepts, whilst they were allowed to add up to 5 new concepts). Data Collection and Analysis. Students pre, group and post CMaps were collected, analysed and compared. In particular, each student constructed two individual CMaps (one pre-map at Phase A and one post-map at Phase C) and a group CMap with the other group members in the collaborative session of Phase B (see in Fig. 1 the Pre and Post Maps constructed by Student 2 of Group 7 and the corresponding Group 7 Map). Similarities and differences among students pre and post maps and between both (pre and post) maps with the group map were identified and characterised, e.g. propositions of a pre-map that appear also in the group map were characterised as Transferred in group map, or propositions of a post-map that appear only on the pre and post maps of a student but not on the group map were characterised as Individual persistence. This way, we identified several influences of the individuals on the group product (comparing the pre map of each student with their group map) and vice-versa (comparing the group map to the pre and post maps of each student member of the group), which reflect on the different categories of propositions on the pre, group and post maps acknowledged. Moreover, we assessed students beliefs (i.e. propositions) appearing on the maps based on an assessment scheme with three different characterizations: False Belief (FB) for non-scientific beliefs, Complete & Accurate Belief (CAB) for scientific beliefs, Partial Accurate Belief (PAB) for toward-scientific beliefs i.e. beliefs that could not characterized as FB or CAB (e.g. the proposition depicted on the map is correct but a more appropriate relationship between the concepts was expected or the concept(s) of 42

43 the proposition are not placed at the correct position on the map). Based on the above analysis, quantitative data about the different types of propositions appearing on the pre, post and group maps for each student and group of students were derived (as these shown in Tables 2 and 3). Also, qualitative data about students and groups outcomes were derived since each map reflects students knowledge on the specific domain concepts represented and their relationships, e.g. a proposition on the post map that is characterised as FB and Individual persistence reflects a specific resistance to learning that may need special treatment, since the student insist on his false belief even if he didn t manage to transfer it on the group map. Results Which factors may reflect the Individual to Group and the Group to Individual influence? Based on the above data analysis, we tried to group the different types of propositions with respect to the influences they describe, and identify a set of factors (involving also individuals and group characteristics) that reflect (a) the impact that an individual may have on group achievement based on his/her influence on the group (Individual to Group Influence), and (b) the impact that the group may have on individual achievement (Group to Individual Influence). In Table 1, the proposed factors are presented. According to Table 1, the I-to-G influence is related with the impact that the initial state of the group (including group synthesis and group pre-overlap) and the degree of interaction between the individual and the group (individual-to-group impact) may have on group achievement (in terms of group outcome and group creativity). Accordingly, the G-to-I influence is related with the impact that the group in terms of its state, outcome, creativity, and interaction (group-to-individual impact) may have on individual achievement (in terms of individual outcome, individual creativity, and individual persistence). Table 1. Factors involved in I-to-G and G-to-I influence while working on concept mapping tasks 1. Individual to group influence. Factors identified: Group synthesis: (a) Individuals styles: H&M model, Felder & Silverman model (Table 2, Initial state - Style I, Style II ) (b) Individuals background: number of (FB, CAB, PAB) propositions of individuals pre maps (Table 2, Initial state, Background ) Group pre-overlap: number of identical (FB, CAB, PAB) propositions among group members pre maps which were also transferred in the group map (Table 3, Group pre-overlap ) Individual-to-Group impact: number of (FB, CAB, PAB) propositions of the individual s pre map that were transferred in the group map (Table 2, Social state (related or inspired by group work), Transferred in group (In-G) ) Group outcome: number of (FB, CAB, PAB) propositions of group map (Table 3, Group outcome ) Group creativity: new (FB, CAB, PAB) propositions of the group map, not appearing on group members pre maps (Table 3, Group creativity (G-Cr) ) 2. Group to individual influence. Factors identified: Group state: (a) Group style based on a synthesis of the individuals styles (Table 3, Group style ), (b)group overlap: (i) Group pre-overlap: number of identical (FB, CAB, PAB) propositions among group members pre maps which were also transferred in the group map, and (ii) Group post-overlap: number of identical (FB, CAB, PAB) propositions among group members post maps which also appear on group map (Table 3, Group post-overlap ) Group outcome: total number of (FB, CAB, PAB) propositions of the group map Group creativity: new (FB, CAB, PAB) propositions of group map not appearing on members pre maps Group-to-Individual impact: (a) number of new (FB, CAB, PAB) propositions inserted in post map (compared to the individual s pre map) that also appear on the group map (Table 2, Social state (related or inspired by group work), Inserted in Post (Ins-G) ), (b) number of (FB, CAB, PAB) propositions of post map that also appear on the individual s pre map but in another place, i.e. they usually substitute one or more propositions of the pre map (Table 2, Social state (related or inspired by group work), Changed in post (Ch-G) ), (c) number of (FB, CAB, PAB) propositions of the pre map not appearing on the individual s post map, i.e. rejected in post map (Table 2, Social state (related or inspired by group work), In pre not in post (Not-in-Post) ), (d) number of (FB, CAB, PAB) propositions of the individual s pre map not appearing on group map (Table 2, Social state (related or inspired by group work), In pre not transferred in group (Not-in-G) ) Individual outcome: total number of (FB, CAB, PAB) propositions of post map (Table 2, Final state, Individual outcome ) Individual creativity: number of (FB, CAB, PAB) propositions of the post map that neither appear on the individual s pre nor on the group map (Table 2, Final state, Individual Creativity (I-Cr) ) Individual persistence: number of (FB, CAB, PAB) propositions of the post map that also appear on the individual s pre map but not on the group map (Table 2, Final state, Individual Persistence (I-Pers) ) 43

44 In Fig. 1, the propositions of the three maps were characterised following the notation of Table 1 and the CAB, FB, PAB assessment characterizations. Additional characterisations were used in Fig. 1 on (a) group map: In x map(s) y post map(s), proposition appearing on x pre maps and y post maps, (b) post map: In-Pre, proposition appearing also on the individual s pre map. Which individual and group characteristics are related to these factors and may inform the learner and group models? The factors, presented in Table 1, involve several characteristics of the individuals consisting a group, of the group as an entity as well as of the group interaction during a collaborative task. These characteristics were separated in individual and group characteristics and organised as depicted in Tables 2 and 3 respectively, aiming to become meaningful for learner and group modelling purposes. In Tables 2 and 3 (see Table 1 for the notation used in the column titles), we present the data analysis results for the CMaps of three groups, Group 6, 7, and 12. More specifically, Table 2 presents data for the 9 individuals participating in these groups and Table 3 presents data for the three groups. We selected the particular groups as representative of different types of groupings (see Table 3, Group synthesis), that is Group 6 is considered as heterogeneous, Group 7 as homogenous and Group 12 as (average homogenous). The specific characteristics included in Table 2 depict the state of the individuals through the three phases, whilst those of Table 3 depict the state of the group as an entity before the whole procedure and its achievement after collaboration. These characteristics may be used to inform the learner and group models with quantitative and qualitative data reflecting students work at individual and group level. In more details, Table 2 consists of three layers that correspond to the three phases of the procedure: (a) Initial state includes the initial student characteristics as these are evaluated through specific questionnaires (i.e. style) and the task (i.e. background knowledge) at phase A, (b) Social state (related or inspired by group work) includes student characteristics derived by the collaboration process at phase B, (c) Final state includes those characteristics identified through the individual work of the student at phase C. In a learner model, quantitative (as those presented in Table 2) and qualitative data may be represented. In particular, the qualitative data will represent a student s initial state and achievement on the specific goals of the task and the domain model (assessing pre and post maps including concepts and their interrelations) such as prior knowledge (e.g. Section Initial state Background ), misunderstandings/deficiencies (based on the assessment scheme adopted), and knowledge/skill development through the three phases (e.g. Section Social state (related or inspired by group work) Changed in post & Inserted in post, Section Final state Individual creativity & Individual persistence ). The group characteristics, presented in Table 3, are derived through the three phases, such as the group state (group style and group overlap) and group achievement (group outcome and group creativity). In particular, the group style is characterised based on the level of homogeneity or heterogeneity of the group according to specific style categorisations. The group overlap refers to the shared knowledge of the individuals before (pre-overlap) and after (post-overlap) collaboration. Specially, the group post-overlap may reflect strong influences of the group adopted by all members and could be further exploited in case the particular individuals work together in the future. Discussing the I-to-G and G-to-I influences and the factors identified, as well as the learner/group characteristics involved Regarding the factors involved in I-to-G influence of Table 1, data derived from analysing the individual and group maps showed that the students background, style, overlap, may affect group interaction and achievement. The particular factors are reflected in (a) the learner characteristics of the Initial state section as well as the Transferred in group characteristic of the Social state section of Table 2, and (b) the group characteristics of Table 3. For example, the student that mainly influenced the Group 7 outcome is student G7.2. This is obvious if we compare the Background (Table 2, section Initial state ) of the students of Group 7 with their Transferred in group characteristic. Although student G7.1 has the highest background (in terms of the total number of concepts on his pre-map), student G7.2, seems to have mostly influenced the group map as the twenty of the twenty five propositions of the group map (including eightteen CAB, one FB and one PAB) appear also on the student s G.7.2 pre-map. Thus, the factor Individual to-group impact (linked with the Transferred in group (In-G) learner characteristic) provides evidence about the importance of the influence of each individual on the group map. Additionally, comparing the background of the three students presented in Table 2, it is obvious that the 44

45 knowledge level of student G7.2 is high, while the knowledge level of the other two students, G.7.1 and G.7.3, is quite low (student G7.2 has the most CAB beliefs on her pre-map whilst G.7.1 has the maximum number of propositions on his pre-map but with most of them characterized as FB and PAB). Moreover, the Group 7 overlap is quite low (see Table 3). Thus, in Group 7 the student with the highest knowledge level seems to influence mostly the group product (she managed to transfer also a FB to the group map). On the contrary, group maps of Groups 6 and 12 (both have higher group overlap than Group 7 and consist of students with similar background) were mainly influenced by each of the students of the group (only student G12.1 had an average impact on the group map although a high knowledge background and individual outcome). Thus, the group synthesis in terms of knowledge background and overlap seem to be important factors influencing group interaction. Moreover, Group 6 seems to have better results on group outcome and creativity compared to the other two groups. The particular group has high group pre-overlap (see Table 3, Group overlap ) and is an heterogeneous group based on the styles involved (see Table 3, Group Style ) but with students of high knowledge level (see Table 2, Background ). However, the impact that the group synthesis (in terms of the individuals characteristics) may have on group interaction and achievement is a challenging goal that needs further investigation. Results of a preliminary research on the effects of group synthesis on the group outcome in a collaborative setting, are presented in [6]. Regarding the G-to-I influence factors of Table 1, these reflect to the student characteristics Group-to-post map influence and In pre not transferred in group (both link to the Group-toindividual impact factor see Table 2, Social state section), and those of the Final state section of Table 2. In general, all the students seem to have increased their CAB, reduce their FB and PAB after the collaborative task at Phase B. This results by comparing students background with their final outcomes (Table 2, Section Final state, Individual outcome ). Moreover, students knowledge/skills development or deficiencies are reflected in the student characteristics Groupto-post map influence ( Inserted in post, Changed in post, In pre not in post ) and In pre not transferred in group. In particular, comparing students post maps with their pre maps and the group map, we derive specific quantitative data as well as qualitative data about students skills and knowledge on the particular domain concepts. For example, the post map of student G7.2 differs from her pre map in several places: four additional CAB propositions appear (reflected in the Inserted in post characteristic), one CAB proposition substitutes two PAB propositions of her pre map (reflected in the Changed in post characteristic), and two CAB propositions of her premap were rejected (reflected in the In pre not in post characteristic). Concerning the qualitative data resulted, the reader may notice on Student G7.2-post map depicted in Fig. 1 the following: (i) the propositions Hard disk may be accessed Randomly and Tape may be accessed Serial are annotated as Ins-G, as they do not appear on student s pre-map but they both appear on the group map. So, the insertion of these propositions in the post-map of Student G7.2 is considered as a positive influence of the group on the individual, (ii) the proposition Secondary storage units have lower than the main memory Data access speed is annotated as Ch-G, i.e. changed at postmap based on group influence. Actually this proposition appears also on the group map and seems to substitute two different propositions that appear on her pre-map, that is Magnetical storage units have lower than the main memory Data access speed and Optical storage units have lower than the main memory Data access speed. Thus, we suppose that this student, due to group influence, managed to go from the specific cases of storage units (Magnetical and Optical) to the general rule about the speed characteristic of the Secondary storage units using inductive reasoning skills, and (iii) the proposition Format partitions the unit in Sectors of her pre-map, although it is characterized as CAB, it neither appears on the group nor on her post map, i.e. rejected due to group influence. A particular interesting issue to further investigate is that students after a collaborative session seem to evolve their beliefs or some times abandon even the complete and accurate ones or change them to false ones. This is obvious, if we compare the characteristics In pre not transferred in group with Individual persistence of Table 2. For example, student G.12.1 did not manage to transfer 8CAB out of his 16CAB to the group map, and he didn t also put them in her post map as her Individual persistence is zero. Moreover, the characteristics Inserted in post and Changed in post reflect the group influence on the individuals outcome. These characteristics show that although in most cases there is a positive influence making students correct their FB, there are also some cases where students insert FB in their post maps (e.g. students of Group 6 inserted one FB in their post maps based on the group influence - see Table 2, Social state, Inserted in post ) or 45

46 replace correct and accurate beliefs with false ones (e.g. student G12.2 replaced a CAB of his premap with a PAB in the post map based on group influence - see Table 2, Social state, Changed in post ). Group creativity, as a group characteristic, seems to influence individuals outcome. For example, Group 6 showed increased group creativity compared to the other groups. But the FB inserted in the map of Group 6 due to group creativity (see Table 3 - Group Model, Group creativity ), appears also in the post overlap characteristic of Group 6, i.e. all three students adopted the particular FB and inserted it in their post maps. Moreover, the students of Group 6 do not have developed individual creativity as only one student (G.6.1) inserted a totally new concept in his post map (see Table 2, Final state, Individual creativity ). Actually, the investigation of relationships among individual and group creativity with group synthesis and individual characteristics is a challenging research goal. Individual persistence relates to the Group-to-Individual influence, as it denotes that a student insists on his/her beliefs that s/he didn t manage to transfer to the group product. The particular factor may also reflect students progress, resistances to learning as well as interaction behaviour. For example, student G7.1 insists on five beliefs appearing on his pre- and post-maps but not on the group map. Actually, he insists on 1FB and 4PAR instead of 8FB and 15PAR that initially appeared on his pre map, i.e. although some improvement is obvious, the particular student insists on a false belief. In another case, students of Group 12 do not show Individual persistence although they didn t manage to transfer all their beliefs to the group map (see Table 2, Section Social state, In pre not transferred in group ). This means that the particular students abandoned those beliefs of their pre maps that they didn t manage to transfer to the group map. Summarizing, interesting directions for personalization are to investigate how learner and group models extended with the above information may be exploited to support collaboration (e.g. promote student participation in case of a student with high background knowledge and low influence on the group product), adaptation (e.g. support students in adopting CAB beliefs of the group map or acknowledging their CAB beliefs that they rejected on their post maps), feedback provision, or enhance reflection (e.g. inform learners about their evolution through the procedure). Table 2. Individuals characteristics that reflect students state through the three phases: data for students of Groups 6, 7, 8 are depicted. Note that Gx.y: Student y of Group x, Act: Activist, Refl: Reflector Individual Characteristics Initial state StyleI StyleII Background CAB FB PAB TOTAL G6.1: Activist High Visualiser G6.2: Reflector Low Visualiser G6.3: Theorist High Visualiser G7.1: Act&Refl Low Visualiser G7.2: Act&Refl Low Verbaliser G7.3: Act&Refl Low Visualiser G12.1: Activist High Visualiser G12.2:Activist Low Verbaliser G12.3:Activist High Visualiser social state (related or inspired by group work) Individual-togroup impact Transferred in group (In-G) G6.1: 18(CAB) G6.2:15CAB G6.3: 22CAB Group-to-individual impact Group-to-post map influence Inserted in post (Ins-G) Changed in post (Ch-G) 4CAB 1FB 5CAB in post replace 2FB of pre-map 3CAB 1FB 1CAB in post replace 1FB 2PAB of pre-map 6CAB 1FB 1PAB in post replace 1PAB of pre-map In pre not in post (Not-in-post) 1CAB 2CAB 3FB 2PAB 1CAB 2PAB In pre not transferred in group (Not-in-G) 4CAB 2FB 2PAB 3CAB 4FB 2PAB 2CAB 3PAB 46

47 G7.1: 9CAB G7.2: 18CAB 1FB 1PAB G7.3: 5CAB G12.1: 8CAB G12.2: 19CAB 3CAB 4CAB 7CAB 9CAB 5CAB 1CAB in post replace 5PAB of pre 2CAB in post replace 1FB of pre 1CAB of post replace 2PAB of pre map 4CAB in post replace 2FB & 3PAB of pre map 1CAB in post replace 1PAB of pre-map 1CAB of post replace 1FB of pre-map 1PAB of post replace 1CAB of pre-map 4FB 8PAB 2CAB 3CAB 4FB 2PAB 7CAB 2PAB 2CAB 4PAB 2CAB 8FB 15PAB 3CAB 2PAB 3CAB 6FB 5PAB 8CAB 3PAB 1CAB 1FB 4PAB G12.3: 17CAB 6CAB3PAB CAB 4FB 2CAB 4FB Final State Individual outcome Individual creativity (I-Cr) Individual persistence (I-Pers) CAB FB PAB TOTAL G6.1: 1CAB 2CAB G6.2: 0 1CAB 1FB G6.3: G7.1: 2CAB 1FB 1PAB 1FB 4PAB G7.2: 1CAB 1CAB G7.3: 1CAB 1FB 4PAB G12.1: 2CAB 2PAB G12.2: 1CAB 1FB G12.3: 6CAB Table 3. Group characteristics of Groups 6, 7, 12. Note: Act&Refl: a student with strong preferences on both Activist and Reflector styles, Theor:Theorist, Vis/Verb:a group that consists of Visualiser(s) & Verbaliser(s) Group Characteristics Group state Group achievement Group Style Group overlap Group outcome Group creativity Pre-overlap Post-overlap CAB FB PAB TO G6: Heterogeneous 4CAB 1FB (Act / Refl / Theor, 11CAB 20CAB 1FB PAB High/Low Visualiser) G7: Homogenous (Act & Refl, Low Vis/Verb) 4CAB 13CAB CAB G12: Middle Homogenous (Activist, High Vis/Low Verb) 6CAB 15CAB CAB References 1. Brodbeck, F. C., Greitemeyer, T.: Effects of individual versus mixed individual and group experience in rule induction on group member learning and group performance. Journal of Experimental Social Psychology, 36 (2000) Gouli, E., Gogoulou, A., Papanikolaou, K., and Grigoriadou, M.: COMPASS: An Adaptive Web-Based Concept Map Assessment Tool, in Proc. of the 1 st Int. Conf. on Concept Mapping, Spain (2004) 3. Honey, P. & Mumford, A.: The manual of learning styles, Maidenhead: Peter Honey Publ. (1992) 4. Novak, J., Gowin, D.: Learning How to Learn, New York: Cambridge University Press (1984) 5. Olivera, F. and Straus, S.G.: Group-to-Individual Transfer of Learning: Cognitive and Social Factors. Small Group Research (2004) DOI: / Papanikolaou, K., Gouli, E., Grigoriadou, M.: Group Formation For Collaborative Concept Mapping. 2nd International Conference on Concept Mapping (CMC2006), Costa Rica, 5-8 September (2006) 7. Soloman, B.A., Felder, R.M.: Index of Learning Styles Questionnaire (1999). Available: 8. Stoyanova, N., Kommers, P.: Concept mapping as a medium of shared cognition in computer supported collaborative problem solving, Journal of Interactive Learning Research, 13(1/2) (2002)

48 Fig. 1. Pre and post maps of student 2 of Group 7 and the Group 7 map. 48

49 Adaptation of Feedback in e-learning System at Individual and Group Level Ekaterina Vasilyeva 1,2, Mykola Pechenizkiy 2, and Paul De Bra 2 1 Department of Computer Science and Information Systems, University of Jyväskylä, P.O. Box 35, Jyväskylä, Finland 2 Department of Computer Science, Eindhoven University of Technology, P.O. Box 513, 5600 MB Eindhoven, the Netherlands ekvasily@cc.jyu.fi, m.pechenizkiy@tue.nl, debra@win.tue.nl Abstract. This paper discusses the issue of feedback personalization and adaptation in e- learning systems, and distinguishes between adaptation to the individual user and to groups of users. We briefly review the scope of feedback use in e-learning systems and outline the necessity of feedback adaptation in an e-learning system. We are trying to answer the following question: what is the difference between personalization of feedback at the individual and the group level. Particularly, we are focusing on the analysis of the differences in user modeling for feedback personalization/adaptation in e-learning systems. We discuss what kind of characteristics could be included in the learner model for individual and group personalization. The main purpose of this paper is to attract the attention of the research community to the problem of the feedback personalization and to be helpful in further studies and implementation of individual and group personalization of feedback in e-learning systems. We present some recent feedback adaptation experiments that try to determine the usefulness of certain types of adaptation. 1 Introduction Feedback is an important part of the learning and interaction in e-learning systems. In this paper we consider feedback as information the user receives from the system as the result of his/her action. Thus, feedback in e-learning is the mechanism that tends to replace the teacher who provides comments, advice, and explanations and evaluates the students in traditional learning environments. In general, the feedback in e-learning occurs not only in the assessment process, but can be provided to a student during navigation through learning materials, communication and collaboration with other students, in the process of work with personal information and managing the courses (planning, enrolling, completing), etc. Even the alerts and reminders that often appear in the e-learning system can be considered as feedback. However, in this paper we focus on analysis of feedback that the user receives during the assessment in an e-learning application. According to [13] the feedback mechanisms that are used by students have changed with the advances and growth of web-based learning systems. Bischoff [3] argued that students need regular feedback in order to know how their performance was evaluated, and how they can improve it, and also how their grades are calculated. The effective elements of online teaching include frequent and consistent online feedback, diplomatic online feedback, and evaluative online feedback [3]. It was suggested in [13] that feedback in a web-based learning system should have the following qualities: (i) prompt, timely, and thorough online feedback; (ii) ongoing formative feedback about online group discussions; (iii) ongoing summative feedback about grades; (iv) constructive, supportive, and substantive online feedback; (v) specific, objective, and individual online feedback; and, (vi) consistent online feedback. The following problems with feedback design in e-learning systems can be outlined: (i) feedback representation (what should be included into feedback and what kind of structure should it have); (ii) time of feedback presentation (either immediate or delayed feedback); (iii) distraction of students from the learning by feedback. In [16] we argued that the problems of feedback listed above could be partially solved by adaptation of feedback to the tasks and to the characteristics of an individual user or the group of 49

50 users. In [10] the authors also emphasized the necessity of feedback personalization in e-learning systems and proposed adaptive feedback framework. In this paper we distinguish between feedback adaptation in e-learning systems from the perspective of individual and/or group adaptation. Individual adaptation means that feedback is adapted to each student and his/her individual (combination of) characteristics. For example, the individual characteristics could include the user s knowledge of the subject being studied (knowledge of the main concepts, formulas, etc) and the number of mistakes the user makes during the testing. The time and the way of feedback presentation could be personalized to these individual characteristics. For example, if the user has started to make some mistakes more often the system can present the feedback more often and include more detailed explanations in the feedback (compared to the feedback given to a user who only occasionally makes a mistake). The information that is presented in the feedback can be also personalized by relating to concepts that are already mastered by the user. Group adaptation supposes that the system adapts the feedback to common characteristics of a group of users. For example, they can be grouped according to their learning style. Immediate feedback could be presented in a brief form for active users, while detailed elaborated feedback could be presented to reflective learners. Another example of the group adaptation of feedback is personalization of the feedback to students who have passed the same courses before. The feedback could include references to the previous course (if the student has passed it) or the detailed explanation (in the case the material might be unknown to the user as s/he has not taken the course). The current paper analyzes what and how can be adapted in feedback of an e-learning system at the individual and group levels. The rest of the paper is organized as follows. In Section 2 we analyze the specific aspects of feedback personalization to the individual users in e-learning systems. Section 3 discusses the issue of the group adaptation of feedback. Section 4 summarizes the differences between individual and group personalization of feedback in e-learning system. In Section 5 we describe recently completed experiments. In one experiment we used feedback to multiple-choice tests in the Moodle learning system, and in another experiment we are analyzing the suitability of different types of feedback for field-dependent and field-independent users, using the AHA! system [7]. We briefly conclude with the summary of the paper and outline the directions of the further work in Section 6. 2 Individual Feedback Adaptation in e-learning System In this section we discuss individual feedback adaptation in e-learning systems, by analyzing the following main questions: (1) to which individual characteristics of the user (learner model parameters) feedback can be adapted, (2) how those learner model characteristics could be acquired by the system, (3) what can be adapted to the individual users in the feedback of e- learning systems?. Adaptation to the user s individual characteristics is traditionally organized on the basis of a user model [11, 2]. A user model determines the user s goals, tasks, beliefs, and characteristics, which are important for adaptation [11]. In the adaptive e-learning systems a user model is traditionally called student or learner model (or profile). According to Brusilovsky [4] a hypermedia application can be adapted to the following characteristics of the user: knowledge, goals, background and experience, preferences, interests, individual traits and environment. Besides the parameters, which are traditionally included into the user (learner) model that are listed above, the following groups of individual user characteristics could be important for the adaptation of e-learning systems: personal data (including demographic characteristics such age, data, culture); psychological, cognitive and physiological parameters such as user s attention (simple and complex reaction time), memory (verbal working memory, longterm memory), cognitive abilities (spatial arrangement, etc.), user s internality/externality, cognitive and learning styles, and personal decision abilities. Recent investigations of these features have shown that the cognitive characteristics have an appreciable impact on hypermedia and web-systems adaptation (for example, [6]). However, we defer their discussion to the next 50

51 section as cognitive styles, as well as user preferences, lead to a stereotypical form of adaptation that we could consider a form of group adaptation. We will try to outline the characteristics that can be important for individual feedback adaptation in e-learning system: 1) Personal data Personal data typically includes parameters such as age, gender, etc. that lead to grouping users and performing stereotype adaptation. The only real individual use of personal data is to give a personal touch to the application, for instance by including the user s name in the feedback. (Example: Sorry, this answer is incorrect, Paul. You should revisit ) 2) Knowledge Of course the user s answers to a test provide information about the user s knowledge, but individual feedback adaptation means that other parts of the user s knowledge play a role in the feedback the learner receives. The feedback to a (wrong) answer can be more informative when it refers to knowledge the learner already has, perhaps about related topics. We plan to perform an experiment to verify the usefulness of this type of adaptation. 3) Interaction Parameters For the purposes of feedback adaptation the following interaction parameters, grouped according the taxonomy suggested in [15], can be taken into consideration: (i) knowledge data (discussed above); (ii) chronometric data (time spent viewing pages with learning materials, time spent for passing the question in the tests and the total time spent on the assignment, the time of idle intervals); (iii) try data (the number of attempts to pass the tests or assignment, the number of times needed to give the correct answer for the certain question); and, (iv) navigation data (visited links and pages, number of visits, the frequency that specific selections have been made). The way and form of feedback presentation can be adapted to the listed characteristics. For example, feedback could be presented more frequently for the users who have started to make more mistakes, and feedback can be delayed to slow down students who are answering too quickly and sloppily. The user s features which are important for individual feedback adaptation can be collected in several ways depending on the nature of the e-learning system. First, they can be collected using separate tasks (for example, small test to evaluate the user s primary knowledge and interaction skills) or they can be derived from the performance of the user in the actual e-learning task. Secondly, the system can use some general prototypical or stereotype user profile or previous information of the user s performance as a starting point and after that obtain more accurate information about the user and gradually move from stereotypical feedback to more individually personalized feedback. True individual feedback based on an individual s evolving characteristics like knowledge, chronometric, try and navigation data requires a tight coupling of the part of the adaptive system that provides learning material and the part that provides feedback. We are planning an experiment using a small extension to the AHA! system [7] to apply the same form of content adaptation (using conditionally included fragments) to feedback as it is done in a course text. 3 Personalization of Feedback to a Group of Users or a Stereotype Adaptation to a group of users is traditionally performed on the basis of a group (or stereotype) model. The main purpose of stereotype modeling is to model a group of users in order to adapt to them as a group of users, not as individuals 3. This approach has been proven to be very useful for application areas in which a quick but not necessarily completely accurate assessment of the user s background knowledge is required. It is one of the first mechanisms that have been used in adaptation of recommender systems [14]. According to Lock [12] stereotype and group modeling are not synonymous. The user is 3 Sometimes in the literature it is possible to find another meaning of the group user modeling and group personalization: group modeling is used for the collaborative environments in which group members must work together to reach joint decisions or, in entertainment settings, in which media are enjoyed simultaneously by a group of users. So the system is adapted to the group of users who are performing the collaborative task together. 51

52 considered to belong to a single group in group modeling adaptive systems, whereas in stereotypebased modeling applications a user belongs to multiple groups [12]. In this paper we are analyzing the group adaptation in the context where the user is assigned to one or several groups and the adaptation is performed according the characteristics of those groups (and thus we do not emphasize the difference between group and stereotype adaptation). For example, we can have in the system the following two group (stereotype) models: (i) the first one characterizes the user s knowledge Kn by the courses s/he has already passed and classifies the user into the several groups (A students that have passed course A, B students that have passed course B, C students that have passed both courses A and B); (ii) the second one defines the user cognitive style according to the Field Dependent/Field Independent Scale (FD/FI) [18]. Thus the student who has passed the course B and has field independent cognitive style could be assigned to the two groups Ui {B, FI}. The adaptation should be organized according to those two groups. Typically the different partitionings in groups lead to independent (or orthogonal) kinds of adaptation. Having taken course A previously can be taken into account to choose between explanations, whereas adaptation to FD/FI learning styles can be done independently from that (by changing recommended navigation paths for instance). In case there are conflicts the system should have a weighting scheme, where the priority of adaptation should be given to the each of the stereotype models. In this section we will analyze the possibilities of feedback adaptation in e-learning systems to group user models by answering to the questions, similar to the questions from Section 3: (1) according to which characteristics could the students be grouped for adaptation of feedback in e- learning system, and (2) what can be adapted to a group model in feedback of e-learning systems? We suggest that the users should be divided to the stereotype groups according long-term characteristics (characteristics that do not frequently change). (Shorter-term, more detailed and individual changes in characteristics lead to individual adaptation, not group adaptation.) The following groups of parameters or what can be adapted to can be outlined: 1) Personal data (Demographical Characteristics) Personal data include parameters that are relatively stable, such as the user s age, gender, time zone, language and cultural properties. They lead to a clearly defined partitioning of the whole user population. Age is important in the e-learning applications, to adapt differently to small children, grade school, high school or adults. For example, an e-learning system can be used in a school, where for the smallest children feedback should include entertainment and motivational components. Informational aspects of feedback are more important for the elder student. Audio feedback could only annoy and distract the elder students, while it could be quite positive for the young children, especially if they do not have enough reading skills. The gender of the user can influence to the adaptation process as males and females differ in terms of navigation support, attitudes, information seeking strategies and media preferences [1, 8]. 2) Professional Among the professional characteristics of the user the following are especially important for the feedback adaptation: the user s skills and experience (mainly in interaction with the e- learning system: familiarity with its structure, ease of navigation within it) and the user s background (his/her knowledge of the subject that s/he is studying and of related subjects). Novice users should receive the maximal guidance and support via feedback while this is not required for expert users. Professional experience can also be used: when the user has a certain job for a considerable time this also leads to experience, perhaps equivalent to having taken certain courses although the courses do not appear in the user s learning history. 3) Psychological and cognitive The following characteristics that are important for the feedback adaptation can be outlined among this group: (i) the user s attention (reaction time, types of errors, and omitted contents). The feedback as well as its timing could be adapted to the changes in the user s attention and help the user to focus on the task. (Like with age, reaction times and errors can be partitioned so as to obtain a few groups of users rather than considering all individual values.); (ii) the user s memory (verbal, visual memory); (iii) the user s cognitive abilities (intelligence, educational level, verbal or spatial skills, etc. as estimates of these abilities); and (iv) the user s cognitive and learning styles. The knowledge of the learning styles allows 52

53 determining the best learning strategies that can be used for the certain user. Thus, the feedback could be presented in the way that facilitates the learning. 4) Physiological This group of characteristics could include personal abilities/disabilities. For the feedback vision and hearing characteristics of the user could be particularly important. The system could provide audio feedback for visually impaired users and vice versa verbal and visual feedback for hearing impaired students. 5) Environment As e-learning applications become ubiquitous, adaptation of feedback to environmental settings should be provided to the users. The system can adapt feedback to the user s access platform. For example, student can use mobile phone to pass the test in the e-learning system and receive the feedback that can be presented by phone. The network connection (bandwidth limitation) should be also taken into consideration. 6) User Preferences This group of characteristics includes the parameters that characterize the user s personal preferences, interests, goals, habits and mood. Users may prefer some links or parts of the pages over others and this can influence the adaptation of the feedback in e-learning. For example, the user can prefer to receive feedback in a pop-up window. It is unlikely that feedback adaptation systems will be developed in the near future that can take all the above mentioned characteristics into account. We have to be realistic, especially considering that the feedback to answers on tests as part of a course is a relatively small part of the entire interaction between the learner and the system. So instead of looking at all these possible parameters that influence the adaptation we can look at the taxonomy for what can be adapted, as suggested in [16]. The feedback in e-learning systems can vary according to its complexity (nofeedback, knowledge-of-response, knowledge-of-correct-response, answer until correct, elaborated), form of presentation (textual, visual, audio, video), user s progress within a task (immediate, continuous, summative), function (confirmation, informing, correcting the user, explaining, evaluating, motivating, rewarding the user, and attracting his/her attention) and time of presentation (immediate and delayed). In the next section we analyze the advantages and disadvantages of individual versus group personalization of feedback in e-learning systems and suggest our answer to the question, in which situations it is better to use individual personalization and in which situations it is better to adapt feedback to group model in e-learning system. 4 Individual vs. Group Personalization of Feedback The main difference between individual and group personalization of feedback is in the way the user modeling and user identification are organized as well as what parameters are included into the model. Individual adaptation is performed on the basis of a detailed individual model of the user, while group personalization is performed on the basis of group (or stereotype) models to which the user is assigned according to the value of one or several parameters. This is the traditional explanation of the differences between individual and group user modeling [14]. It is also important to emphasize that the group adaptation is relatively easy to implement when based on long-term user models, when there is no need to track changes of characteristics in the shortterm (i.e. potentially rapidly changing characteristics). On the contrary, since the short-term user models include many parameters that may change even during the same interaction session, it is unrealistic to think that these parameters will change in a similar way with different individuals and therefore, the tracking of such changes at a group level is hardly possible and needs to be implemented at the individual level. The problem of tracking and handling changes in user models may become even more severe when hidden (not directly observable) contexts change over time. In Table 1 we summarize the most obvious advantages and disadvantages of the individual and group personalization of feedback in e-learning systems. From the table it is clear that the purely individual or purely group-based approaches both have shortcomings. However, a combined approach is possible that combines the advantages of both approaches: the group (stereotype) model can be used to classify new users and set initial values of the user model, whereas the 53

54 individual user model could be used while the user interacts with the system to correct and enlarge the information about the user that can be used in adaptation. The user model could include both short-term (for individual adaptation) and long-term characteristics (for group adaptation). The truly individual adaptation could become sufficiently unobtrusive that it does not hinder users discussing their experience with the e-learning application. Proc Cons Table 1. Individual vs. Group Feedback Adaptation Individual The user can receive the feedback that more exactly corresponds to her/his individual characteristics. The feedback can be dynamically adapted to the changes of the user individual characteristics and performance (attention, number of mistakes, etc.). Difficulty of implementation as the system should observe the changes of the interaction parameters and support the dynamic adaptation to these changes. Difficulties in collaborative learning. Group The implementation of group adaptation is easier than supporting the individual adaptation (especially dynamic adaptation to the changes of the user characteristics). The users will more probably receive the same feedback and could refer to it during the discussions. The mistakes in initial assignment of the user to the certain group could lead to the problem that the user experience difficulties and interaction and the feedback is distracting the user instead of helping him/her. In our opinion this combined approach is the most promising for organizing the feedback adaptation in e-learning systems. This approach can allow taking into account both short-term and long-term user preferences, needs, interests and the parameters of interaction. 5 Feedback Personalization Experiments with Moodle and AHA! In this section we will discuss a few experiments with feedback adaptation, using the Moodle e- learning system 4 and using the adaptive hypermedia system AHA! 5 [7]. The first experiment is finished and analyzed, the second experiment is completed and is being currently analyzed. We have recently used Moodle for a pilot experiment that was aimed at studying the interrelations between personal learning styles and reaction to immediate and delayed feedback presentation. The main goal of the experiment was to discover whether there was a difference in the performance of the students with various learning styles (active/reflective, global/sequential, sensitive/intuitive, visual/verbal) in the tests, where brief and detailed immediate feedback were presented. The learning styles of the users were acquired using the index of learning styles test that allows to determine the learning style (LS) according to the Felder-Silverman model [9]. Two performance parameters were evaluated: the score of the user on the test and the time used for completing the test. In this paper we are presenting several outputs of the experiment that can illustrate the stereotype feedback adaptation (see [17] for more results). The following main tendencies were discovered with respect to the active/reflective LS and brief/elaborated immediate feedback: (i) active learners performed better (with the higher score and less time) on average in the test where brief immediate feedback was presented; (ii) the performance of the reflective learners was better in the test with detailed immediate feedback; (iii) the users that were balanced between active and reflective LS performed better in the tests where brief immediate feedback was presented; and, (iv) the users with the tendency to active LS have shown the same score in the tests with brief and detailed immediate feedback. Although the obtained results are not statistically significant due to the limited number of the participants of the experiment (12 students participated in the pilot experiment), they allow to conclude that the users could be assigned to the three stereotype groups according to their active/reflective learning style. The immediate feedback should be adapted in the following way:

55 brief immediate feedback should be presented to the active learners and the learners who are well balanced between active and reflective LS, while detailed (elaborated) immediate feedback should be presented to the students with reflective LS. In the experiment with the AHA! system the possibilities of adaptation of the elaborated immediate feedback to field dependent/field independent cognitive styles [18] were analyzed. The students (that have taken the Adaptive Hypermedia course at Eindhoven University of Technology) answered to 4 tests (of 10 questions each) on the course materials in AHA! system. In the first two tests the presentation of immediate feedback alternated in the questions: the direct elaborated (detailed) feedback was provided to the even questions and the pointing to elaborate feedback was presented after submission of the answers to the odd questions. The direct elaborate feedback included the detailed explanations of the answers (why the answer was correct or incorrect) and information about the correct variant of the answer. The pointing to elaborate feedback was used to present a brief explanation of the answer together with a hyperlink to the original explanation (on slides of the presentation, where the corresponding material was discussed). The users also answered to the question about their preference of direct elaborated vs. pointing elaborated feedback after passing the test. In the second two tests the presentation of delayed (after all questions) feedback alternated in the same way: either direct elaborated feedback or pointing to elaborate feedback was presented. The users were provided with the feedback only after they had answered all 10 questions of the test. The preferences of the users were also assessed by the additional question. Field dependent users require additional reinforcements [18] and need reducing complexity of navigation. The original hypothesis for our experiment was that the field independent users would prefer and have better performance with the direct elaborated feedback, while the pointing elaborated feedback is better for field dependent students (as it provides the context they need more strongly). We are currently gathering the final data and will analyze the results before the workshop. The responses of 15 students (7 FI users, 3 with the tendency to FI, 3 mixed FD/FI and 2 with the tendency to FD cognitive style) have already been analyzed. Most of the students preferred direct elaborated immediate feedback. However two FI users, one student whose cognitive style was balanced between FD and FI and one student with the bias to be FD preferred to have feedback with the hyperlinks ( pointing elaborated feedback). In the tests where delayed elaborated feedback was presented three of seven FI students preferred the feedback with hyperlinks to the slides instead of direct elaborated feedback, while the students, whose cognitive styles were balanced between FD and FI, and the student with the bias to be FD preferred to have direct elaborated feedback. Thus, the obtained results did not demonstrate the strong relations between user s cognitive styles and their preferences of direct or pointing elaborated feedback: whether the feedback is immediate or delayed seems to influence the result. Further experimental research on the interrelations between the personal LS and the adaptable feedback parameters such as presentation of direct elaborated and pointing elaborated immediate and delayed feedback is needed. The differences found thus far are small enough to indicate that a larger experiment is necessary in order to come up with truly convincing results. 6 Discussion and Further Work Feedback that is provided to the user by the system is an integral part of e-learning. Feedback adaptation seems to be a promising aspect of the e-learning systems personalization. In this paper we have analyzed personalization of feedback from the perspective of individual and group adaptation. We discussed what characteristics are important for the individual and group adaptation. We overviewed the main parameters of the feedback that could be adapted to the user (group) model characteristics. The differences between individual and group feedback adaptation were underlined. We considered the combined group and individual feedback adaptation to be perspective for the implementation in e-learning system. In our paper we proposed a number of hypotheses about the interrelation between the user (group) model characteristics and adaptable parameters of the feedback. Those hypotheses are based on the existing knowledge about the characteristics and the results of several pilot experiments. For further research an experimental study of the interrelations between the user 55

56 (group) model characteristics and the adaptable feedback parameters is necessary. It is important to investigate what user characteristics have an influence to the adaptable feedback parameters and how do they affect interaction and learning process. The experiments should allow obtaining knowledge about the interrelations between user model characteristics and adaptable parameters of feedback that could be used by an adaptation engine of a e-learning system. In a meantime we continue the experimental research of feedback adaptation to the user (group) model in AHA! and Moodle. In the future we are planning an experiment on individual feedback adaptation and the combined (group and individual) personalization using a small extension to the AHA! system. We hope that the paper will attract the attention of the research community to the problem of the feedback personalization and will be helpful in further studies and implementation of individual and group personalization of feedback in e-learning systems. Acknowledgments. This research is partly supported by COMAS Graduate School of the University of Jyväskylä, Finland. We are thankful to the students participated in the experiments. References 1. Arroyo, I., Beck, J., Beal, C., Wing, R., and Woolf, B.: Analyzing students response to help provision in an elementary mathematics Intelligent Tutoring System. In R. Luckin (Ed.), Proc. of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) Benyon, D.R.: Accommodating Individual Differences through an Adaptive User Interface. Schneider- Hufschmidt, M. et al. (Eds.):Adaptive User Interfaces-Results and Prospects. Elsevier Science Publications (1993) 3. Bischoff, A.: The elements of effective online teaching: overcoming the barriers to success. White K.W., Weight B.H. (Eds.), The online teaching guide: A handbook of attitudes, strategies, and techniques for the virtual classroom. Boston: Allyn&Bacon (2000) Brusilovsky, P.: Adaptive hypermedia. User Modelling and User Adapted Interaction, V. 11, N. 1/2 (2001) Brusilovsky, P., Maybury, M.T.: From adaptive hypermedia to adaptive Web. Communications of the ACM, Vol. 45, N. 5, Special Issue on the Adaptive Web (2002) Chen, S., Macredie R.: Cognitive styles and hypermedia navigation: development of a learning model. Journal of the American Society for Information Science and Technology, Vol. 53, N. 1, (2002) De Bra, P., Smits, D., Stash, N.: The Design of AHA!. Proc. of the ACM Hypertext Conf., Odense, Denmark (2006) , Fan, J.P. Macrediel, R.D.: Gender Differences and Hypermedia Navigation: Principles for Adaptive Hypermedia Learning Systems. In Magoulas G.D. & Chen. S.Y. (Eds.) Advances in Web-based Education: Personalized Learning Environments, Idea Group Inc (2006) 9. Felder, R.M. Silverman, R.M.: Learning and teaching styles in engineering education. Journal of Engineering Education, Vol. 78, N. 7 (1988) Gouli, E., Gogoulou, A., Papanikolaou, K., & Grigoriadou, M.: An Adaptive Feedback Framework to Support Reflection, Guiding and Tutoring. In Magoulas G.D. & Chen. S.Y. (Eds.) Advances in Webbased Education: Personalized Learning Environments, Idea Group Inc Kobsa, A.: User modelling: Recent work, prospects and hazards. Schneider-Hufschmidt et al. (Eds.) Adaptive User Interfaces: Principle and Practice, Elsevier (1993) 12. Lock, Z.: Performance and Flexibility of Stereotype-based User Models. PhD Thesis, Univ. of York, UK (2005) 13. Mory, E.H.: Feedback Research Revisited, Jonassen J.H. (Eds.) Handbook of Research on Educational Communications and Technology. New York: MacMillian Library Reference (2003) Rich, E.: Users are individuals: individualizing user models. J. of Man-Machine Studies, Vol. 18 (1983) Stathacopoulou, R., Magoulas, G.D., Grigoriadou, M. Samarakou, M.: Neurofuzzy knowledge processing in intelligent learning environments for improved student diagnosis. Information Sciences 170 (2-4) (2005) Vasilyeva E., Puuronen S., Pechenizkiy M., Räsänen P.: Feedback adaptation in web-based learning systems. Special Issue of Int. J. of Continuing Engineering Education and Life-Long Learning (to appear) (2007) 17. Vasilyeva E., Pechenizkiy M., Gavrilova, T., Puuronen, S.: Adaptation of Feedback to Learning Styles in Moodle: Pilot Experiment Results. Int. IEEE Conf. ICALT 2007 (to appear) (2007) 18. Witkin, H.A., Moore, D.R., Goodenough, Cox, P.W.: Field-dependent and field-independent cognitive styles and their educational implications. Review of Educational Research, Vol. 47, N. 1 (1977)

57 Investigation of Group Formation using Low Complexity Algorithms Christos E. Christodoulopoulos 1 and Kyparissia A. Papanikolaou 2 1 Technology Education and Digital Systems University of Piraeus, Greece 2 General Department of Education School of Pedagogical and Technological Education, Greece {christos.c@ieee.org, spap@di.uoa.gr} Abstract. Designing tools that support group formation is a challenging goal for both the areas of adaptive and collaborative e-learning environments. Group formation may be used for a variety of purposes such as for grouping students that could potentially benefit from cooperation based on their individual characteristics or needs, for mediating peer help by matching peer learners, for facilitating instructors proposing an initial grouping approach. In this paper, we discuss several factors that need to be considered when assigning learners to groups. We also investigate the use of the c-means family clustering algorithms and uniform distribution, for group formation. The fuzzy c-means is compared to (a) the k-means algorithm for homogenously grouping students, and (b) a random selection algorithm (based on the uniform distribution) for formulating heterogeneous groups. Preliminary results from grouping 36 students based on 2 and 3 criteria, indicate the potential of the fuzzy c-means algorithm for homogenously grouping students, and the random selection algorithm as a low complexity approach for achieving a significant level of heterogeneity. 1 Introduction Research on peer influences on learning suggests that students who form a group create a setting that facilitates or impedes learning above and beyond what would be expected. This denotes that how to form an effective group may have an impact to the educational benefit of group interaction. Support for group formation may be based on learner profile information [11], [7], [10] such as ability, prior knowledge, learning style, browsing behaviour, or learner context [8] such as location, time, and availability. Group formation may be used for a variety of purposes in different contexts such as (i) in a Computer-Supported Collaborative Learning (CSCL) context for grouping students that could potentially benefit from cooperation based on their complementarity of knowledge/skills or competitiveness, or for forming groups around problems with specific requirements [5], (ii) in a web-based learning environment for mediating peer help by matching peer learners based on their individual characteristics and/or learning needs on a particular subject/task [4], (iii) in a classroom-based context to facilitate instructors in formulating effective learning groups proposing an initial grouping approach [6]. Critical open issues in the area remain (a) the criteria based on which learners that should maximally benefit from each other when working together are grouped, and (b) the computational issues arising in the implementation of group formation support. In this study we discuss different factors that need to be considered when assigning learners to groups and focus on the computational problem of selecting appropriate to effectively operate on small data sets algorithms for formulating groups. In particular, in Section 2 we suggest specific factors influencing the group formation process, present a brief literature review on algorithms used in group formation and introduce the c-means family algorithms. In Section 3 we compare the k-means to the fuzzy c-means algorithm for group formation purposes. Moreover, a random algorithm based on the uniform distribution is proposed as an alternative approach for generating heterogeneous groups. Preliminary results provide evidence for the effectiveness of the fuzzy c- means algorithm in formulating homogenous groups and the appropriateness of the uniform distribution in heterogeneous groups; however in order to reach a safe conclusion a series of tests should be performed in a real context. 57

58 2 Algorithms for Assigning Learners to Groups Support for group formation aims to facilitate the process of assigning students to groups and increase the possibility that groups will satisfy specific criteria. In particular, specific factors that need to be considered when assigning learners to groups concern: the criteria used for effective grouping: the number and type of criteria that the group members should satisfy; in an educational context these criteria may reflect specific learning characteristics such as ability, prior knowledge, style, competence, or context such as problem requirements, learners location or availability the level of homogeneity/heterogeneity of the groups may be considered as a group characteristic or reflect the status of the group based on specific characteristics of the individuals, resulting in: (a) homogenous/heterogeneous groups such as a group of students with complementary knowledge/skills, or (b) groups that are homogenous according to specific criteria and heterogeneous based on others such as groups consisting of learners with same ability and mixed styles, the size of the groups in terms of the number of members included in each group. Different algorithms have been used for formulating homogenous and heterogeneous groups. In particular, recent studies propose the use of optimization algorithms as an effective solution for assigning groups. Cavanaugh et al. [2] propose the repeated use of the hill climbing optimization algorithm with weighted criteria defined by the instructor, for assigning homogeneous and heterogeneous groups in a web-based environment. Bekele focuses on heterogeneity proposing a mathematical approach which uses the Ant Colony Optimization algorithm for maximizing group heterogeneity [3]. However, the crucial parameter of low complexity remains an open issue. In this study we focus on less complex and more time-saving approaches yet sufficiently effective algorithms such as clustering algorithms for homogeneous grouping and a simple random algorithm for heterogeneous grouping. Clustering algorithms are a category of optimization algorithms designed to discover groups in data. They try to minimize an objective function which is derived from (dis-)similarity measures (usually distance) between the data. The c-means algorithms belong to the partitional clustering algorithms as they try to form clusters by dividing the data. They present a series of advantages compared to other clustering and even most optimization algorithms. First of all they take as input the desired number of clusters to be found, which is a drawback for real-life data mining, but essential to our application. Moreover, they are easy to implement in scripting languages (PHP, JavaScript). Finally one of the main reasons for their popularity (especially for k-means) is the fact that they converge extremely quickly. Their computational complexity is Ω(n) where n is the number of data points. However, the use of clustering algorithms for group formation presents also several disadvantages: (a) they form only homogeneous groups: clustering algorithms are used for grouping similar data, so it is not possible for them to create clusters that maximize the dissimilarity measure, (b) inability to evenly distribute the data points along the clusters: they take as input the desired number of clusters/groups to be found whilst they are not interested in the number of group members, and (c) limited advantages in comparison to simple sorting algorithms when used with one criterion. In this research, we compare two clustering algorithms of the c-means family, the k-means and fuzzy c-means algorithms, in grouping students based on specific criteria. Both algorithms have been extendedly used in application areas such as image processing or data mining in large sets of data. In group formation where data sets are usually small, the performance of both algorithms needs to be re-examined. k-means was proposed by McQueen in 1967 and since then it has become one of the most commonly used clustering algorithms. It is also referred Hard C-Means (HCM) in comparison to the Fuzzy C-Means (FCM) algorithm. The simplicity and the speed of HCM are obvious since it is based on the Euclidian distance that can be estimated by a series of multiplications. However its main drawback is the inability to evenly distribute the data points along the clusters. Interchanging members between neighbor clusters can face this problem, but the complexity of the process is greater than that of the main algorithm. Fuzzy c-means algorithm also known as Fuzzy ISODATA was proposed by James Bezdek in 1973 and is basically an extension of the k-means algorithm to fuzzy sets [1]. Although FCM is more complex than k- means it is still reported to have linear complexity (Ω(n)) making it as fast as k-means. Besides being fast, FCM seems to perform better than k-means when they were both evaluated with standard data mining quality measures [9]. The main advantage of FCM for group formation 58

59 derives from the membership function. In FCM a data point may belong to more than one cluster with a different probability. This feature, allows us to address the problem of inequality of the clusters in a more effective way, as we can exchange data points between clusters based on their membership probabilities. This information could be also provided to the expert-teacher as a useful aid to support final decisions on grouping students. 3 Formation of Homogenous & Heterogeneous Groups Group homogeneity: Comparing FCM with HCM. FCM and HCM have been tested in forming homogenous groups with a set of 36 students based on 2 and 3 criteria that assess students style in 2 or 3 different style categories respectively. In our case, homogenous groups consist of students with similar characteristics. The algorithms used were the standard MATLAB s implementations. FCM and HCM were compared based on their effectiveness, which was evaluated according to specific cluster validity measures. We decided to use such general measures since there is no advanced validity measure that apply to both fuzzy and non-fuzzy algorithms. A commonly used measure is Squared Sum Error (SSE) (see equation 3). Since SSE describes the coherence of a given cluster, we expect that better clusters give lower SSE values. c 2 SSE( C) = d x, v (1) j= 1 x Cj ( j ) In Table 1 we can see that in most cases the FCM algorithm gives lower SSE values than the HCM producing more coherent or homogeneous clusters/groups. For example, in rows 1 and 4, the value of the SSQ for the FCM is and and for the HCM and accordingly. Only in the 2 nd row the HCM appears slightly better. Lastly, in three different groupings (rows 3, 5 and 6), the HCM could not respond to the input conditions. Table 4. SSE values of FCM and HCM in varying number of groups and criteria. Number of groups Number of criteria SSE of FCM SSE of HCM not responding not responding not responding One major advantage of FCM, which emerged from our tests, is its ability to work in spaces that contain a limited amount of data (i.e. students in class ~20-100) and with small groups (the number of students per group decreases when the number of groups increases), whereas HCM seems to be unstable under these conditions. In our data space that contains 36 students, HCM performed well in 9 groups (4 students per group) when used with 3 criteria, but it was unable to produce clusters with 2 criteria (see in Table 1 rows 4 and 3 respectively). Moreover if we downsize even more the number of students per group (in cases where the algorithms should generate 12 groups), the standard implementation of HCM stops responding. Based on these results, and taking into account that usually group formation apply to small data sets whilst groups consist of a few members, we conclude that the classic HCM seems not to be a viable solution. Group Heterogeneity: the standard random algorithm. Heterogeneity in group formation is a relatively vague term. For example in [3] heterogeneity refers to mixed ability groups and as the authors suggest a reasonably heterogeneous group refers to a group where student-scores reveal a combination of low, average and high student-scores. Thus, a heterogeneous group might be defined as a group in which all the different values of the data space can be found. However in cases where more than one criterion is used, with a range of values for each one, then it becomes even harder to define group heterogeneity. In general, when defining the dissimilarity measure then it is a typical optimization problem to maximize its value. Another interesting approach to investigate, due to its low complexity, is to form heterogeneous groups by applying a uniform distribution on the data space. To this end a random algorithm may be used in order to achieve some level of heterogeneity. Especially in cases where the level of group heterogeneity needs not to be the maximum possible, the random algorithm could be effectively used. Moreover, compared 59

60 to most optimization algorithms, the standard random algorithm is by far faster, making it even more appealing as a choice. For validation purposes, we used MATLAB s implementation of random selection which follows the uniform distribution without replacement. We also use the cluster dispersion as a measure of heterogeneity in order to validate this approach. Cluster dispersion is defined as the cluster s diameter which is the maximum distance of any two data points belonging to the same cluster. In particular, we compare the maximum and mean diameters of all the clusters created by the random algorithm and FCM. The results are presented in Table 3, where in every grouping (each one corresponds to a different row), the maximum diameter of the clusters generated by the uniform distribution appears significantly greater than that of clusters generated by the FCM, and close enough to the maximum ones ( 7 for the squared space - 2 criteria - and 8.6 for the cubed space - 3 criteria). Table 5. Cluster dispersion generated by FCM and uniform distribution algorithms. Number of groups Number of criteria FCM Uniform distribution Max. Diam. Mean Diam. Max. Diam. Mean Diam Conclusions and Further Research In this study we investigated the potential of the k-means and fuzzy c-means algorithms for assigning homogenous groups of students and the random selection algorithm for heterogeneous groups. Preliminary experiments in a simulated environment indicate the appropriateness of the fuzzy c-means and uniform distribution algorithm for assigning groups. Especially, the output of the FCM algorithm may also be used to support instructors in group formation or students in identifying appropriate peers, by providing valuable information about the different groups that a student might better fit based on specific criteria. References 1. Bezdek, J.C.: Pattern Recognition with Objective Function Algorithms, Plenum Press, New York (1981) 2. Cavanaugh, R., Ellis, M.G., Layton, R.A., Ardis, M.A. Automating the Process of Assigning Students to Cooperative-Learning Teams. In Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition (2004) 3. Graf, S., Bekele, R.: Forming Heterogeneous Groups for Intelligent Collaborative Systems with Ant Colony Optimization, In Proc. of 8th International Conference in ITS, Taiwan (2006) 4. Greer, J., McCalla, G., Cooke, J., Collins, J., Kumar, V., Bishop, A., and Vassileva, J.: The Intelligent Helpdesk: Supporting Peer - Help in a University Course,Proceedings of ITS 1998: 4 th Int Conference on Intelligent Tutoring Systems, San Antonio, Texas, Springer - Verlag: Berlin (1998) Hoppe, H.U.: The use of multiple student modeling to parameterize group learning, In J. Greer (Ed), Proceedings of AI-ED 95, Washington D.C., USA (1995) 6. Inaba, A., Supnithi, T., Ikeda, M., Mizoguchi, R., & Toyoda, J.: How Can We Form Effective Collaborative Learning Groups?, Proceeding of ITS 2000, , Montreal, Canada (2000) 7. Martin, E., Paredes, P.: Using Learning Styles for Dynamic Group Formation in Adaptive Collaborative Hypermedia Systems. International Workshop on Adaptive Hypermedia and Collaborative Web-Based Systems, International Conference on Web Engineering, Munich (2004) 8. Muehlenbrock, M.: Learning group formation based on learner profile and context, Int Journal on E- learning, 5(1) (2006) Serban, G., Moldovan, G.S.: A Comparison of Clustering Techniques in Aspect Mining, Informatica, vol LI, no 1, Studia University (2006) 10. Tang, T., Chan, K., Winoto, P. and Wu, A.: Forming Student Clusters Based on Their Browsing Behaviors. Proceedings of the 9th International Conference on Computers in Education (ICCE 2001), Seoul, Korea (2001) Wilkinsona, I.A.G, Fung, I.Y.Y.: Small-group composition and peer effects, International Journal of Educational Research 37 (2002)

61 Evidential Multiple Choice Questions Javier Diaz, Maria Rifqi, and Bernadette Bouchon-Meunier Université Pierre et Marie Curie Paris 6 UMR 7606, DAPA, LIP6 104 Avenue du Président Kennedy, Paris, F-75016, France {javier.diaz,maria.rifqi,bernadette.bouchon-meunier}@lip6.fr Abstract. One of the most common and computably tractable ways of evaluating the knowledge of a student is through the use of questionnaires with multiple choice questions (MCQ), where students must express a precise choice to answer a question, without leaving room for ambiguities or doubts. The problem is that sometimes the student doesn t really know the answer or cannot decide between the possible choices, even if he is able to discard some of them. We propose an alternative MCQ that, using belief function theory, allows the student to state his answer in an imprecise way, indicating to which degree each possible choice represents the correct answer. This way we get to model the ignorance and uncertainties of the learners, allowing an Intelligent Tutoring System (ITS) to gather a richer student model. 1 Introduction Thanks to their intuitive interaction and computability, MCQs are arguably the most common method used by ITSs to measure student knowledge acquisition. They are seldom used by educational adaptive hypermedia systems [1] to update the knowledge model of the students and adapt the content and presentation of a learning object to the specific state of a user (student) model. Overlay Models are used to represent the mastering and misconceptions of the domain concepts studied [6, 7]. Even if MCQs have been widely studied in psychometrics and several theories have been proposed (Classical Test Theory, IRT [5]) in order to optimize trait estimation and their use in CAT (Computer Aided Testing), their interaction prevent students from stating their hesitations. When passing a MCQ test, a student has to choose among the possible answers presented in a question. The choice has to be precise even if, as sometimes occurs, the student is not entirely convinced by his own answer. It is normal for a student to find himself in a situation where a question appears to be ambiguous according to the options presented. He may be able to recognize some of the options as incorrect, but not be able to establish the correctness of all of them. He can also be facing a question to which he is certain he does not know the answer, or to which, in his opinion, the possible choices presented do not seem to answer the question. When in doubt, a student must make a blind choice among the answers that, in his view, are not completely wrong. This situations cannot be treated accordingly by classical MCQs, they lack of a way for students to express doubt, ignorance, and uncertainty. By constraining the students to precise answers, noise is gathered and considered as an input to update the student model. Invaluable information as to the real state of knowledge of the student is then lost. The concepts that have not been fully acquired by the students, the concepts that the students themselves are certain they possess and the concepts that the students have wrongfully considered as acquired could have been identified. Some authors have proposed the extension of classical MCQs by allowing the student to specify a degree of global confidence on their answers (by choosing a label) or select alternative possible choices (by suggesting an alternative choice that could be the correct answer to the question) [2]. These approaches provide some flexibility, but they are still too restrictive to represent imperfect degrees of knowledge. We propose evidential multiple choice questions (ev-mcqs), an alternative to classical MCQs, that use belief function theory to allow students to state their answer denoting their imprecise knowledge of the evaluated subjects, indicating to which degree each possible choice can represent the correct answer. ev-mcqs can then be used to diagnose knowledge acquisition and misconception. This way, students are directly involved in the evaluation and further modeling of 61

62 their own knowledge, and ITSs can then be able to gather a richer learner model closer to the actual state of the student knowledge. 2 Belief Function Theory Belief Function Theory (or Dempster-Shafer s evidence theory) [3] is a mathematical theory that generalizes probability theory by abandoning the additivity constraint. Instead of assigning a probability mass to atomic elements only, belief masses can also be associated to subsets of elements. The frame of discernment Θ is the set of all possible elements. A mapping m : 2 Θ [0, 1] assigns a belief mass to the subsets of Θ and is called a basic belief assignment (BBA). The value m(a) represents the fraction of the belief mass unit assigned to the subset A Θ, with the constraint Σ A Θ m(a) =1. The subsets A of Θ such that m(a) > 0 are called focal elements. The case where all focal elements are singletons is the bayesian belief assignment, the particular case of probabilities. Perfect knowledge of the state A Θ of the world is represented by assigning the unit of belief mass to A. On the other hand, total ignorance is represented by the vacuous belief function, an assignment of the total mass unit to the frame Θ. In Dempster-Shafer s interpretation, the frame of discernment is considered to be complete (closed world assumption) and m( ) = 0. In Smets interpretation, called the Transferable Belief Model (TBM), belief masses are considered as numerical representations of the state of belief of a rational agent [4] and m( ) > 0 can model the fact that the current state of the world is not among the elements of the frame Θ (open world assumption). We follow Smets interpretation of belief functions. 3 Evidential Multiple Choice Questions (Ev-MCQs) 3.1 Specification of a MCQ question A multiple choice question can be described by a triplet < Θ, R, r ok >, where Θ is a set of propositions that can be considered as answers to the question, R is a set of subsets of Θ representing the possible choices that the student can make to answer the question such that R 2 Θ, and r ok is the correct answer for the question, r ok R. We will note as r i the possible choices presented in R. Fig. 6. Examples of MCQ. 62

63 We recognize 2 types of MCQ questions. MCQ-SR (Single Response) consider only singletons of Θ as members of R (R = Θ); True or false questions are part of this category. MCQ-MR (Multiple Response) have R made up of choices composed of subsets of elements of Θ (see Fig. 1(a)); r ok is then a subset of Θ. We can see MCQ-MR questions as a collection of MCQ-SR, where each proposition in Θ can be considered separately as a true or false question (as we will explain in the next subsection). 3.2 Belief assignment on an ev-mcq question A student answers a question by distributing a belief mass unit among the options presented (R); he is not forced to assign all of his mass to a single option r i, he can distribute it as he wants. Also, he can leave some mass unassigned indicating his ignorance or lack of confidence on his answer. The acquired freedom allows the student to involve some of the choices that otherwise he wouldn t consider, enriching his answer. We will first explain the belief assignments for the particular case of a MCQ-SR question, where there is only one correct answer r ok Θ. R = Θ is the frame of discernment of MCQ-SR questions (see Fig. 2(a)). We can represent an answer as a BBA among the elements θ j Θ. The unassigned mass can then be associated to Θ itself. We will only have singletons and the frame as possible focal elements of the BBA that represents the answer. Fig. 7. Frames of Discernment. The approach for the belief assignment of MCQ-SR questions cannot be followed on MCQ-MR questions, since multiple propositions in Θ can be part of the correct answer (e.g. r ok on the MCQ- MR shown on Fig. 1(a) is composed of a, b and d). For this type of questions, we consider each proposition θ j Θ separately, taking a MCQ-MR as several virtual true or false MCQ-SR questions having an independent frame of discernment θ j with only two elements stating that the j th proposition is either true or false. The example shown on Fig. 1(a) will induce 5 independent frames of discernment as shown on Fig. 2(b). In MCQ-MR questions, the student will assign a mass m(r i ) (possibly 0) to every r i R. These options contain some elements θ j Θ. The presence of a proposition θ j r i will be interpreted as the statement that that proposition is a correct answer for its question, the mass assigned to r i by the user will be associated to the true singletons in the frames of each θ j r i and to the false singletons in the frames of each θ k r i. If a student doesn t think the answer to the question is represented by any r i, he won t assign any mass. If he is sure the answer is among some of the choices, he will distribute all his belief among them, not leaving any mass unassigned. If he is not totally confident of his answer, he can leave some portion of mass unassigned, representing in fact his ignorance of the actual answer (we will note this ignorance mass u = m(θ)). The value of c = 1 u = Σ ri R m(r i ) represents the confidence of the student that the answer resides among the chosen options. Some MCQ-MR questions have as possible choices the statements all of the above and none of the above. The assignment of belief mass to these options will be interpreted as the assignment to the true singletons of all θ j Θ, and to the false singletons of all the θ j Θ respectively. 63

64 3.3 Example ITS have always used traditional widgets (as radio buttons and checkboxes) to present the MCQ questions. These controls are too restrictive, and they force the students to give a precise answer. We needed to implement a novel way to control the amount of mass that could be assigned to every choice r i. An example of an answer of an evidential MCQ question can be seen in Fig. 1(b). It is the case of a MCQ-SR question, where R = {r 1, r 2, r 3, r 4, r 5 }, and Θ = R. On the right side of the lemma of the question we can see a bar that shows the mass that has been assigned by the student (c) and the mass that has not been assigned (u), this bar only shows these mass values and cannot be modified by the student. These values are updated automatically when the student changes his answer by interacting with the bars that control the mass assigned to every choice r i. The amount of mass that a student can assign to every choice is limited by the amount of mass left (he only has one unit of mass to distribute). In Fig. 1(b), the student is almost sure that the answer to the question resides on r 1, leaves a small chance to r 2 and is almost sure that the last three options are incorrect. Still, he indicates that he is not completely confident on his answer (u > 0). 4 Conclusion We propose ev-mcqs, an extension of classical MCQs that use belief function theory to allow students to express an imperfect answer. The acquisition of noise resulting from the constraint of giving a precise answer is prevented and a richer, closer image of the student s current state of knowledge can be acquired. We are currently organizing a set of experiments to validate the use of ev-mcqs in a CAT environment in order to diagnose student knowledge acquisition and misconception, by taking into account the different degrees of certainty, doubt and ignorance stated by the imprecise answers on ev-mcqs. References 1. Brusilovsky, P.: Adaptive hypermedia. User Modeling and User-Adapted Interaction 11 (2001) Bush, M.: A multiple choice test that rewards partial knowledge. Journal of Further and higher Education. 25(2) (2001) Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, London (1976) 4. Smets, P., Kennes, R.: The transferable belief model. Artificial Intelligence 66 (1994) van der Linden, W. and Hambleton, R. (eds.): Handbook of Modern Item Response Theory. Springer (1997) 6. Wenger, E.: Artificial Intelligence and Tutoring Systems. Morgan Kaufmann Publishers, Inc. (1987) 7. Winkels, R.G.F.: Explorations in Intelligent Tutoring and Help. IOS Press, Amsterdam (1992) 64

65 How to Adapt the Visualization of Programs? Andrés Moreno 1, Roman Bednarik 1, and Michael Yudelson 2 1 Dpt. of Computer Science and Statistics, University of Joensuu, Finland 2 School of Information Sciences, University of Pittsburgh, USA 1 Introduction Adaptation becomes a common resource in education. In educational environments, it is understood that as the student progresses, the learning material should be adapted to the student's changing needs, both knowledge-wise and cognitive-wise. Many different techniques to adapt educational materials have been proposed based on the student's knowledge, but they mostly address the textual content issues, that is, which material should be presented to a student at a given time. In this report we consider the case of adaptation of program animations. We also present a prototype implementation of the adaptive visualization of programs, and we discuss future directions of this research. 2 Program Animation Research Program animation and algorithm animation are both active sub-fields of software visualization research. They aim to leverage the difficulties inherent to computer science education, and learning programming in particular. Hundhausen et al. [1] performed a meta-study that revealed that algorithm animation has not be shown to be generally effective. Unfortunately, algorithm animation tools have not been widely adopted by educators yet. Naps et al. [2] report the findings of a survey carried out amongst educators to determine the factors of this low adoption level. In the survey, educators listed the benefits and disadvantages of visualization tools. Two of the factors they mentioned were 1) the time it takes to develop visualizations (90% responses), and 2) the time it takes to adapt visualizations to their teaching approach and/or to their course content (79%). Program animation tools often visualize programs as graphical and animated representation of program execution. The graphical depiction of the execution can be based on metaphors, e.g., a robot moving according to the method calls of the students' code. It also can be faithful representation of what happens in a computer or virtual machine when that code is executed. Jeliot 3, the tool we discuss in this paper, is an example of the latter approach. 2.1 Jeliot 3 Jeliot 3 [3] animates execution of almost any code written in Java, an imperative object-oriented language. Jeliot 3 produces automatically the animation. This animation, in the present version of the system, will always be the same for given source code. Jeliot 3 is the successor of Jeliot 2000 [4]. Jeliot 3 and Jeliot 2000 share the same visual animations, but Jeliot 3 includes support for objects and other Java constructs. Furthermore, Jeliot 3 features a new architecture design that permits to add new characteristics the system [5]. Jeliot 3 animations consist of frames representing each step in the execution of an objectoriented program, see Fig. 1. Jeliot 3's animation canvas consists on four parts: the evaluation area, the method frame area, the object area, and the constant area. Values move from one area to another as the program advances. For example, a simple expression like a = b +c; will be animated by moving the values of b and c from the method frame area to the evaluation area. The resulting value of the addition will be animated to a box that stores the value of a. 65

66 Fig.1. Screenshot of a running program in Jeliot 3. Ben Bassat et al. [4] carried out a long term experiment with high school programming students. The study compared the effects of a traditional IDE, TurboPascal, to Jeliot According to the study, Jeliot 2000 was effective to teach some programming constructs, but it also helped students to better verbalize the actions of the program. Mediocre students benefited from Jeliot 2000 the most, the tool leveraged their knowledge to that of other students [4]. Kannusmäki et al. [6] introduced Jeliot 3 to CS2 students, and the feedback obtained discouraged the use of animations with advanced students. However, mediocre students were the most positive about the tool. Moreno and Joy [7] reported on an experiment carried out in a programming course with Jeliot 3 for math students. Students were free to select whether to use the tool or not. The minority who used the tool were enthusiastic about the tool, but they still exhibited misunderstanding of some of the concepts that were animated. Moreover, all of the students complained that the animations took too long to finish. Summarizing, Jeliot 3 has shown to be effective in a limited scenario: mediocre students learning the basics of programming. However, the potential of visual representation of running programs should be useful in other common scenarios in education. We consider presentation of visualization to more demanding students, either the very knowledgeable or the very weak students. We believe that adaptation should enable a better presentation for the whole range of students. 3 Jeliot Adapt Program visualization developers have striven to produce automatic animations of running programs. However, these animations have not yet taken into account the student's goals, needs, and knowledge at a given moment. Thus, current visualizations are not adapted to the student, a fact that could partly explain the problems of students using program animation tools. Students may feel a lack of ownership in the tool and they might not be motivated to use it. Jeliot Adapt is intended to be the next iteration of the Jeliot family and the first to include adaptive technologies. We have discussed the need for adaptation before [8], and in this paper we present a working prototype ready for testing and evaluation. Jeliot Adapt addresses the problems found in previous experiments [6,7].First, it will incorporate question generation during visualization [9] that will monitor the student's knowledge, and it will provide explanations. Questions and explanations will be generated automatically from the code the student is working with. Questions, for example, ask the result of the next statement to be animated, be it either a simple assignment or a loop condition being evaluated. If the students answers wrong, the feedback will consist from the actual animation of the statement and an explanation where necessary. On the other hand, if the students answers correctly the animation may not be shown at all, or shown at a faster speed, depending on students' previous knowledge or learning skills. This feature should help the students who still have misconceptions after watching the animations [7]. 66