Content-free collaborative learning modeling using data mining

Transcription

1 User Model User-Adap Inter DOI /s z ORIGINAL PAPER Content-free collaborative learning modeling using data mining Antonio R. Anaya Jesús G. Boticario Received: 23 April 2010 / Accepted in revised form: 20 December 2010 Springer Science+Business Media B.V Abstract Modeling user behavior (user modeling) via data mining faces a critical unresolved issue: how to build a collaboration model based on frequent analysis of students in order to ascertain whether collaboration has taken place. Numerous humanbased and knowledge-based solutions to this problem have been proposed, but they are time-consuming or domain-dependent. The diversity of these solutions and their lack of common characteristics are an indication of how unresolved this issue remains. Bearing this in mind, our research has made progress on several fronts. First, we have found supportive evidence, based on a collaborative learning experience with hundreds of students over three consecutive years, that an approach using domain independent learning that is transferable to current e-learning platforms helps both students and teachers to manage student collaboration better. Second, the approach draws on a domain-independent modeling method of collaborative learning based on data mining that helps clarify which user-modeling issues are to be considered. We propose two data mining methods that were found to be useful for evaluating student collaboration, and discuss their respective advantages and disadvantages. Three data sources to generate and evaluate the collaboration model were identified. Third, the features being modeled were made accessible to students in several meta-cognitive tools. Their usage of these tools showed that the best approach to encourage student collaboration is to show only the most relevant inferred information, simply displayed. Moreover, these tools also provide teachers with valuable modeling information to improve their management of the collaboration. Fourth, an ontology, domain independent features A. R. Anaya (B) J. G. Boticario Artificial Intelligence Department, E.T.S.I.I., UNED, Ciudad Universitaria, c/juan del Rosal, S/N, Madrid, Spain arodriguez@dia.uned.es J. G. Boticario jgb@dia.uned.es

2 A. R. Anaya, J. G. Boticario and a process that can be applied to current e-learning platforms make the approach transferable and reusable. Fifth, several open research issues of particular interest were identified. We intend to address these open issues through research in the near future. Keywords Collaborative learning Collaboration modeling Data mining Open models Collaboration evaluation Meta-cognitive tools in collaborative learning 1 Introduction It is commonplace that students are currently able to take advantage of e-learning environments to support their learning. Moreover, throughout the students learning period they use different e-learning environments either because their educational institution does so (such is the case at UNED (National University for Distance Education), one of the largest universities in Europe with over 200,000 students, which has been using WebCT and dotlrn ever since 2000, and Moodle recently for some open courses), or because they are enrolled at different institutions throughout their learning period (especially in the Lifelong Learning Paradigm (LLL), which is becoming mainstream (Field 2006)). In this context, student modeling has contributed to improving learning by inferring characteristics of the user in order to adapt to the individual student (Kobsa 2007), but still should take into account transferability issues and be independent of the learning environment to be reusable. To tackle transferability, some strategies have been used, such as distributed models (Brooks et al. 2004), educational standards (Baldiris et al. 2008), and semantic web technologies (Denaux et al. 2005). Collaborative learning entails active learning and encourages social interactions (Barkley et al. 2004), and it has been the main contribution to improve learning in these e-learning environments. Nowadays these e-learning environments support collaborative features. However a collaboration analysis is strongly necessary to ascertain whether collaboration takes place (Johnson and Johnson 2004). Also collaboration analysis helps students and teachers manage the collaboration process. Having this in mind, our research objectives are twofold: first, to improve collaborative learning via collaboration analysis, focusing on features that make it easy to transfer the modeling system and its outcomes to other environments; second, to provide timely information to users to help them improve their collaboration management. Both objectives must be addressed, bearing in mind the whole modeling process from data acquisition to pedagogical strategy. Collaboration analysis should focus on analyzing student interactions in the e-learning environment. To this end, Data Mining (DM) techniques can be used to identify student collaboration (SC) based on their interactions (Romero and Ventura 2010). However, because methodologies and comparative studies are scant, the most suitable technique to analyze collaboration is still not known (Strijbos and Fischer 2007; Bratitsis and Dimitracopoulou 2006). In addition, Baker (2010) highlights two issues that DM systems in education should take into account: (1) the possibility of using them in several environments and (2) comparing and analyzing different DM techniques to solve the same educational problem. Both issues are considered in this paper.

3 Content-free collaborative learning modeling Moreover, to convert a modeling system into a transferable one, some important points must be considered: the data to model students should also be found in other environments, the methods to analyze these data should be applicable to other environments, the result of the modeling -the model itself- should be understood by others, and the model s purpose should be useful in common e-learning environments. If the modeling system has all the aforementioned features, we can say that the system can be easily transferred to other environments. Data, derived from active student interaction during their collaboration, were used to model and analyze SC (Gaudioso et al. 2009). We used student interactions in the forums of a collaborative learning environment. Forums are a communication service widely used in e-learning environments. To analyze forums and their interactions, some researchers have focused on encoding forums and messages (Patriarcheas and Xenos 2009) or mining them to obtain student characteristics (Dringus and Ellis 2005; Cocea and Weibelzahl 2009), even using time variables (Dringus and Ellis 2010). The literature encourages analysis of forum interactions to discover student characteristics and behavior. In addition, analysis should be applied to other e-learning platforms and different courses. For this reason, we did not encode the messages or analyze the message content (e.g. lengthy messages can be affected by the domain). A freecontent analysis and standard processes supported by current e-learning platforms are the bases for the transferability feature of our approach. With respect to the analysis methods, DM techniques can be easily used in different environments as long as these environments can store common data permanently and the techniques can infer student features: assessments of student actions (Romero et al. 2009). Other researchers analyzed SC using DM techniques and considered student interactions as a sufficient data source (Talavera and Gaudioso 2004; Perera et al. 2007; Gaudioso et al. 2009). As comparative studies are scant, we investigated different alternatives and finally proposed two different DM approaches to analyze SC, and a simple method to compare both approaches (Anaya and Boticario 2011). The two approaches consist of: (1) grouping students according to their collaboration using unsupervised classification techniques (clustering approach) (Anaya and Boticario 2009); (2) constructing collaboration metrics using supervised classification techniques (metric approach), which assign a collaboration value to each student so that learners can be compared (Anaya and Boticario 2010). As for the modeling result, a student model stores information (user, usage and environment data (Kobsa et al. 2001)) in a stable structure and this information can be dynamically updated and used by inferring methods (Redondo et al. 2003; Duque and Bravo 2007; Baghaei and Mitrovic 2007). We have noted that previous researchers used different strategies to identify transferable features (Brooks et al. 2004; Baldiris et al. 2008; Denaux et al. 2005). This problem can be minimized by using open student models (Bull and Kay 2008). Open student models store and structure information (user, usage and environment data) but the main characteristic is that these models must be managed by students. Thus, the models must be meaningful to the students. Consequently, these open student models should be independent of the system or the learning platform. The responsibility for learning decisions lies with the learner (Bull et al. 2009). Accordingly, the transferability feature advocated in our research is intended to support both the independence of the model from the open

4 A. R. Anaya, J. G. Boticario student viewpoint (i.e. students access their model and manage their own collaborative learning) and its applicability in current learning systems, drawing on features and processes that are common to these systems. From the pedagogical standpoint, the open model strategy encourages active learning, which theoretically may help increase student motivation (Hummel et al. 2005), something always recommendable in educational environments. The open model strategy has been used positively in the educational context (Bull et al. 2009) and we researched this issue and also other issues relating to using meta-cognitive tools in collaborative learning, as will be described later on. In this research a collaboration model was built with information on SC following the open model strategy. The information was selected so that students could use modeled information to improve collaboration process management. We followed different ideas from some researchers to model SC. Johnson and Johnson (2004) explained that, in order to know whether collaborative learning takes place, an analysis of collaboration or group performance is necessary and appropriate. Additionally, monitoring student interactions offers information on their collaboration (Johnson and Johnson 2004; Steffens 2001), and knowledge of the collaboration context can offer useful and essential information to collaborate (Muehlenbrock 2005; Durán 2006). These ideas suggest that collaboration model information can be divided into the following types: SC context, SC process to monitor SC, and SC assessments. To improve learning, different strategies such as providing meta-cognitive information on SC or recommendations on student behavior can be implemented drawing on tools to monitor student interactions. In this respect, the open model strategy provides a method to achieve the objective and increase student activity. This approach stores information so that students are able and are being encouraged- to manage it. Therefore, the information should be understandable. The open model strategy recommends using scrutable tools to achieve the educational objectives (Bull and Kay 2008). Scrutable tools are tools that display students own models. The tools enable students to use and manage the models and the model information (Kay 1999). However, the open model strategy has not yet been used in an evaluation study that considers both hundreds of students and DM techniques to disclose new knowledge on SC. Our research offers four tools. These tools obtain different types of information collected in the collaboration model and they use two displaying strategies: the simplest approach -so that students can use and understand the information- (Barkley et al. 2004) and the scrutable strategy (Kay 1999). These tools were offered to identify and assess the most useful type of information and displaying strategy to achieve the objectives. Other researchers have provided tools that used the modeling approach (Gaudioso et al. 2009), and the tool even had scrutability features (Bull et al. 2009). These researchers, nevertheless, did not compare the scrutable strategy with other strategies in the same context. This was done in our research. Our research collaboration modeling approach uses student information on collaboration and their interactions as the data source. Two DM methods are used to infer information on SC, an open model strategy is used to model SC, and meta-cognitive tools are used to achieve their expected pedagogical advantages. The approach poses a modeling system, which analyzes the collaborative learning process in terms of features that are domain independent and can be transferred to current e-learning

5 Content-free collaborative learning modeling environments. An innovative feature of our approach is to provide an evaluation study that combines the open model strategy and SC assessments with DM methods. The approach was evaluated in a long-term collaborative learning experience. All students of Artificial Intelligence and Knowledge-based Engineering (AI-KE) at UNED were invited to participate, and more than one hundred completed the collaborative learning experiences during three consecutive academic years: , and The experiences were divided into two phases: the shorter, initial phase was an introduction to the work. Students had to complete the initial phase individually. In the longer second phase, students were grouped into three-member teams to work together. The results of the collaborative learning experiences were used to evaluate the whole approach. During the collaborative learning experiences in the academic years , and , student interactions were analyzed using two different DM approaches to obtain SC assessments (Anaya and Boticario 2011). The collaboration modeling approach was applied during the collaborative learning experience in the academic year Students were modeled and the four tools were offered to different collaborative teams. The tools were evaluated from the students answers in a final questionnaire, their evaluations of the collaborative learning experience, and their marks in the AI-KE examination. This research analyzed other collaboration modeling systems to frame the problem, and this is described in the next section. Then, the collaboration modeling approach is explained along with the experimentation carried out to test the approach. Later, the results of the experiment are discussed. The final section of this paper is devoted to the conclusions and future works. 2 Key modeling issues in collaborative learning systems There are several issues affecting the modeling of collaborative learning systems. Developing successful collaborative environments is not trivial and several conditions have been identified to make collaborative learning better than individual or competitive learning (Johnson and Johnson 2004). User Modeling (UM) and student modeling systems have not focused on collaboration in depth (Kobsa 2007). Nevertheless, different researchers have studied collaboration and proposed collaboration modeling. Moreover, we should highlight the lack of standards and methodology in the collaboration analysis field (Strijbos and Fischer 2007), and the lack of comparative studies (Bratitsis and Dimitracopoulou 2006). The collaboration modeling field should be framed to clarify the modeling issues. We propose the following points to frame the modeling process: (1) the information used to model SC, (2) the methods used to acquire this information, (3) the types of collaboration model that have been proposed, (4) modeling, analyzing and evaluating SC, and (5) the strategies that have been applied to achieve the objectives of the modeling systems. In addition, we also take into account the transferability of the modeling systems to other e-learning environments. There are several information sources to model SC. The literature has noted that a group performance analysis can help in finding out whether collaboratvie learning

6 A. R. Anaya, J. G. Boticario takes place (Johnson and Johnson 2004). Other researchers have considered the collaboration context or circumstances as significant features (Durán 2006; Muehlenbrock 2005), because the collaboration context informs other students about the aptitude and capacity of their fellow students to collaborate. Student interactions play an important role in monitoring and analyzing SC (Steffens 2001; Talavera and Gaudioso 2004; Perera et al. 2007; Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006; Gaudioso et al. 2009; Gómez-Sánchez et al. 2009). Regarding information acquisition methods, depending on the context, the e-learning environment and the context of the courses, the collaboration modeling systems obtained information on students in different ways. On the one hand, students and teachers were asked for information on student work and features in questionnaires or reports (Collazos et al. 2007; Soller 2001; Park and Hyun 2006; Meier et al. 2007; Kahrimanis et al. 2009). On the other hand, student interactions were stored to model their collaboration. For instance, Redondo et al. (2003) and Duque and Bravo (2007) stored student interactions in relation to the course content. Talavera and Gaudioso (2004); Perera et al. (2007) and Gaudioso et al. (2009) stored student interactions in the communications means (chat, forums). Bratitsis et al. (2008), Martínez et al. (2006) and Daradoumis et al. (2006) stored the data according to a social network analysis. Communication plays an important role in collaborative learning, to such an extent that without communication there is no collaboration. The aforementioned researchers studied this issue. One of the most important parts of any UM system is the model and there are several types of collaboration models. There are two types of users in collaboration systems: teachers and students. The latter can be provided with regulation functionalities and the former with evaluation, although students may also participate in the evaluation process (see below). The student model is used to monitor student behavior (Bratitsis et al. 2008), which helps students and teachers alike discover the degree of collaboration, or generate recommendations, which are meant to improve student collaboration Baghaei and Mitrovic (2007). Some researchers have established a relationship between the model s attributes or the collaboration model itself and other models that describe other student features or the learning context. Redondo et al. (2003) and Duque and Bravo (2007) compared the student interaction model with another expert-designed model created prior to student activity by means of fuzzy algorithms. These researchers proposed a set of attributes related to student communication (e.g. number of messages, number of instant messages, conversation depth) from which they inferred some of the student features (e.g. initiative, creativity, agreement, disagreement) by means of fuzzy logic. Once the student model had been created, it was compared with another model that described the best way of collaborating according to experts. This evaluation was performed by means of fuzzy logic. Baghaei and Mitrovic (2007) proposed a constraint-based model. When the student interaction model did not satisfy all the constraints of some rule, this rule was applied and a recommendation was sent to the student. Other researchers have studied the relationship of the collaboration model with the characteristics of the collaborative learning course where the collaboration took place or with other parts of the student model. Martínez et al. (2003) proposed a structure of the collaboration model s attributes, which they used to describe the collaboration

7 Content-free collaborative learning modeling actions in Computer-Supported Collaborative Learning (CSCL) systems. Vidou et al. (2006) suggested a student model that established relationships with other models; the relationships refer to other parts of the course components (course activities, resources, students). Barros et al. (2002) proposed an ontology that did not describe collaboration, and the ontology modeled activities that encouraged students to collaborate. Another example is the collaboration model by Durán (2006) who proposed the main characteristics that a collaboration model should have: collaboration context, ability to mediate in conflicts, ability to motivate, manage information and seek others work, ability to delegate, recognize others in conversation, maintain group cohesion and switch tasks during the conversation. From these researchers we can deduce that the collaboration context plays an important role in collaboration analysis. There are also researchers who have focused on the model s attributes or collaboration indicators to model collaboration. Collazos et al. (2007) proposed five indicators: selected strategy, intra-group coordination, review of the success criteria, monitoring and development. Soller (2001) proposed a Collaborative Learning Model where a set of indicators described effective collaboration in an educational setting (participation, basic social knowledge, discussions on active learning, development analysis and processing groups, and interaction promotion). Park and Hyun (2006) encouraged students to evaluate their peers (team) and themselves using the following indicators: interaction, collaborativity, and accountability. This collaborativity represented the group s activities to achieve the team s learning objectives. Meier et al. (2007) and Kahrimanis et al. (2009) attempted to find general indicators to identify collaboration. They conducted a literature review and proposed five aspects of the collaboration process to model collaboration (Communication, Joint information processing, Coordination, Interpersonal relationship and Motivation). These indicators could be used after teachers or students had observed the completed interactions. Researchers within the Kaleidoscope network of excellence used student interactions as the main source of indicators to model collaboration (Gómez-Sánchez et al. 2009). The researchers within that network used the same methodology and proposed a set of similar indicators (Martínez et al. 2006; Daradoumis et al. 2006; Bratitsis et al. 2008). In particular, Martínez et al. (2006) and Daradoumis et al. (2006)) proposed a number of indicators divided into three layers. The top level contained evaluations of the tutors and students. The middle level presented the Social Network Analysis (SNA). The attributes of the quantitative analysis of the lower level are related to social network analysis and these attributes are network density, degree of actor centrality (indicates prestige or acknowledgment) and degree of network centralization (the dependence of the network on a small group of actors). By contrast, Bratitsis et al. (2008) used only two levels: SNA and quantitative analysis. This research proposed four categories to assist users with collaboration (indicators of such concepts as site visits, access to resources, resource manipulation and user behavior). These researchers used the student interaction attributes to conduct an analysis of the social networks, after measuring the number of communications between the sender and the receiver. In addition to the types of models and modeling methods cited above, one of the most critical tasks in collaborative learning systems is collaboration evaluation itself. The aforementioned researchers proposed different approaches for teachers or students, in some cases, to evaluate collaboration. Only Redondo et al. (2003)

8 A. R. Anaya, J. G. Boticario and Duque and Bravo (2007) analyzed student collaboration without any human intervention. The researchers proposed a collaboration model that was intended to be the best way of collaborating, and this model was compared with student interaction models by means of fuzzy logic. The researchers established the best collaboration behavior before the students began the activity and this approach is an a priori judgment and therefore a debatable approach. It should be observed that the expert-based analysis has some problems. Experts, teachers or tutors cannot control what students do and have difficulties when analyzing their interactions, due to student heterogeneity and the large number of students usually involved in e-learning environments (Barkley et al. 2004; Boticario and Gaudioso 2000; Gaudioso et al. 2003). An expert-based oriented analysis takes into account specific features from the given context, which minimizes the transferability of the model. For instance, Meier et al. (2007) and Kahrimanis et al. (2009) used a similar approach to analyze collaboration, but not the same indicators. Kahrimanis et al. (2009) used the indicators: Collaboration Flow, Knowledge Exchange, Argumentation, Structuring the Problem Solving Process, Cooperative Orientation, Sustaining Mutual Understanding and Individual Task Orientation. Only the last two indicators were also used by Meier et al. (2007). There are other techniques that can help when assessing collaborative learning. In e-learning environments, analysis of student interactions can be conducted with DM techniques (Romero et al. 2009). DM techniques can be divided into three parts (Romero and Ventura 2010): the pre-process (data gathering, cleaning, filtering and arranging), the DM process (the result inference), and the post-process (results validation and use). Some researchers have applied DM processes to assess collaboration. We have described the research by Redondo et al. (2003) and Duque and Bravo (2007), but others have analyzed collaboration and obtained assessments (Gaudioso et al. 2009; Talavera and Gaudioso 2004; Perera et al. 2007). All these researchers proposed inferring methods that offered collaboration assessments with Machine Learning (ML) technologies, which are appropriate in these contexts because they can be applied regularly and frequently to support an automated process of analysis (Russell and Norvig 1995). Talavera and Gaudioso (2004) and Gaudioso et al. (2009) applied a quantitative method to obtain the data. They used student interactions as the data source. However, Perera et al. (2007) used interactions and expert-based analysis as data sources. The Machine Learning techniques used were clustering (Gaudioso et al. 2009; Talavera and Gaudioso 2004; Perera et al. 2007), sequential pattern mining (Perera et al. 2007) and decision tree algorithms (Gaudioso et al. 2009). The approaches by Talavera and Gaudioso (2004), Perera et al. (2007) and Gaudioso et al. (2009) can be potentially transferred to other environments because they only took into account student interaction and excluded content information, unlike the authors who proposed a coursecontent-dependent model (Redondo et al. 2003; Duque and Bravo 2007; Baghaei and Mitrovic 2007). The final issue that affects the modeling process in collaborative learning systems is the strategy that is applied to improve learning, which is a shared ultimate goal. Different strategies have been applied to date. Some researchers have offered monitoring tools, which students used to watch their activity, or experts to analyze collaboration. For instance, Collazos et al. (2007), Park and Hyun (2006), Meier et al. (2007) and

9 Content-free collaborative learning modeling Kahrimanis et al. (2009) offered the modeling results in a simple attribute-value set. Martínez et al. (2006), Daradoumis et al. (2006) and Bratitsis et al. (2008) presented or proposed presenting the results graphically. Other researchers, in addition to monitoring, have offered tools that provided meta-cognitive information on collaboration. Redondo et al. (2003) and Duque and Bravo (2007) integrated the results into the student model, which tutors could see. Talavera and Gaudioso (2004), Perera et al. (2007) and Gaudioso et al. (2009) proposed simple and usable visual tools. These tools met the condition suggested by Johnson and Johnson (2004), which was to discover whether collaborative learning takes place. Finally, only one research study has proposed a recommendation system (Baghaei and Mitrovic 2007) offering recommendations on collaboration using a constraint-based model, but without collaboration inferences or assessments. Collaborative learning strategies may impose modeling requirements. Thus, the student model could be designed so that both students and teachers can use it for regulating and evaluating respectively. This issue is supported by the open model strategy (Bull and Kay 2008). The open model establishes that students can use and manage their models. This should increase accountability and, accordingly, motivation and active learning (Hummel et al. 2005; Burleson 2005; Boticario and Gaudioso 2000). The open model requires students to understand and use the information collected in the model. For this reason, the model structure and syntax must be clear enough for a non-expert user. A meta-cognitive tool that uses the open model strategy should achieve all these features. In this section collaboration modeling has been framed. Information on student interactions, mainly communications, has been proposed as the main source to model student collaboration. However, the lack of standards, methodologies and comparative studies has prevented the collaborative model structure or the collaboration indicators from being firmly established (Strijbos and Fischer 2007; Bratitsis and Dimitracopoulou 2006). Additionally, the most common technique to use the collaboration model or its results has been to show them to students and teachers so that they can evaluate student collaboration. Moreover, collaboration assessments are an important issue in the collaborationmodeling field (Johnson and Johnson 2004), but only some researchers have proposed an inferring method for this task (Redondo et al. 2003; Duque and Bravo 2007; Talavera and Gaudioso 2004; Perera et al. 2007; Gaudioso et al. 2009). These assessments can be achieved using DM processes. Applying DM methods to predict student behavior, features or skills, or to group students according to some features is not a new paradigm. However, it is used more and more (Romero and Ventura 2010) because of the good results with minimal expert intervention or an expert-based analysis (Baker 2010). Only a few researchers, as we have already mentioned, have used DM techniques in the context of collaborative learning, so the appropriate DM technique for collaborative learning has not been established. Baker (2010) highlights two points that DM systems in education should take into account: (1) the use of the DM system in other environments and (2) comparative analysis of different DM methods to discover the appropriate technique in a specific context. These two points were considered in our research. The collaboration modeling approach, which will be explained in the next section, uses two different DM techniques. Both techniques are compared, and

10 A. R. Anaya, J. G. Boticario our approach looks for some features that make it easier to transfer the approach to other environments. 3 Collaboration modeling approach After reviewing the related research in the previous sections, we now describe our approach proposed to model collaboration. Our research takes into account: (1) the importance of analyzing student collaboration (Johnson and Johnson 2004); (2) the close relationship between student collaboration and interactions, mainly regarding communication (Martínez et al. 2006; Daradoumis et al. 2006; Bratitsis et al. 2008), (3) the recommended feature of transferability, (4) minimal participant intervention in the analysis to ensure that it can be conducted in other contexts and environments, and (5) the lack of standards, methodology and comparative studies in the collaboration analysis field (Strijbos and Fischer 2007; Bratitsis and Dimitracopoulou 2006). Obviously, the objective of the research was to improve collaborative learning and the constraints were that students in most distance education (DE) settings should control their own learning processes (especially in LLL) and that different DE environments could come into play. Thus, the approach must adapt to these circumstances, i.e. be meaningful to users and transferable to other DE environments. According to the aforementioned collaboration modeling issues, our approach proposes the information for modeling SC, the source of this information, how to infer collaboration assessments, and the strategy to use to improve collaborative learning. We can summarize the main points as follows: - The approach focuses on student interactions and divides the information that it uses to model student interactions into: student context, SC process and SC assessments. This information should be obtained from the given e-learning environment to support the transferability of the approach. - The approach analyzes the interactions of the different communication means during communication in the e-learning environment because of the close relationship that exists between student collaboration and interactions. - The approach uses a DM process with ML technologies to facilitate transferability and analysis without human intervention. - We have mentioned the advantages of the open model strategy. The approach offers a collaboration model that can be managed and understood by students. Different tools are proposed. One of them supports all the features of the open model strategy and is a scrutable tool for evaluating the advantages of metacognition and scrutability itself. In the next subsection the collaborative learning experiences are explained. These experiences were used to evaluate the approach. Here, the evaluation is divided into two experiments. Firstly, the DM process to infer an assessment of student collaboration, which was implemented during three experiences. Secondly, the evaluation of the whole approach and the relation with improved collaborative learning, which was tested during the last collaborative learning experience. The paper now explains in depth the type of information used to model collaboration. The processes for obtaining the information are clarified. Then we describe the

11 Content-free collaborative learning modeling Fig. 1 Collaboration modeling approach schema DM processes and a method to compare them. A very important part of this research is the collaboration model itself. Therefore, one of the subsections specifically defines and illustrates the model. Finally, the tools used to achieve the objective are described. The collaboration modeling approach, which has been proposed in this paper, can be summarized in Fig. 1. Student interactions are collected and structured using the DM approach, which provides ML methods to infer assessments of student collaboration. The structured information on collaboration is used to build the student Collaboration Model, which generates the meta-cognitive tools that students can use to improve collaboration process management. 3.1 Collaborative learning experience This research was developed over three consecutive collaborative learning experiences during the academic years , and All students of AI-KE at UNED were invited to participate. These students were characterized by their heterogeneity and large number (Boticario and Gaudioso 2000), and they had to control their own learning process to be able to take part in a distance education experience (Gaudioso et al. 2003). Thus, the collaborative experience was designed in such a way that students had control of their own learning and collaboration process, so most of the students did not have any problem in participating. The collaborative learning experience had a long-term approach and students had to perform a number of individual and collaborative tasks with no deadlines, except for the final work. The experiences were divided into two phases, and they lasted for about 3 months. - 1st phase: individual work, where students had to answer an initial questionnaire on collaboration context information, and complete a mandatory task. This phase took approximately 3 weeks. Figure 2 shows those students who began and finished this phase over the 3 years. The purpose of this phase was to help students learn how to use the e-learning platform and understand the workload of the collaborative learning experience. Those students who accomplished the mandatory task and agreed to go on with the experiment could start the 2nd phase. - 2nd phase: work in three-member teams. Depending on the student collaboration context or circumstances, which were obtained from questions in the initial questionnaire, students were divided into small three-member teams, following the

12 A. R. Anaya, J. G. Boticario Fig. 2 Collaborative learning experience schema recommendations for this type of experiences (Johnson and Johnson 2004). The teams had to perform a series of consecutive tasks with an increasing degree of collaborative work, and they were asked to answer a final questionnaire. The 1st task aimed to put team members in contact with one another because the team had to choose one problem from three to solve. The 2nd task required individual work. For this, a problem was divided into three parts and each member of the team was given one part to solve individually. In the 3rd task the team members had to join their individual solutions and discuss the number of variables, type of action, etc., in order to reach a team solution to the given problem. Fellow students communicated in the team private space via platform forums so that the tutor could identify any problems. Then, the team members had to propose variations of the solved problem together following the scientific method and solve them collaboratively in the 4th task.the5th and final task asked for a report of the completed work. The objective of this phase was to make students work in teams to tackle the problems that usually arise in collaboration environments. A space on the dotlrn platform ( was used for the collaborative learning experience, widely used at UNED and in the adenu research group (Santos et al. 2007). A general space with the forum services, documentation, questionnaires and news was enabled for all students. For the second phase a space where only team members could enter was enabled. This workspace had forum services, chat, documentation, questionnaires and a task manager. The rationale behind the setting up of the collaborative experience was to provide a flexible organization of student tasks, including individual and collaborative tasks. Moreover, as it will be discussed later on, we aimed to encourage student accountability and improve their collaboration with three meta-cognitive tools with different feedback. 3.2 Information to model SC The literature points out that an evaluation of group performance is necessary (Johnson and Johnson 2004). Some researchers have specifically considered the collaboration context or circumstances (Durán 2006; Muehlenbrock 2005), because the

13 Content-free collaborative learning modeling collaboration context informs other students about the aptitude and capacity of fellow students to collaborate, whereas other researchers have focused on student interactions (Steffens 2001; Talavera and Gaudioso 2004; Perera et al. 2007; Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006). We propose three types of information to model SC. This information is used to improve student awareness of the collaboration process and consequently their collaborative learning management. Moreover, in order to fulfill the research objectives, the modeling system must be able to be used in other e-learning environments. Since we consider the common features of current collaborative e-learning environments, the barriers should stem from the courses themselves, i.e. course contents. For this reason, we propose information that is unrelated to course contents to model the collaboration: - Collaboration context or circumstances. Those students who agreed to participate in the collaborative learning experience were asked in the initial questionnaire how they would be able to collaborate. - Collaboration process. Since the students communications were supported mainly by forum messages, the collaboration process should be related to student interactions in forums, which are the most important communication means in most current e-learning environments. - Collaboration assessment. The assessment can be useful to identify SC behavior, and thus students can be aware of their own behavior and that of their fellow students. This collaboration assessment should stem from SC interactions. 3.3 Acquisition and indicators The first step in a modeling system is to collect, gather and arrange the data, which should be used to model students, their features and behavior. We propose three types of information to model collaboration: collaboration context or circumstances, process and assessments. The collaboration context or circumstances explain student potential and capacity to collaborate. This means that the information can come from data related to both students and the environment, which should be relevant to student teamwork skills. For this reason, this information can be collected in the collaborative learning experience from an initial questionnaire. The questions asked students for personal, academic and work-related data, and study preferences. We considered this information appropriate for SC, because students themselves requested or provided this information in the forums of previous collaborative learning experiences when they contacted and communicated with their peers. Information on the collaboration process relating to features such as activity, initiative or acknowledgment, can be obtained by analyzing student interactions in forums (Gaudioso et al. 2009; Talavera and Gaudioso 2004; Perera et al. 2007; Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006; Redondo et al. 2003; Duque and Bravo 2007). One of the issues that the aforementioned researchers left open was time variables. Perera et al. (2007) proposed pattern mining where experts had to consider time variables. However, an automatic time analysis was not conducted. Dringus and Ellis (2010) found that conversation features, which can be used to improve learning,

14 A. R. Anaya, J. G. Boticario Table 1 Statistical indicators of student interactions in forums Forum conversations started Forum messages sent Replies to student interactions N_thrd = n i (x i ); x number of threads started on day i and n a set of days in the experience M_thrd = average (N_thrd) = (1/N)( n i (x i )); N number of days in the N_msg = n i (x i ); x number of messages sent on day i and n a set of days in the experience M_msg =average (N_msg) N_reply_thrd = number of messages in the thread started by user M_reply_thrd: N_reply_thrd/N_thrd experience V_thrd = variance (N_thrd) V_msg = variance (N_msg) N_reply_msg = number L_thrd = N_thrd / V_thrd L_msg = N_msg / V_msg of replies M_reply_msg: N_reply_msg/N_msg could be deduced by analyzing time variables or variables that were measured over a period of time. These variables were grouped as: temporal transitions, density, intensity, latency, and number of replies. We proposed a statistical analysis of the interactions in forums to discover some features that make students suitable for collaboration (Santos et al. 2003): student initiative, activity and regularity, and perceived reputation by their peers. Student regularity indicators involve time variables because the interactions are considered over a period of time. We proposed twelve statistical indicators, which are listed in Table 1. We believe that student initiative is related to the number of conversations started (N_thrd), which is also connected to a period of time (one day). For this reason, the variance (V_thrd) is related to the regularity of initiative. If all students had the same value of N_thrd, lower values of V_thrd would indicate a higher regularity of student initiative. Since students may not have the same values of N_thrd, we propose an additional indicator L_thrd, which connects student initiative to student regularity. The same rationale supports the student activity indicators (i.e. N_msg, V_msg, L_msg). We analyzed the time variables using these indicators, which are underlined in Table 1. In the case of the indicators measuring the replies to student interactions, acknowledgment can be measured by the replies to student initiative (N_reply_thrd and M_reply_thrd) or activity (N_reply_msg and M_reply_msg). We did not perform any semantic analysis to label or code the messages (Patriarcheas and Xenos 2009) for two reasons. Firstly, free-content interaction variables enable the same variables to be used in other environments. In other words, although the educational domain may vary, the indicators provided can be obtained from user interactions in other environments. Secondly, messages or conversations about off-topics can help students use the platform, discover new knowledge, get to know their partners, etc. (Barkley et al. 2004). Thus, all kind of conversation may help students to collaborate or learn. We consider that the preceding twelve indicators are relevant to the collaboration process. These indicators also identify different student features during communication. At the beginning of our research more than twelve indicators were proposed (e.g. number of tutor interventions, number of messages sent to the tutor), but as our research focused on communication among students to analyze their collaboration,

15 Content-free collaborative learning modeling we only used the indicators necessary for this purpose. However, collaboration assessments require a deeper, expert-based analysis (Perera et al. 2007; Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006) or one performed using automated methods (Talavera and Gaudioso 2004; Perera et al. 2007; Redondo et al. 2003; Duque and Bravo 2007; Gaudioso et al. 2009). Researchers have proposed different student interaction indicators to analyze collaboration. Bratitsis et al. (2008),Martínez et al. (2006) and Daradoumis et al. (2006) used interaction indicators derived from a social network analysis. Perera et al. (2007), Redondo et al. (2003) and Duque and Bravo (2007) analyzed student interactions statistically in the environment to obtain collaboration indicators, and Talavera and Gaudioso (2004) and Gaudioso et al. (2009) focused on forum interactions to obtain collaboration indicators. The last approach was followed in our research. We suggest DM processes to analyze collaboration automatically. However, when the DM process uses ML techniques it needs two data sources. Firstly, a data source from which the results are obtained, and then a second data source to evaluate or infer the results. Usually the second data source is derived, when available, from an expert-based analysis, and then the approach is considered as supervised. One or more experts analyze the data, and label or classify the instances according to criteria based on their knowledge of the issue. Accordingly, the results obtained via ML techniques can be compared with the labels or classification assigned by the experts to validate the usefulness of the technique. However, once the technique is validated, it will be used again without any expert involvement. This research applied the twelve statistical indicators as the main data source and an expert-based analysis to validate the approaches. We proposed a supervised approach in which an expert read all the collaborative learning experience messages and labeled students according to their collaboration, students initiative to collaborate, and their capacity to maintain and structure the teamwork. In our experience the expert used nine collaboration labels ranked from 1 (very collaborative) to 9 (not collaborative). The first three labels (1, 2 and 3) represented high collaboration, the middle labels (4, 5 and 6) represented medium collaboration, and the last three labels (7, 8 and 9) represented low collaboration. The expert labeled only one student with label 1 and no one was labeled with label 9. Thus, labels 1 and 9 were not used in the analysis. The expert used labels rich enough to make the subsequent statistical analyses easier. 3.4 Data mining processes to assess SC We know from other researchers that student features can be assessed with ML techniques in the DM process (Romero et al. 2009). Additionally, DM processes minimize human intervention and can be used in a wide range of environments. We propose different DM processes using different ML techniques and a comparative method to deduce the usefulness of each approach (Anaya and Boticario 2011). The first approach is clustering. Clustering technologies have been used by Talavera and Gaudioso (2004), Perera et al. (2007) and Gaudioso et al. (2009). Grouping instances without prior knowledge of the most relevant attributes (from an

16 A. R. Anaya, J. G. Boticario expert-based analysis) can be done by applying unsupervised ML techniques like clustering (Gama and Gaber 2007), provided that the volume of data is sufficient. Since the instance attributes were related to the collaboration process in our research, the groups or clusters obtained might be related to the level of SC. However, this inference should be proved by comparing clustering outcomes with the SC level, i.e. the label provided by the expert-based analysis. The second approach consists in a collaboration metric or distance function. The objective is to prove that the distance is related to SC. As already stated, given the approximate nature of this problem, a key issue was to be able to obtain the classification results frequently and regularly. As collaboration communication in the experience was held through the team forums, forum interactions were the support and the objective of the analysis for measuring student collaboration. Twelve statistical indicators for student interactions in forums have already been described in this section. The approach consisted of establishing a relationship between indicators and collaboration, and selecting those indicators most related to collaboration. Thus, metrics could be constructed to provide students with an approximate collaboration value so that they could compare themselves with one another Clustering approach This approach consisted of the following stages: 1. Building datasets with the statistical indicators of student interactions in forums for every experience. 2. Running the EM (Expectation-Maximization) clustering algorithm with every dataset to group the instances into three clusters. 3. Comparing the clusters obtained with the expert-based analysis. This research was conducted using student interaction data from the 2nd phase of three collaborative learning experiences. Their interactions in forums were analyzed statistically and twelve indicators were obtained for each student. Nevertheless, oblivious to course instructions some teams seldom communicated in forums. For this reason, in this research two types of datasets were proposed to discover the effect of the low activity of some students in the collaborative learning experience. Thus, some teams were removed from the datasets. Eventually, two datasets were built every year, a first dataset with all instances (D-I-06-07, D-I and D-I-08-09), and a second dataset without the teams whose members seldom communicated (D-II-06-07, D-II and D-II-08-09). The EM clustering algorithm groups instances into clusters using unsupervised learning, which determine how the instances are organized. A clustering algorithm groups instances according to their similarity by applying the Euclidean Metric. The EM is used for finding maximum likelihood estimates of parameters in probabilistic models and the improved accuracy of EM clustering has been proved in the context of web mining (Mustapha et al. 2009), and this is the context that was used for this research. This algorithm has been used by other modeling systems (Perera et al. 2007; Talavera and Gaudioso 2004; Teng et al. 2004; Gaudioso et al. 2009) to group students according to some significant characteristics (for instance, student activity or

17 Content-free collaborative learning modeling Table 2 Number of instances (No.) in every dataset, the log likelihood (L_l) of the clustering process, and the average indicator values n_msg (N_m) and n_reply_msg (N_r_m), and the collaboration label (L) in each cluster Dataset No L_l Cluster-0 Cluster-1 Cluster-2 N_m N_r_m L N_m N_r_m L N_m N_r_m L D-I D-II D-I D-II D-I D-II misunderstanding). As other researchers have used the EM in similar research, we use the EM as an inferring algorithm to derive SC. The relationship between the clusters and SC can be inferred by comparing the clusters with the collaboration label, which was established by the expert-based analysis. We proposed three clusters for three SC levels (Low, Medium, High), which is easy to understand for students (bear in mind that the outcomes will be eventually displayed to the students). As the collaboration label was a number, the collaboration label average was calculated in every cluster, and the results are displayed in Table 2, where the clusters are ordered by the average indicator values. For this reason, cluster-0 groups students with low indicator values, cluster-1 groups students with medium indicator values, and cluster-2 groups students with high indicator values. In Table 2 only two indicators are listed (N_msg, i.e. N_m and N_reply_msg, i.e. N_r_m ) because they were the most representative. Students were grouped primarily according to these two indicators. However, the cluster algorithm used all the statistical indicators in the process. The column labeled L shows the average collaboration label in the cluster according to the expert-based analysis. It is important to note that the expert used an inverse numeric scale (this was an arbitrary decision taken by the expert). The high values in column L mean low collaboration level, while low values in that column mean high collaboration level. From the values of the variable log likelihood L_l ( in the case of D-II-08-09) it follows that the clusters obtained were not perfectly defined, i.e. the groups obtained had fuzzy borders. However, the relationship between the clusters and collaboration can be deduced, according to the L column values in Table 2 (cluster-2 has the lowest values of L and cluster- 0 the highest). Thus, the clusters were related to SC, although with uncertainty. From these results we can confirm that cluster-0 collects students with a low interaction and collaboration level (i.e. D-II N_m = and L = 4.51 in Table 2), cluster-1 collects students with a medium interaction and collaboration level (i.e. D-II N_m = 34 and L = 3.80 in Table 2), and cluster-2 collects students with a high interaction and collaboration level (i.e. D-II N_m = and L = 3.57 in Table 2). Although the clusters were not perfectly defined, there are no overlaps between cluster-0 and cluster-2. Thus, it can be deduced that a member in cluster-0 is not a student

18 A. R. Anaya, J. G. Boticario with a high collaboration level and a member in cluster-2 is not a student with a low collaboration level. In Table 2 we can observe that there are no noticeable differences between the non-filtered dataset (D-I- ) and filtered dataset (D-II- ). The teams whose members seldom communicated were mainly collected in cluster Metric approach With the clustering approach it is possible to group students according to their collaboration using labels that provide information on the collaboration level (high, medium, low). With these levels, collaborative learning can be improved, as will be discussed later on. However, additional, more specific information on collaboration can be inferred, so that students can compare themselves with one another. We propose an approach defined as a metric that provides the value of the student collaboration. Therefore, the metric variables must be related to this collaboration so that the metric can provide assessments on the collaboration level. For this reason, the metric variables that we used were the statistical indicators of student interactions in forums. The problem was to select a specific small set of relevant indicators as the metric variables. The method to select the variables was as follows: 1. Building datasets with the statistical indicators of student interactions in forums for every experience. These datasets were the same as those built for the clustering approach, but every instance of the dataset was labeled using the expert-based analysis. 2. Running a set of decision tree algorithms to identify the most common indicators in the classification learnt by the algorithms. We explain below why decision tree algorithms were selected. 3. Selecting the most common indicators as the metric variables. This approach was implemented with the student interaction data from the three collaborative learning experiences. We were looking for a method that could identify the most appropriate indicators to develop the same classification. ML certainly offers different technologies to classify instances, such as decision tree algorithms, which support the classification with logical trees. These logical trees provide information on the instance attributes used in the classification, and this was a required support for this approach. We needed a method that provided information on the relationship between the instance attributes and the instance labels. In our research the instances were student interactions, the instance attributes were the statistical indicators and the instance labels were the labels from the expert-based analysis. A decision tree algorithm offers a logical tree that proposes attributes ordered logically to learn the classification given by the labels. Thus, the logical tree can connect the attributes (statistical indicators) to the label (collaboration label by the expert-based analysis). Only Gaudioso et al. (2009) used decision tree algorithms to research student interactions, which is an objective of our research. An example, from those obtained in this research, is the logical tree produced by the algorithm REPTree, fast decision tree learner algorithm provided by WEKA, with the dataset D-II As can be seen in Fig. 3, the algorithm REPTree

19 Content-free collaborative learning modeling Fig. 3 REPTree logic tree used just three quantitative statistical indicators to classify the instances according to their collaboration level (L_msg, N_reply_msg and N_thrd). Other technologies could have also been used, such as bagging (Breiman 1996). Bagging uses a grouping meta-learning algorithm whose aim is to improve the automatic learning of classification and regression models in terms of stronger certainty and stability. However, the bagging method was not used here because the aim of our research was not to improve the model for classifying students according to their collaboration, but to obtain an explicit relationship between the dataset statistical indicators and student collaboration. Instead of choosing only one decision tree algorithm, the selection method used a set of algorithms. Every decision tree algorithm has a bias inherent to its functioning. Thus, our analysis approach considered a number of decision tree algorithms large enough to minimize the bias problem. The decision trees used were: Best first decision tree, DecisionStump, Functional trees, J48, Logistic model trees, Naïve Bayes tree, Random tree, REPTree, Simple Cart. These decision tree algorithms were used with the datasets because they provided a logical tree that showed the indicators used. This constraint was not obeyed by all the decision tree algorithms provided by Weka. We selected the algorithms that could be used with most of the datasets in our research. Then, the decision tree algorithms were trained with the datasets and the number of algorithms that used a specific indicator was measured. Finally, a list was built with this number of algorithms for every dataset. We proposed three different mathematical methods to deduce which indicators had been used more frequently. The mathematical methods are additions. Thus, three different additions were used to identify those indicators most relevant to collaboration: a normal addition (Addition I, which adds the number of uses in the datasets), a normalized addition according to the maximum value in the same dataset (Addition II, the number of uses was divided by the maximum value), and a normalized addition according to the number of algorithms that were run in the same dataset (Addition III, the number of uses was divided by the number of algorithms running with that dataset). We note that some decision tree algorithms did not run with some datasets. Figure 4 shows the number of uses for each indicator according to the three different additions. For instance, the same quantitative statistical indicator, L_msg, obtained the highest values according to the three different additions (Addition I L_msg = 30, Addition

20 A. R. Anaya, J. G. Boticario Fig. 4 Statistical indicator use according to different additions II L_msg = 4.25, Addition I L_msg = 5.17). We note that the six most used indicators were, in descending order: L_msg, N_reply_msg, L_thrd, N_thrd, M_reply_msg, N_msg. Three metrics with the most used indicators over the 3 years were thus proposed. A metric is a function that defines a distance between the elements in a set. The idea is that the distance, which the metric measures, is related to SC. The metrics were: - Metric I = (L_msg/max(L_msg)) + (N_reply_msg/max(N_reply_msg)) + (L_thrd/ max(l_thrd)). This metric measures SC using the regularity of activity (L_msg), regularity of initiative (L_thrd) and student acknowledgment (N_reply_msg) indicators. - Metric II = Metric I + (M_reply_msg/max(M_reply_msg)). This metric measures SC using the acknowledgment average indicator (M_reply_msg) and Metric I indicators. - Metric III = Metric I + (N_thrd/max(N_thrd)). This metric measures SC using the student initiative indicator (N_thrd) and Metric I indicators. The metrics were normalized so that all indicators had the same importance in the metrics. However, the relationship between metrics and collaboration was not established. We used the expert-based analysis to prove the relationship. Students were grouped according to the label assigned by the expert-based analysis. Then, the metrics were measured for all students in every group and the metric average and variance were calculated for every group. Figure 5 presents the metric average for each group (collaboration level) in the dataset D-I Figure 5 shows the metric average for a group of students who were labeled with the same collaboration label according to the expert-based analysis. The maximum variance values were 0.18 (metric I), 0.22 (metric II) and 0.32 (metric III). We can observe that the group with the highest collaboration level (2) has the highest metric values,

21 Content-free collaborative learning modeling Fig. 5 The metric average according to the collaboration level assigned by the expert-based analysis and the group with the lowest collaboration level (8) has the lowest values. This same behavior can also be observed in all datasets. Thus, the relationship between metrics and collaboration was identified. Owing to variance, it can be deduced that a student with low metric values should be a bad collaborative student and a student with high metric values should be a good collaborative student. The results are approximate but suffice to improve SC (see Sects. 4 and 5 below), and the method offers flexible and fast results that can be used in other environments Comparative study Both approaches (i.e. clustering and metric) can infer approximate collaboration assessments, but a method to compare them should identify the most appropriate ones in our research. The lack of comparative studies of different collaboration analyses (Bratitsis and Dimitracopoulou 2006) has already been noted. We propose two variables to compare both approaches: - The variable ( ), which is the difference in average values of the collaboration levels, to discriminate or differentiate between two consecutive inferred collaboration levels. This variable indicates how different or similar an inferred collaboration level is, compared with another inferred collaboration level. As the difference between levels is more noticeable, we can say that the approach can identify the different levels of student collaboration more accurately. In other words, ( inference is more predictive. Mathematically, it can be represented as n 1 ) = i (x i+1 x i )/n 1, where n is 8 in the metric approach and 3 in the clustering approach, i takes values in the metric approach from 2 to 8 (level

22 A. R. Anaya, J. G. Boticario Table 3 Average difference and variance % for the different approaches Metric I Metric II Metric III Clustering E% was used only once and 9 was never used) and in the clustering approach from 1 to 3, and x are the averages of the inferred collaboration level for each level i. - The error (E%) is the average percentage of the variance for the metric value in each instance. Mathematically, it can be represented as E% = ( n i (variance i / average i )/n) 100, where n is 7 (number of level used) in the metric approach and 3 (number of cluster) in the clustering approach, i is the level and is assigned values ranging from 2 to 8 in the metric approach and from 1 to 3 in the clustering approach. Table 3 presents the average differences ( ) between levels according to the metric and the clustering method, in addition to the error (E%). *It can be observed that when the approach using clustering techniques is applied the results are divided into three levels and the metric results are divided into seven levels. In Table 3 the scale was changed for clustering (from three to seven levels) to compare this approach with the metric one. It can also be observed that the measurements were calculated with the results obtained over the 3 years. Obviously, the best approach will be the one with the largest difference ( ) between levels and the least error (E%). The conclusions obtained from comparing both approaches are as follows: - The metrics had less error than the clustering approach and their results were as predictive as those of the clustering approach. - A student s characteristic degree of regularity of activity (L_msg) and regularity of initiative (L_thrd) (both indicators used in the metrics) identified SC more accurately than the degree of activity (N_msg, which characterizes the clusters in the clustering approach). - A student s characteristic degree of activity generated (N_reply_msg, which is used in both approaches) is also an appropriate indicator of SC. From the results we deduce that the metric approach is more appropriate for the aims of our research and that in similar collaborative learning environments, regularity of student initiative and activity should be fostered to attain higher collaboration. 3.5 The collaboration model A model in the context of modeling systems is an object where useful information is stored to achieve the objective of the system (Kobsa 2007). To develop collaboration modeling systems several models have been proposed: constraint-based models (Baghaei and Mitrovic 2007), structured models to obtain inferences (Redondo et al. 2003; Duque and Bravo 2007), and models where students and teachers can do the

23 Content-free collaborative learning modeling evaluation (Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006). We have noted that most collaboration modeling systems display the student model information to students or teachers using appropriate tools (Redondo et al. 2003; Duque and Bravo 2007; Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006). We proposed following a similar strategy and, accordingly, the student model, which in our case includes information derived from data mining, should be easy to understand for students and teachers alike. The pedagogical advantages of the tools developed with such a modeling approach, whose usage will be discussed below, should increase according to the open model strategy, which sets up student models so that students can use and manage their own models (Bull and Kay 2008). The model s objectives are to match the model s format and structure. Some researchers have proposed using ontologies, because they structure the contents in an understandable, stable and transferable way (Heckmann 2006; Mizoguchi 2005). Ontologies have been used to model students in other educational environments, for instance, Barros et al. (2002). Our research aimed to support the understandable, stable and transferable features of ontologies. Ontologies are usually configured to enable an inferring system to operate with them and they can be designed to enable students and teachers alike to understand them. Moreover, the model s format may prevent it from being used correctly in other environments. For instance, it may be wrongly interpreted. OWL-LD has been used by other modeling systems that considered educational environments (Denaux et al. 2005; Huang et al. 2005). Further, OWL is used extensively to model the Web environment, and interpreters can be easily developed. The purpose of the collaboration model developed in our research was to enable students to understand their own collaboration behavior and that of their fellow students in order to improve collaboration process management. The collaboration model that we propose is an OWL-LD-based ontology. The information collected was structured so that students were able to understand the contents and navigate across them. We used the Protégé application ( to build the model. Since the collaboration model had to be usable and understandable, we propose a collaboration model with a hierarchical structure, where the information is grouped into classes according to their content. The collaboration model structure is shown in Fig. 6. The hierarchical structure of the SC model can be seen on the left-hand side of Fig. 6. As aforementioned the SC model consists of context, process, and assessment information. On the right-hand side the class User properties and attributes are displayed. The properties link to other classes to navigate across the model from the class User. In particular, the attributes name and store class User information. The other classes collect the defined types of information. Some classes group information on the collaboration context or circumstances. These classes hang from the class Static_Data, because the information does not usually change. This information was asked at the beginning of the collaborative learning experiences in the initial questionnaires. The classes store personal data (the class Personal), academic data (the class Academic), working data (the class Working) and study preferences (the class Preferences). The information on the collaboration process is stored in the

24 A. R. Anaya, J. G. Boticario Fig. 6 Collaboration model structure Fig. 7 Collaboration model class of the collaboration assessments collaboration model by classes that hang from the class Dynamic Data, because this information changes due to interactions. The student interactions that we used to analyze SC were the forum interactions, which are described in the class Forums. We proposed twelve indicators to describe the collaboration process. When the aims were to display the indicators, other researchers selected a small group of indicators so that students or teachers could use them to evaluate student collaboration (Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006). A large number of indicators might affect student comprehension and appropriate management. For this reason, and taking into account the data analysis performed over the first 2 years of the experience, we select only four statistical indicators to model the collaboration process: N_thrd, N_msg, N_reply_thrd and N_reply_msg. These indicators are stored in two classes, total and week, depending on whether the indicator values represent the total period of the collaborative learning experience or just 1 week. Figure 7 presents the assessments of student collaboration. In other words, the information inferred with DM processes from student interactions. The class Collaboration collects two types of assessments. The assessments of collaboration evidence is collected by the attribute collaboration_level, which represents a qualitative indicator that the clustering approach infers, and the attribute collaboration_grade, which represents a quantitative indicator that the metric approach infers.

25 Content-free collaborative learning modeling We proposed a collaboration model defined as a container of suitable information on student collaboration. Since students and teachers were the users of the collaboration model, the structure and attributes were created so that they could be easily understood. 3.6 Using the collaboration model In collaborative learning environments, the tools using the model can be divided into monitor tools, meta-cognitive tools and guide systems (Soller et al. 2005). Monitor tools display information on the student (Bratitsis et al. 2008; Martínez et al. 2006; Daradoumis et al. 2006) but do not offer collaboration analysis results. Therefore they cannot ensure that collaborative learning takes place (Johnson and Johnson 2004). Only one research study was found which proposed a guide or recommendation system (Baghaei and Mitrovic 2007) to achieve the research objectives, but the researchers did not analyze collaboration. In addition, the use of these researchers model in a different environment requires a collaboration analysis process to build a specific recommendation model. The literature acknowledges the advantages of using meta-cognitive tools, which must show the results of the collaboration analysis in collaboration environments (Dimitracopoulou 2009). This idea is compatible with the open model strategy (Bull and Kay 2008) and both share the same aims, namely improved learning through reflection of learners self-related features. In this research we propose four tools: one monitor tool (Portlet I), which showed collaboration process information, and three meta-cognitive tools. The first metacognitive tool (Portlet II) displayed only the collaboration assessments, the second meta-cognitive tool (Portlet III) presented collaboration process information and assessments, and the third meta-cognitive tool (Web Application) provided the collaboration model that students could manage, i.e. it supports all the features of the open model strategy. We propose four tools to identify the most useful collaboration information (collaboration process or collaboration assessments) and the usefulness of the scrutable strategy. The portlets were small windows displayed on one of the team s virtual space pages. These windows were therefore perfectly integrated into the dotlrn platform, whose interface is portlet-based. The information in all portlets (Portlet I, II and III) was updated once a week, every Monday during the 2nd phase of the collaborative learning experience. The rationale behind updating every Monday was to offer students meaningful information on their processes on a regular basis, thereby meeting the requirements of frequency and regularity that a collaboration analysis should have (Johnson and Johnson 2004). An example of Portlet III, which showed the information displayed in Portlet I and II, is depicted in Fig. 8. Information on student indicators from forum interactions on a weekly basis (the student Mariano Paredes sent 5 messages in the first week) and the total value of those indicators (Mariano Paredes sent 28 messages) are displayed in the tables on the right-hand side of Fig. 8, which were similarly displayed in Portlet I. The SC levels (Mariano Paredes had medium collaboration level), which were inferred by the clustering approach, are shown in the table on the right-hand side of Fig. 8, which were similarly displayed in Portlet II.

26 A. R. Anaya, J. G. Boticario Fig. 8 Example of Portlet III, translated from Spanish into English The Web Application, which presented the collaboration model and allowed students to manage the collaboration context information, was not integrated into the web platform dotlrn so that other environments could use it. However, students had to log in to use it. Thus, the web application was not as easy and immediate to use as the portlets, which were set in the private spaces of the e-learning platform, enabling students to use and see them easily. The Web Application was absolutely new and students had to quickly become familiar with how to use it. Figure 9 displays the main page of the Web Application. As presented in the left frame, the application structures the classes containing information on the collaboration context, the collaboration process and the collaboration assessments hierarchically. The class currently navigated is marked in green, in this case the class User. In the central frame the instances contained by the class User are visible, in this case, those users who are allowed to navigate across the application. The instance that the user is navigating at this moment is marked in green (nuria-fuentes-11). The right frame shows the instance data, in this case, the data referring to context such as , name, the student s team (Team_25), a link to academic details, data relevant to the process such as the links to the student s weekly interaction statistics, and the data referring to the inferred information, i.e. the collaboration assessments (Medium). Users, as owners of the information, i.e. to whom the information refers, can edit the collaboration context information if they select the link in the top right-hand corner of the right frame (Edit link). 4 Experiment The DM processes were implemented with the data generated by the collaborative learning experiences over three consecutive academic years , and to verify that the inferred values were meaningful as collaboration

27 Content-free collaborative learning modeling Fig. 9 Scrutable Web Application screen showing the data for one sample of the class User assessments. The results described in the section entitled Comparative Study prove that the DM process collaboration assessments are related to student collaboration, although with some degree of uncertainty. In the academic year collaborative learning experience, all parts of the collaboration modeling were completed and we proposed an experiment to test the approach. The experiment focused on the 2nd phase of the collaborative learning experience, where 112 students took part from to The students who participated in the 2nd phase were monitored. Their answers to the initial questionnaire were used to fill the SC context in the collaboration model. The students worked in groups to do the different tasks. The forums were a useful communication means where fellow students could collaborate. Their interactions in team private space forums were used to measure the statistical indicators. The students were grouped into teams and the teams were selected randomly so that between six and nine teams used each of the tools, leaving just a few teams without any tool as the control group (Kirk 1995). The number of students who were offered the tools and participated in the control group is shown in Table 4. SC assessments were inferred after the third week of the 2nd phase so that there were enough data. The 2nd phase allowed teams to start in two different but consecutive weeks. The collaborative learning experience had