Recommending Collaboratively Generated Knowledge

DOI: 10.2298/CSIS111129017C Recommending Collaboratively Generated Knowledge Weiqin Chen 1,2 and Richard Persen 1 1 Department of Information Science and Media Studies, University of Bergen, POB 7802, N-5020 Bergen, Norway {weiqin.chen, richard.persen}@@infomedia.uib.no 2 Faculty of Technology, Art and Design Oslo and Akershus university college of applied sciences POB 4, St. Olavs plass, NO-0130 Oslo, Norway Abstract. With the development and adoption of information technologies in education, learners become active producer of knowledge. There is an increasing amount of content generated by learners in their learning process. These emerging learning objects (ELOs) could potentially be valuable as learning resources as well as for assessment purpose. However, the potentials also give rise to new challenges for indexing, sharing, retrieval and recommendation of such learning objects. In this research we have developed a recommender system for emerging learning objects generated in a collaborative knowledge building process and studied the implications and added values of the recommendations. We conducted two evaluations with learners to assess and improve the system s design and study the quality and effects of the recommendations. From the evaluations, we received generally positive feedback and the results confirm the added values of the recommendations for the knowledge building process. Keywords: Recommender systems, collaborative knowledge, knowledge building, emerging learning objects. 1. Introduction Traditionally, learning objects refer to resources that are created mainly by teachers or course designers. These learning objects are mostly selfcontained and vary in granularity. For example, a course, a simulation, or a piece of text can all be learning objects. Learning objects can be aggregated into a larger collection of content. Substantial effort has been made in annotating (with metadata), sharing, and reusing these learning objects. The benefits of reusing and sharing learning objects have been studied intensively in the last decade [27, 31, 35]. Recommender system for learning objects or learning materials have been developed [2, 42, 52, 54]. These systems are based on traditional learning objects which are either standard-compliant or

Weiqin Chen and Richard Persen have been annotated by teachers or learners. In addition, because of the educational potentials that information on the Internet presents, efforts have been made to take advantage of the online information. For example, some systems directly search the Internet using keywords for learning material recommendation [40]. Some use data mining techniques to automatically extract meta-data and relations among learning objects [54]. When it comes to evaluation of the systems, the main focus has been on the algorithm and technical measurement. Few of the systems for recommending learning objects have been evaluated through trials with human learners although the importance of such evaluation has been highlighted [34]. Subjective criteria (e.g. usability) in standard human-computer interaction research are rarely applied to evaluate adaptive systems including recommender systems [34, 53]. With the development and adoption of digital technologies in learning, there is an increasing amount of material generated by learners in their learning process. For example, in a knowledge building process, there are a large number of messages posted by learners and learning groups, including problems, hypothesis, and scientific evidence [48, 49]. In an inquiry learning process, learners generate many different materials (data collected, pictures taken, models created, and hypotheses generated) [19, 23]. In contrast to traditional learning objects, these learner-generated contents, also called emerging learning objects, are not pre-fabricated or authored [23]. They represent the growing knowledge and experience of learners and groups. The emerging learning objects could potentially be valuable as learning resources as well as for assessment purpose. For example, learners can build new knowledge based on existing learning objects created by themselves or other learners. Adaptive learning can be generated based on these objects. However, the potentials also give rise to new challenges for indexing, sharing, retrieval and recommendation of such learning objects which call for intelligent support for dynamic annotation and modelling of learning contexts on a semantic level [23]. In addition, it is essentially important to study the effect of these emerging learning objects on the learning process and outcome [17, 41, 55]. 2. Recommender Systems This section presents an overview of general recommender systems and recommendations in learning environments and describes the methods for creating recommendations. Special attentions are paid to the systems that recommend emerging learning objects. 872 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge 2.1. General Recommender Systems Recommender systems have been an important research area over the last decade [44]. The aim of recommender systems is to help users to deal with information overload and provide personalized recommendations, contents and services [1]. In recent years a number of recommender applications have been implemented and applied in different fields. For example, Amazon recommends books, CDs and other products to the users [30] and Google News recommends news stories to readers. Some systems recommend research papers to the users based on the users' profiles [5, 36]. In order to provide recommendations to users, it is crucial for the systems to represent the user behaviors and the information about the items to be recommended. Many of current recommender systems include a user profile (user model) that contains information about the user's tastes, preferences and needs. This information can be elicited from the users explicitly, e.g. through questionnaires or user's rating of items, or implicitly, e.g. learned from the user's activities over time. In addition, contextual information has also been incorporated into the recommendation process. Based on the methods used, recommender systems are usually classified into three categories: Content-based recommendation recommends items similar to those the user preferred in the past [4]. Collaborative recommendation recommends items that other people with similar tastes and preferences liked in the past [28, 45]. Hybrid recommendations combine the content-based and collaborative methods [10, 37]. Early research in adaptive hypermedia [9] focused mainly on contentbased adaptation, which is to adaptively present a selection of links or content most appropriate to the current user. Part of this branch of study can be considered alternatively as content-based recommendations when the users' past reading items/pages are recorded and analyzed. 2.2. Recommendations in educational settings Some of the techniques in recommender systems have been adopted for educational purposes. However, making recommendations in learning environments is different from that in other domains (e.g. movies, news) because of the particular pedagogical considerations [15, 51]. Learning objects/items liked by learners might not be pedagogically appropriate for them. Therefore a recommendation based purely on the learner's interest may not be suitable from pedagogical point of view. Other factors such as the learning context should be taking into consideration as well. In learning environments, the recommendation mechanisms focus on providing personalized advice to the learners which are pedagogically suitable to the learner and learning activities. Some systems can recommend knowledgeable people or possible collaborators who share similar interests or are working on similar topics or problems [3, 6]. Some systems provide ComSIS Vol. 9, No. 2, June 2012 873

Weiqin Chen and Richard Persen learners with navigation support or advice on suitable learning activities to follow [15, 39]. Some systems recommend learning resources [34]. Santos [47] defined eight types of recommendations in lifelong learning including motivation, learning style, technical support, previous knowledge, collaboration, interest, accessibility and scrutability. An educational recommender system normally keeps a learner model which includes the learner's preferences, learning history and current learning activities and context. Recommendations can be initiated by the learner implicitly or explicitly. For example, a learner is carrying on his/her learning activities in a certain location. The system presents context-based recommendations. The recommendations are triggered by the activity of the learner. A learner can also explicitly require recommendations from the system by e.g. initiating a query for possible collaborators. Among other elements in learner model, recommendations of learning objects are mainly based on three types of information about learning activities: context: Current location of the learner or learning activities. Many learning environments with handheld devices provide just-in time recommendations (learning objects and/or peers) based on the learner's location and activities [12, 26]. For example, descriptions of a painting are given to learners on their PDAs or mobile phones when they walk towards the painting in a museum. similarity: The ranking of the learning objects by the learner. The learning environment analyzes the rankings and builds a model of the learning objects based on the ranking [29, 51]. The environment then recommends relevant learning objects (based on clustering of objects) and possible collaborators (based on clustering of learners). topic: The current topic on which the learning activity is focused. By analyzing the current learning activities, the learning environment can identify the topics in focus. These topics are used to retrieve relevant learning objects as recommendations [20, 56]. The learning objects are classified into different categories using clustering techniques beforehand. For an extensive review of recommender systems in technology-enhanced learning, see [34]. 2.3. Recommending Emerging Learning Objects With the rapid growing access of teachers and learners to the Internet, Web resources are becoming an important aspect in educational settings. In order for such resources to be used in a productive, educational relevant ways, recommenders systems have been developed to provide adaptive support based on web resources [16]. According to [9], the Web educational resources or dynamically expanding educational repositories fall into the open corpus category where it is not possible to manually structure and index them with domain concepts and metadata. Traditional learning objects 874 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge which are annotated and maintained systematically by teachers, designers and institutions can be considered as in the category of closed corpus because the semantic description of these objects are predefined. Lying between the open corpus and closed corpus is the emerging learning objects which are created by learners or learning groups for a certain learning goal/task with a certain tool and in a certain learning environment. On the one hand, these objects are growing in volume and quality and they are not annotated or indexed at design time. On the other hand, they carry with them context information as part of its meta-data. Therefore emerging learning object can be considered as semi-open corpus. Efforts have been made in providing recommendations based on learner generated content in a community setting [16, 25, 33, 43]. Tang and McCalla [51] describe a system that recommends research papers based on learners' explicit rating of the papers. ReMached [16] mashes data of learners from various Web2.0 services in order to recommend learning resources. The majority of the systems are based on tags and ratings that learners provide. This implies the use of collaborative filtering or hybrid method for recommendations and requires explicit feedback from the learners. Explicit feedback could be difficult to obtain, so data mining techniques have been developed to mine the usage data [25]. Another line of research in recommending learner generated content is based on the current topic on which the learning activity is focused. Ye [57] describes the CodeBroker which can predicate the learner's information needs and recommend code examples/software components created by other learners based on the task being performed, the knowledge of the learner performing it and the information used. Hartmann and colleagues [20] developed HelpMeOut, a social recommender system that aids the debugging of error message by suggesting solutions that peers have applied in the past. 2.4. Recommending Collaborative Knowledge in Educational Discussion Forums Discussion forums and bulletin boards have been widely used in web-based education and computer supported collaborative learning (CSCL), in order to assist learning and collaboration. Learners use discussion forums to discuss course-related issues, such as topics in their courses, learning tasks, and projects, etc. These discussion forums include questions and answers, examples, articles posted by former learners, and thus they are potentially useful for future learners [22]. There are different variations of the educational discussion forum based on different pedagogies for collaborative learning. For example, some require learners to specify categories for their messages, and others use sentence openers to help learners with scientific thinking and message writing. By identifying relevant messages and reusing ComSIS Vol. 9, No. 2, June 2012 875

Weiqin Chen and Richard Persen them as new learning resources, future learners can benefit from former learners knowledge and experiences. However, it is not an easy task to identify relevant information from discussion forums given the thread-based structure of them. Messages posted in a discussion forum are usually organized as a tree structure with each branch as a thread. In each thread, the messages are presented in a temporal sequence. It is usually not so easy to decide whether the message is relevant by looking at the title alone, because it is not always informative. It is possible to use a full text search within the discussion forum based on keywords. However, there are always some irrelevant messages that are included in the search result. The modern information retrieval techniques and methods [50] are rarely adopted in the search for information in discussion forums. A few efforts on indexing and reusing the messages in educational discussion forums have been made. The main method is to create a predefined structure for a discussion forum, where the structure reflects a conceptual schema of the subject domain [14]. Helic and his colleagues [21, 22] described a tool to support conceptual structuring of discussion forums. They attached a conceptual schema to a discussion forum, and the learners had to manually assign their messages to the schema. Their study shows some limitations with this method. First, some messages could be assigned to more than one concept in the schema. Second, the learners were not motivated enough to make the extra effort in assigning their messages to concepts, although it may have been beneficial to those learners to do so. Our own experience confirms the second point. We developed a plug-in for FLE3 (see section below), where students could choose relevant topics when preparing their messages, but they could also chose to ignore this feature. Very few students made use of this function to specify relevant topics for their message. In our research we have developed a recommendation system, AnnForum, which can identify relevant existing message from threaded discussion in FLE3 and recommend them to learners. For each message in current or previous discussion, the system associates it with a certain topic (defined in a domain model) using text analysis and clustering techniques, with each association a value describing the relevance between the message and the topic. This process can be automatic or semi-automatic where the teacher is provided with a tool to check and manually specify the association generated by automatic mechanism. When a learner is reading or writing a message, the system identifies the current topic and retrieves relevant messages with a matching mechanism. The relevant messages (each with a relevant value) are provided to the learner as recommendations. The intention is to provide just-in-time recommendations that are meaningful to the current problem so that the learners can look at the problem from a different viewpoint, evaluate their own thoughts, and be inspired by and build upon the ideas in the recommendations. 876 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge 3. Collaborative Knowledge Building with FLE3 FLE3 [38] is web-based groupware for computer supported collaborative learning (CSCL). It is designed to support the collaborative process of progressive inquiry learning. The basic idea of progressive inquiry is that learners gain a deeper understanding by engaging in a research-like process where they generate their own problem, make hypotheses, and search out explanatory scientific information collaboratively with other learners. Fig. 1. Knowledge Building in FLE3 To support the collaborative progressive inquiry process, FLE3 provides several modules, such as virtual WebTop, a Knowledge Building module, and an Administration module. The Knowledge Building module is considered to be the scaffolding module for progressive inquiry, where learners post their messages to the common workspace according to predefined categories. The categories they can use are Problem, My Explanation, Scientific Explanation, Comment, and Summary. These categories are defined to reflect the different phases in the progressive inquiry process (Fig. 1). FLE3 has been used as a knowledge building tool in our Introductory Artificial Intelligence course for discussing issues such as what is intelligence and can machine think. Students follow the progressive inquiry ComSIS Vol. 9, No. 2, June 2012 877

Weiqin Chen and Richard Persen process and collaboratively build knowledge and understand the concepts in artificial intelligence. 4. Design and Implementation of AnnForum In order to recommend relevant messages from the knowledge building process, a conceptual domain model is constructed first. Based on this model, the messages posted in previous knowledge building processes are annotated and classified into different categories corresponding to different concepts in the model. The teacher is responsible for constructing and managing the domain model, as well as for validating (adding/removing) annotations. The messages (including those in the current and previous knowledge building process) that are relevant to the current topic under discussion are gathered and presented to learners. The learners can then read through the messages and rank them according to their degree of relevance. Fig. 2 shows the use cases for the recommender system. Construct Domain Model View ranked list of relevant messages Teacher Manage Domain Model Learner Read relevant messages Manage Annotation Vote for relevant messages Fig. 2. Use cases The following two scenarios explain how AnnForum is used by learners. Scenario 1: A learner writing a new message After the learner has finished writing their message, it is submitted and appears in the knowledge building interface. AnnForum automatically annotates and classifies this message based on the domain model. In the meantime, it finds a list of existing messages that are relevant to the learner s message by computing the relevant values. The ranked list of relevant messages is presented to the learner. The annotation and relevant values are stored into AnnForum s database. Scenario 2: A learner reading an existing message in the knowledge building interface When the learner is reading the message, they click on a button called show relevant messages. The relevant interface appears, containing a ranked list of relevant messages. These messages are retrieved from the database. 878 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge 4.1. Main Components Fig. 3. Main Components in AnnForum AnnForum is a plug-in to the FLE3 environment. It is a domain-independent tool. As shown in Fig. 3, AnnForum takes the domain model and the messages as input, and puts the classified messages into the database. When a new message comes, the Classification module decides its relevant topics. Then it searches for the relevant messages in the database, computes the relevant values based on the relevance of the messages, and stores them in the database. The Controller for relevant messages module retrieves the relevant messages and sends them to the interface in FLE3. The learners can then read through the messages and rank them according to whether they think it is relevant or not. In the current design, the learners do not need to explicitly associate their message to a topic in the domain model. However, if they are willing to do so, they can use the choose topics function when preparing a message, which is provided by the plug-in to FLE3. The Controller for feedback module learns from the feedback of the learners and adjusts the weights used in the matching algorithm accordingly. ComSIS Vol. 9, No. 2, June 2012 879

Weiqin Chen and Richard Persen 4.2. Conceptual Domain Model and Annotation of Messages A conceptual domain model is used to describe the domain concepts and the relationships among them, which collectively describe the domain space. This domain model is usually represented by an ontology. A simple conceptual domain model can also be represented by a topic map. Topic Maps ISO/IEC13250 [24] is an ISO standard for describing knowledge structures and associating them with information resources. It is used to model topics and their relations in different levels. The main components in Topic maps are topics, associations, and occurrences. The topics represent the subjects, i.e. the things, which are in the application domain, and make them machine understandable. A topic association represents a relationship between topics. Occurrences link topics to one or more relevant information resources. Topic maps provide a way to represent semantically the conceptual knowledge in a certain domain. Fig. 4. Topic and association management In AnnForum we use a topic map to represent the domain model of Artificial Intelligence. This domain model includes AI topics and their relations such as machine learning, agents, knowledge representation, searching algorithm, etc. These topics are described as topics in the topic map. Relations between these topics are represented as associations. The occurrence describes the links to the messages where the topic was discussed in the discussion forum. The occurrence is generated by the automatic classification algorithm presented in next subsection. 880 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge In the earlier prototypes of the system, teachers had to write XML in order to create Topic maps for their course domains and when a message is posted, associated topics to this message have to be selected manually by the contributors (learners/teachers). These have been proved rather tedious. In the newer versions AnnForum provides a graphical interface for teachers to create a domain topic map interactively (Fig. 4). Using AnnForum, teachers can create Topic maps for their course domain and load /reload them into FLE3. Because the topic map is written in XML format (XTM), it is easy for teachers to understand and maintain the topics, and the domain model can also be easily reused in other contexts. Fig. 4 also shows the associations between the messages and the related topic ( machine learning ) using automatic classification techniques. Teachers can also use this tool to edit and verify the associations. Fig. 5. Manual annotation of messages Fig. 5 shows an interface where teachers can manually associate messages to topics. The left panel shows the threads of the knowledge building forum. The * in front of the message title means that this message has been classified automatically. In the right panel, teachers can view the information for each message in the discussion thread (including title, knowledge building category and content) and the topics this message is related to. In the message content, the identified topics, such as Turing Test was listed with a relevance value. They can also add or remove the related topics by clicking on the buttons at the bottom of the right panel. ComSIS Vol. 9, No. 2, June 2012 881

Weiqin Chen and Richard Persen 4.3. Message Classification Since the messages can be seen as a kind of document collection, we investigate the methods for classifying documents from a document collection to a conceptual model. There are various approaches in information retrieval that deal with the problem of document classification. For example, ontology-based classification of unstructured/semi-structured documents goes beyond the use of keywords and classifies documents into categories that are meaningful to users [11, 13]. In AnnForum we use an approach that combines a conceptual model (represented by a Topic Map) and a Vector Space Model based classification to determine the relevance of a message to a concept in the domain model. It could be considered a simplified ontology-based text classification because in our Topic Map, the associations between concepts are limited to is-a relations. The core of our approach can be seen in three steps: 1. Generating term vector Vt: We use Apache Lucene to create the TF- IDF (term frequency-inverse document frequency) weighted term vector which consists of words and phrases in the document ordered by their relative importance [46]. For example, a message (id: m-26) in the knowledge building process has Vt: <Turing test: 0.043, production rules: 0.025, machine learning: 0.024, color: 0.009 > 2. Mapping the term vector Vt to Topic Map: For those terms that are in both Vt and the Topic Map, they form a new term vector Vtt. The basenames and variants of the basenames in the Topic Map are used to normalize the vector to account for synonyms. Vtt for message m-26 becomes <Turing test: 0.043, production rules: 0.025, machine learning: 0.024, >. Color with its weight has disappeared from the vector because it is not in the Topic Map. 3. Enhancing the term vector by the associations in the Topic Map and adjusting the weights of terms in Vtt: For every term Ti in Vtt, we use this term to find the associated topics in the Topic Map. If an associated topic Tj is not already in Vtt, Vj will be added to Vtt and the weight for Tj is 25% of the weight of Ti. If an associate topic Tj exists in Vtt, both the weight of Ti and Tj will increase 50%. Vtt for message m-26 becomes <Turing test: 0.043, production rules: 0.025, machine learning: 0.024, knowledge representation: 0.013, >. Knowledge representation has been added to the vector because it is associated with production rules in the Topic Map. Note that the weight adjustment factors (25% and 50%) are rather arbitrary in the current version and some heuristics and empirical data will be used in the future to fine tune these values. The classification results are stored in a MySQL database. The database includes both the messages (title, author, timestamp, and thread information), and the domain topics they are related to, with values of relevance. 882 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge 4.4. Relevant Message Recommendations and Learner s Feedback The Controller for relevant messages module in AnnForum retrieves the relevant messages and sends them to the interface in FLE3. Fig. 6 shows the interface where learners get the relevant message recommendations. They can click the title link to read the message. After reading they can also vote for or against the messages using the Thumb up/down buttons. The voting will affect the relevance value later. The Controller for feedback module adjusts the weights based on the votes. Teachers can also use the vote function to change the relevance value. Fig. 6. Recommendation interface 5. Evaluation We have conducted two formative evaluations with improvement in between. In contrast to most recommender systems which evaluate the algorithms, our evaluations focus mainly on subjective criteria based on principles in human computer interaction. 5.1. Formative Evaluation 1 This evaluation focused mainly on usability issues. It had three main goals: to assess the extent of the system s functionality, to assess the effect of the ComSIS Vol. 9, No. 2, June 2012 883

Weiqin Chen and Richard Persen system on the learner, and to identify any specific problems with the system. More specifically, the evaluation aims to answer the following three questions: How well does the system annotate and classify existing messages based on the domain model? How useful are the relevant messages to the learners? What improvements need to be made to the system? Six university students taking an information science major participated in the evaluation. All of them were familiar with the AI domain and had experience with discussion forums. The evaluation was carried out in a controlled environment where only one participant and one researcher were present. After a short introduction about the system, the participant was given a set of tasks to carry out. Data were collected by observation and interview after the tasks. The questions in the interview reflected the three goals of the evaluation. Before the evaluation, the researchers prepared two messages and posted them into the knowledge building module in FLE3. One of the messages concerned the Turing Test and the other concerned Machine Learning. Both are important topics in AI. These two messages were posted in the category of Problem, and served as the starting point for the discussion. Each participant was asked to read existing messages, use the relevant message interface to check out relevant messages, and post at least two messages of their own responding to existing messages. This way, the number of existing messages grew as the evaluation progressed the first participant had 2 existing messages to read and the last one had 10. This could be considered a simulation of a real knowledge building process, and it also made the dynamic nature of the system (annotating, classifying, and matching) more realistic. The current messages in the knowledge building module, as well as the 237 messages from previous knowledge building processes, were the source of the relevant messages. The data from observation and interview show that all participants were very positive toward the system, and saw the added value of the relevant messages in their knowledge building process. They used the relevance value, or a combination of relevance value and the title of the messages, to decide which recommended relevant messages to read. After reading some of the messages, all six participants thought the message with the highest relevant value was the most relevant, while the one with the lowest relevant value was not quite relevant. Some also used the thumb up and thumb down buttons to vote for the recommended messages they read. Half of the participants responded by stating that the relevant messages they read affected the formulation of their own messages. Perceived relevance of the recommended messages. The relevance values of recommended relevant messages were found to reflect the actual relevance to the current discussion. This indicates that the performance of the annotation, classification, and matching mechanism is acceptable. The automatic process is important because it saves learners from having to manually annotate their messages. 884 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge Perceived usefulness. The most positive aspects about providing the relevant messages include: The learners can build more in-depth knowledge about the discussion topic. The relevant messages provide them with different viewpoints. It gives the learners a feeling that the discussion is more alive, which motivates them to formulate good messages. The ranking list of the relevant messages is a better alternative than searching the discussion forum. It can reduce the possible duplication of information. Duplication of information is a problem in most big discussion forums. The dynamic nature of the relevant messages prevents earlier messages that are buried deep down in the thread from being ignored. Improvements. The evaluation also resulted in a few ideas for improvements: Allowing learners to see the thread to which the recommended message belongs. Allowing learners to see more information about each relevant message. In this version the system shows the title and the relevant value. The feedback from the participants indicates that showing a few opening sentences of each relevant message would help the learners to make a better judgment before going on to read the whole message. This could be implemented as a mouse-over event, which means the opening sentences of the message will be shown in a floating box near the title whenever the learner moves the mouse over the title of the message, and that the floating box disappears when the mouse is moved away from the title. Making the relevant messages interface a part of the FLE3 interface with the same look-and-feel. In this version the relevant interface is implemented as a pop-up window, which, according to the participants, disturbed the workflow. 5.2. Formative Evaluation 2 This evaluation focused on the quality and effect of the recommendations in the context of learning. It was conducted after further development based on the feedback. The system was evaluated throughout the fall semester with 35 Information Science students in the Introductory AI course. Data were collected by an online questionnaire with open end questions after the course. The questions reflected the main focus of the evaluation. Some of the questions used in the questionnaire are as follows: Which relevant messages you read or voted for/against? Why did you choose these to read or vote/against? Do you think the relevant value in percentage is correct for the messages you read? Why? ComSIS Vol. 9, No. 2, June 2012 885

Weiqin Chen and Richard Persen Have any of the relevant messages you read affect how you formulate your own messages? What do you think of the idea of recommending relevant messages? Before the evaluation, the researcher prepared two questions about intelligence and Turing Test and posted them into the knowledge building module in FLE3. These two questions served as the starting point for the discussion. Each participant was asked to read the messages posted by others, use the relevant message interface to check out relevant messages, and post at least four messages (two in each question) of their own responding to others' messages. The current messages in the knowledge building module, as well as the 249 messages from previous semesters' knowledge building processes, were the source of the recommendations. Perceived quality of the recommendations. The data from the questionnaire shows that the students used the relevance value, or a combination of relevance value and the title of the messages, to decide which recommended relevant messages to read. After reading some of the messages, about half (18) of the participants thought the message with the highest relevant value was the most relevant, while the one with the lowest relevant value was not quite relevant. This indicates that the performance of the classification mechanism is acceptable. About half of the participants responded by stating that the relevant messages they read affected the formulation of their own messages. Few (4 out of 36) used the thumbs up and thumbs down buttons to vote for/against the recommended messages they read because most of the students didn t notice the buttons. Effect of the recommendations. The data shows that more than half (20 out of 35) of the students were very positive toward the recommender, and saw the added value of the relevant messages in their knowledge building process. For example, to the question of what do you think of the idea of recommending relevant messages, here are some of the responses: This is a good idea, possibilities to inspire new ideas. Continuity is important both to access a first hand community history, as well as an opportunity to see what previous ideas and thought people had earlier. A student can learn a lot about the views of others. It is very important because it is information and knowledge that is acquired. The possibility to see relevant message is so useful It can help you to find a new ideas, to see if your previous colleagues think more or less the same of you or completely different. To the question of Have any of the relevant messages you read affect how you formulate your own messages, here are some of the responses: I read this message and he (the author of the message) remembered an article which I read some years ago, so I read the article again and I was able to put my ideas. Of course, I found anyone posted more or less the same as I intended, so I had to reevaluate my post and take in account the insights the post brings. One of the most positive aspects about the recommender is that the learners can build more in-depth knowledge about the discussion topic. The 886 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge relevant messages provide them with inspirations, different viewpoints and additional resources. 6. Conclusion and Future Work In this paper we have presented a recommender system for collaborative knowledge. It can be seen as a content-based recommender based on topics on which the learners are focusing. We have also conducted formative evaluations of the system. The general response is positive and provides a confirmation that the recommended messages may be useful. Further research is needed to judge the effect of the recommendations on learning. Content analysis based on logs of learners activities with timestamps may provide evidence on whether the recommended messages actually influence the formulation of the new messages. Moreover, qualitative evaluation of the algorithm can also provide us insight on whether the technical decisions are appropriate. Some possible improvements were also identified through the evaluation. One feedback is to allow learners to see the thread to which the recommended message belongs. This will give the learners context information regarding the message. Context information allows learners to have a feeling of presence, that is, that they are collaborating with previous learners [18]. In the current version in order to help the learners to make better judgment before going on to read the whole message, we show a few opening sentences of each recommended message. The approach presented in this paper is primarily for recommending messages in educational discussion forums. But it has implications for searching in traditional discussion forums and for organizational knowledge management. The problem with traditional discussion forum is that it is difficult to find useful information about a certain topic, especially when the number of messages grows. It becomes impossible to have an overview of threads. In addition, the titles of messages usually use RE: XXX and do not tell the users much about the content of the message. The results from keywordbased search are not always satisfactory. The method presented in this paper, including the dynamic annotation, classification and matching, will be able to help users in finding relevant information from traditional discussion forums by providing them a ranked list of relevant messages. This will also help to reduce the number of duplicated messages. Because the process is automatic, it does not give users overhead when they post messages. One important research area in knowledge management is to look for better ways to handle large amount of organizational information and knowledge so that it is easy to represent, organize, maintain, search and reuse them. Annotation and information retrieval have played important roles in knowledge management. We believe that knowledge management can benefit from our research in two aspects: ComSIS Vol. 9, No. 2, June 2012 887

Weiqin Chen and Richard Persen Dynamic annotation and classification allows each new piece of knowledge to be automatically annotated and classified immediately when it is stored in the organizational knowledge repository. Dynamic matching provides users with ranked list of relevant information and knowledge, which saves the users from having to formulizing queries by themselves. The approach presented in this paper can also be used for retrieval and suggestion of any unstructured/semi-structured documents as learning resources. Recently wiki becomes popular as a pedagogical tool to support learning [7] where new knowledge is generated collaboratively. The approach we have used in our research can also be used to analyze wiki pages created by learners and provide recommendations to the learners based on the topics they are working on. References 1. Adomavicius, G., Tuzhilin, E.: Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17, 734-749 (2005) 2. Al-Khalifa, H. S.: Building an Arabic learning object repository with an ad hoc recommendation engineproc. of the 10th international Conference on information integration and Web-Based Applications & Services (iiwas'08). ACM, Linz, Austria (2008) 3. Anjewierden, A., Chen, W., Wichmann, A., Van Borkulo, S. P.: Process and product analysis to initiate peer-topeer collaboration on system dynamics modellingworkshop on Intelligent and Innovative Support for Collaborative Learning Activities (WIISCOLA), Rhodes, Greece (2009) 4. Balabanovic, M., Shoham, Y.: Fab: Content-based, collaborative recommendation. Communications of the ACM 40, 66-72 (1997) 5. Basu, C., Hirsh, H., Cohen, W.: Recommendation as classification: using social and content-based information in recommendationthe Fifteenth National Conference on Artificial Intelligence (1998) 6. Behama, G., Kumpa, B., Leya, T., Lindstaedta, S.: Recommending knowledgeable people in a work-integrated learning system Procedia Computer Science (2010) 7. Bruns, A., Humphreys, S.: Wikis in teaching and assessment: the M/Cyclopedia projectthe 2005 international Symposium on Wikis. ACM, San Diego, California (2005) 8. Brusilovsky, P.: Adaptive hypermedia. User Modeling and User Adapted Interaction 11, 87-110 (2001) 9. Brusilovsky, P., Henze, N.: Open corpus adaptive educational hypermedia. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The adaptive web Springer-Verlag (2007) 10. Burke, R.: Hybrid recommender systems: survey and experiments. User- Modeling and User-Adapted Interaction 12, 331-370 (2002) 11. Castells, P., Fernandez, M., Vallet, D.: An Adaptation of the Vector-Space Model for Ontology-Based Information Retrieval. IEEE Transactions on Knowledge and Data Engineering 19, 261-272 (2007) 888 ComSIS Vol. 9, No. 2, June 2012

Recommending Collaboratively Generated Knowledge 12. Chen, G. D., Chang, C. K., Wang, C. Y.: Ubiquitous learning websites: scaffold learners by mobile devices with information-aware techniques. Computers & Education 50, 77-90 (2006) 13. Chen, W., Persen, W.: Reusing Collaborative Knowledge as Learning Objects -- The Implementation and Evaluation of AnnForum. In: Dillenbourg, P., Specht, M. (eds) The Third European Conference on Technology-Enhanced Learning (EC- TEL). Springer-Verlag (2008) 14. Cheng, C. K., Pan, X., Kurfess, F.: Ontology-Based Semantic Classification of Unstructured DocumentsAdaptive Multimedia Retrieval. Springer Berlin / Heidelberg (2004) 15. Craven, M., DiPasquo, D., Freitag, D., McCallum, A., Mitchell, T., Migam, K., Slattery, S.: Learning to extract symbolic knowledge from the World Wide WebProf. of the 15th National Conference on Artificial Intelligence, Madison, Wisconsin (1998) 16. Drachsler, H., Hummel, H. G. K., Koper, R.: Personal recommender systems for learners in lifelong learning: requirements, techniques and model. International Journal of Learning Technology 3, 404-423 (2008) 17. Drachsler, H., Pecceu, D., Arts, T., Hutten, E., Rutledge, L., Van Rosmalen, P., Hummel, H., Koper, R.: ReMashed Recommendations for Mash-Up Personal Learning Environments. In: Cress, U., Dimitrova, V., Specht, M. (eds) Proceedings of the Fourth European Conference on Technology-Enhanced Learning (EC-TEL 2009). Springer-Verlag (2009) 18. Falloon, G.: Learning objects and the development of students' key competencies: A New Zealand school experience Australasian Journal of Educational Technology 26, 626-642 (2010) 19. Garrison, D. R., Anderson, T., Archer, W.: Critical thinking in text-based environment: Computer conferencing in higher education. The Internet and Higher Education 2, 87-105 (2000) 20. Hanauer, D. I., Jacobs-Sera, D., Pedulla, M. L., Cresawn, S. G., Hendrix, R. W., Hatfull, G. F.: Inquiry Learning: Teaching Scientific Inquiry. Science 314, 1880-1881 (2006) 21. Hartmann, B., MacDougall, D., Brandt, J., Klemmer, S. R.: What Would Other Programmers Do? Suggesting Solutions to Error MessagesProceedings of the 28th international conference on Human factors in computing systems (CHI'10). ACM (2010) 22. Helic, D., Maurer, H., Scerbakov, N.: Learning to extract symbolic knowledge from the World Wide Web. ACTA Advanced Technology for Learning 1, 8-16 (2004) 23. Henze, P. B. a. N.: Open corpus adaptive educational hypermedia. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web. Springer-Verlag (2007) 24. Hoppe, H. U.: Integrating Learning Processes Across Boundaries of Media, Time and Group Scale Research and Practice in Technology Enhanced Learning 2, 31-49 (2007) 25. Janssen, J., Tattersall, C., Waterink, W., Van den Berg, B., Van Es, R., Bolman, C., Koper, R.: Self-organising navigational support in lifelong learning: how predecessors can lead the way. Computers & Education 49, 781-793 (2007) 26. Khribi, M. K., Jemni, M., Nasraoui, O.: Automatic Recommendations for E- Learning Personalization Based on Web Usage Mining Techniques and Information Retrieval. Educational Technology & Society 12, 30 42 (2009) ComSIS Vol. 9, No. 2, June 2012 889

Weiqin Chen and Richard Persen 27. Klamma, R., Spaniol, M., Cao, Y.: Community Aware Content Adaptation for Mobile Technology Enhanced Learning In: Nejdl, W., Tochtermann, K. (eds) Proc. of the First European Conference on Technology Enhanced Learning (EC- TEL 2006). Springer (2006) 28. Klebl, M., Krämer, B. J.: Editorial: Learning ojbects in progress. British Journal of Educational Technology 41, 869-872 (2010) 29. Kobsa, A., Koenemann, J., Pohl, W.: Personalized hypermedia presentation techniques for improving online customer relationships. The Knowledge Engineering Review 16, 111-155 (2001) 30. Konstan, J. A., Miller, B. N., Maltz, D., Herlocker, J. L., Gordon, L. R., Riedl, J.: GroupLens: Applying collaborative filtering to Usenet news. Communications of the ACM 40, 77-87 (1997) 31. Linden, G., Smith, B., York, J.: Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Computing Jan./Feb.(2003) 32. Littlejohn, A.: Reusing Online Resources: A Sustainable Approach to e-learning, Routledge (2003) 33. Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., Koper, R.: Recommender Systems in Technology Enhanced Learning In: Ricci, F., Rokach, L., Shapira, B., Kantor, P. B. (eds) Recommender System Handbook. Springer (2011) 34. Manouselis, N., Vuorikari, R., Assche, F. V.: Simulated Analysis of MAUT Collaborative Filtering for Learning Object RecommendationProceedings of the 1st Workshop on Social Information Retrieval for Technology-Enhanced Learning & Exchange (2007) 35. Manouselis, N., Vuorikari, R., Assche, F. V.: Collaborative recommendation of e- learning resources: an experimental investigation. Journal of Computer Assisted Learning 26, 227 242 (2010) 36. McGreal, R.: Online Education Using Learning Objects. Routledge2004) 37. McNee, S. M., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S. K., Rashid, A. M., Konstan, J. A., Riedl, J.: On the recommending of citations for research papersacm International Conference on Computer Supported Collaborative Work (2002) 38. Melville, P., Mooney, R. J., Nagarajan, R. P., 2002.: Content-boosted collaborative filtering for improved Recommendationsthe 18th National Conference on Artificial Intelligence (2002) 39. Muukkonen, H., Hakkarainen, K., Lakkala, M.: Collaborative technology for facilitating progressive inquiry: future learning environment toolsproc. of International Conference on Computer Supported Collaborative Learning (CSCL'99), Palo Alto, CA (1999) 40. Mödritschera, F.: Towards a Recommender Strategy for Personal Learning Environments. Procedia Computer Science (Proc. RecSysTEL 2010) 1, 2775-2782 (2010) 41. Nitchot, A., Gilbert, L., Wills, G.: A Competence-based System for Recommending Study Materials from the Web (CBSR)Proc. 9th International Conference on Information Technology Based Higher Education and Training (2010) 42. Northrup, P. T.: Learning Objects for Instruction: Design and Evaluation. Information Science Publishing (2007) 43. Ochoa, X., Duval, E.: Use of contextualized attention metadata for ranking and recommending learning objectsproc. of the 1st international Workshop on 890 ComSIS Vol. 9, No. 2, June 2012