Version DRAFT 1.0 Evaluation of Learning Management System software Author: Richard Wyles Date: 1 August 2003 Part II of LMS Evaluation Open Source e-learning Environment and Community Platform Project
1 Document Control Abstract: This document describes the findings of an initial evaluation of Open Source Learning Management System software. It forms Part II of the evaluation methodology to select the core LMS as part of the overall Platform architecture. Version: 1.0 Status: Draft Date of Issue: Author: Richard Wyles Authorisation: Ken Udas Signature: Date: Summary of Changes: Version Date Description 2
2 Table of Contents 1 DOCUMENT CONTROL...2 2 TABLE OF CONTENTS...3 3 CONSIDERATIONS FOR ONLINE PEDAGOGY...4 3.1 Seven principles of pedagogy and technology selection...4 3.2 A Framework for the Pedagogical Evaluation of elearning Environments - (Britain and Liber, Feb. 2004)...6 3.2.1 The Viable System Model...6 3.2.2 Conversational model...6 3.3 Summary conclusions for online pedagogy...6 3.3.1 Programme level management...6 3.3.2 Resource management...7 3.3.3 Monitoring...7 3.3.4 Learner centred...7 3.3.5 Flexibility / Adaptability...7 3.3.6 Conversations / Communication tools...7 4 PEDAGOGICAL FRAMEWORK QUESTIONNAIRE...8 4.1 Programme level management...8 4.2 Resource management...8 4.3 Monitoring...8 4.4 Learner centred...9 4.5 Flexibility / Adaptability...9 4.6 Conversations / Communication tools...10 4.7 General Comments... 10 5 CONSORTIUM USER GROUPS...11 5.1 Context of User Group Participation...11 5.2 Purpose of User Groups... 11 Terminology LMS: Learning Management System. LMS typically contain features for administration, assessment, course management, collaboration. LCMS: Learning Content Management System: In addition to typical LMS features, a LCMS includes content management/authoring features typically found in a Content Management System (CMS). 3
3 Considerations for online pedagogy What are the issues or considerations for online pedagogy that impact on the selection of an e-learning platform? The purpose of this document is to propose a framework for evaluating the short-listed Learning Management Systems. The pedagogy considerations are based on recommendations from several key discussion papers published in the U.S., and most recently A Framework for the Pedagogical Evaluation of elearning Environments (Britain and Liber, Feb. 2004). Ragan (1998) noted that the resulting principles and practices that represent effective teaching apply to face-to-face instruction as well as the distance education delivery model. Furthermore, what makes for an effective educational experience is independent of where or when it is delivered. When considering online pedagogical issues, two questions need to be addressed (Chizmar and Williams, 1996): What pedagogy should be used? Will the pedagogy work over the Internet using a variety of Internet delivery techniques? The second question is subordinate to the first. The pedagogy must drive the choices of instructional technology. In choosing answers to the first question, whether on campus courses or Internet courses, effective teaching is guided by Seven Principles (Chickering and Gamson, 1987), these principles are independent of and beyond teaching and learning styles. Chickering and Ehrmann s (1996) added technology to the Seven Principles with the statement, If the power of the new technologies is to be fully realised, they should be employed in ways consistent with the Seven Principles. These are examined and summarised in the table below: 3.1 Seven principles of pedagogy and technology selection Seven Principles 1. Encourage contacts between learners and faculty. Frequent student-faculty contact in and out of class is a most important factor in learner motivation and involvement. 2. Develop reciprocity and cooperation among learners. Good learning, like good work, is collaborative and social, not competitive and isolated. 3. Use active learning techniques. Technology As Lever (Chickering and Ehrmann, 1996) Asynchronous means of communication allows for greater contact between faculty and learners with varying schedules. Learner faculty contact can also be enhanced by synchronous means such as chat. Learner to learner interaction, separated by geography and time is enhanced using Internet tools to create a sense of community, both within the institutions and without. Learners can use various tools in order to become more active Tools for evaluation (examples only) Email, bulletin boards, forum, chat. Chat, forums, instant messaging, blogging, resource pooling/sharing, online community sites and resources. Online games, simulations, interactive tools, quizzes. 4
Learning is not a spectator sport. Learners must talk about what they are learning, write reflectively about it, relate it to past experiences, and apply it to their daily lives. learners. Searching the internet for information, simulations that can replace dangerous or expensive experiments that often were not done but just 'talked about', and simulations that teach concepts all make the learner a more active learner. 4. Give prompt feedback. Knowing what you know and don't know focuses your learning. In getting started, learners need help in assessing their existing knowledge and competence. Then, in classes, learners need frequent opportunities to perform and receive feedback on their performance. Computer programmes can be used to provide immediate feedback to learners in areas such as testing. Online tutorials, quizzes, online assessments, self-assessment tools. 5. Emphasise time on task. Time plus energy equals learning. Allocating realistic amounts of time means effective learning for learners and effective teaching for faculty. 6. Communicate high expectations. Expecting learners to perform well becomes a self-fulfilling prophecy. 7. Respect diverse talents and ways of learning. Many roads lead to learning. Different learners bring different talents and styles. Learners need opportunities to show their talents and learn in ways that work for them. Using new technologies to allow learners to learn when they are ready without having to travel, or work at specific times increases the productivity of learners. Many learners 'feel stimulated by knowing their finished work will be "published". If they know other learners will see their work, learners usually set higher goals for themselves. Also, learners are able to use the published portfolios to get job interviews as part of their C.Vs. The new technologies can provide learners with multiple ways of learning. Learners are able to pick the one that suits them best and are therefore more likely to succeed. Then they can be pushed to learn in new ways that do not come so easily. Flexible and intuitive course design, scheduling and completion, online monitoring tools for student s progress. Online Web tools for content creation/management/publishing for learners; e.g. Web Blogs, Wikis. Personalisable online environment providing a variety of content media and interactive choices. 5
3.2 A Framework for the Pedagogical Evaluation of elearning Environments - (Britain and Liber, Feb. 2004) Britain and Liber amalgamate two models to provide an evaluation framework. 3.2.1 The Viable System Model The key premise of the Viable System Model (VSM) in relation to VLEs is that the variety or options for e-learning provision must be increased to match the increasing diversity of requirements from learners. The model explores platform criteria from different levels; Programme, Module, and Individual. Criteria include resource management, system flexibility / adaptability, ability for learners to self-organise, monitoring functions, and individualisation. 3.2.2 Conversational model The Conversational model draws a strong distinction between essentially a one-way transmission of information compared to the interactivity implicit in a conversation. Because it is far easier, the transmission model is commonplace in current VLEs. Workflow actions can be described as follows. However, they do not necessarily occur in a linear fashion interactive loops or conversations may evolve freely and should be accommodated by the VLE. 1. Teacher presents / re-describes conception 2. Learner presents / re-describes conception 3. Teacher sets up micro-world activities 4. Learner interacts with micro-world activities 5. The system provides feedback on the action 6. Learner modifies actions in light of feedback 1 Presentation, interactivity, flexibility, personalisation and communication tools are key areas for evaluation criteria. 3.3 Summary conclusions for online pedagogy There are certainly some common themes from the Chickering and Ehrmann s (1996) and Britain and Liber (2004) research. Core to both is that technology should adapt to fit pedagogy not the other way around. Educational institutions need more flexibility and control over their e-learning environments to enable different schools, programmes, course, or instructors to select and deploy the most appropriate e-learning tools suited to the pedagogy. Interoperability standards and modular, extendable architectures hold the promise of delivering the desired flexibility and ensuring greater future proofing in a technology environment that is fast evolving. By enabling configuration at multiple levels, alternative visualisations of what a VLE should be like, can be accommodated. There is a strong link between the way that a VLE is designed and the way that it is typically used. 3.3.1 Programme level management Different programmes or departments may have very different needs for e-learning tools. An interface with student management systems is highly desirable. 1 The Conversational Framework A Framework for the Pedagogical Evaluation of elearning Environments (Britain and Liber, 2004) pp25. 6
Monitoring or reporting should be available at the programme management level i.e. programme managers have access to drill down into the courses to the individual level. 3.3.2 Resource management Establish a methodology to communicate the limits of the instructor s responsibility. Enable equitable access to the instructor s time. Enable co-ordination of activities to balance system demands 3.3.3 Monitoring In addition to tracking activities, there needs to be self-assessment and reflective practice tools. Learning conversations between instructor and learner are the best way to ensure learning is actually taking place as intended. The system needs to facilitate the scalability of conversations. The instructor should be able to link detailed feedback for a learner depending on their actions e.g. contexturalised discourse. Consistency of experience must be weighed up against the variety of experiences possible. 3.3.4 Learner centred Incorporate personal development planning (PDP) tools. Allow self-organising activities e.g. clubs, study groups and unmonitored communications. Develop a time management tool for learners. A VLE should allow the learners to restructure the presented material, add resources of their own, annotate material, launch and run simulations etc. In other words the learner should not merely be a passive observer of the micro-world constructed by the teacher, but should be pro-active in shaping the world. 2 This indicates a need for wiki or blogging tools to be integrated into the VLE. 3.3.5 Flexibility / Adaptability Develop ability to incorporate new materials and processes on the fly. If you can move the chairs, you have more choice in a classroom; if you can adapt the workflow of a VLE, you can provide more flexibility in your learning opportunities. 3 Activities associated with a learning topic should be able to be adapted according to the needs of an individual or learner sub-group as revealed by course interactions between the learner and their peers, system or instructor. Sub-groups must therefore be identifiable within the VLE once the learning activity is in progress. 3.3.6 Conversations / Communication tools Many VLEs rely heavily on asynchronous discussion groups using forums, and email contact. The conversation tools should be integral to the learning i.e. accessible directly to/from the topic area. Content should be able to be extracted and embedded into the course structure, including learning goals where applicable. 2 Interactivity - A Framework for the Pedagogical Evaluation of elearning Environments (Britain and Liber, 2004) pp26 3 Coordination Channel A Framework for the Pedagogical Evaluation of elearning Environments (Britain and Liber, 2004) pp17. 7
4 Pedagogical Framework Questionnaire The following questionnaire will be placed online accessible from Eduforge. 4.1 Programme level management Rate the system for its management at a programme level. For example, does the system allow specification of programme rules for delivering a module? 5pt High / Low Is the programme within the system a coherent entity rather than just a collection of unrelated modules? Y/N Rate the system for its reporting at a programme level. For example, can the programme manager monitor the performance of a module? 5pt High / Low Rate the system for its workflow tools. Does the system provide tools for new modules to go through design, development and validation? 5pt High / Low Rate the system for its support of instructors working on different modules to coordinate their activities and assist each other. 5pt High / Low 4.2 Resource management Is there functionality to enable equitable access to the instructor s time? Y/N Does the system co-ordinate activity to manage demands on the system? Y/N 4.3 Monitoring Rate the system for its support of learner self-assessment tools. Specifically is there the opportunity to set up Right student on right course pre-enrolment assessment and recommendations? 5pt High / Low Rate the facilities for monitoring how well learning is progressing on a module. 5pt High / Low Rate the system for learners to monitor their own activity. 5pt High / Low Learning conversations between instructor and learner are the best way to ensure learning is actually taking place as intended. The system needs to facilitate the scalability of conversations. Rate the facilities for the instructor to link detailed feedback for a learner depending on their actions e.g. contextualised discourse. 5pt High / Low Can students provide feedback on the quality of the module? Y/N 8
4.4 Learner centred Personal Development Planning describes the incorporation of self-assessment, reflection and action planning for learning. It facilitates continuing personal and professional development. Rate the system for the learner to undertake Personal Development Planning (PDP). 5pt High / Low Does the system allow self-organising activities e.g. clubs, study groups and unmonitored communications? Y/N Can learners locate others with similar interests outside of their own module or course, with the appropriate permission from the learner? Y/N Is information about people available? Y/N Do learners have their own file stores or repositories? Can they create a personal homepage? Y/N Do learners have access to online publishing tools? Y/N Does the system provide time management / planning / organisation tools for the individual student to organise their work? For example can they obtain statistics on what they are spending most time on, whether their time is being evenly shared or not Y/N Can the learner make notes or annotate course material? Y/N 4.5 Flexibility / Adaptability If required, are there permissions that can be enabled for instructors and learners to extend/change their presentations during the module s time period? Y/N Please rate the ability to incorporate new materials and processes on the fly. 5pt High/Low scale Can you add / change / delete resources? Y/N Can you add / change / delete fragments of module structure? Y/N Activities associated with a learning topic should be able to be adapted according to the needs of an individual or learner sub-group as revealed by course interactions between the learner and their peers, system or instructor. Sub-groups must therefore be identifiable within the VLE once the learning activity is in progress. Can you add / remove people? Y/N Can you split them into different groups (whole group, small groups, individuals)? Y/N Can you create and assign resources or learning activities to individuals? Y/N Can you alter the sequencing of a module? Y/N Rate the system for supporting different pedagogical models or approaches 5pt High/Low scale 9
4.6 Conversations / Communication tools Many VLEs rely heavily on asynchronous discussion groups using forums, and email contact. Conversation tools should be integral to the learning. Are the conversation tools accessible directly from the topic area? Y/N Can content from a conversation be extracted and embedded into the course structure, including learning goals where applicable? Y/N Please rate how well the system encourages contacts between learners and faculty. 5pt High / Low Please rate how well the system provides instructors with tools to present/express their ideas to students? 5pt High / Low box. Please rate how well the system provides learners with tools to articulate their ideas their ideas to other learners and the instructor? 5pt High / Low box. 4.7 General Comments Comments box 10
5 Consortium User Groups 5.1 Context of User Group Participation The NZ Open Source VLE Project needs to take users requirements and expectations, from across the consortium, into consideration. This can occasionally be challenging in an open source community or consortium context as we try to cater to all the requirements of particular sub-groups. To address this we are endeavouring to ensure that the platform framework adopts open standards and is modular, providing institutions to interface various open source, proprietary, and in-house applications that meet their institution s unique needs. It is also important to appreciate that we will be selecting and adopting e-learning software from existing, well established open source communities. Value for money and fitness for purpose are underlying principles of the project. It is our preference to avoid forks in the open source communities. A fork is when a sub-group of a community charts a different course to the rest in the system s application development. Inherently a fork results in a smaller community of users to support the application s development. In this project, the Moderator will operate similarly to a traditional project management role by working with the Change Control Board (representatives of User Groups) to ensure all major features and selection of development pathways aligns with the objectives of the overall project and requirements of consortium partners. The moderator also ensures there is good communication between the project team and the open source communities from which code is being adopted. 5.2 Purpose of User Groups User Groups at each consortium partner organisation should represent the key stakeholder groups within the organisation. User Groups contribute to core requirements specifications and requirements related to deployment at each TEO. 11