e-learning Roadmap Evaluation Rubric (as of 12/17/2010) Framingham State University elearning evaluation criteria are modified from the original Longsight document according to the Creative Commons license referenced below. 1 The criteria also incorporate elements from the Duke University e-learning Roadmap Website. Information was modified to meet Framingham State learning platform requirements. A. Evaluation Criteria Categories are rated on a scale from 0 to 3; 0 = NA (unable to comment; need more information), 1 = Limited functionality, 2 = Acceptable functionality and 3 = Recommended. The evaluation criteria fall into these categories: Teaching and Learning (faculty core functionality) User-interface (Navigation, overall use of the system) Administrative Functionality (care and feeding of the LMS) E-Learning Services (e-tutoring, e-commerce, 24-7 support etc.) Vendor Services and Support Miscellaneous (there just isn t a place for this) 1 This work is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 2.5 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/2.5/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.
I. Guiding Principles The elearning Roadmap Task Force assumptions and guiding principles form the foundation by which the LMS Review project technology and planning decisions are made. It is based on a similar document posted to the Duke University e-learning Roadmap Website and was modified to meet the needs of the FSU learning environment. These statements do not attempt to specify technical or functional requirements but suggests guidelines that can be applied to any institutionally supported LMS solution. They form the basis by which all elearning platform recommendations are made. A. Assumptions 1. A set of core, baseline requirements form the decision process. The solution must; - meet institutional goals and directives, - support and/or advance the student and faculty teaching and learning experience, on-campus or online, - scale, so that it supports all constituencies at the college, from students enrolled in the day program, Graduate and Certificate programs, international program or continuing education students or faculty and staff committee and research needs - be reliable, with a good record of stability and minimal service interruptions in context with our current and proposed future use of the system - adhere to security and privacy policies - meet industry standards so that it integrates with existing systems or future unanticipated needs 2. There is no agenda to move to open source versus a proprietary system. The determination to move to a new platform will be based on the following contingencies: - Impact Significant, Minimal, None - Support (Particularly among Faculty) Acceptance, Unknown, Rejection - Resources Available, Possible, Not Possible - Funding Available, Possible, Not Possible - Feasibility Very Feasible, Feasible, Not So Feasible 3. It is understood that user needs and desires, as well as teaching and learning methodologies change overtime, and that the e-learning solution will need to adapt frequently to meet these evolving needs.
B. Principles 1. Favor a best of breed approach that allows interoperability which will permit: The ability to swap out components as needed; The option to use specialized tools without requiring a major overhaul of the system or expensive consulting services; and The use of open standards and APIs, which will make it easier to integrate with different services and resources 2. Consider solutions which have evolved as net-scale cloud services (if they meet the contingencies outlined above) 3. Analyze how to use our limited resources to achieve our overarching project goals 4. Provide effective identity and group management 5. Consider tools that have a broad user and developer community 6. Integrate strategies that accommodate change and adapt to the evolving e-learning environment 7. Accommodate and encourage multiple modes of access C. Deal Breakers The following list of deal breakers are integrated into the evaluation criteria and must be present to move forward with the evaluation process. The University elearning Task Force recommendation will; Support University ELearning Roadmap guiding assumptions and principles listed above. Integrate with Banner and the Luminis portal Support an automated course creation, course merge enrollment process Provide a documented migration/upgrade strategy Provide single sign-on capabilities with Active directory Support an E-Reserve document management strategy Support an e-portfolio assessment strategy Integrate with Web 2.0, 3.0, and other third party tools (e.g. Elluminate, Respondus, Learning Objects)
D. Evaluation Standards: Rating Scale: 0 = NA (unable to comment), 1 = Limited functionality, 2 = Acceptable functionality and 3 = Recommended. TEACHING AND LEARNING CORE FUNCTIONALITY NA LIMITED ACCEPTABLE RECOMMENDED Pedagogical Design Provides basic access to organized materials but few opportunities for interaction, constructivist or engaging methods. Provides basic access to content as well as tools for engaging students, interactive learning. Provides access to content that integrates well with interactive tools, and new pedagogical tools are being routinely added to the system. Content Authoring Provides a basic means for uploading and storing content in a hierarchical manner to support teaching and learning. Allows basic content to be uploaded or created within an authoring system that is part of the LMS Provides a suite of tools for authoring media-rich content, importing content, drag-and-drop interfaces, as well as uploading rich content types such as podcasts, video clips, etc. Allows metadata creation for easier/better management. Also allows for a modular design. Copyright Management No specific tools for either authoring or consuming intellectual property. A fixed set of copyright options is available to the content author Authors are given choices (such as Creative Commons) for their content, and consumers (students) are reminded of their responsibilities. (Ability to publish open courseware) Communication LMS provides secure access to the email addresses that comprise the class roster, but individuals may not be selectable for private email. Both asynchronous (email) and synchronous communication tools are present. LMS provides a high level of flexibility for the use of email (asynchronous by roster, individual or group; opt-in options) as well as instant messaging, chat and threaded discussions. File Exchange LMS provides secure dropbox functionality so that students can exchange materials with instructors. LMS provides drop-box and ability for students and faculty to upload resources to a central course repository. LMS provides secure shared folders for file exchange among students as well as instructors and allows for bulk downloads of attached files. (Drag and drop feature)
NA LIMITED ACCEPTABLE RECOMMENDED Sections and Groups LMS allows instructors to define sub-groups of students within the class roster for purposes of communication and collaborative work. LMS allows sub-groups but allows the instructor the choice of interacting with only the sub-group or the entire course in all available tools. LMS provides the hierarchy to support sections within a single course so that course content is shared among sections. Instructors can define sub-groups of students which then link to separate content repositories and tools. LMS also supports shared groups among multiple (organization) sites. Discussion Tools Adequate speed and functionality with the ability to attach files Quick and functional with user profiles or pictures, file attachments and html interface. Extremely fast and highly functional with user profiles and pictures, files attachments and easy html and web 2.0 interface. (Reporting capabilities; easy navigation; voice threads) Testing and Assessment Tools A simple test generator with the ability to add multiple choice, true/false, short answer and essay questions. More than a simple test generator, this system provides tools for creating assessments with images or other attached files. More than a simple test generator, this system provides tools for creating assessments with multimedia, learning games, and other interactive tools such as polls. Tests can provide immediate feedback with tips for remediation. (Integration with test banks and publisher content) Grading Functionality Moderately functional gradebook that is relatively easy to use. Functional gradebook that is easy to use. Grades can be exported to a spreadsheet. Highly functional, user defined, gradebook that is easy to use. Grades can be exported/imported to/from to a spreadsheet or to a student information system or the LMS. Student Assessment Tracking Minimal student tracking tools. Student tracking tools give the instructor some information about student progress Student tracking tools give the instructor information about what pages the student has viewed and what tasks have been completed. The student can be automatically emailed when their participation is substandard. Calendar Functionality Basic calendaring functionality Basic calendar with pop-up announcements. Collaborative calendar with pop-up announcements. Integration with external calendaring systems; ICAL; Integrated with academic calendar; campus events
NA LIMITED ACCEPTABLE RECOMMENDED Conditional (content) release functionality Selective release is possible (of course content) but somewhat difficult to set up. Release of course content and assessments can be scheduled for student access with moderate effort. Release of course content and assessments can be easily scheduled (and is flexible) for student access based on time and situational conditions. (Adaptive release) Presentation Tools LMS makes it possible to store PowerPoint or Keynote files for download. LMS makes it possible to store PowerPoint or Keynote files and provides a mechanism to display the files in-browser. The LMS provides access to PowerPoint and Keynote presentations through the web interface but also provides tools for creating and editing presentations within the LMS framework to leverage use of the LMS content repository. Tool Set integration (3 rd Party Tools) Limited integration with Web 2.0 and 3 rd party tools Extended integration with some external tools. The LMS provides extensive options to integrate Web tools, Mashups, widgets and gadgets; integrates with third party tools (Elluminate, Respondus, Learning Objects Blog and Wiki tools; Plagiarism tools (TurnitIn, SafeAssign)
User Interface NA LIMITED ACCEPTABLE RECOMMENDED Design and layout Functional interface with decent layout but somewhat complex and counterintuitive. Aesthetics are bland, distracting or cartoon-like. Good functional interface that can be navigated with minimal training. Good look and feel. Simple, intuitive interface with minimal clicks to access materials, little or no training needed to get started, and the look and feel is inviting. Ease of Use Content Organization LMS provides a basic repository for course content. LMS provides a repository for content and basic tools for content organization. LMS provides a framework for diverse storage and use strategies, from public, private and shared workspaces, to subscription-based content (e.g., podcasts and feeds) to archival content Browser setup and support Supports the most popular browsers with end user set up and installation of necessary components. May have a preferred browser for proper operation. Supports most browsers with minimal effort from the user. Supports all browsers and platforms with no special setup requirements for the user. Is able to render the LMS experience in most browsers with consistency. Usability Course environment meets usability standards, but requires some customization and training. Users are able to complete most tasks with limited instruction or training. Supports drag and drop; customized interface; ease of use; limited time ; number of clicks; consistency of the tools throughout the system; intuitive Learnability Users need technical assistance to upload course content or build a basic course environment. Users are able to complete basic tasks with little training the first time they enter the platform. Users are able to complete basic tasks with little or no previous technical knowledge the first time they enter the platform.
NA LIMITED ACCEPTABLE RECOMMENDED Efficiency Basic functionality is remembered, but requires user to reference documentation. Basic functionality is easily remembered but more complicated functions require assistance from support. Basic functionality is easily remembered and repeated with little or no assistance from support. Memorability Proficiency is established after being away from the system for an extended period of time but requires help from support. Proficiency is established after being away from the system with some assistance from support. Proficiency is easily established after being away from the system for an extended period of time. Errors If errors are made (by the end user) they require assistance from support to correct the problem. If errors are made (by the end user) they require some assistance from support to correct the problem If errors are made (by the end user) they require little or no assistance from support to correct the problem. Satisfaction The environment is somewhat pleasant. The environment is mostly pleasant. It is pleasant to use the design. Accessibility 2 Meets minimal ADA requirements. Meets some accessibility standards set by the ADA Meets accessibility standards set by the ADA. Quality Matters Provides a basic means for implementing Quality Matters course design principles. Learning platform may be customized to implement Quality Matters course design principles. Learning platform supports and is easily used to implement Quality Matters course design principles. 2 Ask LaDonna Bridges (CASA) to weigh-in on the Accessibility requirement.
Administrative Functionality NA LIMITED ACCEPTABLE RECOMMENDED Analytics Some reporting tools exist but the tools and documentation is inadequate or limited. Reporting tools are provided and well documented, but limited. Self-service analytics are available and well documented. Additional functionality may be added or modified. Course /Organization Management Limited or no automated process exists to update or create course shells or manage enrollment (faculty and students) Automated course creation process integrates with University SIS seamlessly. Course shells, faculty assignments and student enrollments are modified from a script. (e.g. (LDI and/or Snapshot) Automated course/organization creation process integrates with University SIS seamlessly. Course shells, faculty assignments and student enrollments are modified in real time. (LDI and/or Snapshot) Course Copy Limited or no automated process exists to copy course content between semester shells. Course content is easily copied between old and new course spaces. Course content is easily copied between one or multiple course sections. User interface and course tool settings are maintained. Upgrade Strategy Limited or no upgrade strategy is proposed but may be defined by user requirements. The upgrade strategy is well defined, but faculty and students may be disrupted if not planned well in advance. Faculty and students are minimally disupted by vendor upgrades, bug fixes and other maintenance solutions. The upgrade strategy is well defined. Migration of existing courses Some migration tools exist but the tools and documentation are either inadequate or difficult to use. Good tools are provided and well documented, but the migrated material will need additional formatting. Excellent migration tools with great documentation which maintains the integrity of the original course. All migrated course materials are ready for use. Course Export Permits course content to be exported and re-imported into the LMS itself but may have limited ability to export to other LMS systems. Allows course structure and content to be exported but in formats that constrain how the exported content may be imported elsewhere. Exports course structure and content, as well as selected sub-elements of a course, using an industry-standard such as IMS Content Packaging so that courses can be imported into other LMS.
NA LIMITED ACCEPTABLE RECOMMENDED Course Archives Some archival tools but much of the process is manual. Archived courses are not available to be viewed by the instructor. Good archival tools that support backup of completed courses with student submissions and discussions intact. The LMS administrator must set up instructor access to the completed course. Powerful archive tools that support automatic backup of completed courses with student submissions and discussions intact. Instructors have full access and control of completed courses. Integration with Student Information System (Banner)* Integration is possible but will require large level of product customization. Tools for integration are available but some tasks will need to be completed manually or in a batch process. Seamless integration with automatic updating of courses, student and faculty lists and all rosters. Students can be automatically emailed course access information. Student and faculty profiles with pictures and syllabi can be shared between the LMS and the SIS. (Merge course management; role assignment; organization management) Single sign-on Ability to batch load users from a campus central identity system. Ability to batch load users but also to integrate a campus single sign-on system like CAS or Pubcookie. A real-time connection with a campus central identity system (Active Directory) that avoids the need for batch processes. Integration with campus single sign-on. Security Limited support for campus wide and industry (TEACH) standards LMS Platform supports to campus wide and industry (TEACH) standards LMS Platform adheres to campus wide and industry (TEACH) standards Integration with campus portal (Luminis) LMS is accessible through the campus portal but only by linking that requires a separate authentication by the user. LMS is linked with the portal via single sign-on, but the only level of integration possible is the iframe. LMS and portal share single sign-on and select tools can be integrated with the portal via industrystandard integrations (Luminis Database). System speed and reliability* Course material access times are adequate on high speed connections but frustrating for dial-up users. Access times are very good for students on high speed connections and adequate for dial-up users. The fastest system available with support for streaming media and/or offline companion materials to better serve dial-up users. System is rarely off-line.
NA LIMITED ACCEPTABLE RECOMMENDED Server requirements* LMS only operates on one operating system and requires special configurations of hardware or supporting software. LMS is available on multiple platforms but does not offer compatibility with an implementer s choice of application server or database. Server software operates on a wide variety of operating systems (Windows, Linux/Unix, Mac) using commodity hardware and industry-standard web servers. Vendor offers hosted solution. Scalability LMS has no problem meeting demands of a small institution on a single server. LMS supports clustering and the ability for multiple servers to act in unison, but there are few installations supporting over twenty thousand users. LMS clusters well and has been known to support installations well over one hundred thousand users. LMS allows for multiple modes of access (mobile, traditional, technologies not on the market). Impact Adoption of the LMS will have Significant ramifications to the university Adoption of the LMS will have Minimal ramifications to the university Adoption of the LMS will have Little or No ramifications to the university Support Rejection Unknown Acceptance Resources Not possible Possible: Resources may be leveraged to achieve project overarching project goals including user community at large. Available: Resources are leveraged to achieve project overarching project goals. Large user and developer community exists from which to draw solutions. Funding Not possible Possible Available Feasibility Not so feasible Feasible Very Feasible
ELearning Services (Extended Functionality and Resources) NA LIMITED ACCEPTABLE RECOMMENDED E-portfolio Basic tools allow students and instructors to gather student work products for assessment and presentation. Tools allow students and instructors to create ad-hoc or structured presentations of resources. A full-featured e-portfolio tool is integrated into the LMS and makes possible the gathering, review and presentation of work products to support any portfolio strategy (resume, learning, tenure, etc). Reporting tools allow for individual, departmental or institutional assessments. (Integration with SIS; External access; Life-long access) Course Evaluations Basic survey tools for capturing student reflections on course, instructor Anonymous evaluations that can be gathered by the faculty including question pools and templates. Hierarchical and flexible system for anonymous evaluations at course, department and institutional level for either summative or formative purposes. Includes item pools, templating, announcements, reminders, and tools to easily target different audiences. University Collaboration Allows shared access to files among users and some tools for asynchronous collaboration. Provides access to shared files and some tools for asynchronous and synchronous collaboration and communication. Limited group functionality. Provides a campus-wide framework that supports collaborative work such as wiki with version tracking, threaded discussion, instant messaging and chat, whiteboard, web conferencing (audio and video). Enables subgroups to be defined within courses for collaboration. Provides non-course sites to support special project work among small groups. Integration with library resources and e-reserves* Limited to no integration with locally licensed library content. Ability to create resources that can be resolved to librarycontrolled databases. Tools are present that allow faculty to find and reference both public and licensed library materials, including full texts. Students are able to access these materials once logged into the system from any location. Textbook, Publisher content integration Some textbook materials but difficult to find, request or install. Several supported texts with good materials that can be installed with moderate efforts. Many supported texts, excellent well-organized materials, easily installed and based on industry or community standards (e.g., Common Cartridge)
NA LIMITED ACCEPTABLE RECOMMENDED Ecommerce Limited or no capabilities exist to integrate with student ID services. System may be customized to integrate with e-commerce student ID services. Product functionality includes e-commerce capabilities or integration with student ID card services. Messaging Limited or no messaging capabilities exist to send messages to a selected audience. Some messaging capabilities exist to send information to specific user groups enrolled in the system. Automated messages (email, text, announcements) may be delivered to specific user groups enrolled in the LMS across courses. Branding Limited or no capabilities exist to brand sections of the LMS platform to reflect divisions within the university. Some capabilities exist to brand sections of the LMS platform to reflect divisions within the university. The Learning Platform is easily branded to reflect divisions within the university.
Vendor Services and Support NA LIMITED ACCEPTABLE RECOMMENDED Vision and product roadmap The vendor or developer community does not make public their technical or pedagogical vision for the LMS, and the timing of future releases may be unclear. Vision and roadmap are loosely available via conferences or insider knowledge but not published for public view or critique. LMS vendor or developer community has published their vision for both the technical and pedagogical aspects of the product, and timing of releases is clear and adhered to. Supports a partnership philosophy. Adaptability Vendor culture is somewhat willing to change to adapt to change based on faculty and student needs and the evolution of the e-learning (Blended, online, webenhanced) environment for a price. Vendor culture is willing to change to adapt to change based on faculty and student needs and the evolution of the e-learning (Blended, online, web-enhanced) environment for a price. Vendor culture is willing to change to adapt to change based on faculty and student needs and the evolution of the e-learning (Blended, online, web-enhanced) environment. Support Email support only. Email support and limited phone support. 24/7 phone and email support with tracking system to follow the progress of issue resolution; Solution provider assigns the university with an account manager who may be notified is problems need escalating or attention. Training Materials Fair printed materials, minimal online training or classroom training sessions available. Good printed materials, some online training or classroom training sessions available. Excellent printed/online (video) materials and many opportunities for online and classroom training sessions. System Resources A users manual is accessible online. Help files are accessible at each step of a process, and system documentation is accessible online. Contextually-appropriate help files are accessible from all pages and provide assistance for students, faculty and system administrators as appropriate. Pop-ups or rollovers provide just-in-time information for specific actions. Use of Open Standards Standards are seen as a goal, but the implementation of standards is missing or incomplete. Open standards (IMS CP, QTI, etc) are used in the LMS but are incomplete or are built in combination with proprietary methods that create product lock-in and inflexibility. Open standards are incorporated wherever appropriate in the LMS and are leveraged to provide as many options as possible. No proprietary components are present that require separate licensing or lock in data.
Miscellaneous NA LIMITED ACCEPTABLE RECOMMENDED Math Equation Editor Minimal math equation functionality Some math functionality incorporated into the system. Full functioning math equation editor is built into the LMS. Ability to enter complicated functions, graphs and graphics exists. Adaptive Release Minimal functionality to hide or reveal content based on user roles Some functionality exists to hide or reveal content based on user roles Functionality exists to hide and reveal content based on student roles. (e.g. Different test, surveys, grad vs undergrad syllabi) Student Dashboard metrics Dashboard metrics may be integrated with the SIS as a customization for a price. Dashboard metrics may be integrated with the SIS as a customization. Dashboard metrics are integrated with SIS. Warnings may be automated to alert student if they are falling behind or failing a course. Interoperability A few solutions may be integrated with learning environment; Cloud solutions, e-portfolio, Card Access, Analytics, etc. Ability to swap out solutions as requirements evolve and/or change. Some solutions may be integrated with learning environment; Cloud solutions, e-portfolio, Card Access, Analytics, etc. Ability to swap out solutions as requirements evolve and/or change. Multiple solutions may be integrated with learning environment; Cloud solutions, e-portfolio, Card Access, Analytics, etc. Ability to swap out solutions as requirements evolve and/or change. University Goals and Objectives Some university goals and objectives are supported. Most university goals and objectives are supported. Platform supports university goals and objectives Teaching and Learning experience LMS Platform somewhat supports and/or advances the student and faculty teaching and learning experience, oncampus or online. The Platform meets the needs of the digital immigrant but not the millennial student. Features of the LMS supports and/or advances the student and faculty teaching and learning experience, oncampus or online. The Platform meets the needs of the digital immigrant but not the millennial student. LMS Platform supports and/or advances the student and faculty teaching and learning experience, on-campus or online. Platform meets the needs of the millennial as well as digital immigrant.