Automating Outcome Based Assessment Suseel K Pallapu Graduate Student Department of Computing Studies Arizona State University Polytechnic (East) 01 480 449 3861 harryk@asu.edu ABSTRACT In the last decade, web-based instruction was established as one of the popular trends in instruction. Colleges and universities across the USA are using the web for delivering instruction and for providing student support (distribution of materials, communication with teacher and peers). Presently in the market there are software available for course management, e-learning, and web-based evaluation. Some of the popular software used for this purpose are Blackboard, WebCT, and TrueOutcomes. BlackBoard is an e-learning and Course Management System used in most universities in the USA. TrueOutcomes is a webbased assessment and evaluation software based on the standards specified by the accreditation agencies. These two software operate independently, without any option of importing a BlackBoard course into TrueOutcomes. Outcome-based assessment and evaluation process is mandatory for universities to get accreditation. This paper discusses the process of automating outcome-based assessment, where an existing course in BlackBoard is exported into TrueOutcomes. Some of the issues involved during this process are (i) how to create and manage a course in BlackBoard, (ii) what files needed to be exported, (iii) how the extracted information should be arranged, (iv) what format to store information, (v) how assignments tie with the program outcomes, and (vi) what additional information is required. The results of this research has been used by the TrueOutcomes development team for developing a module to import a course from BlackBoard. 1. INTRODUCTION Accreditation Board for Engineering and Technology (ABET) made it mandatory for universities to follow the outcome-based assessment and evaluation process for accreditation purposes. Most universities are trying to get accreditation, for the degrees they offer, to increase the credibility and accountability of the degree. One of the criteria for accreditation purposes is that the university has to design, develop and assess a degree based on outcomes. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Conference 05, Janaury 3 6, 2005, Island of Hawaii, USA. Copyright 2004 ACM 1-58113-000-0/00/0004 $5.00. 1.1 Outcomes Willam Spady and Kit Marshall (1994) wrote, Outcomes are clear, observable demonstrations of student learning that occur after a significant set of learning experiences. They are not values, attitudes, feelings, beliefs, activities, assignments, goals, scores, grades, or averages, as many people believe. Typically, these demonstrations, or performances, reflect three things: (i) what the student knows; (ii) what the student can actually do with what he knows; and (iii) the student s confidence and motivations in carrying out the demonstration. A well-defined outcome will have clearly defined content or concepts and be demonstrated through a well-defined process beginning with a directive or request such as explain, organize, or produce [1]. The definition states outcomes are observable through demonstration or performance; it means they can be assessed based on a criteria. Education that is based on outcomes is called Outcome-based Education (OBE). 1.2 Outcome Based Education (OBE) The focus of OBE is to design and teach a course based on outcomes. James Towers (1996) defines OBE as Education that is outcome-based is a learner-centered, results-oriented system founded on the belief that all individuals can learn. Towers listed four points that are necessary to make OBE work: (i) What the students are to learn must be clearly identified; (ii) Student s progress is based on demonstrated achievement; (iii) Multiple instructional and assessment strategies need to be available to meet the needs of each student; and (iv) Adequate time and assistance need to be provided so that each student can reach the maximum potential [2]. Design and developing outcome-based course is a three step process: (i) Identify the outcomes; (ii) Decide the contents and teaching strategies; and (iii) Developing assessments based on the outcomes. In this process the central focus is on outcomes not on the course content. Before deciding on the content the outcomes of the course need to be decided. 1.3 Assessment in OBE Assessment in OBE is more than memorization, the student should be able to demonstrate the skills acquired. It means student s performance can not be judged based on exams and quizzes alone. There should be other ways to determine the knowledge/skill gained by the student. The skills/knowledge of students can be assessed by giving them more challenging tasks like writing papers, project proposals, designing, developing and implementing technologies, projects, assignments, presentations etc. These tasks makes a student improve his/her thinking, researching, analyzing, decision making, and presenting abilities.
Besides this group projects can also help students acquire team skills. ABET document Criteria for Accrediting Engineering Programs - 2003 state that and an engineering program graduate should demonstrate the following outcomes: (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs (d) an ability to function on multi-disciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global and societal context (i) a reorganization of the need for, and ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice [3]. Historically, assessment has not been a subject area covered in the education of engineers... Hence, few engineering educators were prepared, by either by education or experience, to set up and manage the programmatic evaluation or experience, to set up and manage the programmatic evaluation and outcomes assessment processes mandated engineers are by nature problem solvers who do much, if not most, of their best work by banning together to analyze problems, identify and evaluate candidate solutions, and develop plans for implementing a recommended solution. Cooperation and collaboration are elements of engineers professional modus operandi, [4]. Assignments and projects can help in assessing student s ability to apply the knowledge acquired, analyze and solve problems, design, develop, and implement solutions. Group projects/assignments can be used to assess student s ability to work in teams. Project presentations and paper presentations can be used for assess the communication capabilities of the student. Including the current research topics in the curriculum would see that the students acquire contemporary knowledge. One way of keeping track of student performance is maintaining a student portfolio for each course, which contains the papers he/she presented, projects completed, assignments, exams, quizzes etc. By assessing the entire student portfolio one can evaluate the performance of the student. These assessments also make sure that the assessment is learner-centered and result-oriented. Willies & Kissane (1995) suggested two techniques for assessing students learning outcomes: (i) Standard-referenced assessment (similar to criterion-referenced assessment but with a clearer description of expected performance), and (ii) Student portfolios documenting their progress. 1.4 Automating Outcome-based Assessment Developing software that automates outcome-based assessment involves integration of two existing systems, course management system (CMS, e.g. BlackBoard) and Outcome-based assessment and evaluation system (e.g. TrueOutcomes). At present no single software has the ability to manage courses as well as assess them. This paper discusses the issues involved in the integration of CMS with AMS. This completes the automation process of transforming information from one system to another. 2. BACKGROUND Course management systems became an important part of universities. The primary use of CMS is to distribute course materials, collect student assignments, and maintain student grades, etc. Outcome-based Assessment and Evaluation System is a tool used for assessing and evaluating a program based on program outcomes. These outcome standards are specified by various accreditation agencies like ABET engineering; NLNAC nursing; AACSB business; and NCATE education; and regional accrediting agencies such as NEASC, MSACS, SACS, WASC, NCCS, and the American Association for Higher Education. 2.1 Course Management Systems CMS is a software system designed to help faculty and students in teaching and learning. Most popular CMS include Blackboard, WebCT etc. Most CMS include tools for content management, file sharing, online assessments (quizzes), student tracking, assignment and portfolio management, and virtual collaboration. Faculty provided options to manage their own internet-based file space on a central system, which helps to collect, share, and manage important materials from articles and research papers to presentations and multimedia files. CMS also has features for online collaboration between students and faculty, chat and a message board. It also provides faculty with tools for course management, content management, user management, and assessment Management. Some of these tools are export and import course, export and import grade book, submit student work, upload course material for distribution, creating assessments, quizzes etc. Students are able to access course materials, do submissions, check grades, etc., once they log into the system [5]. Faculty started using CMS in their courses they realized the potential of technology to improve the teaching and learning experience. Faculty is finding additional uses for the CMS as the get comfortable with the system. CMS enhances the learning experience of the students as the course materials are available for them all the time. For example, if the face-to-face class is recorded and the recording is distributed using CMS, the students can go back and listen to the lecture any time during the semester in case they need to understand concepts discussed in the class. It would be helpful for the students who missed the class to refer the recording and understand the concepts taught. This would definitely enhance the learning experience of the students. The use of CMS is best described in the research paper by Glenda Morgan (2003) Faculty use course management systems in teaching and learning to achieve a number of teaching goals. The most important goals include supplementing lecture materials, increasing transparency and feedback, and increasing contact with and among students. CMS helps faculty to include more interactive materials, allowing them to address diverse learning styles. increase the amount of feedback they are able to provide to students and to improve the promptness of the feedback online gradebook, which many faculty described as altering their relationship with their students and students relationship with their own work [6].
Another good use of CMS is importing and exporting a course. For example, the BlackBoard provides faculty an import and export course option. This is very convenient for faculty as it reduces the process of recreating a course every semester and arranging materials from scratch. From the above paragraphs it is clear that CMS stores course materials like syllabus, quizzes, discussions forums, student grades, and student submissions, etc. In BlackBoard two important options available for the faculty are exporting grade book and exporting student submissions. 2.2 Outcome-based Assessment Software Outcome-based Assessment and Evaluation Software designed based on the outcome specifications developed by popular accreditation agencies like ABET engineering, and American Association for Higher Education etc. The goal of the software is to create a web-based file system that stores information about a program offered by the university and the courses in that program. The purpose is to use this stored information to assess and evaluate the program. The course information stored contains the details of the course, student grades, student submissions, and activity details etc. The information available is used for creating portfolio for students as well as faculty. One of the best examples available for this outcome-based assessment software is TrueOutcomes. TrueOutcomes is web-based assessment software. The goal of TrueOutcomes is to provide a single platform that brings together administrators, faculty, students, alumni, and external stakeholders using flexible assessment tools. It includes the student s professional portfolio; faculty maintained juried portfolio, curriculum record, and survey. TrueOutcomes is designed for institution-wide assessment that includes programlevel assessment and course-level assessment. All the instruments in TrueOutcomes (the professional portfolio, juried portfolio, survey, and curriculum record) can be used to assess co-curricular activities. There are many characteristic differences between cocurricular activities and academic activities. But TrueOutcomes can help in creating a process that can be customized for each situation [5]. A general outcome-based assessment system stores information about a program and course. A course contains grade book details, student details, and activity details etc. Activities are exams, quizzes, assignments and projects that are used for assessing a students performance. Creating a course in TrueOutcomes involves entering course details, course outcomes, assignments or projects given, exams conducted, student submissions, and student grades. Universities that are using both a CMS and outcome-based assessment system can autome this process of creating a new course in outcome-based assessment system. The next section addresses the issued involved in the transition of information from a CMS into Outcome-based assessment system. 3. INTEGRATION From the discussion about the course management system and outcome-based assessment system, it is clear there is an overlap of information stored. But the information is used for different purposes. The files that overlap in both systems are grade book, student submissions, course details, faculty details etc. This following sections discusses the process followed for automating the transforming a course from CMS to Outcome-based Assessment System. BlackBoard and TrueOutcomes are the example software used. For a successful transition of information, it is important to identify the information that can be extracted from BlackBoard; rules to be followed in creating a course in BlackBoard; other required information about a course by TrueOutcomes; and the structure of file arrangement and storage format. 3.1 Required Files Two dumps are identified as required files after comparing the course information stored in BlackBoard and TrueOutcomes. They are grade book and activity dump. An activity dump is an assignment or a project dump. The grade book has student name, login ids for the CMS, grade for each activity, and activity name. An activity dump has all the student submissions for a single activity. Other required files are abstract and rubric for each activity. The abstract and rubric can be a word document, text document or a PDF document. 3.2 Constrains in creating a course in CMS For the successful transformation of course content from CMS into Oucome-based assessment system following constrains are there while creating a course. Conventions followed are: 1. Naming Conventions followed are: a. Syllabus is named as Syllabus. The syllabus is a word document or a PDF document. b. Assignments are named as Assign followed by the number of the assignment. (e.g. Assign1, Assign2) c. Projects are named as Project followed by the number of the project (e.g Project1, Project2). d. Quizzes are named as Quiz followed by the number of the quiz (e.g. Quiz1, Quiz2). e. Exams are named as Exam followed by the number of the exam (e.g. Exam1, Exam2). f. Abstract file is named as Assign followed by number of the assignment and the word Abstract (e.g. Assign1Abstract). This file can be a word document, text document, or PDF document. g. Rubric file is named as Assign followed by number of the assignment and the word Rubric (e.g. Assign1Rubric). This file can be a word document, text document, or PDF document. 2. To create an activity dump, all student submissions for that activity should be downloaded as a single file. For example, in BlackBoard student submissions could be accepted two different ways. First one is through digital dropbox where in student submissions are accepted as individual submission. Second one is against activity; faculty can create an activity that accepts student submissions. The second option is used for creating activity dump. The created dump is a ZIP file. 3. Student submissions should be named uniquely (Unique ID + Assignment name). For example, At Arizona State University every student has a unique user identification called ASURITE which is added to the appropriate assignment/project name for form unique file names.
Another option is to use the student last names, provided there are no two students with the same last name. The following steps explain the procedure followed for creating a course according to the conventions. 1. Syllabus file is uploaded under the option Course Information, this is done by going to the control panel of the course, select course information and then section item option. Under specify your own name, type syllabus, in file to attach, select the file named syllabus.doc or syllabus.pdf. 2. Assignments are created under Assignments option. Select the control panel and then select the assignments option. In the drop down box that says Learning Unit, select Assignment and press the Go button next to it. This will create a new assignment against which the students can submit their work. 3. Abstract and Rubric files can be uploaded under the Assignments, these will be later downloaded. 3.3 Additional required information Even though grade book and student submissions form the crux of the information required, TrueOutcomes requires some additional information. These include course details, program outcomes, and activity details. Course details contain course code, course description/name, schedule line number, instructor name, semester and login name instructor uses for TrueOutcomes. The course code, description, instructor name and semester details are as per the university catalog information. Activity details are rubric, abstract, and outcomes. These outcomes are program outcomes that are achieved on completion of the activity by the student. Rubric file has the criteria used for grading the student work. Abstract file contains the brief description and the goals of that activity. 3.4 Arrangement of files File structure followed during this project was developed with the help of TrueOutcomes development team. The following figure shows the example structure of the file arrangement in the course dump (ZIP file). Figure 1: Directory Structure The above example the gradebook.xml file in the root directory contains all the information required by TrueOuctomes about a course. The subdirectories 001, 002, 003 contains student submissions for the first, second and third assignments respectively. If the activity is an exam or a quiz, no directory will be created. 3.5 XML schema An XML Schema was created based on the input from TrueOutcomes development team. An XML schema specifies the structure to be followed by and XML file. This schema is used for validating the gradebook.xml file. The figure 2 shows the XML schema used during the development of this project. Figure 2: XML Schema Even though the XML schema presented looks very complicated, it will make sense once the example XML file is presented. The following section explains the XML file used in this development.
3.6 XML File The XML file generated is gradebook.xml. The following figure shows a sample of generated XML file. Figure 4: Course Details Screen In the above figure there are five steps that are required to be done to complete the creation of course dump that would be used by the TrueOutcomes software. The second step is to enter course outcomes. Each outcome has a code and description, once the user finishes entering all the outcomes, he/she is provided with the option of saving the outcomes. The default file name used for storing outcomes is outcomes.xml. Third step is to process the grade book file to create gradebook.xml. The grade book exported from the BlackBoard is a CSV file. If the user decides to use the default grade book file name by BlackBoard, user does not need to select the file name. If the default file name is not used then the user is provided the option for choosing the file. Forth step is to map the student submission to the student grade. Figure 5 shows the activity details screen. Figure 3: Sample XML file As you can see from the above figure most of the element names are explanatory. Information about the course is stored as individual element values and attribute for element. ID attribute in the activity is the subdirectory name where all student submissions are located for that particular assignment/project. Even though an ID number given to exams/quizzes, no directory will be created during the execution of the software. Mime-type attribute for the work stores the type of file submitted by the student. This helps TrueOutcomes during import process. So far only simple text information stored in abstract, outcome, and rubric elements while testing process. 3.7 Software The software developed during this research uses Java and XML technologies. A simple graphical user interface is developed for the software. Self explanatory names are used in each screen to guide the user in executing the software. User input is accepted for course details, outcomes, and activity details. User is provided helpful hints where ever possible. Figure 4 shows the details accepted for the course details. The semester option is hard coded into the code, to make it easier for the user select semester information. Figure 5: Activity Details Screen User needs to select the appropriate ZIP file for an activity and select to which activity name (the list box is populated while processing the grade book), outcome (outcomes list box is populated when the user enters the outcomes), abstract and rubric can be simple text or file names. User can select the files by clicking on the Browse button next to the text box. Fifth is the final step where the user is given some directions and asked to click on the Finish button. When the finish button is clicked the program checks appropriate activity directory and maps the student submissions to the student grade and Zip all the files together into course dump with a default name truocdump.zip. Once the dump is generated the user needs to upload the file in TrueOutcomes to create a new course.
4. CONCLUSIONS This research was successful in automating the information transfer from BlackBoard to TrueOutcomes. The research results are used by the TrueOutcomes development team to come up with a new version of TrueOutcomes software that has the capability of importing a ZIP file generated at the end of the process explained in the previous paragraphs. At present there is no provision made for storing course outcomes. Without the course outcomes it would be difficult to judge the activity simply based on program outcomes. Course outcomes should be mapped program outcomes. These course outcomes should be designed in such a way that each outcome is realized by a single activity. This would help the accreditation agencies understand how the program outcomes are realized using the course outcomes. Another suggestion is to save the level of skills achieved by the student after each assignment. Bloom (1956) categorized the skill level achieved by the students in to six categories. The are as follows: Knowledge: arrange, define, duplicate, label, list, memorize, name, order, recognize, relate, recall, repeat, reproduce state. Comprehension: classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, restate, review, select, translate, Application: apply, choose, demonstrate, dramatize, employ, illustrate, interpret, operate, practice, schedule, sketch, solve, use, write. Analysis: analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test. Synthesis: arrange, assemble, collect, compose, construct, create, design, develop, formulate, manage, organize, plan, prepare, propose, set up, write. Evaluation: appraise, argue, assess, attach, choose compare, defend estimate, judge, predict, rate, core, select, support, value, evaluate [8]. Other enhancement is improving the CMS to provide option for downloading the required resources. This may require development of a plug-in for the CMS to download all required information as a single ZIP file. 5. ACKNOWLEDGMENTS I sincerely thank my advisor Dr Harry Koehnamann for the support and guidance provided to me throughout the project. I want to thank him for kindling my independent thought process while designing and developing the project. I would also like to thank Mr. Paul Mendelson, TrueOutcomes team, for his invaluable input for the success of this project. I would also like to thank my committee members Dr Timothy Lindquist, and Dr Bruce R. Millard. 6. REFERENCES [1] Spady, W., Marshall, K., Light, not heat, on OBE. The American School Board Journal, 181 (November. 1994), 29-33. [2] Towers, J.M., An elementary school principal s experience with implementing an outcome-based curriculum. Catalyst for changes, 25 (winter), 19-23. [3] ABET, Criteria for Accrediting Engineering Programs, Baltimore, MD. 2003. [4] Michael S.L., Eleanor W.N., ABET Outcomes-based Assessment: Providing Opportunities to Cooperate and Collaborate Across the University Campus and Among Universities. [5] Glenda Morgan., Faculty Use of Course Management Systems. Available at HTTP: http://www.educause.edu/ir/library/pdf/ecar_so/ers/ers0302 /ekf0302.pdf [6] BlackBoard website. Available at HTTP: http://www.blackboard.com [7] TrueOutcomes website. Available at HTTP: http://www.trueoutcomes.com [8] Bloom's Taxonomy. Available at HTTP: http://www.dlrn.org/library/dl/guide4.html