10: The use of computers in the assessment of student learning

Transcription

1 10: The use of computers in the assessment of student learning Nora Mogey & Helen Watt Increased numbers of students in Higher Education and the corresponding increase in time spent by staff on assessment has encouraged interest into how technology can assist in this area. Ensuring that the assessment methods adopted reflect both the aims and objectives of the course and any technical developments which have taken place is becoming increasingly important, especially as quality assurance procedures require departments to justify the assessment procedures adopted. Introduction and definition of terms Assessment consists of taking samples of behaviour at a given point in time and estimating the worth of those behaviours. Thus the underlying assumption of assessment is that it provides a representative sample of behaviour of the person being assessed. On the basis of the kind of sample taken inferences are made about a persons achievements, potential, aptitudes, intelligence, attitudes and motivations. All forms of assessment provide estimates of the persons current status. Before performing any assessment it is important to ask ourselves three questions: What is this assessment for? Who is it for? What is the context? Criterion referenced assessment refers to assessment based on explicit criteria and targets. Does the student have the necessary mastery of identified components of the course of instruction? In this type of assessment the requirements of a pass level are defined in advance, as learning objectives. Norm referenced assessment refers to assessment which is used to establish a rank order of achievements or aptitudes with a group of students, across &/or through time, providing a continuum of performance. Standards could be established partly by controlling the percentage of students awarded each grade. The three functions of different types of assessment are: Diagnostic:- to identify strengths and weaknesses. Formative:- to provide feedback to students. Summative:- to estimate performance for the purpose of (formal) assessment at the end of a course or unit of study. In practice the types and functions of tests are not clear cut. For example, mid course/unit formative assessments are often used as an element within multiple summative assessments. Overview: Using IT in assessment Technology can be used for assessment purposes at various levels ranging from the management of the assessment information to a fully automated assessment system. Using technology for the management of assessment information can enable information to be presented in different ways to meet the needs of different audiences (such as teachers, students, course organisers and external examiners). Not only the quality of presentation of reports but more importantly the range and scope of their content can be improved by utilising technology for this purpose. At the other extreme, in a fully automated assessment system all aspects of the system from the assessment which the student completes to the processing and administration of the marks, including the overall management of assessment information, is technology-based. Assessment strategy should be considered during Course Design, here it is useful to ask ourselves: 50

2 What kind of things do we want our students to learn? What opportunities will be provided? What assessment tasks will be set? What methods of assessment will be used? Analysing the types of learning which we require to take place has significant implications for the instructional design, assessment strategies and methods. Since assessment involves measuring it gives rise to problems in: Choosing a valid assessment instrument; Finding a suitable unit of measurement; Ensuring the test is measuring what it is supposed to measure; Scorer reliability, especially if more than one marker is involved; Using valid statistical methods and drawing valid inferences from measures. Why change current practice? Drive to find effective new and improved assessment methods. Focus on Quality Assurance. The changing aims of HE (mix of academic, vocational and general transferable skills). Emphasising deep learning rather than shallow. Interest in using new teaching methods. Need for greater cost effectiveness. To save staff time. Advantages in the use of IT for Assessment More frequent formative and summative assessment. Staff can be alerted sooner to adapt their teaching. Can spend less time marking. Self-assessment; in the student's own time, at their own pace, when they are ready. Increased student confidence. Students like rapid results. Electronic delivery of tests Objective tests There is growing interest and increasing practical experience in the use of computers to deliver objective tests. Objective testing is often taken to imply the use of multiple choice questions (MCQs). However, objective tests can incorporate a wide range of question styles in addition to standard multiple choice questions; for example multiple response, word entry, number entry, gapfill and free-format where student entry is compared to a correct solution using a keyword search. Objective tests, and MCQs in particular, are generally considered to be an efficient method of testing factual knowledge, enabling a wide syllabus to be examined in a relatively short time. It is, however, important for academic staff to be aware of the limitations of objective tests (especially MCQs), particularly in their inability to indicate higher level and process skills. Objective tests can be used for both formative and summative assessments, and a variety of scoring systems can be applied, tailored to the importance of discouraging students from guessing answers. Several packages are available which are designed for the electronic delivery of objective tests (e.g. Question Mark, Examine, EQL Assessor), all of which support the delivery of a variety of question types. Entering questions is generally straightforward, requiring minimal experience with the package. Although the design of questions for computer based delivery is no more difficult than for paper based objective tests, this remains non-trivial and the most time consuming part of the whole objective testing process. Difficulties can sometimes arise where subjects require the use of specialised notation (such as for mathematics, chemistry or linguistics) but these can almost always be overcome by the use of appropriate specialised fonts, or the inclusion of small 51

3 graphic objects. When the test has been completed the students responses are marked automatically, quickly and consistently. Computerised delivery of objective tests offers interesting possibilities not available within paper based systems: The creation of a bank of questions invites the possibility of each student being presented with a paper made up of different questions, but of an equivalent standard. (Some words of caution need to be associated with such an approach - the identification of questions which are truly of an equivalent standard is not a trivial task.) Instant computerised marking facilitates immediate feedback for the students. Students can be invited to sit tests as frequently as they find useful. Computerised recording of results facilitates the analysis of groups responses to questions. Structured questions Electronic delivery of tests need not be restricted to MCQs, and the results recorded need not only be whether a response is right or wrong. Consider the situation where a question has been posed, but the student is unable to get started. In a traditional setting (for a summative test) this student would be forced to omit this question and try another. In an electronic setting it is possible to include an option to provide a hint for the student. Most computer based assessment packages can operate in a choice of modes, perhaps described as tutorial or exam mode. The exact number of modes available varies from package to package, but four modes is not uncommon, each offering a different level of detailed feedback for the student, to assist them in reaching the correct answer, or explaining why their response was not correct. Questions can be split up into several stages, and marks can be awarded at interim stages before a final answer is determined. Marks can be deducted if hints have been provided. For students who have made a slip, the computer can identify an error and offer the possibility of going back, correcting an interim answer and hence enabling the student to successfully reach the final stages of a question. Electronic generation of tests In addition to using electronic packages to create unique tests, it is possible to use the computer to generate different tests automatically. Question banks : Electronic selection of questions from a bank has already been mentioned as one possibility for the electronic generation of tests. The creation of a question bank is a demanding task for a single individual, however, where several members of staff (possibly from different institutions) collaborate to share questions a large bank can be established relatively quickly. From this a huge number of different tests can be generated. Randomisation of parameters : An alternative method of generating questions electronically is the use of parameters. The format of the question will be identical on every occasion, but one or more variables in the question is selected from a list of permitted values. These values may be entered when the question is created, or they may be generated by the computer, either randomly or according to some formula. Feedback from students indicates that the opportunity to work through questions is often considered to be very helpful in identifying areas of weakness in their knowledge, or in developing a confidence in their understanding of a subject. Computerised delivery from a bank of questions or of randomly generated problems, supported by automated marking and feedback to the student is a flexible and efficient method of providing formative assessment, particularly where factual knowledge is an important component of the course. Electronic recording and analysis of results Perhaps the most immediately obvious and most easily accessible use of technology to assist the assessment process is in the recording, analysis, general storage and management of results. A wide range of spreadsheets, 52

4 statistical packages and database packages are available (e.g. Excel, Lotus 1-2-3, Dataease, SPSS, Minitab, Access), into which it is easy to enter data manually if results are not already in electronic form, though enormous care must be taken to avoid transcription errors when generating the data files. Most of these packages readily accept the transfer of electronically stored data from other applications, aiding data acquisition and increasing the potential data analysis that can be carried out. Results from several assessments, courses or modules can be collated quickly, easily and accurately for discussion at examination boards, and the volume of paper required for long term storage can be dramatically reduced. Further, any trends within the data can be fully explored, which in turn provides valuable feedback for the academic team. Final scores vs. other information It is sometimes useful to record data other than how many right answers the student is able to achieve, particularly when the assessment is formative. The use of interim stages in a problem to provide feedback and guidance to students has already been mentioned. Some computer based assessment packages record all the interim responses from the students into a file, which is available to the tutor for diagnostic purposes, if required. Other packages require students to log on before using a package, so that frequency of usage of the package can be monitored. Another practice is to record the length of time for which students are logged onto the system. This can be helpful in identifying students who achieve high scores, but only when they have a lot of time to do so, compared with students who may score less highly, but who spend very little time on the test. As outlined above other information relating to the path a student followed through a package, the frequency of usage and the time taken to complete sections can be stored. Although this can provide teachers with useful information it can often cause concern to students if they have had to enter identifying information before accessing the package. Some students are anxious about the use to which such information will be put. However if students are informed when given operating instructions for the package of the use to which information will be put then this anxiety is usually alleviated and students will feel happier and more motivated to use the software particularly for formative assessment purposes. Electronic scoring tools The use of electronic methods to store and manipulate data becomes pointless if the integrity of the data cannot be guaranteed. The manual entry of marks is particularly susceptible to error, time consuming and costly to check thoroughly. The use of data capture devices, such as an Optical Mark Reader (OMR) connected to a computer, can vastly reduce input errors, particularly the problem of number transposition on data entry, e.g. typing 45 instead of 54. Standard pre-printed OMR forms can be a cost-effective way of collecting student responses to questions. A pencil or pen mark is made on the form by the student to indicate each selected response, i.e. their answer to a particular question. No special training is required for this just some simple instructions re entering responses clearly in the designated check areas. The completed forms can then be scanned by an OMR to detect the presence of a mark (usually by measuring reflected light levels). The pattern of marks and spaces is interpreted by the reader, following instructions provided by the operator, and is stored in a data record and sent to your computer file for storage. Thus large quantities of information can be entered onto your computer without the need to use a keyboard. Hence increasing accuracy and saving time. One example of the use of these is for students to record their answers on the pre-printed OMR response sheets although the test has been presented in the form of MCQs on paper. Software packages are available which allow you to design and print customised forms using a personal computer and a laser printer. An additional feature used in some HE institutions is to allocate an individual barcode to each student which can be attached to their form. This can be read by the OMR thus decreasing the chance of mistaken identity due to students incorrectly entering their personal identification number, e.g. due to transposing characters or checking the wrong boxes when entering their matriculation number. Once the student answers have been stored for a test, the responses can not only be scored but can be analysed in a number of different ways, e.g. by individual question, groups of questions, all questions. Thus a variety of reports can be produced such as: the results of individual students; the results of groups of students including 53

5 the mean, median and modal scores; graphs of results; analysis of each question including its reliability, facility value and discrimination factor. Security considerations One obstacle which can prevent teaching staff from utilising technological solutions to administer student assessment is the worry of security. It is hoped that the following practical suggestions will help to put the reader s mind at rest on some of the issues which may concern them. When considering delivering assessment on computer, it is possible to password protect the file containing the test and also to disallow access until after a particular date. To overcome the possibility of students copying from each other s visual display screens, the possible responses to each test question can be displayed in a different order. Also databanks of questions can be used which means that each student is sitting a comparable but not identical test. There is of course the problem of obtaining access to sufficient equipment to deliver a test to all students in a class at the same time. This does require careful preplanning to book the facilities and to ensure that the test is available on all the machines simultaneously. Usually this means that additional support staff, such as computer suite managers, will be involved in assisting the course teaching staff in organising the delivery of the test. Ensuring that the person completing an assessment is who s/he says they are (authentication) is an issue in all examinations and is not a security issue specific to computer based assessment. However, in a computer environment impersonation may be perceived as a greater risk. The most obvious way to control this issue is to set summative assessments in exam conditions, checking identities against matriculation cards etc. and checking these against the candidates computer log-on information. Authentication of remotely sat examinations is likely to remain a problem - as it is with paper based assessments. Sometimes due to class size and the availability of equipment it is not possible to arrange for all the students to sit the test simultaneously. One solution to this problem which has been successfully used is to divide the class into two sections. The second section sits the test immediately after the first. In order to eliminate any possible transfer of answers from students in section one to section two, the first group leaves the computer cluster by a different route than that taken by the second group to enter, e.g. via the fire-escape door rather than the main entrance. However students themselves are self-motivated not to transfer any information to their peers as it may disadvantage their own showing in a test. Plagiarism should present no greater problem in an electronic environment than in a more traditional context. However, the cut and paste facility of word processors and information retrieval systems may tempt students to copy sections of text directly into their work. This is a potential that has to be guarded against in the same way as text manually copied from other sources - largely though the style and quality of English used. Peer group assessment where each member of the group awards a share of the total marks to their group collaborators may present another practical guard against plagiarism and could appropriately be employed in some situations. Students should also be made aware of the law relating to copyright, especially when they are asked to compile portfolios or other multimedia presentations. Storing student marks on a computer file is not necessarily more prone to student access than traditional means. It is possible to password protect a file to stop unauthorised access. If the machine on which the file is stored is not networked then any potential infiltrator would require to obtain access to the room in which the computer is situated as well as cracking the password. If the file is stored on a floppy disk rather than on the computer's hard disk then this can be locked in a filing cabinet in the same way as paper-based marks but will in fact be more secure than the paper-based version as in order to view the data the person seeking access to it would require to know the name of the file, have access to appropriate hardware and software and be able to enter the correct password. Electronic support as a tool in the assessment process There are many other ways in which computers can assist in the assessment process. A few of these are outlined below. Of course, not every example will be applicable in every subject area or every course. 54

6 Technology as an administrative and management tool There are a number of examples of Integrated Learning Systems (ILS), where entire course structures, lecture, practical and assignment schedules and supplementary resources are held electronically and available for student consultation whenever needed. The management of assessment schedules is just one small part of an ILS, and an equivalent benefit is available through the use of other, less holistic tools. For departments where electronic mail is available this offers both tutors and students an efficient and straightforward means of communication. One application is to use to remind students of impending deadlines. Even students who are absent when the message is first posted are certain to receive the message when they next access their mail. Similarly any changes, revisions to the assignment, or hints can be delivered to all students. Assignments can be submitted electronically. For distant learners this avoids the need to rely on the speed of postal services, and for all students the date and time of delivery is automatically attached, so meeting deadlines can be monitored accurately. Computers are an ideal tool to track attendance or achievement records, allowing monitoring of trends, for individuals and cohorts, comparison between years or classes and early identification of problems. Word processing and direct presentation tools Encouraging or insisting that students word process assignments is now widespread. A similar strategy is to promote the use of IT tools (e.g. Powerpoint, Freehand) to support oral presentations. In both cases students can be encouraged and assisted in the development of a professional attitude, in addition to academic considerations. Word processors are invaluable to those with poor spelling or who would produce an illegible script. If these skills are important, then perhaps use of technology should be actively discouraged. Insistence on all assignments being word processed may effectively penalise students who possess fewer keyboard skills, or those who have more difficulty in gaining access to the necessary equipment. (Access issues are discussed more fully in chapter 8 ). The correct balance must be sought between the time invested in the appearance of an assignment and the time invested in consideration of its content; students, particularly early in their courses, may require clarification and assistance with these issues. There are a number of collaborative writing tools now available, at least one of which, Common Space, has been written and designed specifically for educational use. Common Space could be used by a group of individuals involved in a collaborative creative process, where each can comment on and revise the ideas of others in the group, or by tutors to give feedback on early drafts of a document. Comments can be as text or as audio recordings. It offers a number of features to promote flexibility and adaptability in integrating comments and revisions from a range of different sources. Similar annotation, comment and revision marking features are provided in most quality word processing packages. Electronic seminars Electronic mail has already been mentioned as a useful administrative tool. It can also play a supporting role in the whole process. Examples exist of courses where seminars are presented electronically, rather than in a tutorial room as is conventional. The student leading the seminar prepares a paper and submits this by to the group for consideration. Some electronic discussion follows, prompted where necessary by the tutor. After a few weeks the student 'presenting' summarises the discussion, and presents a revised paper. Such a structure is extremely flexible in making demands on the student's time. The contribution of all students to the discussion is marked by the tutor, hence all students are encouraged to contribute, however those who might naturally be more reserved have the opportunity to consider their contributions rather than being forced to make them as soon as an idea or question occurs to them. Management of such seminars does require skill from the tutor in judging just when to contribute a comment in order to keep discussion going. It also offers possibilities for peer group assessment and other less traditional methods. 55

7 Portfolios and other simple authoring tasks Technology can be important in the assessment of transferable skills, and can be used to promote collaborative assessments, shared between more than one subject area. Although it is not advocated that students should as a rule be encouraged to be authoring or creating packages, this can be a useful means of promoting a deeper understanding of many of the issues relating to the use of and evaluation of technology. A similar idea is the development of computer based portfolios where students can create multimedia packages with simple links between different resources. These could be used for the creation of portfolios where students have gathered a range of resources, such as graphics, audio and video clips in addition to text and links to relevant pages from the world wide web. Students could also be assessed on their ability to create a relevant and linked portfolio of World Wide Web resources, which can quite simply be created by writing Web pages in HTML (hyper text mark-up language), possibly using the conversion and editing tools provided for several standard word processing packages. The structure of the package/portfolio and relationships between the links included in the materials can be a reliable indicator of the depth of learning. Electronic portfolios would be plausible for a wide range of academic subject areas, and could also be used for group or peer assessments. Many students will be expected to use technology in future careers, and perhaps to compare and contrast the features offered by different packages. Asking students to create a simple package or portfolios of linked resources, to evaluate each other's products and produce a critical report on their own can foster a range of transferable skills, and promote a deeper level of learning. Packages can be targeted at a vocational use, or could illustrate and explore a single academic concept. In some situations the products created by one cohort may be suitable for use by following years. Simulations Simulations have been in use for many years to assess likely performance in hostile environments, particularly, but not exclusively, as a formative assessment tool. A simulated environment can never be quite like the real thing, as assumptions, simplifications and restrictions will have been programmed into any simulation - creating problems of realism. However, assessment problems associated with experiments in the real world, which sometimes behaving unpredictably, can be avoided in a simulated environment. Simulations can be used to investigate problem solving skills, perhaps allowing the student to explore a range of options. They provide a range of flexible assessment tools, suitable for individual or group exercises, under open or closed conditions. In addition to assessment of the academic content of the simulated environment, they can be used as a tool in the assessment of group interaction processes. Diagnostic assessment A range of diagnostic tools exist, both for student and tutor use and information. Two specific examples are introduced here to give some indication of the variety of tools available. DIAGNOSYS is a knowledge based package to investigate mathematics skills on entry to university. The questions have been designed with great care, it is not intended that users of the package should alter them. The tutor can identify which topic areas are relevant for investigation, and the student then sits a test, delivered and assessed by the computer. The test is adaptive, that is, not every student gets the same set of questions. As students answer correctly or incorrectly the computer selects a new question from its bank in order to check more fully on the particular skills of that student. Hence students are not faced with a whole series of questions that they cannot attempt to answer, or a whole series that trivial are to them. The diagnostic report can be made available for either the student or the tutor. Individual or class profiles can be provided. THESYS is a package designed as a formative self assessment tool for students preparing a project report. Thesys presents the student with a series of questions examining the structure and content of their report, and then provides detailed advice and suggestions on where additional information would be required in order to achieve a higher grade. An estimate of the grade that this project would achieve can also be provided. Summary The overall balance of assessments in a course is of vital importance, and although computerised testing facilities can provide a rapid means of assessing and providing feedback to large numbers of students it is 56

8 essential to consider their use as part of the overall course/unit strategy, especially as multiple choice/limited response type questions can lead to an emphasis on shallow learning. Electronic assessment tools are unlikely to reduce significantly the burden of assessment, but they can be used to promote deeper and more effective learning, by testing a range of skills, knowledge and understanding. Using computers in assessment does not have to mean more multiple choice testing to the exclusion of other assessment techniques. A wide range of innovative assessment methods lend themselves to computer based implementation. Further reading CTISS (1995) Computer-Assisted Assessment, Active Learning Issue No.1 CTISS NCET (1994) Using IT for Assessment: Going Forward, NCET NCET (1994) Using IT for Assessment: Directory of Software, NCET NCET (1995) Using IT for Assessment: Key Issues, NCET NCET (1995) Using IT for Assessment: Case Studies Reports, NCET HMSO (1993) Directory of Computer-Assisted Assessment Products and Procedures, Employment Department ALTER (1993) Workshop on Assessment of Learning in Higher Education, TLTP Project ALTER, Nov ALTER (1993) Using Technology to Assess Student Learning, TLTP Project ALTER, Dec ALTER (1993) Quality and Information Technology in Assessment Procedures, Occasional Green Paper No.5, TLTP Project ALTER, Oct Thorndike, Robert L (1971) Educational Measurement (2nd Edition), American Council for Education, Washington DC CTI Computing (1996) Monitor Issue No.6, CTI Computing, Jan TLTP Project 7 (1995) It in Teaching and Learning: A Staff Development Pack, TLTP Project 7, University of Durham 57