Developing and Using Classroom Assessments Albert Oosterhof Fourth Edition

Developing and Using Classroom Assessments Albert Oosterhof Fourth Edition

Pearson Education Limited Edinburgh Gate Harlow Essex CM20 2JE England and Associated Companies throughout the world Visit us on the World Wide Web at: www.pearsoned.co.uk Pearson Education Limited 2014 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without either the prior written permission of the publisher or a licence permitting restricted copying in the United Kingdom issued by the Copyright Licensing Agency Ltd, Saffron House, 6 10 Kirby Street, London EC1N 8TS. All trademarks used herein are the property of their respective owners. The use of any trademark in this text does not vest in the author or publisher any trademark ownership rights in such trademarks, nor does the use of such trademarks imply any affiliation with or endorsement of this book by such owners. ISBN 10: 1-292-04115-3 ISBN 10: 1-269-37450-8 ISBN 13: 978-1-292-04115-5 ISBN 13: 978-1-269-37450-7 British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Printed in the United States of America

92 PART II How to Develop, Administer, and Score Written Tests Limitation: Limited quantity of content can be assessed Explanation: Because of time required to write out answer, few essay questions can be included within a test Limitation: Scoring problems Explanation: Different raters often assign different scores, and a greater amount of time is required to complete scoring 7.2 1. Neither the total number of points nor the attributes to be evaluated is specified. 2. Neither the total number of points nor guidelines for awarding points is specified. 7.3 1. A; 2. B. Within the first pair, item A is so general that a knowledgeable student could cite a variety of correct answers that were not included in the scoring plan. Within the second pair, the correct answer for item B is largely a matter of opinion or circumstance. SOMETHING TO TRY If you have access to some previously written essay items, use the qualities listed in Figure 7.4 to evaluate these items, including the scoring plan. Prepare an essay item with a scoring plan that measures the following objective: Information: Using an actual example, describe the procedure Congress is to follow for attempting to override a presidential veto. Use Figure 7.4 to evaluate this item. An essay item written to measure this objective probably can be answered using two to four sentences. Although this represents a minimal essay question, it provides a useful context for writing a concise essay item and scoring plan. Prepare an essay item with a scoring plan that measures the following objective: Information: State the relative advantages of various types of motor vehicles such as fullsize cars, compact cars, SUVs, and light trucks, including a statement concerning the cause of each advantage. Construct the essay item so it requires approximately 5 minutes to answer and involves a comparison of two specified types of vehicles. Use qualities listed in Figure 7.4 to evaluate the item. This includes being sure that the task specified by the essay question closely matches the scoring plan. Similarly write and evaluate essay items, with scoring plans, for an objective within your academic specialization. ADDITIONAL READING Chung, K. W. K., & O Neil, H. F. (1997). Methodological approaches to online scoring of essays. CSE Technical Report 461, Los Angeles: Center for the Study of Evaluation, University of California. Available online at http://www.cse.ucla.edu/products/reports/tech461.pdf. Computer scoring of essays is an emerging capability that presently is used with some standardized tests. This report discusses two approaches to computer-scored essays, describing what they do and how effectively they work. Coffman, W. E. (1971). Essay examinations. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.). Washington, DC: American Council on Education. This chapter, written almost 50 years ago, provides a thorough discussion of advantages, limitations, and research issues related to essay questions as well as a description of procedures for improving the development and scoring of essay questions. 75

76

Multiple-Choice Items 8 What multiple-choice items can and cannot do Qualities multiple-choice items should have Determining the optimal number of options How to ensure their validity Useful variations of the multiple-choice format The multiple-choice format is one of the most popular item formats used in educational testing. Many group-administered standardized tests consist entirely of multiple-choice items. The multiple-choice format is also used extensively in classroom tests, particularly in the middle grades through college. The multiple-choice item traditionally consists of a stem that describes a problem and a series of options, or alternatives, each representing possible answers to the stem. Normally, one option is correct, with the remaining alternatives referred to as distractors, or foils. The suggestion is often made that the multiplechoice format is limited to testing recall of information. Because the correct response is always included among the item s alternatives, this type of item sometimes has been nicknamed multipleguess and assumed to be unable to measure skills assessed by the essay and completion formats. The multiple-choice format does have distinct limitations; however, effectively constructed multiple-choice items also have significant advantages. Many multiple-choice items used in classroom tests could achieve these advantages if they were constructed more effectively. Based on your own experience with multiple-choice tests, you probably will anticipate many of the qualities desired in multiple-choice items, such as a clearly expressed statement of the problem in the stem and judicious use of the all of the above option. This chapter will help you identify the qualities inherent in the better multiple-choice items. Later in the chapter you will be asked to use these qualities to evaluate a series of multiplechoice items. Considerable flexibility exists within the multiple-choice format. This flexibility will be easier to anticipate by broadening the traditional definition of multiple-choice options. Although the stem presents a problem to be addressed by students, the options should not be thought of as possible solutions to this problem. Instead, these options are the means through which students transform proposed solutions to a mark recorded in the test booklet or on an answer sheet. Look at the multiple-choice items illustrated in Figure 8.1. The directions serve as the stem in that they present the problem to be addressed. The multiplechoice options are the lines into which the individual sentences are formatted. These options do not represent alternative solutions to the problem presented by the stem. They do control how students mark their answer sheets. The options are designed so that students who are unable to solve the problem will typically mark a different answer than will those who understand the concept. Again, think creatively and more broadly, using multiple-choice options to structure how students transform their proposed solutions to a mark in the test booklet or answer sheet. For instance, multiple-choice items that are free of written material, such as the item illustrated in Figure 8.2, are appropriate for very young children. 93 From Chapter 8 of Developing and Using Classroom Assessments, 4/e. Albert Oosterhof. Copyright 2009 by Pearson Merrill. All rights reserved. 77

94 PART II How to Develop, Administer, and Score Written Tests Directions: For each sentence, mark on your answer sheet the letter identifying the line that contains an adjective. Mark E if the sentence contains no adjective. 1. A. Krueger National B. Park is a C. game reserve D. in South Africa. E. (No adjective) 2. A. This park was B. established in 1898 C. to protect wildlife D. in the region. E. (No adjective) FIGURE 8.1 Illustration of multiple-choice items using embedded options With this and similar test items, students are orally instructed to draw a line joining the two circles containing squares arranged exactly the same way. FIGURE 8.2 Illustration of a multiple-choice item containing no written text Although standardized tests are increasingly using performance assessments and various construct-response formats, the multiple-choice item still dominates and is likely to continue to do so for some time. This encourages introduction of multiple-choice items in early grades. Computeraided lessons often incorporate multiple-choice items because it often is difficult for a computer to adequately process answers students provide with other item formats. This chapter helps you achieve five skills: Identify what multiple-choice items can do, recognizing issues related to their validity Identify qualities desired in multiple-choice items Evaluate multiple-choice items for these qualities Identify variations of multiple-choice items Determine the optimal number of options to be included within a multiple-choice item WHAT MULTIPLE-CHOICE ITEMS CAN DO Multiple-choice is quite adaptable with respect to the type of capabilities it can measure. Certainly multiple-choice items can measure declarative knowledge or information. Sometimes it is assumed that the multiple-choice format is limited to cognitively lower levels of information, although we will find that is not the case. Multiplechoice is also able to measure procedural knowledge involving concepts and rules. Recall that concepts, for instance, are assessed by providing students a variety of illustrations and asking them to indicate whether each is an example or non-example of the concept. To effectively assess knowledge of a concept, a series of correct and incorrect illustrations must be provided students to ensure they have learned the complexities and subtleties of the concept. Unlike with the essay format, a test can include a number of multiple-choice items, which is essential to broadly assessing knowledge of a particular concept or rule. Multiple-choice items cannot assess problem solving. Remember that in this book, problem solving does not refer to solving math problem such as multiplying pairs of three-digit numbers or even more advanced procedures such as working 78

CHAPTER 8 Multiple-Choice Items 95 with integrals. Those math skills involve procedural knowledge, specifically rules. Once the relevant rule is learned, one can apply the procedure to other similar multiplication problems or integrals. In cognitive psychology, a problem to be solved exists when one has a goal and has not yet identified a means for reaching that goal. Problem solving involves a sequence of steps, beginning with establishing a representation of the problem, then selecting a strategy that seems appropriate for resolving the problem, followed by evaluating the results of employing that strategy. Multiple-choice items cannot emulate this sequence of steps, nor can other written test formats, including essay. Performance assessments can assess problem solving, which is why later chapters look closely at that assessment option. In addition to being able to broadly sample content, and to measure both declarative and procedural knowledge, multiple-choice items are particularly effective at structuring the task to be performed, particularly when compared to completion items. Consider the following items. Completion: When backing a car out of a garage, the first thing one should do is. Multiple-choice: Which among the following should be done first when backing a car out of a garage? A. Lock the car doors B. Place the foot on the break pedal before shifting into reverse C. Start the engine D. Walk behind the car looking for obstacles A knowledgeable student answering the completion item might anticipate several actions, any of which could occur first when backing a car out of a garage, such as opening the garage door. The context provided by the options makes it easier to structure the task with a multiple-choice item than with other written formats. Multiple-choice items can be quickly and objectively scored. Because students respond to each item with a single mark, multiple-choice items are scored very quickly. Except with students in early elementary grades, scoring efficiency can often be improved by having students mark responses in a blank to the side of each item, or on a separate answer sheet. When responses are marked on a separate answer sheet, tests can be scored by machine. These machines score answer sheets for an entire class in a couple of minutes. Scoring machines often print each student s score on the answer sheet and mark incorrect answers. Many of these scoring machines will enter each student s responses into a computer so that patterns in students answers can be quickly tabulated and used diagnostically. Reliability can suffer greatly from inconsistencies in scoring students responses to completion and particularly essay items. Inconsistency in scoring is negligible with multiple-choice items. When given the answer key, two individuals are likely to assign the same score to a given student s responses. Because multiple-choice items are objectively scored, students, teacher aides, or machines can score them. Although objectively scored, multiple-choice tests contain other sources of inconsistency. We will look carefully at this issue later in the chapter. 8.1 Apply What You Are Learning Which item format has each of the following qualities? For each item, identify the one item format that best exemplifies the quality being described. A. Completion B. Essay C. Multiple-choice 1. Students answers can be quickly scored. 2. Test items can be quickly constructed. 3. Skills can be more adequately sampled within a single test. 4. Students ability to express ideas in writing can be evaluated. 5. Students responses can be scored with minimal error. Answers can be found at the end of the chapter. 79