Language Testing and Validation

Language Testing and Validation

Research and Practice in Applied Linguistics General Editors: Christopher N. Candlin and David R. Hall. All books in this series are written by leading researchers and teachers in Applied Linguistics, with broad international experience. They are designed for the MA or PhD student in Applied Linguistics, TESOL or similar subject areas and for the language professional keen to extend their research experience Titles include: Cyril J. Weir LANGUAGE TESTING AND VALIDATION Forthcoming titles: Martin Bygate and Virginia Samuda TASKS IN LANGUAGE LEARNING Francesca Bargiela, Catherine Nickerson and Brigitte Planken BUSINESS COMMUNICATION Sandra Gollin and David R. Hall LANGUAGE FOR SPECIFIC PURPOSES Sandra Hale COMMUNITY INTERPRETING Geoff Hall LITERATURE IN LANGUAGE EDUCATION Richard Kiely and Pauline Rea-Dickins PROGRAM EVALUATION IN LANGUAGE EDUCATION Martha Pennington PRONUNCIATION Devon Woods and Emese Bukor INSTRUCTIONAL STRATEGIES IN LANGUAGE EDUCATION Tony Wright LANGUAGE EDUCATION AND CLASSROOM MANAGEMENT Research and Practice in Applied Linguistics Series Standing Order ISBN 1 4039 1184 3 hardcover Series Standing Order ISBN 1 4039 1185 1 paperback (outside North America only) You can receive future titles in this series as they are published by placing a standing order. Please contact your bookseller or, in case of difficulty, write to us at the address below with your name and address, the title of the series and one of the ISBNs quoted above. Customer Services Department, Macmillan Distribution Ltd, Houndmills, Basingstoke, Hampshire RG21 6XS, England

Language Testing and Validation An Evidence-based Approach Cyril J. Weir Centre for Research in Testing, Evaluation and Curriculum (CRTEC) Roehampton University

Cyril J. Weir 2005 Softcover reprint of the hardcover 1st edition 2005 978-1-4039-1188-9 All rights reserved. No reproduction, copy or transmission of this publication may be made without written permission. No paragraph of this publication may be reproduced, copied or transmitted save with written permission or in accordance with the provisions of the Copyright, Designs and Patents Act 1988, or under the terms of any licence permitting limited copying issued by the Copyright Licensing Agency, 90 Tottenham Court Road, London W1T 4LP. Any person who does any unauthorised act in relation to this publication may be liable to criminal prosecution and civil claims for damages. The author has asserted his right to be identified as the author of this work in accordance with the Copyright, Designs and Patents Act 1988. First published 2005 by PALGRAVE MACMILLAN Houndmills, Basingstoke, Hampshire RG21 6XS and 175 Fifth Avenue, New York, N.Y. 10010 Companies and representatives throughout the world PALGRAVE MACMILLAN is the global academic imprint of the Palgrave Macmillan division of St. Martin s Press, LLC and of Palgrave Macmillan Ltd. Macmillan is a registered trademark in the United States, United Kingdom and other countries. Palgrave is a registered trademark in the European Union and other countries. ISBN 978-1-4039-1189-6 ISBN 978-0-230-51457-7 (ebook) DOI 10.1057/9780230514577 This book is printed on paper suitable for recycling and made from fully managed and sustained forest sources. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress. 10 9 8 7 6 5 4 3 2 1 14 13 12 11 10 09 08 07 06 05

To Shigeko and Jamie, and to my friends

This page intentionally left blank

Contents General Editors Preface x Acknowledgements xi Abbreviations xii Introduction 1 Part 1 Testing as Validity 3 1 Language Testing Past and Present 5 1.1 The Cambridge Proficiency Examination 1913 1945: The Garden of Eden, the pre-scientific era 5 1.2 Developments in the 1960s: the move towards a language-based examination 7 1.3 The 1975 and 1984 revisions: The Promised Land? 8 2 The Nature of Test Validity 11 3 Before the Test Event: A Priori Validity Evidence 17 3.1 Theory-based validity 17 3.2 Context validity 19 4 After the Test Event: A Posteriori Validity Evidence 22 4.1 Scoring validity 22 4.2 Criterion-related validity 35 4.3 Consequential validity 37 Part 2 New Frameworks for Developing and Validating Tests of Reading, Listening, Speaking and Writing 41 Introduction 43 5 Test Takers 51 5.1 Physical/physiological characteristics: making accommodations 52 5.2 Psychological characteristics: affective schemata 53 5.3 Experiential characteristics: familiarity 54 6 Context Validity in Action 56 6.1 Task setting 57 6.2 Task demands 68 6.3 Setting and test administration 82 vii

viii Contents 7 Theory-based Validity in Action 85 7.1 Reading 87 7.2 Listening 95 7.3 Speaking 102 7.4 Writing 108 8 Response Formats 119 8.1 Techniques for testing reading comprehension 119 8.2 Techniques for testing listening comprehension 132 8.3 Techniques for testing speaking 143 8.4 Techniques for testing written production 161 9 Scoring Validity in Action 177 9.1 Scoring written production 179 9.2 Scoring speaking tests 191 9.3 Internal reliability of receptive tests 201 9.4 Scores, grading and post-exam validation procedures 205 10 External Validities in Action 207 10.1 Criterion-related validity 207 10.2 Consequential validity 210 Part 3 Generating Validity Evidence 217 Introduction 219 11 Research Methodologies for Exploring the Validity of a Test 221 11.1 An introductory note on research 221 11.2 A priori validation: investigating the specification of the construct and the operationalization of the test 222 11.3 Establishing context validity 224 11.4 Establishing theory-based validity evidence 233 11.5 Establishing scoring validity evidence 247 11.6 Establishing evidence on a posteriori validities 259 Part 4 Further Resources in Language Testing 271 12 Key Sources 273 12.1 Books 273 12.2 Journals 274 12.3 Professional associations 276 12.4 Principal testing conferences 276 12.5 Email lists and bulletin boards 277 12.6 Internet sites 277

Contents ix 12.7 Databases 280 12.8 Statistical packages 280 Postscript 283 References 285 Index 299

General Editors Preface Research and Practice in Applied Linguistics is an international book series from Palgrave Macmillan which brings together leading researchers and teachers in Applied Linguistics to provide readers with the knowledge and tools they need to undertake their own practice-related research. Books in the series are designed for students and researchers in Applied Linguistics, TESOL, Language Education and related subject areas, and for language professionals keen to extend their research experience. Every book in this innovative series is designed to be user-friendly, with clear illustrations and accessible style. The quotations and definitions of key concepts that punctuate the main text are intended to ensure that many, often competing, voices are heard. Each book presents a concise historical and conceptual overview of its chosen field, identifying many lines of enquiry and findings, but also gaps and disagreements. It provides readers with an overall framework for further examination of how research and practice inform each other, and how practitioners can develop their own problem-based research. The focus throughout is on exploring the relationship between research and practice in Applied Linguistics. How far can research provide answers to the questions and issues that arise in practice? Can research questions that arise and are examined in very specific circumstances be informed by, and inform, the global body of research and practice? What different kinds of information can be obtained from different research methodologies? How should we make a selection between the options available, and how far are different methods compatible with each other? How can the results of research be turned into practical action? The books in this series identify some of the key researchable areas in the field and provide workable examples of research projects, backed up by details of appropriate research tools and resources. Case studies and exemplars of research and practice are drawn on throughout the books. References to key institutions, individual research lists, journals and professional organizations provide starting points for gathering information and embarking on research. The books also include annotated lists of key works in the field for further study. The overall objective of the series is to illustrate the message that in Applied Linguistics there can be no good professional practice that isn t based on good research, and there can be no good research that isn t informed by practice. Christopher N. Candlin Macquarie University, Sydney and Open University, UK David R. Hall Macquarie University, Sydney x

Acknowledgements I am especially grateful to Chris Candlin of Macquarie University for his encouraging and insightful comments at each stage in the preparation of the manuscript. His sure and encyclopedic grasp of the field of Applied Linguistics helped in the development of many of the ideas this book contains. A number of my PhD students and attachments at Roehampton University contributed to the conceptualization of testing as evidence-based validity in Part 2 of this book during their studies in the Centre for Research in Testing, Evaluation and Curriculum in ELT (CRTEC) at Roehampton University; in particular, Akmar Saidatul, Tony Green and Xiu Xudong, and Jessica Wu, Rachel Wu and Joyce Shao Chin of the LTTC in Taiwan. My colleague (in CRTEC) Barry O Sullivan contributed substantially to the framework for speaking. Glenn Fulcher was the source for many of the website references in Chapter 12. Jin Yan from the CET in China helped with the proof reading and commented reflectively on the text. Roger Hawkey read the typescript and made a number of useful suggestions. Akmar Saidatul helped with the index. Cambridge ESOL funded part of the research that this book is based on and staff in their Research and Validation Group commented critically on the ms at a number of stages. The publishers are grateful to the following for permission to reproduce copyright material: TOEFL materials are reprinted by permission of Educational Testing Service, the copyright owner. However the test questions and any other testing information are provided in their entirety by Palgrave Macmillan. No endorsement of this publication by Educational Testing Service should be inferred. LTTC GEPT materials are printed by permission of the Language Training and Testing Centre in Taiwan. USAID for the materials from the Student Achievement Test Development Manual, ed. H. Khalifa (2003), produced for use by teachers in Egypt. xi

Abbreviations AAAL ACTFL AERA AERT AILA ALTE ANOVA APA BAAL CALS CB CBT CEF CET CLA CPE CR CRTEC CUEFL EALTA EAP EFL ELBA ELT EPTB EPQ ERIC ESL ESOL ESP ETS FCE FL FSI GCE GCSE American Association of Applied Linguistics American Council on the Teaching of Foreign Languages American Educational Research Association Advanced English Reading Test International Association of Applied Linguistics Association of Language Testers in Europe Analysis of Variance American Psychological Association British Association of Applied Linguistics Centre for Applied Language Studies, University of Reading Computer-based Computer-based Test Common European Framework of Reference for Languages College English Test Communicative Language Ability Certificate of Proficiency in English Criterion-referenced Centre for Research in Testing Evaluation and Curriculum Communicative Use of English as a Foreign Language European Association for Language Testing and Assessment English for Academic Purposes English as a Foreign Language English Language Battery English Language Teaching English Proficiency Test Battery Eysenck Personality Questionnaire Educational Resources Information Center English as a Second Language English for Speakers of Other Languages English for Specified Purposes Educational Testing Service First Certificate in English Foreign Language Foreign Service Institute General Certificate of Education General Certificate of Secondary Education xii

Abbreviations xiii GEPT IATEFL IELTS ILR ILTA IRT JMB KR L1 L2 LTM LTRC LTTC LTU MCQ MFR MTMM NCME NNS OPI PBT PLAB RSA SAQ SATD SD SEM STM SLA SOPI SPSS TEEP TEM TL TLU TOEFL TOIEC TSE TSWE TWE UCLES General English Proficiency Test International Association for Teaching English as a Foreign Language International English Language Testing System Interagency Language Roundtable International Language Testing Association Item Response Theory Joint Matriculation Board (Northern Universities) Kudar Richardson First Language Second Language Long-term Memory Language Testing Research Colloquium Language Training and Testing Centre Language Testing Update Multiple-choice Question Multi-faceted Rasch Analysis Multi-trait, Multi-method National Council on Measurement in Education Non-native Speaker Oral Proficiency Interview Paper-Based Test General Medical Council s Professional and Linguistic Assessments Board (Test of overseas doctors language proficiency), United Kingdom Royal Society of Arts Short-answer Question Student Achievement Test Development Standard Deviation Standard Error of Measurement Short-term Memory Second Language Acquisition Simulated Oral Proficiency Interview Statistical Package for the Social Sciences Test of English for Educational Purposes Test for English Majors Target Language Target Language Use Test of English as a Foreign Language Test of English for International Communication Test of Spoken English Test of Standard Written English Test of Written English University of Cambridge Local Examinations Syndicate

xiv Language Testing and Validation UCRN UK USAID University of Cambridge Research Notes United Kingdom United States Agency for International Development