Language Testing and Validation

Transcription

1 Language Testing and Validation

2 Research and Practice in Applied Linguistics General Editors: Christopher N. Candlin and David R. Hall. All books in this series are written by leading researchers and teachers in Applied Linguistics, with broad international experience. They are designed for the MA or PhD student in Applied Linguistics, TESOL or similar subject areas and for the language professional keen to extend their research experience Titles include: Cyril J. Weir LANGUAGE TESTING AND VALIDATION Forthcoming titles: Martin Bygate and Virginia Samuda TASKS IN LANGUAGE LEARNING Francesca Bargiela, Catherine Nickerson and Brigitte Planken BUSINESS COMMUNICATION Sandra Gollin and David R. Hall LANGUAGE FOR SPECIFIC PURPOSES Sandra Hale COMMUNITY INTERPRETING Geoff Hall LITERATURE IN LANGUAGE EDUCATION Richard Kiely and Pauline Rea-Dickins PROGRAM EVALUATION IN LANGUAGE EDUCATION Martha Pennington PRONUNCIATION Devon Woods and Emese Bukor INSTRUCTIONAL STRATEGIES IN LANGUAGE EDUCATION Tony Wright LANGUAGE EDUCATION AND CLASSROOM MANAGEMENT Research and Practice in Applied Linguistics Series Standing Order ISBN hardcover Series Standing Order ISBN paperback (outside North America only) You can receive future titles in this series as they are published by placing a standing order. Please contact your bookseller or, in case of difficulty, write to us at the address below with your name and address, the title of the series and one of the ISBNs quoted above. Customer Services Department, Macmillan Distribution Ltd, Houndmills, Basingstoke, Hampshire RG21 6XS, England

3 Language Testing and Validation An Evidence-based Approach Cyril J. Weir Centre for Research in Testing, Evaluation and Curriculum (CRTEC) Roehampton University

4 Cyril J. Weir 2005 Softcover reprint of the hardcover 1st edition All rights reserved. No reproduction, copy or transmission of this publication may be made without written permission. No paragraph of this publication may be reproduced, copied or transmitted save with written permission or in accordance with the provisions of the Copyright, Designs and Patents Act 1988, or under the terms of any licence permitting limited copying issued by the Copyright Licensing Agency, 90 Tottenham Court Road, London W1T 4LP. Any person who does any unauthorised act in relation to this publication may be liable to criminal prosecution and civil claims for damages. The author has asserted his right to be identified as the author of this work in accordance with the Copyright, Designs and Patents Act First published 2005 by PALGRAVE MACMILLAN Houndmills, Basingstoke, Hampshire RG21 6XS and 175 Fifth Avenue, New York, N.Y Companies and representatives throughout the world PALGRAVE MACMILLAN is the global academic imprint of the Palgrave Macmillan division of St. Martin s Press, LLC and of Palgrave Macmillan Ltd. Macmillan is a registered trademark in the United States, United Kingdom and other countries. Palgrave is a registered trademark in the European Union and other countries. ISBN ISBN (ebook) DOI / This book is printed on paper suitable for recycling and made from fully managed and sustained forest sources. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress

5 To Shigeko and Jamie, and to my friends

6 This page intentionally left blank

7 Contents General Editors Preface x Acknowledgements xi Abbreviations xii Introduction 1 Part 1 Testing as Validity 3 1 Language Testing Past and Present The Cambridge Proficiency Examination : The Garden of Eden, the pre-scientific era Developments in the 1960s: the move towards a language-based examination The 1975 and 1984 revisions: The Promised Land? 8 2 The Nature of Test Validity 11 3 Before the Test Event: A Priori Validity Evidence Theory-based validity Context validity 19 4 After the Test Event: A Posteriori Validity Evidence Scoring validity Criterion-related validity Consequential validity 37 Part 2 New Frameworks for Developing and Validating Tests of Reading, Listening, Speaking and Writing 41 Introduction 43 5 Test Takers Physical/physiological characteristics: making accommodations Psychological characteristics: affective schemata Experiential characteristics: familiarity 54 6 Context Validity in Action Task setting Task demands Setting and test administration 82 vii

8 viii Contents 7 Theory-based Validity in Action Reading Listening Speaking Writing Response Formats Techniques for testing reading comprehension Techniques for testing listening comprehension Techniques for testing speaking Techniques for testing written production Scoring Validity in Action Scoring written production Scoring speaking tests Internal reliability of receptive tests Scores, grading and post-exam validation procedures External Validities in Action Criterion-related validity Consequential validity 210 Part 3 Generating Validity Evidence 217 Introduction Research Methodologies for Exploring the Validity of a Test An introductory note on research A priori validation: investigating the specification of the construct and the operationalization of the test Establishing context validity Establishing theory-based validity evidence Establishing scoring validity evidence Establishing evidence on a posteriori validities 259 Part 4 Further Resources in Language Testing Key Sources Books Journals Professional associations Principal testing conferences lists and bulletin boards Internet sites 277

9 Contents ix 12.7 Databases Statistical packages 280 Postscript 283 References 285 Index 299

10 General Editors Preface Research and Practice in Applied Linguistics is an international book series from Palgrave Macmillan which brings together leading researchers and teachers in Applied Linguistics to provide readers with the knowledge and tools they need to undertake their own practice-related research. Books in the series are designed for students and researchers in Applied Linguistics, TESOL, Language Education and related subject areas, and for language professionals keen to extend their research experience. Every book in this innovative series is designed to be user-friendly, with clear illustrations and accessible style. The quotations and definitions of key concepts that punctuate the main text are intended to ensure that many, often competing, voices are heard. Each book presents a concise historical and conceptual overview of its chosen field, identifying many lines of enquiry and findings, but also gaps and disagreements. It provides readers with an overall framework for further examination of how research and practice inform each other, and how practitioners can develop their own problem-based research. The focus throughout is on exploring the relationship between research and practice in Applied Linguistics. How far can research provide answers to the questions and issues that arise in practice? Can research questions that arise and are examined in very specific circumstances be informed by, and inform, the global body of research and practice? What different kinds of information can be obtained from different research methodologies? How should we make a selection between the options available, and how far are different methods compatible with each other? How can the results of research be turned into practical action? The books in this series identify some of the key researchable areas in the field and provide workable examples of research projects, backed up by details of appropriate research tools and resources. Case studies and exemplars of research and practice are drawn on throughout the books. References to key institutions, individual research lists, journals and professional organizations provide starting points for gathering information and embarking on research. The books also include annotated lists of key works in the field for further study. The overall objective of the series is to illustrate the message that in Applied Linguistics there can be no good professional practice that isn t based on good research, and there can be no good research that isn t informed by practice. Christopher N. Candlin Macquarie University, Sydney and Open University, UK David R. Hall Macquarie University, Sydney x

11 Acknowledgements I am especially grateful to Chris Candlin of Macquarie University for his encouraging and insightful comments at each stage in the preparation of the manuscript. His sure and encyclopedic grasp of the field of Applied Linguistics helped in the development of many of the ideas this book contains. A number of my PhD students and attachments at Roehampton University contributed to the conceptualization of testing as evidence-based validity in Part 2 of this book during their studies in the Centre for Research in Testing, Evaluation and Curriculum in ELT (CRTEC) at Roehampton University; in particular, Akmar Saidatul, Tony Green and Xiu Xudong, and Jessica Wu, Rachel Wu and Joyce Shao Chin of the LTTC in Taiwan. My colleague (in CRTEC) Barry O Sullivan contributed substantially to the framework for speaking. Glenn Fulcher was the source for many of the website references in Chapter 12. Jin Yan from the CET in China helped with the proof reading and commented reflectively on the text. Roger Hawkey read the typescript and made a number of useful suggestions. Akmar Saidatul helped with the index. Cambridge ESOL funded part of the research that this book is based on and staff in their Research and Validation Group commented critically on the ms at a number of stages. The publishers are grateful to the following for permission to reproduce copyright material: TOEFL materials are reprinted by permission of Educational Testing Service, the copyright owner. However the test questions and any other testing information are provided in their entirety by Palgrave Macmillan. No endorsement of this publication by Educational Testing Service should be inferred. LTTC GEPT materials are printed by permission of the Language Training and Testing Centre in Taiwan. USAID for the materials from the Student Achievement Test Development Manual, ed. H. Khalifa (2003), produced for use by teachers in Egypt. xi

12 Abbreviations AAAL ACTFL AERA AERT AILA ALTE ANOVA APA BAAL CALS CB CBT CEF CET CLA CPE CR CRTEC CUEFL EALTA EAP EFL ELBA ELT EPTB EPQ ERIC ESL ESOL ESP ETS FCE FL FSI GCE GCSE American Association of Applied Linguistics American Council on the Teaching of Foreign Languages American Educational Research Association Advanced English Reading Test International Association of Applied Linguistics Association of Language Testers in Europe Analysis of Variance American Psychological Association British Association of Applied Linguistics Centre for Applied Language Studies, University of Reading Computer-based Computer-based Test Common European Framework of Reference for Languages College English Test Communicative Language Ability Certificate of Proficiency in English Criterion-referenced Centre for Research in Testing Evaluation and Curriculum Communicative Use of English as a Foreign Language European Association for Language Testing and Assessment English for Academic Purposes English as a Foreign Language English Language Battery English Language Teaching English Proficiency Test Battery Eysenck Personality Questionnaire Educational Resources Information Center English as a Second Language English for Speakers of Other Languages English for Specified Purposes Educational Testing Service First Certificate in English Foreign Language Foreign Service Institute General Certificate of Education General Certificate of Secondary Education xii

13 Abbreviations xiii GEPT IATEFL IELTS ILR ILTA IRT JMB KR L1 L2 LTM LTRC LTTC LTU MCQ MFR MTMM NCME NNS OPI PBT PLAB RSA SAQ SATD SD SEM STM SLA SOPI SPSS TEEP TEM TL TLU TOEFL TOIEC TSE TSWE TWE UCLES General English Proficiency Test International Association for Teaching English as a Foreign Language International English Language Testing System Interagency Language Roundtable International Language Testing Association Item Response Theory Joint Matriculation Board (Northern Universities) Kudar Richardson First Language Second Language Long-term Memory Language Testing Research Colloquium Language Training and Testing Centre Language Testing Update Multiple-choice Question Multi-faceted Rasch Analysis Multi-trait, Multi-method National Council on Measurement in Education Non-native Speaker Oral Proficiency Interview Paper-Based Test General Medical Council s Professional and Linguistic Assessments Board (Test of overseas doctors language proficiency), United Kingdom Royal Society of Arts Short-answer Question Student Achievement Test Development Standard Deviation Standard Error of Measurement Short-term Memory Second Language Acquisition Simulated Oral Proficiency Interview Statistical Package for the Social Sciences Test of English for Educational Purposes Test for English Majors Target Language Target Language Use Test of English as a Foreign Language Test of English for International Communication Test of Spoken English Test of Standard Written English Test of Written English University of Cambridge Local Examinations Syndicate

14 xiv Language Testing and Validation UCRN UK USAID University of Cambridge Research Notes United Kingdom United States Agency for International Development