Outcome-Based Evaluation

Transcription

1 Outcome-Based Evaluation

2 Outcome-Based Evaluation Robert L. Schalock Hastings College Hastings, Nebraska Springer Science+Business Media, LLC

3 Library of Congress Cataloging-in-Publication Data Schalock. Robert L. Outcome-based evaluatlon! Robert L. Schalock. p. cm. Includes bibliographlcal references and lndex. ISBN Human servlces--evaluation--methodology. 2. Evaluation research (Soclal action programs) 3. Present value analysls. 1. Title. HV 11. S ' dc CIP ISBN DOI / ISBN (ebook) 1995 by Springer Science+Business Media New York Originally published by Plenum Press, New York in 1995 Softcover reprint of the hardcover lst edition AII rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission from the Publisher

4 To my students, program adminisitrator friends,colleagues, and my wife Susan, who have been receptive and supportive of my ideas and efforts throughout my career. My thanks also to Sharon Enga and Lori Christy for their expert technical assistance provided to me in writing this book.

5 Preface This book is the product of 30 years of experience with program evaluation. During this time, both service recipients and educational and social programs have experienced major cultural and political shifts in service delivery philosophy, including a focus on quality, mainstreaming, deinstitutionalization, community inclusion, and an emphasis on measurable outcomes. Recently stakeholders of these programs have demanded more than just the provision of service, forcing program administrators to evaluate their programs' effectiveness and efficiency. The "era of accountability" is here, and my major goal in writing this book is to help current and future program administrators understand that they need to look beyond simply the provision of service. Indeed, they need to be competent in outcome-based evaluation, which I define as a type of program evaluation that uses valued and objective person-referenced outcomes to analyze a program's effectiveness, impact or benefit-cost. By design, this book can be read from the perspective of a consumer or producer of outcome-based evaluation. As a consumer, the reader will be introduced to the various techniques used in outcome-based evaluation and how to interpret data from outcome-based evaluation analyses. As a producer, the reader will be instructed in how to do outcome-based evaluation analyses, along with how to use and act on their results. For both the consumer and producer, two questions should guide the use of outcomebased evaluation. For what purpose will I use the outcome-based evaluation data? What information will I need for the intended use? Numerous examples of outcome-based evaluations that reflect answers to these two questions will be provided throughout the text. The reader will encounter a number of key terms throughout the text. Chief among these are the following: Valued, person-referenced outcomes that reflect both the results of the intervention provided and an enhanced quality of life for the service recipient. vii

6 viii Preface Performance-based assessment that involves using objective indicators to evaluate a person's adaptive behavior level and role status. Outcome-based analyses that include effectiveness, impact, or benefit-cost. These analyses are used respectively to determine whether the program is meeting its goals, whether the program makes a significant difference, or whether the program represents a reasonable return on investment. Data-based management systems that are used to provide the information necessary for both outcome-based analysis and formative feedback that can be used by program administrators to increase their programs' effectiveness and efficiency. I have attempted to make the book as "user friendly" as possible. I realize that most of the readers are neither program evaluators nor statisticians. As a teacher and program evaluator for these 30 years, I have discovered that outcome-based evaluation requires primarily logical thinking and being clear in the questions asked. Once one knows where he/she is going and the basic road map to get there, then the journey is much easier. Thus, I have attempted throughout the text to provide the reader with easily read and followed tables, graphs, and exhibits that should facilitate both our tasks. For those readers who like to count and keep track, there are 16 figures, 33 tables, 34 exhibits, and 20 guiding principles that summarize key points. For those readers who want to go into greater detail, I have provided study questions and a list of additional readings for each chapter. Whether the book is read from cover to cover or by topical area, my goal has been to make your journey easier by stressing the critical need for thinking clearly and asking specific questions that can then be answered via one or more of the outcome-based evaluation analytic techniques discussed in the text's 13 chapters. The book is divided into five sections, beginning with an overview and concluding with the future of outcomebased evaluation. Working with program administrators for the last 30 years has made me sensitive to the challenging job they face. Their task is probably best exemplified in a recent book by Hammer and Champy (1993) entitled, Reegineering the Corporation: A Manifesto for Business Revolution. In the book, the authors discuss the "three Cs" of current service delivery: consumers, competition, and change. Consumers are asking more from education and social programs; fiscal restraints are forging new, competitive service delivery mechanisms; and change is constant. Thus, any program evaluation

7 Preface ix effort must be designed and implemented within the current zeitgeist that demands accountability within the context of the "three es." My hope and anticipation is that this text will assist present and future program administrators to understand and use outcome-based evaluation to demonstrate their programs' effectiveness, impact, or benefit-cost. If this is the case, the journey has been both beneficial and rewarding.

8 Contents I. AN OVERVIEW OF OUTCOME-BASED EVALUATION 1. The Whats and Whys of Outcome-Based Evaluation Overview... 3 Definitions of Program Evaluation... 4 The Whats of Outcome-Based Evaluation Outcome-Based Evaluation Model Outcome-Based Evaluation Compared to Other Program Evaluation Approaches The Whys of Outcome-Based Evaluation Summary Study Questions Additional Readings Where to Begin and How to Proceed Overview Guiding Principles Mission and Goals Services Provided Person-Referenced Outcomes Data Management System Summary Study Questions Additional Readings xi

9 xii Contents II. OUTCOME-BASED EVALUATION ANALYSES 3. Effectiveness Analysis Overview Steps Involved in Effectiveness Analysis Study 1: Longitudinal Status Analysis Study 2: Post-Secondary Status Analysis Summary Study Questions Additional Readings Impact Analysis Overview The Difference between Outcomes and Impacts Steps Involved in Impact Analysis Study 1: The Impact of Different Training Environments Study 2: The Impact of Transitional Employment Summary Study Questions Additional Readings Benefit-Cost Analysis Overview Historical Approaches to Benefit-Cost Analysis Benefit-Cost Analysis Model Benefit-Cost Analysis Procedural Steps Summary Study Questions Additional Readings III. OUTCOME-BASED EVALUATION DATA 6. Outcome-Based Evaluation Data Collection 99 Overview Recipient Characteristics

10 Contents xiii Core-Service Functions Cost Estimates Person-Referenced Outcomes Measurement Techniques Guidelines for Outcome-Based Evaluation Data Collection Summary Study Questions Additional Readings Outcome-Based Evaluation Data Management Overview Trends Impacting Data Utilization Data Management Model Data Collection Principles Data Management Issues Summary Study Questions Additional Readings Outcome-Based Evaluation Designs and Data Analysis Overview Outcome-Based Evaluation Designs General Principles Regarding Statistical Analyses Common Outcome-Based Evaluation Analyses Summary Study Questions Additional Readings IV. REPORTING AND ACTING ON OUTCOME-BASED EVALUATION RESULTS 9. Contextual Analysis Overview Key Contextual Variables Contextual Analysis Approaches Summary Study Questions Additional Readings

11 xiv Contents 10. Reporting Outcome-Based Evaluation Results 187 Overview Challenges to Reporting Outcome-Based Evaluation Results Establishing Credibility Effective Communication Techniques Summary Study Questions Additional Readings Acting On Outcome-Based Evaluation Results 201 Overview Quality-Management Techniques Staff Training Quality-Enhancement Techniques Shared Quality Assurance Summary Study Questions Additional Readings V. THE FUTURE 12. Factors Influencing The Future of Outcome-Based Evaluation 215 Overview Noncategorical Approach to Services and Supports Accountability Defined on the Basis of Outcomes Multiple-Evaluation Designs Service-Provider Networks Consumer-Oriented Evaluation Linkage of Program Evaluation and Forecasting Summary Conclusion Study Questions Additional Readings

12 Contents xv 13. Those Guiding Principles You Should Not Forget 227 References Author Index Subject Index

13 Outcome-Based Evaluation