Outcome-Based Evaluation

Outcome-Based Evaluation

Outcome-Based Evaluation Robert L. Schalock Hastings College Hastings, Nebraska Springer Science+Business Media, LLC

Library of Congress Cataloging-in-Publication Data Schalock. Robert L. Outcome-based evaluatlon! Robert L. Schalock. p. cm. Includes bibliographlcal references and lndex. ISBN 0-306-45051-8 1. Human servlces--evaluation--methodology. 2. Evaluation research (Soclal action programs) 3. Present value analysls. 1. Title. HV 11. S36 1995 361'.0068--dc20 95-33352 CIP ISBN 978-1-4757-2401-1 DOI 10.1007/978-1-4757-2399-1 ISBN 978-1-4757-2399-1 (ebook) 1995 by Springer Science+Business Media New York Originally published by Plenum Press, New York in 1995 Softcover reprint of the hardcover lst edition 1995 109876543 AII rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission from the Publisher

To my students, program adminisitrator friends,colleagues, and my wife Susan, who have been receptive and supportive of my ideas and efforts throughout my career. My thanks also to Sharon Enga and Lori Christy for their expert technical assistance provided to me in writing this book.

Preface This book is the product of 30 years of experience with program evaluation. During this time, both service recipients and educational and social programs have experienced major cultural and political shifts in service delivery philosophy, including a focus on quality, mainstreaming, deinstitutionalization, community inclusion, and an emphasis on measurable outcomes. Recently stakeholders of these programs have demanded more than just the provision of service, forcing program administrators to evaluate their programs' effectiveness and efficiency. The "era of accountability" is here, and my major goal in writing this book is to help current and future program administrators understand that they need to look beyond simply the provision of service. Indeed, they need to be competent in outcome-based evaluation, which I define as a type of program evaluation that uses valued and objective person-referenced outcomes to analyze a program's effectiveness, impact or benefit-cost. By design, this book can be read from the perspective of a consumer or producer of outcome-based evaluation. As a consumer, the reader will be introduced to the various techniques used in outcome-based evaluation and how to interpret data from outcome-based evaluation analyses. As a producer, the reader will be instructed in how to do outcome-based evaluation analyses, along with how to use and act on their results. For both the consumer and producer, two questions should guide the use of outcomebased evaluation. For what purpose will I use the outcome-based evaluation data? What information will I need for the intended use? Numerous examples of outcome-based evaluations that reflect answers to these two questions will be provided throughout the text. The reader will encounter a number of key terms throughout the text. Chief among these are the following: Valued, person-referenced outcomes that reflect both the results of the intervention provided and an enhanced quality of life for the service recipient. vii

viii Preface Performance-based assessment that involves using objective indicators to evaluate a person's adaptive behavior level and role status. Outcome-based analyses that include effectiveness, impact, or benefit-cost. These analyses are used respectively to determine whether the program is meeting its goals, whether the program makes a significant difference, or whether the program represents a reasonable return on investment. Data-based management systems that are used to provide the information necessary for both outcome-based analysis and formative feedback that can be used by program administrators to increase their programs' effectiveness and efficiency. I have attempted to make the book as "user friendly" as possible. I realize that most of the readers are neither program evaluators nor statisticians. As a teacher and program evaluator for these 30 years, I have discovered that outcome-based evaluation requires primarily logical thinking and being clear in the questions asked. Once one knows where he/she is going and the basic road map to get there, then the journey is much easier. Thus, I have attempted throughout the text to provide the reader with easily read and followed tables, graphs, and exhibits that should facilitate both our tasks. For those readers who like to count and keep track, there are 16 figures, 33 tables, 34 exhibits, and 20 guiding principles that summarize key points. For those readers who want to go into greater detail, I have provided study questions and a list of additional readings for each chapter. Whether the book is read from cover to cover or by topical area, my goal has been to make your journey easier by stressing the critical need for thinking clearly and asking specific questions that can then be answered via one or more of the outcome-based evaluation analytic techniques discussed in the text's 13 chapters. The book is divided into five sections, beginning with an overview and concluding with the future of outcomebased evaluation. Working with program administrators for the last 30 years has made me sensitive to the challenging job they face. Their task is probably best exemplified in a recent book by Hammer and Champy (1993) entitled, Reegineering the Corporation: A Manifesto for Business Revolution. In the book, the authors discuss the "three Cs" of current service delivery: consumers, competition, and change. Consumers are asking more from education and social programs; fiscal restraints are forging new, competitive service delivery mechanisms; and change is constant. Thus, any program evaluation

Preface ix effort must be designed and implemented within the current zeitgeist that demands accountability within the context of the "three es." My hope and anticipation is that this text will assist present and future program administrators to understand and use outcome-based evaluation to demonstrate their programs' effectiveness, impact, or benefit-cost. If this is the case, the journey has been both beneficial and rewarding.

Contents I. AN OVERVIEW OF OUTCOME-BASED EVALUATION 1. The Whats and Whys of Outcome-Based Evaluation......... 3 Overview... 3 Definitions of Program Evaluation... 4 The Whats of Outcome-Based Evaluation...................... 5 Outcome-Based Evaluation Model... 14 Outcome-Based Evaluation Compared to Other Program Evaluation Approaches.................................... 16 The Whys of Outcome-Based Evaluation... 17 Summary... 22 Study Questions... 23 Additional Readings... 24 2. Where to Begin and How to Proceed... 25 Overview... 25 Guiding Principles... 26 Mission and Goals... 28 Services Provided........................................... 30 Person-Referenced Outcomes................................. 31 Data Management System... 31 Summary... 35 Study Questions............................................ 36 Additional Readings... 37 xi

xii Contents II. OUTCOME-BASED EVALUATION ANALYSES 3. Effectiveness Analysis... 43 Overview... 43 Steps Involved in Effectiveness Analysis....................... 44 Study 1: Longitudinal Status Analysis......................... 44 Study 2: Post-Secondary Status Analysis....................... 49 Summary... 53 Study Questions... 54 Additional Readings... 55 4. Impact Analysis... 57 Overview... 57 The Difference between Outcomes and Impacts... 58 Steps Involved in Impact Analysis... 60 Study 1: The Impact of Different Training Environments... 61 Study 2: The Impact of Transitional Employment............... 66 Summary... 72 Study Questions... 73 Additional Readings... 74 5. Benefit-Cost Analysis..................................... 75 Overview... 75 Historical Approaches to Benefit-Cost Analysis................. 76 Benefit-Cost Analysis Model... 77 Benefit-Cost Analysis Procedural Steps........................ 78 Summary... 92 Study Questions... 93 Additional Readings... 94 III. OUTCOME-BASED EVALUATION DATA 6. Outcome-Based Evaluation Data Collection 99 Overview... 100 Recipient Characteristics..................................... 101

Contents xiii Core-Service Functions... 106 Cost Estimates.............................................. 110 Person-Referenced Outcomes................................. 115 Measurement Techniques... 115 Guidelines for Outcome-Based Evaluation Data Collection... 118 Summary... 121 Study Questions... 122 Additional Readings... 123 7. Outcome-Based Evaluation Data Management.............. 125 Overview.................................................. 125 Trends Impacting Data Utilization... 126 Data Management Model.................................... 127 Data Collection Principles... 128 Data Management Issues... 133 Summary... 136 Study Questions... 137 Additional Readings... 138 8. Outcome-Based Evaluation Designs and Data Analysis... 139 Overview... 139 Outcome-Based Evaluation Designs........................... 141 General Principles Regarding Statistical Analyses............... 155 Common Outcome-Based Evaluation Analyses................. 161 Summary... 166 Study Questions... 168 Additional Readings... 169 IV. REPORTING AND ACTING ON OUTCOME-BASED EVALUATION RESULTS 9. Contextual Analysis...................................... 173 Overview... 173 Key Contextual Variables... 174 Contextual Analysis Approaches... 181 Summary... 184 Study Questions... 184 Additional Readings... 184

xiv Contents 10. Reporting Outcome-Based Evaluation Results 187 Overview... 187 Challenges to Reporting Outcome-Based Evaluation Results... 188 Establishing Credibility...................................... 189 Effective Communication Techniques.......................... 192 Summary... 193 Study Questions... 198 Additional Readings... 198 11. Acting On Outcome-Based Evaluation Results 201 Overview... 201 Quality-Management Techniques... 203 Staff Training............................................... 204 Quality-Enhancement Techniques............................. 207 Shared Quality Assurance... 207 Summary... 208 Study Questions... 210 Additional Readings... 211 V. THE FUTURE 12. Factors Influencing The Future of Outcome-Based Evaluation 215 Overview... 215 Noncategorical Approach to Services and Supports............. 216 Accountability Defined on the Basis of Outcomes... 217 Multiple-Evaluation Designs... 218 Service-Provider Networks................................... 219 Consumer-Oriented Evaluation............................... 221 Linkage of Program Evaluation and Forecasting................ 222 Summary... 222 Conclusion................................................. 223 Study Questions... 224 Additional Readings... 224

Contents xv 13. Those Guiding Principles You Should Not Forget 227 References... 229 Author Index............................................... 237 Subject Index............................................... 239

Outcome-Based Evaluation