Syllabus Version 2.0, 10/22/08 SPP636: Program Evaluation Monday, Wednesday 10:00-11:30 am, 1230 Weill Hall Professor Elisabeth Gerber, 5228 Weill Hall, 647-4004, (ergerber@umich.edu) GSI John English, 3202 Weill Hall, 615-4245, (engjohn@umich.edu) Purpose: The objective of this course is to learn the tools that are frequently used to determine whether public policies and programs are achieving their objectives. A great deal of money is spent on evaluations, and they are difficult to conduct successfully. All evaluations have weaknesses, and some have more weaknesses than others; you will learn to distinguish high from low quality evaluations. Policies and programs in a broad range of areas will be examined, including health, criminal justice, education, welfare, and development. Materials will be presented in several ways, including but not limited to lectures, guest lectures, in-class exercises, student presentations, classroom discussions, and problem sets. Syllabus: This syllabus is subject to change at any time. Changes in reading assignments and discussion topics may be announced in class. Students are responsible for all materials on this syllabus unless instructed otherwise by the professor, plus any additional materials assigned in class. readings are available on CTools (https://ctools.umich.edu/portal). Full citations are provided in the bibliography section of this syllabus. Please do the assigned readings before class. Come prepared to discuss them and to provide examples that illustrate points raised in the readings. You will be responsible for all assigned materials, whether or not they are discussed in class. Prerequisites: PUBPOL 529 (Statistics) or its equivalent is required for this course. PUBPOL 569 (Applied Regression Analysis) or its equivalent is strongly recommended. Assignments and Grading: Problem sets must be submitted at the beginning of the class period on the day they are due. Assignments submitted late will be docked one-third of a letter grade (i.e., from A- to B+) for each day they are submitted past the due date. Any questions about grades must be raised in writing within two weeks of the time the test/problem set is returned. Your grade will be determined by the following assignments/activities. 1. Three problem sets (#1=20%, #2=25%, #3=25%). The first problem set is intended to enhance your knowledge of regression analysis. The second and third test your knowledge of experimental and quasiexperimental evaluation concepts, and provide experience analyzing data from real evaluations. Use of Stata is required. 2. Critical assessment (25%). Each Master s student will select a published evaluation and write a 7-10 page memo assessing and critiquing the evaluation methodology. Ph.D. students will write conduct a longer and more detailed assessment. Evaluations must be approved by Professor Gerber by December 1. Memos/ assessments are due at 12:00 noon on Monday December 15 in 5228 Weill Hall. 3. Class participation (5%). Students are expected to regularly attend class, complete all assigned readings before class, and participate actively and intelligently in all class discussions. If you know you are going to have to miss class, please let me and John know beforehand. Office Hours: Professor Gerber will hold weekly office hours Tuesdays from 1:00-3:00 in 5228 Weill Hall or by appointment. John English will hold office hours Fridays from 12:00-2:00 in 3202 Weill Hall or by appointment. 1
Writing Assistance/tutoring: David Morse and Elena Delbanco are available by appointment to providing writing assistance. Please consider making an appointment with them if you want/need help with your critical assessment paper. http://fordschool.umich.edu/current/academic_resources.php (general info on tutors) https://sitemaker.umich.edu/writing.appointment.david.morse/ (appointments) https://sitemaker.umich.edu/writing.appointment.elena.delbanco/ (appointments) Outline of topics Class # Date Topic 1 9/3 Introduction to the course, review syllabus 2 9/8 STRIVE evaluation 3 9/10 Program theory 4 9/15 Regression basics 5 9/17 Multivariate regression 6 9/22 Multivariate regression 7 9/24 Model specification, omitted variables bias, hand out first problem set 8 9/29 Introduction to impact analysis 9 10/1 Experimental designs understanding the concept, first problem set due 10 10/6 Experimental designs applications 11 10/8 Experimental designs applications 12 10/13 Experimental designs randomization exercise 13 10/15 Quasi-experimental designs intro - 10/20 Fall break no class 14 10/22 Quasi-experimental designs pre-/post-intervention w/o control group 15 10/27 Quasi-experimental designs pre-/post-intervention with control group 16 10/29 Quasi-experimental designs interrupted time series, hand out 2 nd problem set 17 11/3 Quasi-experimental designs regression discontinuity 18 11/5 Quasi-experimental designs regression discontinuity, second problem set due 19 11/10 Quasi-experimental designs matching 20 11/12 Quasi-experimental designs matching 21 11/17 Quasi-experimental designs propensity scores 22 11/19 Quasi-experimental designs propensity scores 23 11/24 Process and implementation analysis methods - 11/26 Thanksgiving break no class 24 12/1 Process and implementation analysis, hand out 3 rd problem set 25 12/3 Process and implementation analysis 26 12/8 Review, third problem set due - 12/15 Critical assessment due 12:00 pm, 5228 Weill Hall 2
Daily Schedule 1: Introduction to the course, review of syllabus. (September 3) We will begin with a review of the syllabus. We will then provide a broad picture of program evaluation, addressing questions such as: What is program evaluation? What is evaluated -- programs? projects? policies? laws? Why are programs evaluated? What is the history of program evaluation? 1(a): Rossi et al, Chapter 1 1(b): Rossi et al, Chapter 2, pp. 62-78 2: STRIVE evaluation. (September 8) We will watch a video about a job training evaluation and complete a group exercise that requires you to begin to think critically about programs and policies. none 3: Program theory. (September 10) Before we can evaluate a program or policy, we need to know what outcomes they are expected to alter. What is the theory that guides the design of the program or policy? Are there competing theories that need to be considered? Do these alternative theories imply different effects on the given outcomes, and do they imply that additional outcomes may be affected? A good theory will explicitly identify each of the linkages necessary to translate the program or policy into the anticipated effect on the desired outcome. Reading 3(a): Rossi et al, Chapters 3 and 5 4, 5, 6, 7: Refresher on the basics of regression analysis. (September 15, 17, 22, 24) Regression analysis is used in the vast majority of evaluations. The objective of these four sessions is to bring you up to speed with the basic ideas of regression analysis. At the end of the four classes, you should know how to read and interpret regression output and be familiar with the assumptions of OLS. Moreover, you should understand the concept of omitted variable bias, which will be used extensively during the course. 4(a): Lewis-Beck, Applied Regression. Sage University Papers Series, #22. 4(b): Berry and Feldman, Multiple Regression in Practice. Sage University Papers Series, #50. 8: Introduction to impact analysis. (September 29) This class session will serve as an introduction to and overview of impact analysis. The following topics will be covered: a. Definition of impact analysis and the questions it is intended to answer b. Establishment of terminology used in impact analyses, e.g., treatment, policy, program c. Description of the fundamental problem d. Overview of alternative strategies for impact analysis e. Discussion of construct, conclusion, internal, and external validity 3
Reading 8(a): Rossi et al, Chapter 7. 9: Experimental designs: understanding the concept. (October 1) The topics covered include the following: a. Theory of experimental design b. Advantages of experiments c. Disadvantages of experiments 9(a): Rossi et al, Chapter 8. 9(b): Burtless, Gary. 1995. The Case for Randomized Field Trials in Economic and Policy Research. Journal of Economic Perspectives. 9(2): 63-84. Problem set #1 due at the beginning of class. 10: Experimental designs: applications. (October 6) Throughout the course, we will read and discuss actual evaluations from various fields that employ designs and demonstrate concepts that we are covering in class. The first experimental evaluation is a large-scale HUD evaluation/demonstration program. The full report is 341 pages long. All students are responsible for carefully reading the Executive Summary; group of students will be assigned to lead the class discussion on specific aspects of the methodology. Reading 10(a): U.S. Department of Housing and Urban Development, Office of Policy Development and Research. 2003. MTO for Fair Housing Demonstration Program: Interim Impacts Evaluation. http://www.huduser.org/publications/pdf/mtofullreport.pdf 11: Experimental designs: applications. (October 8) The second experimental evaluation analyzes evidence from a public health field experiment in Malawi. Please read and be ready to discuss the entire study. Reading 11(a): Thornton, Rebecca. 2006 The Demand for and Impact of Learning HIV Status: Evidence from a Field Experiment, working paper. 12. Randomization exercise. (Oct 13) As a class, we will select a program and design a randomized control trial and evaluation. Students should come to class with ideas for programs/treatments. We will also hand back and discuss the first problem set. 13: Quasi-experimental designs: introduction (October 15) This class marks the beginning of our coverage of quasi-experimental program evaluations. This is the most difficult part of the course, and arguably the most important. The vast majority of evaluations use quasiexperimental designs, the quality of quasi-experimental designs varies greatly, and many of the designs use quite sophisticated statistical and econometric techniques. Our goal is to become familiar with the various approaches so that you can understand them and ask critical questions about the evaluations that use these approaches. The first class will provide an overview of quasi-experimental designs. 4
13(a): Campbell, Donald. (1965). Reforms as Experiments, Psychologist, 24(4): 409-429. 13(b): Posavac, Emil J., and Raymond G. Carey, Program Evaluation: Methods and Case Studies, 6th Edition, Chapter 8 13(c): Posavac, Emil J., and Raymond G. Carey, Program Evaluation: Methods and Case Studies, 6th Edition, pages 184-188. 13(d): Judd, Charles M., and David A. Kenny. (1981). Estimating the Effects of Social Interventions, Chapter 6. 13(e): Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi- Experimental Designs for Generalized Causal Inference, Chapter 6. 14: Quasi-experimental designs: pre-/post-intervention observation, without comparison group. (October 22) Reading 14(a): Duckart, J.P. An Evaluation of the Baltimore Community Lead Education and Reduction Corps (CLEARCorps) Program. Evaluation Review, 22: 373-402. 15: Quasi-experimental designs: pre-/post-intervention observation, with comparison group. (October 27) Reading 15(a): Harmon, Michelle Alicia. 1993. Reducing the Risk of Drug Involvement among Early Adolescents. Evaluation Review 17(2): 221-239. 16: Quasi-experimental designs: interrupted time-series. (October 29) Reading: 16(a): Loftin C. et al. 1991. Effects of Restrictive Licensing of Handguns on Homicide and Suicide in the District of Columbia. New England Journal of Medicine. 325(23): 1615-1620. Problem set #2 distributed. 17: Quasi-experimental designs: regression discontinuity. (November 3) Reading 17(a): Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Chapter 7. 18: Quasi-experimental designs: regression discontinuity. (November 5) Reading 18(a): Martorell, Paco and Isaac McFarlin. 2007. Help or Hindrance? The Effects of College Remediation on Academic and Labor Market Outcomes. NBER Working Paper. Problem set #2 due at the beginning of class. 19: Quasi-experimental designs: matching. (November 10) 19(a): Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi- Experimental Designs for Generalized Causal Inference. Page 118-122. 19(b): LaLonde, Robert. 1986. Evaluating the Econometric Evaluations of Training Programs with Experimental Data. American Economic Review. 76(4): 604-620. 5
19(c): Dehejia, Rajeev and Wahba, Sadek. Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs. Journal of the American Statistical Association, December 1999, 94(448): 1053-62. 20: Quasi-experimental designs: matching. (November 12) Reading 20(a): Reville, Robert, and Robert Schoeni (2001). The Effects of Permanent Partial Disability on Employment and Earnings. RAND DRU-2554. 21: Quasi-experimental designs: propensity scores. (November 17) Reading 21(a): Luellen, Jason K., William R. Shadish, and M. H. Clark (2005). Propensity Scores: An Introduction and Experimental Test. Evaluation Review 29: 530-558. 22: Quasi-experimental designs: propensity scores. (November 19) Reading 22(a): Mendola, Mariapia (2007). Agricultural Technology Adoption and Poverty Reduction: A Propensity-score Matching Analysis for Rural Bangladesh. Food Policy 32: 372-393. 23: Process and implementation analysis: methods. (November 24) Process or implementation evaluations answer important questions that inform decision makers are programs being implemented in the way they were intended? Moreover, a high quality process analysis can enhance the impact study. We will spend two class periods talking about the goals of process analyses and the methods used to conduct them. In addition, we will review examples of high quality process studies. Reading 23(a): Rossi et al. Chapter 6. 24: Process and implementation analysis: applications. (December 1) 24(a): Lantz, Paula et al. 1995. Breast and Cervical Cancer Screening in a Low-Income Managed Care Sample: the Efficacy of Physician Letters and Phone Calls, American Journal of Public Health, 85(6): 834-836. 24(b): Lantz, Paula. 1996. Implementation Issues and Costs Associated with a Proven Strategy for Increasing Breast and Cervical Cancer Screening Among Low-Income Women, Journal of Public Health Management and Practice, Summer: 54-59. 25: Process and implementation analysis: applications (December 3) Reading 25(a): Klerman, Jacob, et al. 2000. Welfare Reform in California: State and County Implementation of CalWORKS in the Second Year. RAND Report. http://www.rand.org/pubs/monograph_reports/2005/mr1177.pdf 26. Review (December 8) Problem set #3 due at the beginning of class. 6