THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL SCHOOL OF SOCIAL WORK

Transcription

1 THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL SCHOOL OF SOCIAL WORK COURSE NUMBER: SOWO 910 COURSE TITLE: Research Methods for Social Intervention SEMESTER & YEAR: Fall, 2013 Room: TTK 226 Time: 9:00AM 11:50 AM INSTRUCTOR: OFFICE HOURS: Mark Testa, PhD Spears-Turner Distinguished Professor School of Social Work University of North Carolina at Chapel Hill Tate-Turner-Kuralt Building 325 Pittsboro St., Campus Box 3550 Chapel Hill, NC Tel: (919) Fax: (919) Room 548C Fridays, 12:00 1:30 PM or by appointment COURSE DESCRIPTION: This course provides an introduction to basic research processes and methods for use in planning, implementing, evaluating, and improving social interventions at the explorative, formative, summative, translative, and confirmative stages of program implementation and evaluation. Topics include outcomes monitoring, problem formulation, needs assessment, construct measurement, theory of change, research review, human subjects protection, evaluation design, implementation science, data analysis, and the application of findings to practice improvement and theory refinement. COURSE OBJECTIVES: This course affords students an opportunity to gain knowledge about the following issues in social intervention research: The need for broadly inclusive processes to plan, implement, and evaluate social interventions at the explorative, formative, summative, translative, and confirmative stages of program implementation and evaluation, and how researchers' approaches to these processes can facilitate or impede research; The quantitative-comparative experimental (potential outcomes) paradigm that currently prevails in social intervention research; 1

2 How various policy and implementation constraints sometimes necessitate the use of designs other than fully randomized experiments; Special legal and ethical issues pertaining to the protection of human subjects; and The need for culturally aware social intervention research that is responsive to the diversity of community values and preferences. Students taking the course will be able to: develop well-built research questions for estimating the causal impact of social interventions on desired outcomes for target populations; develop logic and other conceptual models to support proposed social interventions and explicate underlying theories of change; assess the validity and reliability of alternative qualitative and quantitative measures of constructs in conceptual models that guide social intervention research; understand basic aspects of data analysis, sample design and statistical power analysis; critically evaluate various experimental, quasi-experimental, and non-experimental research designs by identifying various threats to implementation integrity and intervention validity; and prepare an application for IRB approval of human subjects research. Recommended Prerequisites SoWo 102 or equivalent SoWo 292 or equivalent Relationship to Other Courses in the Curriculum This course introduces basic concepts and skills that are reinforced and further developed in Development of Social Intervention Models (SOWO 940) and Advanced Research Methods in Social Intervention (SOWO 913) during the 2 nd and 3 rd years of the program. It extends students knowledge about basic research processes and orients them to the variety of professional issues, theoretical perspectives, statistical methods, and advanced methods of intervention research that they will examine in related doctoral-level courses (SOWO 900, 911, & 914) and can pursue in courses outside of the School. REQUIRED TEXTS: Ayres, I. (2007). Super crunchers: Why thinking-by-numbers is the new way to be smart. New York: Bantam Books Shadish, W.R, Cook, T.D., & Campbell, D.T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston: Houghton Mifflin Company. Testa, M. F. & Poertner, J. (2010). Fostering accountability: Using evidence to guide and improve child welfare policy. Oxford: Oxford University Press. 2

3 REQUIRED AND SUPPLEMENTARY READINGS: Required readings outside of the textbook are listed in the course outline. Supplementary readings are assigned at the instructor s discretion. TEACHING METHODS The development of a supportive learning environment, reflecting the values of the social work profession, is essential for the success of this class. A supportive learning environment is fostered by listening to the ideas and views of others, being able to understand and appreciate a point of view which is different from your own, articulating clearly your point of view, and linking experience to readings and assignments. Everyone will appreciate your contributions to making this a safe and respectful class for learning and growth. CLASS ASSIGNMENTS SEMINAR PARTICIPATION 1. Register with JSSR: Sign up as a subscriber to the Journal of the Society for Social Work and Research (edited by Mark Fraser) by registering online. August Obtain Certification in Protection of Human Participants: Students must complete the on-line course developed by the Collaborative IRB Training Initiative (CITI) to certify that they are familiar with the ethical principles and guidelines governing research involving human participants; completing the course requires several hours; each student is required to submit electronically on Sakai or a completion certificate before September 6 (this assignment is required, but is not graded). 3. Identify potential resources and collaborators. Become familiar with potential sources of information about obtaining financial support for your research. Also you may wish to make yourself known to potential research collaborators by registering as a member of the Community of Science. September Participate in seminar: Students are expected to read assignments and come to class prepared to share concepts from the readings, ask questions, and respond to questions about the material. Each student should prepare a 1-page self-evaluation of his or her efforts to make this a productive learning experience. Award yourself 1-5 points for those efforts (see description of grading). This is due the last day of class (November 22) and should be submitted to the instructor electronically on Sakai or by . WRITING ASSIGNMENTS AND IN-CLASS EXAM During the semester, you will write five papers (varying in length from 1 to 7 pages) on different components of the research process. The final assignment will require revision and expansion of each of these components into a full IRB application. There will also be a one-hour, in-class exam. The assignments are as follows: 3

4 1. Research Question: After meeting with the instructor to discuss preliminary ideas, each student will formulate a well-built PICO question that consists of the following components: 1) the target population about which you wish to draw inferences; 2) the outcome you intend to achieve or problem you hope to address; 3) the intervention you are interested in evaluating; 4) and the alternative course of action with which you will draw a comparison (e.g., no intervention, regular services, or different interventions). This one (1) - pager, including a brief statement about the significance of the problem or outcome you are studying, will be due September Research Review: Conduct a computerized search of electronic databases using keywords from your PICO question. Select the strongest four to six (4-6) studies that bear on your topic. Write up a research review using narrative descriptions that also assess the strength of the evidence for supporting the use of your intervention and identify the limitations of these studies and their applicability to your population. This paper (5-7 pages) will be due October Logic Model: Expand your PICO question into a logic model that lays out the expected causal mechanisms and mediating pathways from the intervention to the desired outcome. Your description of the target population should also list any population conditions that you believe may moderate the intervention s impact on the outcome. The model should also enumerate any background factors and external conditions that contribute to the significance of problem you are addressing. It should list the key assumptions of the theory of action you are positing will effectuate the desired change. Finally the model should identify general end-values for reconciling diverse outcomes and evaluating the ultimate worth of the resulting change. A modified Logic Model Template for filling in this one (1) page figure is available under Resources on Sakai. It will be due October Measurement Review: Identify and assess the relative strengths and weaknesses of alternative measures or approaches to measuring a construct (population, intervention, or outcome) referenced in your research question. The purpose of the exercise is to select the measure (or set of measures) that will yield the most valid and reliable data concerning this construct. This paper (5-7 pages) will be due October Evaluation Design: Based on your research question, you will outline the basic features of an experimental or quasi-experimental design for evaluating the impact of the identified social intervention on your outcome construct. The description should identify the unit(s) of analysis, comparison group(s), and how threats to the validity of your research will be addressed. The discussion should provide a rationale for the method you are proposing and how concessions to design constraints may make the research vulnerable to criticism. This 5-7 page paper will be due November In-Class Exam: There will be a one-hour, in-class, multiple-choice exam based on questions that you and your fellow students bring to class each week about Shorthand Abstractions (SHAs) that you found improved or challenged your thinking and which you think could also improve the cognitive toolkit of your fellow classmates and instructor. Professor James Flynn uses the term to refer to concepts drawn from science which make people smarter by providing widely applicable templates. Concepts such as control group, random sample, regression coefficient, the 2SD rule, placebo, and falsifiability are examples of SHAs that make available complex ideas in unified cognitive chunks that 4

5 can be used as elements in critical thinking and debate. As you read the assigned readings for each class, take note of a concept or two that you think could qualify as a SHA. The concept can be understood in a broad sense as a valid and reliable way of gaining knowledge as long as it is a rigorous conceptual tool that may be summed up succinctly in a word or a phrase and has broad application to interpreting the world. Come prepared to each class with a question each week about a SHA. These questions will be sampled for the in-class exam that will be administered on November IRB Application: This final assignment will follow the instructions issued by the UNC Office of Human Research Ethics for application for IRB approval. We will be reviewing various resources throughout the course that will be helpful in preparing it. It will incorporate material from each of the other written assignments as well as a description of risks to human subjects and measures to minimize those risks and a discussion of the benefits to subjects and/or society. The online application form can be accessed from after you enter your Onyen and password. Instructions and training videos can be accessed at by clicking on online submission on the left. This application will be due on November 25. WRITING ASSIGNMENT GUIDELINES: All written assignments must be typed and follow APA format. Several writing resources are posted on the website. Students should also refer to the following: American Psychological Association. (2009). Publication manual of the American Psychological Association (6th ed.). Washington, DC. Note: You can find a self-paced tutorial for APA style at ONLINE RESOURCES: Course materials for SOWO 910 will be accessible to you on In addition, Angela Bardeen, Behavioral and Social Sciences Librarian at UNC, has created a SOWO 910 course library website for finding articles, evidence-based practice resources, measurement tools, and human subjects protection. It can be accessed at COURSE PERFORMANCE ASSESSMENT: Final grades will be determined on the basis of points earned on each assignment and on participation in seminars. Letter grades will correspond to the following point totals: : High Pass 80-93: Pass 70-79: Low Pass < 70: Fail 5

6 Letter grades in this course generally follow a distribution with a mean of 87 and a standard deviation of 6 points. Points are apportioned to individual assignments as follows: 1. Seminar participation: 10 points (5 points by instructor and 5 points by self-evaluation) 2. Research Question: 10 points 3. Research Review: 10 points 4. Logic Model: 10 points 5. Outcome Measurements: 10 points 6. Evaluation Design: 15 points 7. In-Class Exam: 10 points 8. IRB Application: 25 points POLICY ON INCOMPLETES AND LATE ASSIGNMENTS The instructor will only entertain requests to hand in assignments late in circumstances of special hardships or emergencies. The potential for handing in papers late needs to be discussed with the instructor in person at least three days before the assignment is due. If not approved beforehand, late assignments will be graded five points off for each day the assignment is late, including weekends. A grade of Incomplete is given on rare occasions when there is sufficient reason to warrant it. It is the student s responsibility to initiate a conversation with the instructor to request an Incomplete instructors have no responsibility to give an Incomplete without such a request. POLICY ON ACADEMIC DISHONESTY Please refer to SSW Writing Resources and References website for information on attribution of quotes, plagiarism and appropriate use of assistance in preparing assignments. All written assignments should contain a signed pledge from you stating that, "I have not given or received unauthorized aid in preparing this written work". In keeping with the UNC Honor Code, if reason exists to believe that academic dishonesty has occurred, a referral will be made to the Office of the Student Attorney General for investigation and further action as required. POLICY ON ACCOMMODATIONS FOR STUDENTS WITH DISABILITIES Students with disabilities that affect their participation in the course and who wish to have special accommodations should contact the University s Disabilities Services and provide documentation of their disability. Disabilities Services will notify the instructor that the student has a documented disability and may require accommodations. Students should discuss the specific accommodations they require (e.g. changes in instructional format, examination format) directly with the instructor. 6

7 OUTLINE OF CLASS TOPICS Date Topics for Discussion Reading Assignments To Be Completed before Class UNIT I: INTRODUCTION TO SOCIAL INTERVENTION RESEARCH Week 1 (August 23) PICO Questions & Logic Models Start readings and complete PICO & Logic Model exercise for next week. Week 2 (August 30) Overview of Social Intervention Research Testa & Poertner, pp ; Kenny et al., pp ; Testa & White, pp.1- Week 3 (September 6) Scientific Validity & Agency Integrity 35. Testa & Poertner, pp. 3-13; Shadish, Cook & Campbell, pp ; Dane & Schneider, pp UNIT II: EVALUATIVE, FORMATIVE, SUMMATIVE, TRANSLATIVE & CONFIRMATIVE RESEARCH Week 4 (September 13) Outcomes Monitoring Testa & Poertner, pp ; Goodwin, pp Week 5 (September 20) Data Analysis Ayres, pp. 1-45; Testa & Poertner, pp , ; Freedman, et al., pp. Week 6 (September 27) Research Reviews Ayres, pp ; Testa & Poertner, pp ; Shadish, Cook & Campbell, pp Week 7 (October 4) Week 8 (October 11) Week 9 (October 18) Week 10 (October 25) Week 11 (November 1) Week 12 (November 8) UNIT III: INTERVENTION EVALUATION DESIGNS Evaluation Designs Ayres, pp ; Freedman, et al., pp. 3-28; Akin et al., pp Experimental Designs Testa & Poertner, pp ; Boruch, et al. pp Fall Break A Brief Tutorial on DAGs Testa & Poertner, pp ; West et al., pp ; Fleischer & Diez Roux, pp UNIT IV: RESULTS-ORIENTED ACCOUNTABILITY Statistical Precision & Power Ayres, pp ; Testa & Poertner, Analysis pp ; Orme et al., pp Human Subjects Protections The Belmont Report; Homer,

8 Week 13 (November 15) Week 14 (November 22) Confirmative Implementation & Quality Improvement & In-Class Exam Confirmative Research and Reflexive Practice Ayres, pp ; Testa & Poertner, pp , ; Moseley & Solomon, pp Ayres, pp ; Testa & Poertner, pp ; Nijnatten, pp UNIT I. INTRODUCTION TO SOCIAL INTERVENTION RESEARCH Week 1 (August 23) PICO QUESTIONS AND LOGIC MODELS The research process begins with the formulation of a well-built research question that can be parsed into the four components of population, intervention, comparison, and outcome. Grouped together under the acronym PICO, the formulation of a well-built question guides the process of data analysis, computerized research reviews, logic model construction and evaluation design. There are a variety of designs for constructing logic models. The design used in this course elaborates on the PICO framework and visually depicts the mediating activities that link interventions and population conditions to the short-term outputs and proximal outcomes produced by the activities and the longer-term distal outcomes these intermediate actions are expected to produce. A full logic model also identifies the external conditions that prompted concern over the problem, the underlying theory of change, end-values for evaluating the ultimate worth of the resulting change, and any moderating conditions that may affect the intervention s impact. You will spend this first week working on the PICO question (Testa and Poertner, 2010: 81-82) that you think Kenny et al. (2004) addresses in next week s reading assignment. Also construct a logic model of the intervention they studied using the modified template posted on Sakai under Resources and described in Testa and Poertner (2010: 85-98). Submit both the PICO question and logic model electronically on Sakai before 12:00 pm on Thursday, August 29 and bring hard copies to class the next day. Week 2 (August 30) OVERVIEW OF SOCIAL INTERVENTION RESEARCH Social intervention research serves five distinct but interrelated purposes: 1) developing and testing a social intervention to improve practice and policy (formative research); 2) comparing a promising intervention to its counterfactual and learning about its average causal effect (summative research); 3) replicating and adapting evidence-supported interventions to different local contexts and sub-populations (translative research); 4) applying the intervention broadly and improving its integrity and validity to decide whether it should be maintained as is, changed in some way, or discarded completely, with or without replacement (confirmative research) and 5) describing and explaining the effects of a social intervention as a contribution to scientific knowledge and theory (explorative research). 8

9 Figure 1 Cycle of Social Intervention Research Figure 1 depicts these five interrelated purposes as a series of multi-level transactions that move interventions from the micro level through formative implementation and evaluation of practiceinformed innovations to the macro level through summative implementation and evaluation of their average causal effects (ACEs). Further, if the success of the interventions is empirically supported, the process continues and makes linkages back down to the micro level through translative implementation and evaluation that adapt evidence-supported interventions to different populations and local contexts. After an evidence-based practice has been implemented for a period of time, its validity is periodically reexamined through confirmative implementation and evaluation that helps to decide whether the practice or program should be maintained as is, changed in some way, or discarded completely, with or without replacement. The decision to change or replace initiates another cycle of formative implementation and evaluation. 9

10 Surrounding the diagram is a set of explorative research, problem definition, scientific knowledge and theory-building, and research review activities that precede intervention development and implementation and are informed by the results of the process. These activities typically proceeds through a series of steps and can include 1) outcomes monitoring-- defining the desired outcomes; 2) data analysis identifying the potentiating and compensatory factors for attaining the outcomes and determining the extent of need and potential target populations for intervention; 3) research review converting the information into an answerable research question, summarizing the best available evidence for answering the question, and selecting an intervention for implementation; and 4) evaluation design developing a plan for insuring the integrity of implementation and evaluating the efficacy and effectiveness of the intervention. Your engagement with the process of social intervention research begins by identifying yourself to potential collaborators by registering as a member of the Community of Science. Testa & Poertner, pp Kenny, D.A. et al. (2004). Evaluation of treatment programs for persons with severe mental illness: Moderator and mediator effects. Evaluation Review, 28, Testa, M. F. & White, K. R. (In press). Insuring the integrity and validity of social work interventions: The case of the subsidized guardianship waiver experiments. Journal of Evidence-Based Social Work, pp Supplementary Reading: The Evaluation Exchange, 11(2): pp Week 3 (September 6) SCIENTIFIC VALIDITY AND AGENCY INTEGRITY Accountability for social welfare outcomes involves holding practitioners and other change agents answerable both for the integrity of the actions taken on behalf of their clients and other principals and for the validity of those actions in achieving the outcomes valued by their principals and the public (Testa & Poertner, 2010). The success of a principal-agent (clientpractitioner) relationship is a product of the two. Failure of an agency relationship to achieve its intended purposes thus may reflect either a problem with the integrity of the implementation or a problem with the validity of the intervention (Klein & Sorra, 1996). Social intervention research is an agency relationship that involves both people who design and implement interventions (agents) and the people that provide funding for testing them and the participants who are affected by the interventions (principals). As such, agents bear responsibility for abiding by a set of scientific standards and ethical principles in fulfilling the 10

11 interests of their principals. In this respect, research accountability goes beyond accumulating valid evidence of the efficacy and effectiveness of child welfare interventions to ensuring that the implementation process reliably and responsibly serves the purposes valued by clients, research sponsors, and the public at large. To be accountable, social intervention researchers must appreciate the context in which they work and develop the interpersonal skills as well as the technical knowledge and skills that can make them both effective and responsible in their roles. Your socialization into the role of a responsible agent begins with completing the on-line course related to the protection of human participants in research. Each student will submit a CITI certificate indicating their completion of the course. This certification and the automatic registration with UNC's Office of Human Research Ethics satisfy the requirements of many of the research assistantships held by students in the doctoral program. Scientific validity refers to the best available approximation to the truth or falsity of a given hypothesis, inference, or conclusion and provides a set of standards by which the quality of the research can be judged. There are four generally recognized types of scientific validity: 1) statistical conclusion validity that is concerned with whether there is a statistically significant association between an intervention and the desired outcome; 2) internal validity that focuses on whether the statistical association results from a causal relationship between the intervention and the outcome or is a spurious association; 3) external validity that addresses how generalizable the particular causal relationships are over variations in PICO; and 4) construct validity that assesses the degree of correspondence between the observational particulars of a population, intervention, comparison, and outcome (PICO) and their higher-order constructs. The demonstration of scientific validity usually proceeds cumulatively and the order of demonstration varies with the purpose or purposes served by the social intervention research. Testa & Poertner, pp Shadish, Cook & Campbell, pp Dane, A.V. & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23-45, Supplementary Readings: Webb, S.A. (2001). Some considerations on the validity of evidence-based practice in social work. British Journal of Social Work, 31: Gibbs, L. & Gambrill, E. (2002). Evidence-based practice: Counterarguments to objections. Research on Social Work Practice, 12:

12 UNIT II: EXPLORATIVE, FORMATIVE, SUMMATIVE, TRANSLATIVE & CONFIRMATIVE RESEARCH Week 4 (September 13) OUTCOMES MONITORING The construction of a well-built research question begins with the identification and measurement of an outcome or set of outcomes that principals and their agents want to monitor and, if desired, change. Outcome is synonymous with the effect, result or dependent variable of a cause, intervention or independent variable. In mathematical terms, it is y or a function of x: y = f(x). In this course we will also substitute O for y in keeping with the PICO framework. In order to estimate the effect of an intervention on an outcome, it is necessary to translate the higherorder, theoretical construct of the outcome into a lower-order, operational measurement of the variable. Construct validity refers then to the degree of correspondence between the higher order, theoretical constructs and the lower-order, observational particulars. To demonstrate construct validity, you need to show evidence that the observational particulars (data) support the theoretical structure of the construct. How researchers conceptualize this task is evolving, and we will review some of the latest thinking on these endeavors. Some researchers oppose the routine use of outcome indicators to improve public management. They argue that monitoring and evaluating agency units, e.g., schools, departments, and courts, invariably lead to the corruption of the indicators used to monitor results and to the degradation of the agency relationships that program evaluation is supposed to improve. It is important to acknowledge these agency risks and take cognizance of such threats to agency integrity. We will consider the precautions that can be taken which can help decrease these agency risks and increase the opportunities for responsible public policy and management. Testa & Poertner, pp Goodwin, L. (2002). Changing conceptions of measurement validity: An update on the new Standards. Journal of Nursing Education, 41, Supplementary Readings: Pelton, L. (2008). A note contesting Mark Testa's version of national foster care population trends, Children and Youth Services Review, doi: / j.childyouth Testa, M. (2008). How the bear evolved into a whale: A rejoinder to Leroy Pelton's note contesting Mark Testa's version of national foster care population trends, Children and Youth Services Review, doi: / j.childyouth

13 Week 5 (September 20) DATA ANALYSIS Observation of a deterioration in outcomes, which is of both practical importance and statistical significance, does not necessarily indicate that agency performance is deficient or in need of correction. There could be some other antecedent conditions beyond the practitioners and agents immediate control that could be causing the difference. For example, a practically important and statistically significant difference in post-operative mortality rates between two hospitals may not necessarily mean that the hospital with the lower rate is the superior performing hospital. It may instead mean that the hospital with the higher rate admits a sicker group of patients, on average, than the other. Therefore before a summary judgment can be rendered about agency performance or the type of improvement needed, it is first important to identify those factors that may be clouding the comparison with data analysis methods for purging or adjusting for external confounding influences. In this week, we will introduce the regression method of adjusting for confounding variables that can exaggerate or obscure the true causal effect of an intervention on an outcome. Freedman et al. offer a simple description of the correlation coefficient and how it is easily transformed into a regression coefficient. We will then replicate the results obtained by one of the first applications of the regression method to a social policy question: G. Udny Yule s investigation into the causes of changes in pauperism in England published in Ayres, pp Testa & Poertner, pp , Freedman, D., Pisani, R. & Purves, R. (2007). The Regression Line. Statistics- 4 th edition, (pp ). New York: W.W. Norton & Company. Supplementary Reading: Yule, G. U. (1899). An investigation into the causes of changes in pauperism in England, chiefly during the last two intercensal decades (part I.). Journal of the Royal Statistical Society, 62, (2), Hayes, A. F. (2013). Simple linear regression. Introduction to mediation, moderation, and conditional process analysis: A regression-based approach (pp ).New York: The Guildord Press. 13

14 Week 6 (September 27) RESEARCH REVIEWS (Guest Instructor, Qi Wu, Doctoral Student, UNC) Research reviews involve the explicit search and selection of relevant studies, the assessment of their scientific validity, and a critical synthesis of their findings to reach a tentative conclusion about the efficacy and effectiveness of social interventions. This stage of research-oriented accountability is what the field most commonly understands as evidence-based practice (EBP). A commonly cited definition by Sackett et al. (1997) is that EBP is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual [clients]. EBP is emerging also as the guiding paradigm within which social intervention research is pursued by university-based researchers and applied by various actors in the areas of medicine, psychology, public health, criminology, social work and public policy. While paradigms provide a framework within which knowledge can be developed, they can also intellectually straitjacket their adherents. Given this potential agency risk, there is a need to subject EBP to the same methods of critical review and reflective assessment that its adherents espouse for validating clinical practices and public policies. Ayres, pp Testa & Poertner, pp Shadish, Cook & Campbell, pp Supplementary Readings: Chalmers, I. (2003). Trying to do more good than harm in policy and practice: The role of rigorous, transparent, up-to-date evaluations. Annals of the American Academy of Political and Social Science, 589, Littell, J. (2008). Ch. 4: How do we know what works? The quality of published reviews of evidence-based practices. In Lindsey, D. & Shlonsky, A. (Eds.) Child welfare research: Advances for practice and policy (pp ). Oxford: Oxford University Press. Winokur, M., Holtan, A., & Valentine, D. (2009) Kinship care for the safety, permanency, and well-being of children removed from the home for maltreatment. Cochrane Database of Systematic Reviews, 1, Art.No.: CD DOI: / CD pub2. 14

15 UNIT III. INTERVENTION EVALUATION DESIGNS Week 7 (October 4) EVALUATION DESIGNS The question that clients, practitioners and policymakers really want to answer from social intervention research is what would happen if people who are given a certain treatment or intervention option were instead denied this possibility. Of course such potential outcomes can never be compared at the individual level because it is impossible simultaneously to deny and to offer a treatment option to an individual person. Instead researchers have to fall back on high quality approximations to this impossible what if scenario (what statisticians call the counterfactual ) by conducting rigorous studies that allow them to draw causal inferences at the macro level. A compelling case can be made for more routine use of randomized controlled experiments in social work than is currently the practice. But there are situations in which controlled experimentation is inadvisable, unethical, or just plain impossible. Over the past several decades, researchers have made tremendous strides in conceptualizing the assumptions that need to be satisfied in order to draw valid causal inferences from non-experimental research and how best to approximate the necessary conditions using statistical methods. The material in this section will help students acquire technical knowledge related to the development and application of different research designs experimental, quasi-experimental, and nonexperimental. Ayres, pp Freedman, D., Pisani, R. & Purves, R. (2007). Controlled experiments & Observational Studies. Statistics- 4 th edition, (pp. 3-28). New York: W.W. Norton & Company. Akin, B., Bryson, S., Testa, M., Blase, K., McDonald, T. & Melz, H. (2013). Usability testing in the initial implementation and formative evaluation of an evidence based intervention to reduce long-term foster care. Evaluation and Program Planning, 41,(19-30). Supplementary Reading: Testa & Poertner, pp Rubin, D.B. (2004). Teaching statistical inference for causal effects in experiments and observational studies. Journal of Educational and Behavioral Statistics, 29 (3),

16 Week 8 (October 11) EXPERIMENTAL DESIGNS The most rigorous method for drawing causal inferences about the potential effects of a treatment is the randomized controlled experiment. By employing a process such as a lottery, coin flips or a table of random numbers to treat a chance selection of persons or group, we can feel confident that the characteristics of the intervention group (who are offered the option) and the comparison group (who are denied the option) are statistically equivalent on average within the bounds of statistical error. The thinking is that if the two groups start out looking statistically similar at the initiation of the experiment and then if any significant differences later emerge after implementation, it is reasonable to conclude that the cause of the differences is the intervention itself rather than any pre-existing group dissimilarities or concurrent policy changes (which affect both groups). Despite these advantages, some social work practitioners find randomized controlled experiments to be ethically suspect because of its denial of services to the comparison group and to be of limited use because of the lengthy observational period before a summative judgment can be confidently rendered. This section will consider the appropriateness of constraining practitioner discretion by experimental protocols and the conditions under which randomization constitutes a justifiable interference with agent discretion when the empirical evidence for the efficacy or effectiveness of a promising intervention is weak. Testa & Poertner, pp Shadish, Cook & Campbell, pp Boruch, R. F., Victor, T. & Cecil, J.S. (2000). Resolving Ethical and Legal Problems in Randomized Experiments. Crime Delinquency, 46, Supplementary Readings: Testa, M. (2002). Subsidized guardianship: Testing an idea whose time has finally come. Social Work Research, 26 (3), Ludwig, J., Liebman, J.B., Kling, J.R., Duncan, G.J., Katz, L.F., Kessler, R.C. & Sanbonmatsu, L. (2008). What can we learn about neighborhood effects from the moving to opportunity experiment? American Journal of Sociology, 114(1), Prinz, R.J., Sanders, M.R., Shapiro, C.J., Whitaker, D.J. & Lutzker, J.R. (2009). Populationbased prevention of child maltreatment: The U.S. Triple P system population trial. Prevention Science, DOI /s

17 October 18: Fall Break No Class Week 10 (October 25) A BRIEF TUTORIAL ON DIRECTED ACRYLIC GRAPHS (DAGs) (Guest Instructor, Kevin White, Doctoral Student, UNC Although experiential research is the gold standard in social work and many related fields such as medicine, education, mental health, and criminology, oftentimes it is difficult or impossible to randomize clients or other units to intervention and comparison groups. Evaluation methods that approximate the statistical equivalence that is best obtained through random assignment are called quasi-experiments or more broadly observational studies.. Econometricians and statisticians have developed a variety of statistical methods for handling selection biases in order to uncover the genuine causal effect of an intervention on an outcome. The most commonly used approaches for adjusting for selection biases in the absence of randomization include, propensity score matching, instrumental variable analysis, and regression discontinuity designs. Directed Acyclic Graphs (DAGs) are a rigorous, largely graphic approach to causal inference from quasi-experimental and observational data. This approach helps to separate causation from association by identifying what control variables must be included and which variables mustn t be included in order to reduce the observed (conditional) association between intervention and outcome to its purely causal (i.e., non-spurious) component. Testa & Poertner, pp West, S. G., Duan, N., Pequegnat, W., Gaist, P.,Des Jarlais, D.C., Holtgrave, D., Szapocznik, J., Fishbein, M., Rapkin, B., Clatts, M., & Mullen, P.D. (2008). Alternatives to the randomized controlled trial. American Journal of Public Health, 98(8), Fleischer, N.L. & Diez Roux, A.V. (2008). Using directed acyclic graphs to guide analysis of neighborhood health effects: An introduction. Journal of Epidemiology and Community Health, 62(9), Supplementary Reading: Elwert, F. (2013). Graphical causal models. In S. Morgan (Ed.) Handbook of causal analysis for social research (pp ). Dodrecht: Springer. 17

18 UNIT IV: RESULTS-ORIENTED ACCOUNTABILITY Week 11 (November 1) STATISTICAL PRECISION AND POWER ANALYSIS When steady progress is being made toward remedying a problem or attaining a desired outcome, the monitoring process continues another round of assessment and review. Otherwise, the detection of a worrisome gap between a valued result and an observed outcome may signal the need for some corrective action or at least a reasonable accounting for the shortfall. When the gap is of both practical importance and statistical significance, data analysis can be initiated to identify the areas for improvement and the risk factors that may be contributing to the result. The first step in assessing whether a gap is worrisome enough is to decide on the smallest difference or distance that is deemed important enough to matter. This is best done in conjunction with practitioners and administrators who are able to draw comparisons from experience, history, an established standard, or some other reference point. With the availability of large administrative databases, an assessment of practical importance can proceed without much concern for the statistical significance of the conclusion. However when information is based on a sample from a larger population, there is always a chance that the conclusion will be wrong due to the lack of complete information about the population. Sampling theory provides a basis for evaluating the chances of error and for taking those risks into account when designing a study. Statistical power is a concept that has been commanding greater attention in social intervention research. It refers to the probability that a real difference, effect size, or pattern of association of a certain magnitude in a population will be detectable in a particular study. Power analysis proceeds from assumptions about available sample size and the amount of error one is willing to tolerate. It can also be used to determine sample size. To do this, we set an effect size that we want to be able to detect and select a high probability (typically, 80%) that a statistical test will reject the null hypothesis when the hypothesis of no effect is false. In this section, we will look at some statistical power software that is typically used to determine the sample size necessary to demonstrate an intervention s effect. Required Reading: Ayres, pp Testa & Poertner, pp Orme, J. G. & Combs-Orme, T. D. (1986). Statistical power and type II errors in social work research. Social Work Research & Abstracts, 22, Supplementary Readings: Boruch, R. (2007). The null hypothesis is not called that for nothing: Statistical tests in randomized trials. Journal of Experimental Criminology, 3,

19 Week 12 (November 8) HUMAN SUBJECTS PROTECTION This section will provide the nuts-and-bolts on how to prepare a UNC application for IRB approval of research proposals. You will be given an overview of the IRB application process at the School. After this, you should have the tools and knowledge needed to complete the final assignment. The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979). The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. Homer, C.S.E. (2002). Using the Zelen design in randomized controlled trials: debates and controversies. Journal of Advanced Nursing, 38(2), Week 13 (November 15) CONFIRMATIVE IMPLEMENTATION AND QUALITY IMPROVEMENT Confirmative implementation and quality improvement requires that researchers move beyond causal description and acquire a behavioral and deeper interpretative understanding of the intervening processes by which program resources are disbursed within a social system and transformed into program outputs and client outcomes. Initially quality improvement takes the form of repeating a single-loop learning cycle several times as needed to ensure that the intervention is supplied in sufficient dosage and with adequate fidelity to the original program design. If the results of the formative or translative evaluations fall short of the desired targets, show no change, or else run contrary to other end-values, quality improvement takes the form of a double-loop leaning cycle in which the existing theory of action and its assumptions and values are questioned and reflexively changed. Ayres, pp Testa & Poertner, pp , Moseley, J.L. & Solomon, D.L. (1997). Confirmative evaluation: A new paradigm for continuous improvement. Performance Improvement, 35(5),

20 Supplementary Readings: Moynihan, D. P. (2005). Goal-based learning and the future of performance management. Public Administration Review, 65(2), Week 14 (November 22) CONFIRMATIVE RESEARCH AND REFLEXIVE PRACTICE Some of the misunderstanding and distrust that occurs during the conduct of research arises from the failures of researchers to acquire an interpretative understanding of client values and perspectives and to take account of those preferences. To establish the mutual trust upon which a research enterprise must rest, it is important to create recurring opportunities for the impulses, desires, and values of the intended beneficiaries to enter into the design of a social intervention and continuous revision of existing routines in light of new knowledge about the impact of those practices. The coordination of the quantitative results with the qualitative feedback from clients, practitioners, and researchers through peer review and learning forums constitutes an essential ingredient in the integration of ethical, evidentiary, and practical concerns that is necessary for the validity and integrity of social intervention research. Ayres, pp Testa & Poertner, pp van Nijnatten, C. (2010). Children s agency, children s welfare: A dialogical approachto child development, policy and practice. Bristol: The Policy Press, pp Supplementary Readings: Shadish, W.R. (1995). Philosophy of science and the quantitative-qualitative debates: Thirteen common errors. Evaluation and Program Planning, 18 (1), D Cruz, H., Gillingham, P. & Melendez, S. (2007). Reflexivity, its meanings and relevance for social work: A critical review of the literature. British Journal of Social Work 37, November 29 University Holiday No Class 20