Annotated Bibliography 1 Samantha Arzon Applied Investigations Due: 4/21/14 Dr. Sopko Introduction: Extensive research has suggested that incorporating out of school suspensions is an ineffective way to handle behavior indiscretions. As such, many schools have introduced a preventative model that focuses more on positive behavior than negative behavior. The idea is that students will be rewarded for good conduct, which will reinforce those behaviors and make them more likely to re-occur. This preventive model is often referred to as Positive Behavior Interventions and Support (PBIS). Although intuitively this program seems like it would be beneficial, it is important to evaluate its effectiveness in order to work towards more positive school climates that are conducive to learning. Key Terms: School-wide Positive Behavior Interventions and Support (SW-PBIS), Positive Behavior Interventions and Support (PBIS), Positive Behavior Supports (PBS), Office Discipline Referrals (ODR s) Annotations: Miramontes, N. Y., Marchant, M., Heath, M. A., & Fischer, L. (2011). Social validity of a positive behavior interventions and support model. Education and Treatment of Children, 34(4), 445-468. This descriptive empirical article evaluated the social validity of PBIS implementation using a convenience sample of elementary and middle school consumers in a western state. Social validity is defined as the value society places on a product and is useful for gathering information on program drawbacks, implementation barriers, and potential impact. A total of 270 consumers (teachers, administrators, and other school personnel) were administered an anonymous 18-item, closed-question questionnaire at an annual PBIS training conference that assessed perceptions of PBIS implementation, goals, procedures, and outcomes. The questionnaire was developed in accordance with social validity assessment recommendations from the literature. A majority of participants indicated that as a result of PBIS implementation, a significant improvement of their overall school climate ensued, their school was positively impacted, and PBIS was worth the time and effort required. Additionally, the findings suggested that schools that accurately implemented PBIS initiatives, rated the program as more effective. Suggested areas of improvement included procedures of implementation and the amount of paperwork required. Limitations of the study include incomplete data sets (not all participants answered every item), participants (not every PBIS consumer attended the conference), and as such, the sample may not be generalizable.
Annotated Bibliography 2 Molloy, L. E., Moore, J. E., Trail, J., Van Epps, J. J., & Hopfer, S. (2013). Understanding realworld implementation quality and active ingredients of pbis. Prevention Science, 14(6), 593-605. DOI: 10.1007/s11121-012-0343-9 This descriptive empirical article examined differences in PBIS implementation quality and student outcomes of 27, 689 students across 166 primary and secondary public schools and seven states from 2007-2008. The present study aimed to explore which factors of PBIS were related to problem behaviors (aggression, drug use, and defiance) resulting in office discipline referrals (ODRs) and whether quality implementation of those factors differed across schools. Each school had implemented PBIS for at least two years. Quality of implementation was measured against the seven components/factors of PBIS (Expectations Defined, Expectations Taught, Reward System, Violation System, Monitoring & Decision Making, Management, and District-level Support). Results indicated that quality implementation of the Reward System predicted fewer ODRs across all three domains, high quality Violation System indicated fewer defiance and aggression ODRs, and high quality Taught Expectations predicted fewer defiance and drug-related ODRs. Reward Systems and Taught Expectations were two of the most poorly implemented components. Interestingly, implementation quality was less related to students with IEPs. Primary schools, higher SES schools, and smaller schools demonstrated a more quality implementation of PBIS. A significant limitation of the study was that students not receiving ODRs were not included in the sample. Pas, E. T., & Bradshaw, C. P. (2012). Examining the association between implementation and outcomes: State-wide scale up of school-wide positive behavior intervention and supports. Journal of Behavioral Health Services & Research, 39(4), 417-433. DOI: 10.1007/s11414-012-9290-2 This descriptive empirical article evaluated SW-PBIS through a type II translational research framework. The authors used Maryland state evaluation data from 2008-2010 to examine how implementation affected student outcomes when scaled up state-wide. Implementation data included quality of program, and fidelity of 421 traditional elementary and middle schools. Level of implementation was measured by three assessment tools. Student outcome data consisted of achievement on math and reading state exams, truancy, and suspension rates. Contextual factors such as student enrollment, student to teacher ratios, teacher certifications, student mobility, and years since SW-PBIS training were controlled for. The authors hypothesized that higher levels of implementation would be positively correlated with achievement and number of teacher certifications and negatively correlated with truancy, suspension, school size, student to teacher ratios, and student mobility. The results indicated that number of years since training, smaller school size, and low student mobility were significantly associated with math achievement and reading achievement (with the exception of school size). Higher implementation was associated with lower truancy rates and increased
Annotated Bibliography 3 achievement but not with suspension rates. A limitation of the study is that schools not using a SW-PBIS were not included in the sample. Bradshaw, C. P. (2013) Preventing bullying through positive behavioral interventions and supports (pbis): A multitiered approach to prevention and integration. Theory Into Practice, 52(4), 288-295. DOI:10.1080/00405841.2013.829732 This descriptive empirical article explored the effectiveness of integrating bullying prevention programs with PBIS. Fifty-eight high schools in Maryland were included in randomized control trial of PBIS and bullying prevention programs. Half of the schools received the intervention and half did not. Those that did were trained on implementing a PBIS system, using data to determine school needs, and implementing an evidence-based intervention. Interventions included the Olweus Bullying Prevention Program, Life Skills Training for High Schools, Check-In/Check-Out, Check & Connect, and Cognitive-Behavioral Intervention for Trauma in Schools (CBITS). Every school will be evaluated over a three-year period based on a school climate measure, developed by the US Department of Education s Safe and Supportive Schools Initiative. This webbased instrument measures students, parents, and school personnel perceptions of school safety, student engagement, and school environment (i.e. school climate). The interventions that have shown the most promising results are the Check & Connect, Olweus Bullying Prevention, and CBITS. The preliminary results indicate positive effects of the integrated PBIS system on school climate. A major limitation of this article is that the trial study is not yet complete and there are no published statistics to review. Reynolds, H., Irwin, D., & Algozzine, B. (2010). North Carolina Positive Behavior Support Initiative Evaluation Report 2008-2009. Raleigh, NC: Department of Public Instruction, Exceptional Children Division, Behavioral Support Services. This online evaluation report summarized major PBS gains in the state of North Carolina for the 2008-2009 academic year. PBS evaluations are usually conducted by school personnel. Notable gains from the previous year include a marked increase in schools implementing a PBS system, higher implementation quality, a decrease in office discipline referrals, and a decrease in suspension rates. For example, out of 115 school districts, at least one school in 93 of those districts are utilizing a PBS system. Evaluation tools included the Team Implementation Inventory, the School-wide Evaluation Tool (SET) or Benchmarks of Quality (BoQ), and the NC PBS Office Discipline Referral (ODR) Reporting Spreadsheet, the EBS Survey, School Safety Survey, and the Team Implementation Checklist. The results should be interpreted cautiously because the graphs are constructed in a way that leads the reader to believe that differences are bigger than they actually are. Additionally, we cannot assume that these differences are significant. However, because this data was compared against data from the previous year, we can maximize the possibility that the differences were actually due to PBS and not to other variables.
Annotated Bibliography 4 Martens, K., & Andreen K. (2013). School counselors involvement with a school-wide positive behavior support intervention: Addressing student behavior issues in a proactive and positive manner. Professional School Counseling, 16(5), 313-322. This descriptive empirical article reviewed the Check-In/Check Out (CICO) component of PBIS implementation. CICO is a tier-two intervention for individuals whose needs are not being met by tier-one interventions. Students are referred by teachers and/or parents for behavioral concerns and are eligible to participate in the program if they have at least four ODR s. A functional behavioral assessment (FBA) is then conducted to determine the cause of the problem behavior. CICO has been demonstrated to be more efficacious with those that are motivated by attention-seeking. Students that are motivated as a means of escaping are usually recommended for an alternative intervention. Once eligible to participate, students check in with school counselors twice a day to determine if the student s goals for the day were met (as indicated by teacher ratings on a behavior report card). Together, the school counselor and the student discuss daily successes and failures, and revaluate goals. A daily report is then sent home. Students participating in the CICO program at Jefferson Elementary School improved all problem behaviors. A limitation of the study was that because the CICO program is a tier two intervention, few students are included in the sample. Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of school wide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133-148. DOI: 10.1177/1098300709334798. This descriptive empirical article examined the fidelity and effectiveness of School-wide Positive Behavior Interventions and Support (SWPBIS) implementation with a five year longitudinal randomized controlled trial in 37 representative elementary schools in Maryland. Of the selected rural and suburban schools, 21 were randomly selected to participate in the intervention condition and 16 composed of the comparison group. Each school in the intervention group was provided with SWPBIS training at the state level and was separate from the researchers. The evaluation tools used were the School-wide Evaluation Tool (SET), which measures implementation fidelity, the Effective Behavior Support Survey, which measures self-report teacher evaluations of program fidelity, office discipline referrals (ODR s), Student Suspensions, and academic achievement. The results indicated that schools in the intervention group implemented SWPBIS with higher fidelity than schools in the control group. Schools in the intervention group also demonstrated a statistically significant decrease in the amount of ODR s and student suspensions. Additionally, although not statistically significant, schools in the intervention group demonstrated academic achievement gains. A limitation of this study was the small effect and power sizes of the statistics.
Annotated Bibliography 5 Farkas, M. S., Simonsen, B., Migdole, S., Donovan, M. E, Clemens, K., & Cicchese, V. (2012). Schoolwide positive behavior support in an alternative school setting: An evaluation of fidelity, outcomes, and social validity of tier 1 implementation. Journal of Emotional and Behavioral Disorders, 20(4), 275-288. DOI: 10.1177/1063426610389615. This descriptive empirical article sought to evaluate the effectiveness of Tier 1 implementation of School-wide Positive Behavior Interventions and Supports (SWPBIS) at an alternative school setting (Grades 5-12) using a single-subject case study design. Students are referred to this school after being diagnosed with an Emotional Disturbance (ED) or otherwise health impaired (OHI). Measures of evaluation included fidelity measurements (ex. the School Evaluation Tool), student outcomes (ex. Office discipline referrals), and social validity (ex. staff perceptions of SWPBIS). The results indicated high program fidelity measures, a decrease in ODR s, and an increase in academic achievement. Additionally, staff members and students indicated overall satisfaction with the SWPBIS program. A limitation of the study is the small and specific sample size and hence, the generalizability of the study. Conclusion: The current literature review demonstrates the effectiveness of SW-PBIS programs at the elementary, middle, and alternative school setting. Specifically, schools that implement PBIS with the highest fidelity tend to show to most marked student outcomes in relation to office discipline referrals, truancy rates, suspension rates, and achievement data. Moreover, stakeholders are generally satisfied with the results of PBIS. Few research studies, however, have been published on the evaluation of PBIS within the high school setting. Researchers are also in the process of evaluating the effectiveness of PBIS when paired with other interventions, which will be valuable information for all education stakeholders. The research has indicated areas for improvement as well. Research indicates that quality implementation of the Reward System and Taught Expectations components of PBIS predicted better student outcomes. However, the Reward Systems and Taught Expectations were two of the most poorly implemented components, especially after the initial implementation of a SW-PBIS program. Additionally, a missing piece in the literature is student evaluations of PBIS. Further, student evaluations are not required by the U.S Department of Education s Office of Special Education Programs (OSEP), the founders of PBIS. Although the implementation of PBIS likely varies school by school, future research should seek to gain student feedback on programs that demonstrate high social validity.
Annotated Bibliography 6 Type Topic Conclusion Limitations Miramontes, Marchant, Heath & Fischer, 2011 PBIS Social validity evaluations suggest that school personnel who implement PBIS programs at their school are generally satisfied with the results. Incomplete data set; sample may not be generalizable. Molloy, Moore, Trail, Van Epps, & Hopfer, 2013 PBIS Of the seven components of PBIS, significant predictors of ODRs are Taught Expectations, Reward Systems, and Violation Systems. Interestingly, two of three of these factors are of the most commonly poorest of implementation quality. Evidence of Bias (data from study were obtained from the creators of PBIS); only students with ODRs were included in the sample. Pas & Bradshaw, 2012 SW-PBIS A more in depth implementation of SW-PBIS is related to higher student achievement and lower truancy rates but unrelated to suspension rates. Implications of study are that suspension rates may not be related to PBIS implementation and that the Implementation Phases Inventory (IPI) may be a more accurate measurement of SW- PBIS implementation. Schools not using a SW- PBIS were not included in the sample. Bradshaw, 2013 PBIS & Bullying The preliminary results of integrating bullying prevention programs with PBIS programs have revealed positive impacts on school climate. No current statistics to review to evaluate effectiveness of interventions.
Annotated Bibliography 7 Type Topic Conclusion Limitations NC Department of Public Instruction, 2010 Online Report PBS The 08-09 data demonstrated that NC schools are increasingly implementing PBS systems and that there are marked behavior and academic improvements as a result of these changes. The report left out significance values, so the reader cannot conclude that the observed differences in PBS schools are significantly different than non-pbs schools. Martens & Andreen, 2013 Check In/ Check Out (PBIS intervention) Research on the Check In/ Check Out (CICO) system within the PBIS framework has shown to be effective for K-3 students with behavior problems associated with attention-seeking. School counselors are an integral part of its success. Because the CICO program is a tier two intervention, few students are included in the intervention and sample. Bradshaw, Mitchell, & Leaf (2010) Desriptive SWPBIS A 5-year longitudinal randomized controlled effectiveness trial of SWPBIS of 37 schools in Maryland indicated positive outcomes. Specifically, schools in the intervention group demonstrated a statistically significant decrease in the amount of ODR s and student suspensions and gains in student academic achievement. Small effect and power size. Study limited to elementary schools in Maryland.
Annotated Bibliography 8 Type Topic Conclusion Limitations Farkas et al., (2012). SWPBIS A single-subject case study from an alternative 5-12 school indicated positive student outcomes from the implementation of SWPBIS. Because this study was not experimental in nature, we cannot be certain that the outcomes were a direct result of the intervention. However, we can make inferences.