Drilling Down Why and How to Analyze Your Central Oregon PBIS Conference Bend, Oregon April 2017 Presenters Katie Conley Sr. Research Assistant & Doctoral Student kconley1@uoregon.edu Celeste Rossetto Dickey Sr. Research Associate cdickey@uoregon.edu PBISApps is run by Educational and Community Supports (ECS), a research unit at the University of Oregon since 1977. Lead by Kent McIntosh, PhD, ECS focuses on federal and state funded projects supporting research, teaching, and technical assistance through the PBIS OSEP Technical Assistance Center. 2017 Central Oregon PBIS Conference 1
Introductions Purpose To close performance gaps we need the right data, at the right time, in the right format, available to the right people (Gilbert, 1978) Objectives Rationale for building comprehensive data and decision systems in SWPBIS implementation In s and out s of data Core features of data analysis or the drill down approach Example of the drill down approach in SWPBIS Implementation Phase Exploration & Adoption We think we know what we need so we are planning to move forward (evidence based) Installation Let s make sure we re ready to implement (capacity infrastructure) Initial Implementation Let s give it a try & evaluate (demonstration) Full Implementation That worked, let s do it for real and make a plan to implement all tiers across all schools (investment) Let s make it our way of doing business & sustain implementation for at least 10 years (institutionalization) Adapted from Fixsen & Blase, 2005 Considerations What is my role in evaluating SWPBS data? Data analyst generate and summarize data for others Team facilitator use data to drive team meeting time and decisions Direct consumer use data to participate in discussions and decisions Indirect consumer receive data/updates less than monthly What would I like to take away from this session? Participate more effectively in using data for decision making Improve my ability to consume data and apply to my role Improve my data analysis practices 2017 Central Oregon PBIS Conference 2
Information People Time Session Organizer Information What information do we need? In what format? Universal System Fidelity Formative Summative Outcomes Screening Formative Summative People How do we include everyone in ways that are helpful? Time How often do we review and communicate information? Systems Coordinator Administrative Representation Family Representation Content Expertise (social & academic) General School Representation Coaching Expertise Student Representation All Staff (Communication) Team Meeting (monthly) Fidelity Checks (quarterly) Staff/Community (Two Way) Communication: Formative monthly Summative annually Continuum of Decision Making Tertiary intensive, individualized 5% of student population Secondary targeted, small group 15% of student population Universal primary prevention provided to all students, effective for approximately 80% All specialized interventions are more effective & durable with strong universal, Tier I systems as a foundation. 2017 Central Oregon PBIS Conference 3
SWPBIS and Outcomes Desired Outcome: Enhanced social competence & academic achievement for all students. Data Decision Making Supports How do we get to our desired outcome? Systems Staff behavior Supports Practices Student behavior Supports Outcome/Goal SWPBIS Sustainability Research Number of Responses 80 70 60 50 40 30 20 Enablers Admin Support Staff Buy In Fidelity Data Teaming Number of Responses 80 70 60 50 40 30 20 Barriers Resources Turnover Fidelity Staff Buy In Philosophy 10 10 0 0 McIntosh, K., Predy, L., Upreti, G., Hume, A. E. & Mathews, S. (2014) SWPBIS Sustainability Research What is the factor most related to high sustainability of SWPBIS? The frequency that DATA are presented to all school staff! McIntosh, Kim, Mercer, Strickland Cohen, & Horner (2015) 2017 Central Oregon PBIS Conference 4
Identifying Performance Gaps Capacity Knowledge Information Resources Incentives Motives Performance Gap = The difference between where an organization is and where they want to be. Performance Gap & Cause Analysis Gilbert s behavior Engineering Model (BEM) provides a way to systemically identify barriers to individual and organizational performance. Addressing Performance Gaps 1. Information 2. Resources 3. Incentives 4. Motives 5. Capacity 6. Knowledge Gilbert, 1978 2017 Central Oregon PBIS Conference 5
Discuss with a neighbor Where is data in terms of our overall SWPBIS priorities? Are we using DATA to drive decisions about resources, incentives, and individual needs (student or staff)? Right Information Data Based Decision Making Decisions are more likely to be effective and efficient when they are based upon data. The quality of decision making depends most on the first step: defining problems to be solved with: Precision (who, what, where, when, how often, why) Clarity (general agreement across team) 2017 Central Oregon PBIS Conference 6
Connecting Fidelity & Outcomes Lucky Positive Outcomes Low understanding of how they were achieved Sustaining Positive Outcomes High understanding of how they were achieved Outcomes Replication of success unlikely Losing Ground Undesired Outcomes Low understanding of how they were achieved Replication of failure likely Replication of success likely Learning Undesired Outcomes High understanding of how they were achieved Replication of failure unlikely Fidelity It s all about connecting the dots Drilling Down: PBIS Tier ISystems Fidelity Data Team. Surveys, Action Plans *Tiered Fidelity Inventory (TFI Tier I) Staff. Surveys, observations, interviews Student. Surveys, observations, interviews External. Non staff observations, interviews (TFI Walk through) Procedures, Routines, Perceived enablers and barriers to fidelity Outcome Data Overall effectiveness (e.g., % students with 0 1, 2 5, 6+ discipline referrals, academic data) School Climate (surveys, interviews) Context specific problems/progress who, what, where, when, how often, why o Overall Summary for year o Current Summary (month/quarter) o Specific problem contexts o Students to refer for additional supports Continuous Quality Improvement Cycle Reassess and revise solution(s) as needed Identify current status & problems with precision Monitor outcomes and compare to goal(s) Establish goal(s) Develop solution(s) Implement solution(s) with integrity and fidelity 2017 Central Oregon PBIS Conference 7
What are PBISApps? PBIS Applications are educational tools created to support implementation of multi tiered system of supports (MTSS). PBIS Application tools have been utilized in more than 25,000 schools internationally. www.pbisapps.org How are Data Used? Screening. Are we matching supports and services to those who need them? Summative. What is the BIG picture (comprehensive) and what are the major areas of progress and concern that we should find out more about? Formative. How can we quickly identify status, progress, potential problems to make decisions right now? Data Analyst Role Who has access and interest in analyzing data and then summarizing visually and verbally to tell the story? Data analysts need to understand the school culture, the data, and storytelling to be effective! 2017 Central Oregon PBIS Conference 8
Drill Down Approach The Drill Down approach starts with the big picture and looks for potential red flags that might indicate a problem that might need to be addressed. Then we identify and address the problem context. 1. What is our current reality? 2. Is there a problem (red flag)? 3. What is the specific problem context? What, where, when, who, how often, why? Be efficient, don t drown by getting too specific (e.g., red heads with blue shoes) Fidelity Examples from the TFI TFI Total Score (across tiers) TFI Scale (by tier) Fidelity Examples from the TFI Tier I Subscales: 1. Teams 2. Implementation 3. Evaluation TFI Subscale (across tiers and major features) 2017 Central Oregon PBIS Conference 9
Fidelity Examples from the TFI Individual Items Outcome Example from SWIS SWIS Dashboard Shows the core reports for the current school year Average Referrals Per Day Per Month Referrals by Time of Day Referrals by Location Referrals by Day of Week Referrals by Problem behavior Referrals by Grade Referrals by Student Outcome Example from SWIS SWIS Core Reports Allows each report to be tailored for a current status Average Referrals Per Day Per Month Referrals by Time of Day Referrals by Location Referrals by Day of Week Referrals by Problem behavior Referrals by Grade Referrals by Student 2017 Central Oregon PBIS Conference 10
Outcome Example from SWIS SWIS Additional Reports Allows for additional perspectives to be used at least quarterly or as needed Multi year reports Referrals by Staff (for Admin use only) Ethnicity Reports Triangle Data Year End Reports Student Dashboard SWIS Drill Down Tool Use the information from the SWIS Dashboard to drill down and analyze data. Change the graph type to change the analysis. Use the summary to analyze problem size. Precise Problem Statements Does our problem statement answer the following questions? What is the problem behavior? Where is the problem happening? Who is engaged in the behavior (sub groups, not individuals)? When is the problem most likely to occur? How often is the problem happening? Why is the problem sustaining? 2017 Central Oregon PBIS Conference 11
Primary vs. Precise Problem Statements Low fidelity of SWPBS implementation Too many referrals Gang behavior is increasing The cafeteria is a nightmare Staff aren t giving out praise tickets There are twice as many ODRs for aggression on the playground during first recess than last year. These involve a large number of students (32) and staff (8), and it seems to be related to getting access to the new playground equipment. Systemic vs. Individual Student Issues 12 Hallway Referrals by Student ID Sept. 3 Nov. 9 10 8 6 Individual Issue Fewer than students in the same context System Issue 10+ students in the same context 4 2 0 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 #120 #121 #122 #123 #124 #125 Setting Goals & Building Solutions 1. Define the goal What would be good enough to move to another problem? 2. Use the goal to guide the solution How can we move from where we are to where we want to be? Solution Components Prevention Teaching (re teaching) Elevate Recognition Extinction Consistent corrective consequences Data Collection 2017 Central Oregon PBIS Conference 12
Right People Collaborative Problem Solving Nearly every school in the country has teams meeting regularly to make decisions concerning administrative, academic, and social support issues. The assumption is that multiple individuals will provide a more inclusive system of problem solving and decision making in which collective experiences, knowledge, skills, language, and vision contribute to improving the core features of schooling and outcomes for children. Team Initiated Problem Solving Model (TIPS) Todd, et al. (2011) Right People? Everyone is included. administrators, coaches, teams, staff, students, families, community 1. Who needs to know what? Overall status of fidelity and outcomes Detailed status of fidelity and outcomes Specific focus areas (e.g., problems, needs, goals met, new initiatives) 2. Why? Awareness only (shared knowledge) Call to action Celebration of success 2017 Central Oregon PBIS Conference 13
Identifying Stakeholders Data Analyst(s) Core decision makers Related decision makers Implementers Administration Local community members Building Positive Relationships Storytelling with Data (Knaflic, 2015) Relevant Easy to Read Focused Simple Right Time 2017 Central Oregon PBIS Conference 14
Discuss with a neighbor How are we sharing information with our stakeholders (outside the team)? What information? Fidelity? Outcomes? How often? Monthly? Quarterly? Annually? Ever? Continuum of Supports As we increase the intensity (tier) of supports we will also increase the frequency & specificity of our data collection, data analysis, & decision making cycles. When Should Schools Start Screening? 12 Cumulative Mean ODRs Per Month for Elementary Schools Cumulative Mean ODRs 10 8 6 4 Differences become clear in October 2 0 McIntosh et al (2010) 2017 Central Oregon PBIS Conference 15
Are Secondary Schools Different? 12 Cumulative Mean ODRs Per Month for Middle Schools 10 Cumulative Mean ODRs 8 6 4 Still October 2 0 Predy et al (2015) The October Catch 12 Hypothesized Impact of October Catch No Impact (0 1) Hypothetical Cummalitive Mean ODRs 10 8 6 4 2 30% 20% Hypothesized 20% Decrease (2 5) Hypothesized 30% Decrease ( 6) 0 SWPBIS Sustainability Research What is the factor most related to high sustainability of SWPBIS? The frequency that DATA are presented to all school staff! McIntosh, Kim, Mercer, Strickland Cohen, & Horner (2015) 2017 Central Oregon PBIS Conference 16
Routines Per Stakeholder Groups Group Information Schedule Data Analyst(s) & Core Team Related Decision Makers & Implementers Comprehensive Fidelity Checks Comprehensive Outcome Checks Formative Fidelity Checks (based on action plan) Formative Outcome current status, progress reports, new precise problems Brief summary of data reported at core (SWPBS) team meetings Administration This depends on the needs and interests of administrators often a brief summary of fidelity and outcomes Local Community Website Newspaper or local media or local meetings Annually Annually At least monthly At least monthly Monthly (e.g., staff meetings) Quarterly or annually Quarterly As available Let s Practice! Final Reminders Use Data to Make Good Decisions Build Positive Relationships & Tell the Story 2017 Central Oregon PBIS Conference 17