National PBIS Leadership Forum October 27 & 28, 2016 www.pbis.org E2 Data-Based Decision Making for Universal Lead Presenter: Jessica Daily, University of Oregon Key Words: Applied Evaluation, Tier I, Teams Session E2 Purpose & Objectives Purpose Educators make more effective and efficient decisions when they have the right data, in the right format, at the right time (Gilbert, 1978). This session will provide an overview of the (a) critical features of data based decision making, (b) commonly used data sources and their primary purpose, and (c) tools for organizing decision making teams, meeting routines, and data analysis. Objectives Identify the purpose and current practices of school teams. Identify what data teams need for decision making. Identify tools that support efficient and effective team practices. Maximizing Your Session Participation When Working In Your Team Consider 4 questions: Where are we in our implementation? What do I hope to learn? What did I learn? What will I do with what I learned? Where are you in the implementation process? Adapted from Fixsen & Blase, 2005 Exploration & Adoption We think we know what we need so we are planning to move forward (evidencebased) Installation Let s make sure we re ready to implement (capacity infrastructure) Initial Implementation Let s give it a try & evaluate (demonstration) Full Implementation That worked, let s do it for real and implement all tiers across all schools (investment) Let s make it our way of doing business & sustain implementation (institutionalized use) Leadership Team Action Planning Worksheets: Steps Self Assessment: Accomplishments & Priorities Leadership Team Action Planning Worksheet Session Assignments & Notes: High Priorities Team Member Note Taking Worksheet Action Planning: Enhancements & Improvements Leadership Team Action Planning Worksheet Educational and Community Supports Educational and Community Supports (ECS) is a research unit within the College of Education at the University of Oregon. ECS focuses on the development and implementation of practices that result in positive, durable, and scientifically substantiated change in the lives of individuals. Federal and state funded projects support research, teaching, dissemination, and technical assistance. PBIS Applications (PBISApps) is a series of educational tools created within ECS and related to the implementation of multi-tiered systems of support (MTSS). The PBIS Application tools have been utilized in 25,000+ schools both domestically and internationally. PBIS Leadership Forum 2016 1
Agenda People Information Time Tertiary intensive, individualized Secondary targeted, small group Universal primary prevention People Who should be included? Information What information do we need? Time How often do we review and communicate information? Universal Tier Coordinator Administrative Representation Family Representation Content Expertise (social & academic) General School Representation Coaching Expertise Student Representation All Staff (Communication) Universal System Fidelity Formative Summative Outcomes Screening Formative Summative Team Meeting: Monthly Fidelity Checks: Quarterly Staff/Community (Two-Way) Communication: Formative - monthly Summative annually Continuum of Decision Making Tertiary intensive, individualized 5% of student population Secondary targeted, small group 15% of student population Universal primary prevention provided to all students, effective for approximately 80% The Right People All specialized interventions are more effective and more durable with universal, school-wide behavioral expectations as a foundation. Nearly every school in the country has teams meeting regularly to make decisions concerning administrative, academic, and social support issues. The assumption is that multiple individuals will provide a more inclusive system of problem solving and decision making in which collective experiences, knowledge, skills, language, and vision contribute to improving the core features of schooling and outcomes for children. What do we mean by right people? Who needs the information about a specific area (e.g., tier, intervention, student) in order to sustain implementation of good practices? Core decision-makers Related decision-makers Direct implementers/stakeholders Indirect stakeholders Todd, et al. (2011) Team Initiated Problem Solving Model (TIPS) PBIS Leadership Forum 2016 2
The Right People at Universal Tier I Coordinator Administrative Representation Family Member Representation Content Expertise (social & academic) General School Expertise & Representation Coaching Expertise Student Representation All Staff (At Tier I, EVERYONE is involved!) Families/ Community Partners Academic Internal School Staff Core School Team Attendance Social External Educational Staff Performance Gap & Cause Analysis The Right information Environment/ System Individual Persons 1. Information 2. Resources 3. Incentives 4. Motives 5. Capacity 6. Knowledge 1 Information Clear expectations Timely, specific feedback 6 Knowledge Requisite knowledge and skill base 2 Resources Materials, tools Time Processes 5 Capacity Ability to learn and do 3 Incentives Financial & non-financial encouragement 4 Motives Desire to work and excel What do we mean by the right Information? Schools collect lots of data, but information requires that the data become accessible and relevant. What do people need to know about? o Starting performance levels o Goals and strategies (the plan) o Regular progress updates o Specific problems or barriers that need to be addressed Continuous Quality Improvement Adjust plan as needed Monitor progress toward goal(s) Implement solution(s) with integrity Identify current status and precise problems Establish goal(s) Develop solution(s) PBIS Leadership Forum 2016 3
Types of Data Connecting Outcomes & Fidelity Lucky Sustaining Fidelity Data Outcome Data Outcomes Positive outcomes, low understanding of how they were achieved Replication of success is unlikely Losing Ground Undesired outcomes, low understanding of how they were achieved Positive outcomes, high understanding of how they were achieved Replication of success likely Learning Undesired outcomes, high understanding of how they were achieved Did we implement the systems and strategies we agreed upon? Is the plan resulting in progress toward our goals? Replication of failure likely Fidelity Replication of mistakes unlikely The Right Information at Universal Tier I Universal System Fidelity Formative Summative Outcomes Screening Formative Summative What Questions? How are we doing school-wide? What are the red flags? What is a precise problem statement for the priority red flag? Are any students showing signs of needing additional social or academic supports? How are the Tier II and Tier III systems working? What information needs to be communicated? What Data? Fidelity PBIS Assessment data Summative: TFI, BoQ, SET, SAS Formative: TFI, TIC Curriculum or Intervention Evaluations Reading TFI Educator Effectiveness Measures Outcome Discipline/Behavior School Climate Attendance School-Wide Academic Data Universal screening & progress monitoring Aggregated Student Report Cards/Grades/Progress Reports School Performance Reports The Right Time PBIS Leadership Forum 2016 4
Continuum of Decision Making Cumulative Mean ODRs Per Month for 325+ Elementary Schools 08-09 12 Tertiary intensive, As we increase the individualized intensity of supports 5% of student population we will also need to Secondary targeted, intensify the small group frequency of our 15% of student population data collection, Universal primary data analysis, prevention and provided to all students, decision-making effective for approximately 80% cycles. Cumulative Mean ODRs 10 8 6 4 2 0 0-1 2-5 6+ McIntosh, K., Frank, J. L., & Spaulding, S. A. (2010). Establishing researchbased trajectories of office discipline referrals for individual students. School Psychology Review, 39(3), 380. Potential trajectories with the October Catch 12 The Right Time at Universal Tier I Cumulative Mean ODRs 10 8 6 4 2 0 AugSept Oct Nov Dec Jan Feb Mar Apr May Jun 0-1 2-5 6+ McIntosh, K., Frank, J. L., & Spaulding, S. A. (2010). Establishing researchbased trajectories of office discipline referrals for individual students. School Psychology Review, 39(3), 380. Team Meeting: Monthly meetings Quarterly fidelity checks Staff/Community (Two-Way) Communication: Formative - monthly Summative annually Performance Gap & Cause Analysis Analyzing the Data Tier I Universal Summative Environment/ System Individual Persons 1. Information 2. Resources 3. Incentives 4. Motives 5. Capacity 6. Knowledge 1 Information Clear expectations Timely, specific feedback 6 Knowledge Requisite knowledge and skill base 2 Resources Materials, tools Time Processes 5 Capacity Ability to learn and do Formative 3 Incentives Financial & non-financial encouragement 4 Motives Desire to work and excel PBIS Leadership Forum 2016 5
Building Decision Guide teams to build comprehensive decision systems not data systems. Data System Technology that houses data o District student information system (SIS) o Excel spreadsheet o SWIS Suite application Decision System Agreements on the collection and use of data (including a data system) Routines and procedures to embed fidelity and outcome data into decision making Tips for Building Decision Analyzing data in layers to drill down What is our current reality? Is there a problem? (a red flag) Note: It s ok to be doing well! Put the problem into context (get specific) what, where, when, who, how often, why Be efficient, don t drown in the data Problem Solving with Precision The statement(s) of a problem is important for team-based problem solving. Everyone must be working on the same problem with the same assumptions. Problems are often framed in the primary form. Raises awareness Not useful for problem solving Precise problem statements result from a detailed data review and are solvable. Continuous Quality Improvement Adjust plan as needed Monitor progress toward goal(s) Implement solution(s) with integrity Identify current status and precise problems Establish goal(s) Develop solution(s) SWIS Drill-Down Worksheet Tools To Get Organized SWIS Drill Down tool Multi-tiered Data Review and Monitoring Plan Guide PBIS Leadership Forum 2016 6
School-Wide Information System (SWIS) SWIS allows school staff to enter office discipline referrals online. The data are summarized to provide information about the entire student body, groups of students, and/or individual students over any period of time. As a result, the user can: 1. Review school-wide referral patterns 2. Define behavior patterns in greater detail Core Reports Average Referrals/Day/ Month Referrals by Location Referrals by Problem Behavior Referrals by Time Referrals by Grade Referrals by Student The Value of Drilling Down Data Drill Down Drilling down into data allows us to move from general to precise problem statements. What problem behaviors are most common? Where are problem behaviors most likely? When are problem behaviors most likely? Who is engaged in the problem behavior? Why are problem behaviors sustaining (i.e., perceived motivation)? How often are problems occurring? Use the information from the SWIS Dashboard to drill down and analyze data. Change the graph type to change the analysis. Use the summary to analyze problem size. Drill Down Steps Multi-Tiered Data Review and Monitoring Plan Guide 1. Determine your red flag item 2. Disaggregate data related to red flag item 3. Continue to add each clue (filter) or new piece of information to narrow down the context of the problem 4. Summarize the precise problem statement PBIS Leadership Forum 2016 7
Team Organization What Do We Need to Know? Tier 1 Data Review Tier 2/3 Connections Meeting Wrap Up Routine What Next? We gave you some big ideas and a couple of tools, but how can those ideas refine and support schools to start moving toward efficient and effective data collection and organization that gives PBIS Leadership Forum 2016 8
PBIS Applications www.pbisapps.org PBIS Leadership Forum 2016 9