Advances in the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) for Reading First OHIO LITERACY INSTITUTE Assessment, Accountability, and the Learner June 7-9, 4 http://dibels.uoregon.edu Roland H. Good III University of Oregon Beginning Reading Core Components #1. Phonemic Awareness: The ability to hear and manipulate sound in words. #. Phonics: The ability to associate sounds with letters and use these sounds to read words. #3. Fluency : The effortless, automatic ability to read words in isolation (orthographic reading) and connected text. #4. Vocabulary Development: The ability to understand (receptive) and use (expressive) words to acquire and convey meaning. #5. Reading Comprehension: The complex cognitive process involving the intentional interaction between reader and text to extract meaning. National Reading Panel. (). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups. Bethesda, MD: National Institute of Child Health and Human Development. Available: http://www.nationalreadingpanel.org/. Rev. 6/7/4 (c) 4 Model of Big Ideas, Indicators, and Timeline DIBELS Nonsense Word Fluency Adapted from Good, R. H., Simmons, D. C., & Kame'enui, E. J. (1). The importance and decisionmaking utility of a continuum of fluency-based indicators of foundational reading skills for thirdgrade high-stakes outcomes. Scientific Studies of Reading, 5, 57-88. kik woj sig faj yis kaj fek av zin zez lan nul zem og nom yuf pos vok viv feg bub dij sij vus tos wuv nij pik nok mot nif vec al boj nen suv yig dit tum joj yaj zof um vim vel tig mak sog wot sav Here are some more makebelieve words (point to the student probe). Start here (point to the first word) and go across the page (point across the page). When I say, begin, read the words the best you can. Point to each letter and tell me the sound or read the whole word. Read the words the best you can. Put your finger on the first word. Ready, begin. Rev. 6/7/4 (c) 4 3 Rev. 6/7/4 (c) 4 4
http://dibels.uoregon.edu Rev. 6/7/4 (c) 4 5 Reading First: Four Kinds/Purposes of Reading Assessment An effective, comprehensive, reading program includes reading assessments to accomplish four purposes: Outcome - Assessments that provide a bottom-line evaluation of the effectiveness of the reading program. Screening - Assessments that are administered to determine which children are at risk for reading difficulty and who will need additional intervention. Diagnosis - Assessments that help teachers plan instruction by providing in-depth information about students skills and instructional needs. Progress Monitoring - Assessments that determine if students are making adequate progress or need more intervention to achieve grade level reading outcomes. Source: Reading First Initiative: Secretary s Leadership Academy Rev. 6/7/4 (c) 4 6 Using an Outcomes Driven Model to Provide Decision Rules for Progress Monitoring Outcomes Driven model: Decision making steps 1. Identifying Need for Support. Validating Need for Instructional Support 3. Planning and Implementing Instructional Support 4. Evaluating and Modifying Instructional Support 5. Reviewing Outcomes for Individuals and Systems Good, R. H., Gruba, J., & Kaminski, R. A. (). Best Practices in Using Dynamic Indicators of Basic Early Literacy Skills (DIBELS) in an Outcomes-Driven Model. In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology IV (pp. 679-7). Washington, DC: National Association of School Psychologists. Rev. 6/7/4 (c) 4 7 1. Identifying Need for Support Key Decision for Screening Assessment: Which children may need additional instructional support to attain important reading outcomes? Data used to inform the decision: Compare individual student s performance to local normative context or expected performance to evaluate need for additional instructional support. Local normative context: First, choose a percentile cutoff. th percentile seems a good place to start, but a district could choose 15 th percentile or 5 th percentile or other cutoff depending on resources. Expected performance: A deficit in a foundation skill is a strong indicator that instructional support will be needed to attain later benchmark goals. Rev. 6/7/4 (c) 4 8
Beginning of First Grade Rev. 6/7/4 (c) 4 9 Decision Utility of DIBELS Fall of 1st LNF >= 37, DIBELS PSF >= 35, DIBELS NWF >= 4 Instructional Recommendation: Benchmark - At grade level. Effective core curriculum and instruction recommended, Odds of reading 4 or more words correct per minute at the end of first grade: 84% LNF < 5, DIBELS PSF < 1, DIBELS NWF < 13 Instructional Rec: Intensive - Needs substantial intervention: Odds of reading 4 or more words correct per minute at the end of first grade: 18% (unless given intensive intervention) Value of knowing the instructional recommendation and the goal early enough to change the outcome: Priceless. Rev. 6/7/4 (c) 4 1. Validate Need for Support Key Decision: Are we reasonably confident the student needs instructional support? Rule out easy reasons for poor performance: Bad day, confused on directions or task, ill, shy, or similar. More reliable information is needed to validate need for support than for screening decisions. Data used to inform the decision: Repeated assessments on different days under different conditions Compare individual student s performance to local normative context or expected performance to evaluate discrepancy. Rev. 6/7/4 (c) 4 11 Nonsense Word Fluency Validating Need for Support Verify need for instructional support by retesting with alternate forms until we are reasonably confident. Beginning 1st cutoff low risk Beginning 1st cutoff at risk Rev. 6/7/4 (c) 4 1
3. Planning and Implementing Instructional Support Key Decisions for Diagnostic Assessment: What are the Goals of instruction? Where are we? Where do we need to be? By when? What course do we need to follow to get there? What skills should we teach to get there? Focus on the beginning reading core areas: Phonological Awareness, Alphabetic Principle, Accuracy and Fluency with Connected Text Specific skills based on error analysis or additional diagnostic assessment (e.g., CTOPP). How much instructional support is needed? Intensive Instructional Support Strategic Instructional Support Benchmark Instruction Rev. 6/7/4 (c) 4 13 Instructional Goals for Core Components of Beginning Reading Benchmark Goals to be On Grade Level: Middle K: Phonological Awareness with 5-35 on DIBELS Initial Sound Fluency by mid kindergarten (and 18 on PSF) End K: Phonemic Awareness with 35-45 on DIBELS Phoneme Segmentation Fluency by end of kindergarten (and 5 on NWF) Middle 1st: Alphabetic principle 5-6 on DIBELS Nonsense Word Fluency by mid first grade (and on DORF) End 1st: Fluency with 4-5 on DIBELS Oral reading fluency by end of first grade (and RTF 5% or more). End nd: Fluency with 9 + on DIBELS Oral reading fluency by end of second grade (and RTF 5% or more) End 3rd: Fluency with 11 + on DIBELS Oral reading fluency by end of third grade (and RTF 5% or more) Rev. 6/7/4 (c) 4 14 Instructional Goals Establish an Instructional Goal for Alphabetic Principle that will change odds of being a reader Nonsense Word Fluency Mid-year cutoff low risk Mid-year cutoff at risk Rev. 6/7/4 (c) 4 15 Rev. 6/7/4 (c) 4 16
Oregon Reading First Review of Supplemental and Intervention Programs OR Reading First developed review criteria for supplemental and intervention programs and reviewed 16 programs for the percent of criteria met. http://oregonreadingfirst.uoregon.edu/sireport.php Phonemic Awareness Early Reading Intervention 96% Road to the Code 8% Phonemic Awareness in Young Children 75% Phonics or Alphabetic Principle Reading Master Fast Cycle 96% Read Well 94% Voyager Passport 9% Early Reading Intervention 81% Fluency with Connected Text Read Naturally 9% Great Leaps 66% Headsprout 61% Rev. 6/7/4 (c) 4 17 4. Evaluating and Modifying Instructional Support Key Decision for Progress Monitoring Assessment: Is the intervention effective in improving the child s early literacy skills? How much instructional support is needed? Enough to get the child on trajectory for Benchmark Goal. When is increased support needed? Monitor child s progress during intervention by comparing their performance and progress to past performance and their aimline. Three consecutive assessments below the aimline indicates a need to increase instructional support. Rev. 6/7/4 (c) 4 18 Progress Monitoring Repeated, formative assessment to evaluate progress toward important goals for the purpose of modifying instruction or intervention. Frequency of Progress Monitoring 3 times per year for students at low risk (All Students) Benchmark 1 per month for students with some risk Strategic 1 per week for students at risk Intensive Rev. 6/7/4 (c) 4 19 Research on Progress Monitoring Progress monitoring has been extensively researched in Special Education For example: Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional Children, 53, 199-8. With Reading First, progress monitoring is not just for special education any more. Rev. 6/7/4 (c) 4
Effects of Progress Monitoring Fuchs and Fuchs (1986) found the average effect size associated with progress monitoring was: +.7 for monitoring progress +.8 when graphing of progress was added +.9 when decision rules were added A student at the 5th percentile would be expected to move to the 8nd percentile (i.e., a score of 1 would move to a score of 114) Perhaps more important, a student at the 6th percentile would be expected to move to the average range (5th percentile) (i.e., a score of 76 would move to a score of 9) Rev. 6/7/4 (c) 4 1 Progress Monitoring Tools Meaningful and important goals, waypoints, or benchmarks representing reading health or wellness. Meaningful and Important Public and Measurable Ambitious Brief, repeatable, formative assessment of progress toward benchmark goals that is sensitive to intervention. Brief and Efficient Repeatable - weekly or monthly Reliable and Valid indication of risk and growth Rev. 6/7/4 (c) 4 Is Progress is Related to Outcomes? The logic of the Evaluating and Modifying Support step relies on evidence that amount of progress toward goals is related to important reading outcomes. Given or controlling for initial skills, is slope of progress on NWF in the Fall of first grade related to first grade reading outcomes for at risk students? Evaluations of the relation between slope of progress and outcomes must consider the student s initial skills. Rev. 6/7/4 (c) 4 3 Progress GIVEN initial skills. Nora has a slope twice that of Nick, but substantially lower reading outcome because her initial skills are so much lower. NWF Correct Letter Sounds 8 7 6 5 4 3 1 Nick: slope = +.7 Nora: slope = +1.5 Nick Predicted Spring DORF = 7 Nora Predicted Spring DORF = 6 4 8 1 16 4 8 3 36 Week Slope, by itself without considering initial skills is not enough to predict outcomes. Rev. 6/7/4 (c) 4 4
Similar Initial Skills Slope is related to outcomes Nora and Nell have similar initial skills Nell s higher slope predicts higher skills in middle of first grade and higher reading outcomes. NWF Correct Letter Sounds 8 7 6 5 4 3 1 Nick: slope = +.7 Nell: slope = +.56 Nora: slope = +1.5 Nell Predicted Spring DORF = 36 Nora Predicted Spring DORF = 6 4 8 1 16 4 8 3 36 Week Considering Initial Skills, Does Slope Add to Predictions of Outcomes? Students with complete data from -3 in the DIBELS Data System were examined for level of risk, slope of progress, and reading outcomes. Beginning NWF NWF Slope Group N Mean Std Dev N Mean Std Dev At Risk to 1 739 5.46 4.3 739 1.54 1. Some Risk 13 to 3 66 18.8 3.13 66 1.47.97 Low Risk 4 to 49 388 34.6 7.9 388 1.3 1.16 Hi AP 5 to 55 188 7.3.55 188 1.4 1.73 Total 91715 9.9.1 91715 1.36 1.19 Rev. 6/7/4 (c) 4 5 Rev. 6/7/4 (c) 4 6 Utility of Initial NWF Risk Categories Initial skills on NWF are a very strong predictor of reading outcomes. Ending ORF Group N Mean Std Dev Odds of Achieving Benchmark Goal At Risk 739 6.5 1.13 % NWF to 1 Some Risk 66 4.81 4.47 47% NWF 13 to 3 Low Risk 388 6.7 8.74 76% NWF 4 to 49 Hi AP 188 1.19 34.44 97% NWF 5 to 55 Total 91715 55.8 35.68 6% Does Slope Add to the Prediction of Reading Outcomes After Risk Level and Initial Skills? Rules for evaluating effects: 1. Significance. With N >, everything is significant.. Percent of variance explained. More than 1% of variance explained is a good indication of a strong effect. Greater percent is stronger. 3. Educationally meaningful effects. Analysis of outcomes to see if the predicted differences would be educationally important to teachers, students, parents. Rev. 6/7/4 (c) 4 7 Rev. 6/7/4 (c) 4 8
Does Slope Add to the Prediction of Reading Outcomes After Risk Level and Initial Skills? Sequential model predicting first grade DORF reading outcomes from (1) risk category, () initial NWF skill given risk, and (3) slope given risk and initial skill. Source DF R change NWF Risk Category 3.4 Initial NWF Skill Given Risk 1.8 Slope Given Risk, Initial Skill 1.11 Total 91714 1. Risk category, initial skills, and slope combined explain 59% of reading outcomes. Rev. 6/7/4 (c) 4 9 Variance Explained by Slope for Each Risk Category A separate analysis was conducted for each risk category. Group Percent of Risk Category Variance in Reading Outcomes Explained NWF Initial Skills NWF Slope Given Initial Skills At Risk 8% 6% NWF to 1 Some Risk % 1% NWF 13 to 3 Low Risk 8% 1% NWF 4 to 49 Hi AP 5% 11% NWF 5 to 55 But, is the variance explained by slope (given risk and initial skills) educationally important? Rate of progress in alphabetic principle is especially important for students who are at risk for low reading outcomes. Rev. 6/7/4 (c) 4 3 Variability in Slope for At Risk Students About 68% of At Risk student s trajectories are between the low slope and the high slope. Are Differences in Slope Educationally Meaningful for At Risk Students? Yes. Predicted reading outcomes are substantially different. Nonsense Word Fluency Mid-year cutoff low risk Hi Slope Lo Slope Predicted End First DORF 5 45 4 35 3 5 15 1 5 Lo Slope.5.77 1.3 1.8 1.54 1.79.5.3.56 MSlope - 1sd to MSlope + 1sd Hi Slope Rev. 6/7/4 (c) 4 31 Rev. 6/7/4 (c) 4 3
Conclusions: Validity of DIBELS NWF Slope Initial risk status and initial skills on DIBELS Nonsense Word Fluency are very important in predicting reading outcomes in first grade, explaining 48% of variance in outcomes. An increasing pattern of scores through the first semester of first grade on DIBELS Nonsense Word Fluency appears to be a very important predictor of reading outcomes for students who are at risk, and indeed for each risk category. We can be confident that increases in DIBELS Nonsense Word Fluency reflect improved performance on alphabetic principle skills that contribute to important end-of-year reading outcomes. Rev. 6/7/4 (c) 4 33 5. Reviewing Outcomes Key Decisions for Outcome/Accountability Assessment: Does the child have the early literacy skills predictive of successful reading outcomes? Does the school have core curriculum and instruction as well as a system of effective instructional support so their students achieve literacy outcomes? Data used to inform the decision: Evaluate individual student s performance with respect to benchmark goals that with the odds in favor of achieving subsequent literacy goals. Compare school/district outcomes to goals and outcomes from previous year. Evaluate core curriculum and system of additional support for each step to identify strengths and areas for improvement. Rev. 6/7/4 (c) 4 34 Case Example: Rick "Rick" Nonsense Word Fluency Correct Letters per Min 7 6 5 4 3 1 Before Tutoring Tutoring Results at School # Year 3: 1-3-Aug 6-Sep 13-Sep -Sep 7-Sep 4-Oct 11-Oct 18-Oct 5-Oct 1-Nov 8-Nov 15-Nov -Nov 9-Nov 6-Dec 13-Dec Instructional Days Rev. 6/7/4 (c) 4 35 Rev. 6/7/4 (c) 4 36
Kindergarten Benchmark Scores Across Years: Initial Sound Fluency Kindergarten Benchmark Scores Across Years: Phoneme Segmentation Fluency Red = -1 School Year Blue = 1- School Year Red = -1 School Year Blue = 1- School Year Rev. 6/7/4 (c) 4 37 Rev. 6/7/4 (c) 4 38 Kindergarten Benchmark Scores Across Years: Nonsense Word Fluency 1st Grade Benchmark Scores Across Years: Phoneme Segmentation Fluency Red = -1 School Year Blue = 1- School Year Red = -1 School Year Blue = 1- School Year Rev. 6/7/4 (c) 4 39 Rev. 6/7/4 (c) 4 4
1st Grade Benchmark Scores Across Years: Nonsense Word Fluency Reading First in Cincinnati Public: The Multi-Tiered Model of Intervention Support Special Education Red = -1 School Year Blue = 1- School Year Data based decision making at all levels! Intensive Individualized Intervention Efforts: Collab. Prob Solving Targeted Interventions for Struggling Readers: Voyager s In School and Extended Day Interventions DIBELS Weekly Progress Monitoring Effective School and Class-wide Reading Instruction Rooted in Scientifically Based Reading Research: Voyager Universal Literacy System DIBELS Benchmarking Intensity of Resources Based on Student Need Rev. 6/7/4 (c) 4 41 Rev. 6/7/4 (c) 4 4 Model of Big Ideas, Indicators, and Timeline Reviewing Outcomes: Effectiveness of Benchmark Instruction (Core Curriculum) For each step toward literacy outcomes, a school with an effective core curriculum and instruction supports students who are on track (i.e., low risk or benchmark) to achieve the goal. For students with the odds in favor of achieving literacy goals, it is the job of the core to teach the core components so that all students (1%) achieve the goals. Adapted from Good, R. H., Simmons, D. C., & Kame'enui, E. J. (1). The importance and decisionmaking utility of a continuum of fluency-based indicators of foundational reading skills for thirdgrade high-stakes outcomes. Scientific Studies of Reading, 5, 57-88. Rev. 6/7/4 (c) 4 43 Rev. 6/7/4 (c) 4 44
Beginning First to Middle First Middle of first grade outcomes for students with benchmark, strategic, and intensive instructional recommendations in the beginning of first grade Number of Schools 3 5 15 1 5 Intensive Strategic Benchmark - 1 11-1 - 3 31-4 41-5 51-6 61-7 71-8 81-9 91-1 Conditional Percent Reaching NWF Goal A typical (middle) school had 68% of children with a beginning first grade benchmark recommendation achieve the middle of first grade goal, and % of children with intensive support recommendation. Rev. 6/7/4 (c) 4 45 Reviewing Outcomes: Effectiveness of Strategic and Intensive Intervention For each step toward literacy outcomes, a school with an effective system of effective interventions supports students who are not on track (i.e., at some risk or at risk of difficulty achieving literacy goals) to achieve the goal. For students with the odds against achieving literacy goals unless we provide an effective intervention, it is the job of the system of additional support to augment the core curriculum so that all students (1%) achieve the same benchmark goals. Rev. 6/7/4 (c) 4 46 School is the primary level of analysis Students can be placed with different interventionists than their classroom teacher Students with similar and challenging needs are sometimes assigned to the same classroom. Decisions about how intervention is structured and delivered are made at a school level, and the school, at a systems level, needs to plan for and evaluate the instructional support that is provided. The classroom or intervention group level of analysis can inform school level discussions when appropriate. Rev. 6/7/4 (c) 4 47 Evaluation of Effectiveness is Component by Component, Step by Step, Year by Year Component by component Phonemic Awareness Phonics/Alphabetic Principle Accuracy and Fluency with Connected Text Vocabulary and Reading Comprehension Step by Step First half of kindergarten Beg to Mid Second half of kindergarten Mid to End First half of first grade Beg to Mid Second half of first grade Mid to End First half of second grade Beg to Mid Second half of second grade Mid to End Year by year Kindergarten Beg to End First Grade Beg to End Rev. 6/7/4 (c) 4 48
Evaluating Effectiveness I. Outcomes Criterion 95% of students achieve the early literacy goal. II. Adequate Progress Criteria Core: Benchmark students make adequate progress Strategic Support: Strategic students make adequate progress Intensive Intervention: Intensive students make adequate progress I. Outcomes Criterion Strength The schoolwide instructional system is a strength, including research-based effective reading core curriculum and delivery of that curriculum, strategic support, and intensive intervention. Absolute Standard: 95% or more of students achieve the next literacy goal. If outcomes criterion is not met, evaluate the effectiveness of core curriculum, strategic support, and intensive interventions using the Adequate Progress Criteria. Rev. 6/7/4 (c) 4 49 Rev. 6/7/4 (c) 4 5 II. Adequate Progress Benchmark Students Core Curriculum and Instruction Strength Research-based effective reading core curriculum and delivery of that curriculum. Logic: The core curriculum and instruction should support benchmark students to achieve literacy goals. Absolute Standard: 95% of benchmark students achieve the next literacy goal. Relative Standard: Upper third of effectiveness of core curriculum and instruction compared to other schools. Meet either the absolute standard or the relative standard and the effectiveness of the core is a strength for the school. Rev. 6/7/4 (c) 4 51 Needs Support Core Curriculum and Instruction Needs Support School (a) does not meet the Outcome Criterion, (b) does not meet the absolute standard for adequate progress and (c) the school is in the middle third of effectiveness compared to other schools. The school needs support in terms of professional development, curriculum materials, integrity of delivery, or time investment to increase the effectiveness of the core. Rev. 6/7/4 (c) 4 5
Needs Substantial Support Core Curriculum and Instruction Needs Substantial Support School (a) does not meet the Outcome Criterion, (b) does not meet the absolute standard for adequate progress and (c) the school is in the lower third of effectiveness compared to other schools. Schoolwide priority for professional development, curriculum materials, integrity of instruction, and time investment. Rev. 6/7/4 (c) 4 53 II. Adequate Progress Strategic Students Effective Strategic Support Strength Research-based effective reading strategic support and delivery of that support. Logic: Strategic support should be sufficient to change the odds of achieving literacy goals. If students are improving, but not enough to achieve benchmark goals, it isn t enough to change the odds. Absolute Standard: 8% of strategic students achieve the next literacy goal. Relative Standard: Upper third of effectiveness of strategic support compared to other schools. Meet either the absolute standard or the relative standard and the effectiveness of the strategic support is a strength for the school. Rev. 6/7/4 (c) 4 54 II. Adequate Progress Intensive Students Effective Intensive Support Strength Research-based effective reading strategic support and delivery of that support. Logic: Intensive support should be sufficient to change the odds of achieving literacy goals. If students are improving, but not enough to change the odds, they are not making adequate progress. Move from intensive to some risk or emerging Move from intensive to low risk or established Absolute Standard: 8% of intensive students achieve the some risk or low risk levels on the next goal. Relative Standard: Upper third of effectiveness of intensive support compared to other schools. Meet either the absolute standard or the relative standard and the effectiveness of the intensive support is a strength for the school. Rev. 6/7/4 (c) 4 55 Evaluating Effectiveness Worksheet First, clarify the primary goal for the first half of first grade. Core Component: Phonics or Alphabetic Principle DIBELS Indicator: Nonsense Word Fluency (NWF) Goal Skill Level: 5 letter sounds correct per minute with recoding Timeline: by the middle of first grade. Rev. 6/7/4 (c) 4 56
Next, examine the initial skills of students in the school using Distribution Report Beginning 1 st Distribution Report Benchmark: 36 students or 61% of students Odds in favor of achieving goal with effective core curriculum and instruction Strategic: 1 students or % of students Odds are 5 5 of achieving the goal unless we provide effective intervention. Intensive: 11 students or 19% of students Odds are against achieving the goal unless we provide effective intervention. Rev. 6/7/4 (c) 4 57 Rev. 6/7/4 (c) 4 58 Examine the outcomes Outcomes Criterion: 95% Established on DIBELS NWF then Core curriculum and instruction is effective System of additional interventions is effective Established: 4% of students Emerging: 43% of students Deficit: 15% Examine Outcomes 15% Deficit Middle 1 st Histogram Report 4% Established 43% Emerging Rev. 6/7/4 (c) 4 59 Rev. 6/7/4 (c) 4 6
Examine progress of Benchmark Students Are benchmark students reaching goal? Schoolwide Summary of Effectiveness Beginning First to Middle First Phonics/Alphabetic Principle Instruction Effective core curriculum and instruction should support benchmark students to achieve essential early literacy goals. Use Effectiveness Report Focus on schoolwide summary Classroom report illustrates individual children 67% of Benchmark students are reaching the middle of first grade goal. Rev. 6/7/4 (c) 4 61 Rev. 6/7/4 (c) 4 6 Compare to Decision Rules and Other Schools to evaluate effectiveness Effective core curriculum and instruction supports 95% of benchmark students to achieve the goal. Note met. Compared to other schools, the school is in the Upper Third - Strength Middle Third - Support Lower Third Substantial Support Rev. 6/7/4 (c) 4 63 Rev. 6/7/4 (c) 4 64
Beginning First to Middle First Middle of first grade outcomes for students with benchmark, strategic, and intensive instructional recommendations in the beginning of first grade Number of Schools 3 5 15 1 5 Intensive Median School Intensive Strategic Benchmark - 1 11-1 - 3 31-4 41-5 51-6 61-7 71-8 81-9 91-1 Conditional Percent Reaching NWF Goal A typical (middle) school had 68% of children with a beginning first grade benchmark recommendation achieve the middle of first grade goal, and % of children with intensive support recommendation. Rev. 6/7/4 (c) 4 65 Step by Step, Core and Intervention Step Step 1: Phonemic Awareness Step : Phonemic Awareness and Phonics Step 3: Phonics and Fluency Step 4: Fluency and Comprehension Step 5a: Fluency and Comprehension Step 5b: Fluency and Comprehension Step 6a: Fluency and Comprehension Step 6b: Fluency and Comprehension Effectiveness of Core Support Effectiveness of Strategic Support Effectiveness of Intensive Support Strength Effectiveness is at goal or upper third of other schools Support Effectiveness is less than goal, middle third of other schools. Substantial Support Effectiveness is lower third of other schools. Rev. 6/7/4 (c) 4 66 Themes Don t loose track of the bottom line. Are we getting closer to important and meaningful outcomes? Monitor Progress on -- and teach -- what is important: Phonemic Awareness, Alphabetic Principle, Accuracy and Fluency with Connected Text Oral Reading Fluency is an important instructional goal and target of progress monitoring. Use progress monitoring to make decisions that change outcomes for children. Progress monitoring should be efficient and purposeful. Start early! Trajectories of reading progress are very difficult to change. Rev. 6/7/4 (c) 4 67
Evaluating Effectiveness Reports Beginning First Grade Evaluating Effectiveness Report Worksheet 1. Goal: What is the primary instructional goal for the first half of first grade? Core Component or Big Idea: DIBELS Measure: Goal Skills Level: Goal Timeline to Achieve:. Initial Skills: At the beginning of first grade, how many students are Benchmark: Strategic: Intensive: 3. Adequate Progress: Of the students who were Benchmark at the beginning of the grade, what percentage achieved the primary instructional goal for the first half of the grade? 4. Compared to other schools, how effective is the core curriculum and instruction? Lower Third Middle Third Upper Third 5. How would you rate the effectiveness of the core curriculum and instruction? Strength 95% - 1% of benchmark achieve goal or in upper third Support Less than 95% and in middle third of effectiveness Substantial Support Less than 95% and in lower third of effectiveness 6. If effectiveness of core curriculum and instruction is Support or Substantial Support, what does the early literacy team plan to improve the effectiveness of core curriculum and instruction or to supplement the core curriculum and instruction?
Evaluating Effectiveness Reports Page 8 Table 3 School-Based Normative Context for Evaluating the Effectiveness of Core Curriculum and Intensive or Strategic Intervention in the First Half of First Grade in Achieving the Middle of First Grade DIBELS NWF Goal Percent of Students in Instructional Recommendation Category who School- Achieve Mid First NWF Goal Based, Percentile Intensive Strategic Benchmark 5 4 1 45 15 7 5 1 5 5 13 55 3 15 58 35 17 6 4 63 45 3 64 5 5 68 55 4 7 7 6 6 3 7 65 8 31 74 Effectiveness of Core Curriculum and Intensive or Strategic Intervention Lower Third Middle Third 7 11 33 76 75 14 4 78 8 17 43 8 85 47 83 Upper Third 9 7 5 86 95 36 6 9 99 67 81 94 Note. Based on 38 schools with at least 4 participating first grade students in 1.
District: Test District School: Adams Date: September, 1- Class: Adams 1st #1 Dynamic Indicators of Basic Early Literacy Skills Summary of Effectiveness of Core Program Effectiveness of Core Curriculum Effectiveness of Strategic Support Program Effectiveness of Intensive Support Program Students at Benchmark at Beginning of Year Beginning NWF Score Middle NWF Score Check If Reached Middle NWF Benchmark of 5 Students at Strategic at Beginning of Year Beginning NWF Score Middle NWF Score Check If Reached Middle NWF Benchmark of 5 Students at Intensive at Beginning of Year Beginning NWF Score B, Kelly 49 15 B, Erin 8 45 B, KAESY 1 B, PAGE 33 57 C, Shandra 15 4 C, Tasian 6 4 B, Savannah 6 41 H, RAENA 19 66 H, Cassandra 8 34 C, TAYLOR 98 87 J, KYLE 67 H, Cole 57 135 P, Mckenzie 4 67 J, Austin 5 36 K, Trevor 7 45 M, Ivan 4 5 M, Kiersten 6 91 W, Sarah 68 136 Middle NWF Score Check If Reached Middle NWF Benchmark of 5 Count: Percent: 7 / 1 Count: 3 / 5 Count: / 3 7% Percent: 6% Percent: % Summary of Effectiveness Report, 9/16/3, 1
Using DIBELS Data to Guide Decision-making: Implementation of Reading First Ohio in Cincinnati Public Schools Amy Murdoch, Ph.D. Reading First District Coordinator Cincinnati Public Schools Overview of Presentation Components of Reading First Ohio Overarching CPS Model: Three-Tiered Model of Instructional Support Insuring reliability of data collection Monitoring Implementation Family Events and Programs First Year Results Challenges, Successes, and Collaborations Components of Reading First Ohio First Year of a 3 year Grant On-going Professional Development and Technical Assessment Meetings Implementation of Research-Based Core Curriculum and Interventions Required Assessments Screening: TPRI and DIBELS Diagnostic: TPRI Progress Monitoring 3 times a year: DIBELS Outcomes: Terra Nova RF Personnel Building Trio Support Personnel Reading First In Cincinnati Public Schools The Schools: 11 Schools High poverty: 7%-1% Low performance Neighborhood Schools Largely African-American Schools chose to be part of RF Key Components of CPS RF Three Tiered Model Family Involvement and Education Inclusionary Practices Community Involvement and Collaborations
Model Used In CPS RF Project: Three Tiered Model of Support Three Tiered Model of Intervention Support Special Education Tier I: effective school-wide and preventive approaches for academics and behavior Tier II: targeted interventions for students with demonstrated need for intervention Tier III: individualized, more intensive interventions, including eligibility consideration as indicated by data Data based decision making at all levels! Intensive Individualized Intervention Efforts: Collab. Prob Solving TPRI Inventory Targeted Interventions for Struggling Readers: Voyager s In School and Extended Day Interventions DIBELS Weekly Progress Monitoring Effective School and Class-wide Reading Instruction Rooted in Scientifically Based Reading Research: Voyager Universal Literacy System DIBELS Benchmarking, K- TPRI Screener, Terra Nova Intensity of Resources Based on Student Need Tier 1: Core Curriculum Tier 1: Assessments Voyager Universal Literacy System Voyagers Theme Based Kindergarten = Tree House First Grade = Sea Castles Second Grade = Hiding Places Third Grade = Wild Rivers Cooperative Learning and Systematic Explicit Teacher Instruction Small Group Instruction = Differentiated Instruction Reading First Assessments Screening: All children given TPRI and DIBELS (K-) or DIBELS (3 rd ) at the beginning of the year Progress Monitoring: All children receive DIBELS Benchmark assessment 3 times Outcomes: All children receive Terra Nova (1-3 grades) at the end of the year
Tier : Intervention Support and Weekly Progress Monitoring Targeted Interventions for students with demonstrated need for intervention. Demonstrated need is determined through screening assessment and benchmarking data and then validated through baseline assessment. Individual weekly progress monitoring graphs are set up for each child who receives intervention support. Example of Voyager DIBELS Class List Used to Determine Who Needs Intervention Kindergarten Example Letter Naming Fluency Initial Sound Fluency Student Score Status Score Status Overall Status Jane Smith Struggling 3 Struggling Struggling Brian Green 44 On Track 1 Emerging Emerging Jamie Jones 45 On Track 44 On Track On Track Ronald Blue Struggling 9 Struggling Struggling Jose Purple 49 On Track 1 Emerging Emerging Katie Jones 15 Emerging 44 On Track On Track Ryan Red Struggling 19 Emerging Emerging Student Graph Kindergarten Student s Initial Sound Fluency Weekly Progress Monitoring Graph Tier : Intervention For Struggling Readers Number correct per minute 5 45 4 35 3 5 15 1 5 Before Intervention "James": DIBELS Initial Sound Fluency Scores Goal = 5 by /7/4 Peers "James" -Sep 5-Sep 1-Sep 13-Sep 9/1/8 3-Sep 6-Sep 1-Oct 4-Oct 9-Oct 14-Oct 17-Oct 3-Oct 9-Oct 1-Nov 4-Nov 7-Nov 13-Nov 16-Nov 1-Nov 5-Nov -Dec 5-Dec 1-Dec 13-Dec 18-Dec Dec. Break 7-Jan 13-Jan 16-Jan -Jan 3-Jan 8-Jan 31-Jan 5-Feb 1-Feb 13-Feb 19-Feb 4-Feb 7-Feb 4-Mar 7-Mar 1-Mar 17-Mar -Mar 5-Mar 8-Mar 3-Apr 8-Apr 11-Apr spring break -Apr 5-Apr 3-Apr 5-May 8-May 13-May 16-May 1-May 7-May 3-May Research tells us that children who are struggling with reading need more opportunity to practice the skills taught in the classroom AND that additional services should compliment/support classroom instruction Coordination of services is KEY! Date
Tier : Intervention For Struggling Readers The interventions we are using are directly related to the reading instruction children are receiving in the classroom extra dose of key instructional activities. Voyager s In-School Intervention: Done daily for 15-3 minutes each day by classroom teacher (3 rd grade), trained IAs, support staff, and/or trained volunteers. Voyager s After-School tutoring (1 st -3 rd grade): Done 4 days a week for 1 hour each day. Student Level Kindergarten Student s Initial Sound Fluency Weekly Progress Monitoring Graph "James": DIBELS Initial Sound Fluency Scores 5 45 4 35 3 5 15 1 5 Before Intervention Tier Intervention Goal = 5 by /7/4 Peers "James" Tier : Organizing Intervention Efforts Careful Scheduling of Interventions Intervention Binder with Logs and Materials Careful training and monitoring of volunteers and instructional assistants Progress Monitoring Binder in Each Classroom -Sep 5-Sep 1-Sep 13-Sep 9/1/8 3-Sep 6-Sep 1-Oct 4-Oct 9-Oct 14-Oct 17-Oct 3-Oct 9-Oct 1-Nov 4-Nov 7-Nov 13-Nov 16-Nov 1-Nov 5-Nov -Dec 5-Dec 1-Dec 13-Dec 18-Dec Dec. Break 7-Jan 13-Jan 16-Jan -Jan 3-Jan 8-Jan 31-Jan 5-Feb 1-Feb 13-Feb 19-Feb 4-Feb 7-Feb 4-Mar 7-Mar 1-Mar 17-Mar -Mar 5-Mar 8-Mar 3-Apr 8-Apr 11-Apr spring break -Apr 5-Apr 3-Apr 5-May 8-May 13-May 16-May 1-May 7-May 3-May Date Jane s Reading Graph "Jane": 3rd Grade DIBELS Oral Reading Fluency Scores 14 1 1 8 6 4 Before Intervention Tier Interventions Goal = 11 by Spring of 3rd Grade Peers Jane Number correct per minute -Sep 5-Sep 1-Sep 13-Sep 9/18 3-Sep 6-Sep 1-Oct 4-Oct 9-Oct 14-Oct 17-Oct 3-Oct 9-Oct 1-Nov 4-Nov 7-Nov 13-Nov 16-Nov 1-Nov 5-Nov -Dec 5-Dec 1-Dec 13-Dec 18-Dec Dec. Break 7-Jan 13-Jan 16-Jan -Jan 3-Jan 8-Jan 31-Jan 5-Feb 1-Feb 13-Feb 19-Feb 4-Feb 7-Feb 4-Mar 7-Mar 1-Mar 17-Mar -Mar 5-Mar 8-Mar 3-Apr 8-Apr 11-Apr spring break -Apr 5-Apr 3-Apr 5-May 8-May 13-May 16-May 1-May 7-May 3-May Number of words correct per minute Date
Tier 3: Collaborative Problem Solving This is implemented when children continue to struggle Examine progress monitoring graph Decision rules for the data: enough data, trend of data, 3 consecutive points below the aim line. Examine Implementation Was the Tier interventions done as planned: high integrity, child attendance Purpose: To plan a more intensive intervention plan for an individual child Tier 3: Collaborative Problem Solving Researched based 4-step process teams work through to design strong, appropriate, high quality, research-based interventions. Cyclical process that allows us to keep working until we find solutions for our children Meets criteria of new federal and state laws and best practice recommendations Process used in general and special education Involves the expertise of all those involved with a child or group of children Applications for individual child and system concerns Tier 3: The Collaborative Problem-Solving Process Problem Identification What is the Problem? Problem Analysis Why is it Happening? Plan Implementation What can be done to resolve the current problem situation? Plan Evaluation Is the intervention plan working or does it need to be changed? Jane s Reading Graph Number of words correct per minute 14 1 1 8 6 4 Tier 1: Before Intervention "Jane": 3rd Grade DIBELS Oral Reading Fluency Scores Tier : In-School and After-School Interventions Tier 3 Goal = 11 by Spring of 3rd Grade -Sep 5-Sep 1-Sep 13-Sep 9/18 3-Sep 6-Sep 1-Oct 4-Oct 9-Oct 14-Oct 17-Oct 3-Oct 9-Oct 1-Nov 4-Nov 7-Nov 13-Nov 16-Nov 1-Nov 5-Nov -Dec 5-Dec 1-Dec 13-Dec 18-Dec Dec. Break 7-Jan 13-Jan 16-Jan -Jan 3-Jan 8-Jan 31-Jan 5-Feb 1-Feb 13-Feb 19-Feb 4-Feb 7-Feb 4-Mar 7-Mar 1-Mar 17-Mar -Mar 5-Mar 8-Mar 3-Apr 8-Apr 11-Apr spring break -Apr 5-Apr 3-Apr 5-May 8-May 13-May 16-May 1-May 7-May 3-May Peers Jane Date
Three Tiered Model of Intervention Support Special Education District DIBELS Results After Year 1 Data based decision making at all levels! Intensive Individualized Intervention Efforts: Collab. Prob Solving TPRI Inventory Targeted Interventions for Struggling Readers: Voyager s In School and Extended Day Interventions DIBELS Weekly Progress Monitoring Effective School and Class-wide Reading Instruction Rooted in Scientifically Based Reading Research: Voyager Universal Literacy System DIBELS Benchmarking, K- TPRI Screener, Terra Nova Intensity of Resources Based on Student Need Benchmark 1: 9/19/3 Benchmark : 11/1/3 Benchmark 3: /7/4 Benchmark 4: 4/3/4 District Level: KG Phoneme Segmentation Fluency 1 9 Phonem Segmentation Fluency 8 7 6 5 4 3 Goal = 35 by Spring 1 N = 465 456 1 3 4 Benchmark
District Level: Kindergarten Initial Sound Fluency District Level: Kindergarten Letter Naming Fluency 1 7 9 6 8 Initial Sound Fluency 5 4 3 Goal = 5 Letter Naming Fluency 7 6 5 4 3 1 1 N = 46 434 47 4 N = 47 1 435 47 3 456 4 1 3 4 Benchmark Benchmark Cincinnati Public Schools Nonsense Word Fluency First Grade Full Year Students District Level: First Grade Nonsense Word Fluency 1 11 1 Nonsense Word Fluency 9 8 7 6 5 4 3 Goal = 5 by Winter 1 N = 371 377 39 38 1 3 4 Benchmark
District Level: First Grade Reading Connected Text 1 9 8 Oral Reading Fluency 7 6 5 4 3 Goal = 4 by Spring 1 N = 37 387 38 1 3 4 Benchmark District Level: First Grade Phoneme Segmentation Fluency District Level: KG Phoneme Segmentation Fluency 1 1 Phoneme Segmentation Fluency 9 8 7 6 5 4 3 1 N = 37 1 379 39 3 Benchmark 38 4 Goal = 35 by Spring of KG Phonem Segmentation Fluency 9 8 7 6 5 4 3 1 N = Good News for Next Year s 1 st Grade Teachers 1 Benchmark 465 3 456 4 Goal = 35 by Spring
District Level: Second Grade Reading Connected Text Oral Reading Fluency 18 17 16 15 14 13 1 11 1 9 8 7 6 5 4 3 1 Goal = 9 By Spring N = 386 5 th percentile B1 = 1 B4 = 43 1 4 48 3 Benchmark 41 4 District Level: Third Grade Reading Connected Text Oral Reading Fluency 18 17 16 15 14 13 1 11 1 9 8 7 6 5 4 3 1 Goal = 11 By Spring N = 373 383 397 39 1 3 4 Benchmark
Range in School Performance School and Classroom Examples What was happening at schools that made the greatest growth? High level of implementation Full implementation of interventions all who needed it got it Community Involvement Progress Monitoring done weekly Flexible service delivery: Collaboration across general and special education School Level Example Kindergarten Baseline 1: 4% Struggling 8% On Track Average ISF score = 3 Average LNF score = 1 Baseline 4: 11% Struggling 77% On Track 54 Children Classroom Level: Kindergarten Class 1% Literacy! Classroom Level Example: 1 st Grade Reading Connected Text 15 13 11 B1: 41% Struggling % On Track 9 Average ISF score = 6 Average LNF score = 5 B4: % Struggling 1% On Track Oral Reading Fluency 7 5 3 1-1 N = 14 14 14 Goal = 4 by Spring 3 4 17 students BENCHMRK
School Level Example: nd Grade Reading Connected Text School Level Example: 3 rd Grade Reading Connected Text Oral Reading Fluency 18 16 14 1 1 8 6 4 N = 13 14 15 15 Goal = 9 by Spring Oral Reading Fluency 19 18 17 16 15 14 13 1 11 1 9 8 7 6 5 4 3 1 Goal = 11 By Spring BENCHMRK 1 5 th percentile B1 = 6 3 4 5th Percentile B1 = N = BENCHMRK 14 1 16 5 th percentile B1 = 4 16 16 3 4 5th Percentile B1 = 31 B4 = 94 B4 = 6 B4 = 93 B4 = 65 Increasing the Reliability of Our Data DIBELS Reliability Study: PSF DIBELS Certificate prior to being allowed to collect data Use U. of Oregon Integrity check and inter-rater reliability score Review prior to each benchmark Spot check at each benchmark Reliability Study at Benchmark 4 Randomly chose 3 children from each classroom and re-assessed them R eliab ilities 1 9 8 7 6 5 4 3 1 Phoneme Segmentation 99.75 99.5 98.63 97.41 97.31 96.38 95.5 9.9 9.45 87.5 Schools 17.46 n=11 schools Range:.99 to.17
DIBELS Reliability Study: 1 st Reading Connected Text DIBELS Reliability Study: nd Reading Connected Text First Grade Oral Reading Fluency Second Grade Oral Reading Fluency Reliabilities 1 9 8 7 6 5 4 3 1 1 99.7 99.7 99.4 98.74 98.59 98.45 98.36 93.7 9.5 89.1 Schools n=11 schools Range: 1 to.89 Reliabilities 1 9 8 7 6 5 4 3 1 99.96 99 98.6 98.59 97.71 95.7 94.84 93.11 91.4 9.78 85.91 N= Schools n=11 schools Range:.99 to.85 DIBELS Reliability Study: 3 rd Reading Connected Text Monitoring Implementation Keeping Us All On Track Third Grade Oral Reading Fluency Classroom Fidelity Index Reliabilities 1 9 8 7 6 5 4 3 1 99.91 99.68 99.66 99.6 99.7 98.76 97.68 96.73 95.14 94.89 79.61 Intervention Integrity Implementation of the grant activities: Innovation Configuration Schools n=11 schools Range:.99 to.79
Family Involvement Home Connection Family Events Tier 3: Problem Solving Our Home Puts Reading First Challenges, Successes, and Collaboration Challenges Late start Major shift in some schools High mobility Successes All schools are providing interventions to struggling children Family Involvement! Move to Inclusive practices Big Change for some Schools are Excited for Next Year, Speak Very Passionately About RF. We have schools who want to join! Collaboration Within the district University of Cincinnati OhioReads CincinnatiReads Southwest SERRC