School Psychology Review, 2007, Volume 36, No. 1, pp. 159-166 RESEARCH BRIEF Reading at the Frustration, Instructional, and Independent Levels: The Effects on Students' Reading Comprehension and Time on Task Megan A. Treptow, Matthew K. Bums, and Jennifer J. McComas University of Minnesota Abstract. The current study replicated Gickling and Armstrong (1978) by using curriculum-based assessment for instructional design to match reading materials to skill levels of three third-graders who were struggling readers with low levels of on-task behavior. Difficulty levels of the reading passages were grouped according to ratios of known and unknown words into three categories: frustration (80-90% known), instructional (93-97% known), and independent levels (100% known). A single-subject, multielement design was used to examine the effects of these reading activities on individual students' time on task and reading comprehension. Although within-subject variability in responding was observed, results suggested that students demonstrated improved time on task while completing tasks at the instructional level, compared to tasks at the frustration and independent levels. Comprehension was highest at the independent level and lowest at the frustration level. These findings were consistent with but not as robust as those reported by Gickling and Armstrong. The most frequent concerns for which students are referred for special education evaluations include reading difficulties, low task completion rates, and classroom conduct problems (Bramlett, Murphy, Johnson, Wallingsford, & Hall, 2002), three highly interrelated and often co-occurring difficulties (Maughan, Pickles, Hagell, Rutter, & Yule, 1996). In fact, addressing academic difficulties has been shown to improve behavioral outcomes (McComas, Hoch, Paone, & El-Roy, 2000; McCurdy, Skinner, Grantham, Watson, & Hindman, 2001). Betts (1946) hypothesized that when students are presented with tasks that are sufficiently familiar, yet still provide some degree of challenge, optimal learning occurs. Optimally challenging tasks were referred to as tasks at the student's instructional level. Tasks that provided too little challenge or too much The current study was funded by a grant from the Reading Research Center at the University of Minnesota. Correspondence regarding this article should be addressed to Matthew K. Bums, 346 Elliott Hall, 75 East River Road, Minneapolis, MN 55455; E-mail: bums258@umn.edu Copyright 2007 by the National Association of School Psychologists, ISSN 0279-6015 159
School Psychology Review, 2007, Volume 36, No. 1 challenge were referred to as being at the independent or frustration levels, respectively. Gickling and Armstrong (1978) further operationalized instructional level reading material to contain 93-97% known words, frustration level to contain fewer than 93% known words, and independent level to contain greater than 97% known words. They then tested these definitions, by examining students' comprehension of the tasks, percentage of time spent on task, and task completion rates. Students demonstrated high comprehension at both the independent and instructional levels, and low levels of comprehension at the frustration level. However, the most striking results were on students' time on task, with rates much higher during instructional level tasks (84-94%) than during either frustration (approximately 45%) or independent (approximately 53%) level tasks. Curriculum-based assessment for instructional design (CBA-ID), initially proposed by Gickling and Havertape (1981), is an assessment method that focuses on "the instructional needs of a student based upon the on-going performance within the existing course content in order to deliver instruction as effectively and efficiently as possible" (Gickling, Shane, & Croskery, 1989, pp. 344-345). The effective and efficient instruction is obtained through an assessment of individual students' instructional levels by computing the ratio of known to unknown or hesitant words in samples of text and comparing it to the criterion of 93-97% known words. CBA-ID has demonstrated adequate psychometric properties (Bums, 2004), and modifying instruction based upon CBA data has been shown to result in improved student outcomes (Bums, in press; Gickling et al., 1989; Shapiro, 1992). Despite CBA-ID's demonstrated usefulness, it has not been widely implemented (Shapiro, Angello, & Eckert, 2004), perhaps because the research base is relatively limited and derived from a single study that is almost 30 years old (Bums, 2004), which is of special concern given that the study uses research methodology not consistent with current approaches (Kratochwill, 1992). For ex- 160 ample, Gickling and Armstrong (1978) used a single-subject design with four first- and four second-grade students, but they presented their results averaged across the students, by session, as might be done in a large-group design. Presenting the results in this way did not allow the variability within individual students at different sessions and between students within conditions to be evident, nor did it allow for detailed examination of the effects of curricular match for individual students. Moreover, the order of presentation of conditions in their study was not randomized or counterbalanced, which could have created a contagion effect on students' off-task behavior. The purpose of the present study was to use current single-subject methodology to replicate Gickling and Armstrong's (1978) original study, examining the effects of matching reading materials using CBA-ID on the time on task and comprehension of individual students identified as off-task, struggling readers. In a multielement design with counterbalanced order of conditions, each student received reading materials matched according to Gickling and Armstrong's (1978) definitions of the instructional, frustration, and independent levels. The specific hypotheses were as follows: (a) Students typically exhibiting low rates of on-task classroom behavior and reading difficulties will exhibit the highest rates of on-task behavior while working in reading material containing 93-97% known words (instructional level), as compared to passages containing greater than 97% known (independent level) and fewer than 93% known (frustration level) words. (b) These students will exhibit the highest reading comprehension when reading passages containing 93-97% known words (instructional level) or greater than 97% known words (independent level), as compared to passages with less than 93% known words (frustration level). Method Selection of Participants Three students were selected for participation from a third-grade classroom of 23 students based on screenings for both low
The Effects of Instructional Match on-task behavior during reading instruction and reading difficulties. Momentary time sampling procedures were used with 10-s intervals to record the time on ask of each student in the classroom on 2 different days during classroom curricular reading activities, including independent reading, teacher-guided reading, partner reading, and completing reading worksheets. Each observation was approximately 10 min in length, for a total of two 10-min observations of each student. The mean percentage of time on task for all students in the classroom during reading instruction observations was 79.52%, with a standard deviation of 16.13% and a range from 30% to 97%. The results of the observations were then combined with reading fluency assessments collected by the school to select those students who exhibited both lower levels of on-task behavior during reading instruction and lower reading fluency scores than the averages for the class. The six students with the lowest time on task and reading fluency were then screened further using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS; Good & Kaminski, 2002) measure of oral reading fluency and one additional 10-min observation. The three students whose combined levels of time on task and reading fluency were the lowest, and from whom consent was obtained, were selected for participation in the study. Two children who also exhibited low levels of on-task behavior declined to participate in the study. The first participant, assigned the pseudonym of Benjamin, was a Caucasian male diagnosed with a learning disability in reading. Benjamin's mean percentage of time on task during the three screening observations was 68.3% (SD = 26.5%). Screening with the DIBELS measure of oral reading fluency placed him in the at-risk range according to the third-grade end of the year benchmark (Good & Kaminski, 2002), with a score of 17 words per minute on third-grade reading probes. The second student, called Jessica for this article, was an African American student identified by the classroom teacher as experiencing reading difficulties, but was not diagnosed with a disability. Jessica's mean percentage of time on task during screening observations was 78.1% (SD = 10.7%). Screening with the DIBELS measure of oral reading fluency placed her in the at-risk range according to the third-grade end-of-the-year benchmark, with a score of 70 words per minute. Although Jessica's level of on-task behavior was not discrepant from the class average, it was below the mean and she demonstrated the fifth lowest rate of on-task behavior in the classroom. The third student, assigned the pseudonym of Jeremy, was a Caucasian male identified as a struggling reader by his classroom teacher who also was not diagnosed with a disability. Jeremy's mean percentage of time on task during screening observations was 55.0% (SD = 19.7%). Screening with the DIBELS measure of oral reading fluency placed him in the at-risk range according to the third-grade end-of-the-year benchmark, with a score of 75 words per minute. Materials Reading passages used in the study were selected from the reading series, Read Naturally (RN; Read Naturally, 2003), to control the difficulty level of the passages. RN is a reading program that includes a collection of 24 short reading passages at 13 sequenced levels of reading difficulty within Grades 1-8. The passages ranged in length across grades from 50 (Grade 1) to 200 (Grade 8) words, and each individual passage was followed by five comprehension questions, four of which were multiple choice and one open-ended. The questions appeared on the same page as the reading passage. Independent Variable Reading material difficulty served as the independent variable and had three levels: frustration, instructional, and independent. Frustration reading level was defined as 80-90% known words, instructional level as 93-97% known words, and independent level as 100% known words (Gickling & Armstrong, 1978). A CBA-ID was conducted individually with each of the three participants before com- 161
School Psychology Review, 2007, Volume 36, No. 1 mencing the manipulations. Each participant was asked to read aloud for 1 min from two randomly selected third-grade passages, and two at levels above and/or below his or her grade level. The examiner recorded the words as known if the participant read them correctly within 2 s, and unknown if they were read incorrectly, omitted, or if the student hesitated for more than 2 s on the word (Gickling & Havertape, 1981). The percentage of known words for each passage was calculated, and easier and/or more difficult passages were administered to identify the passages within the series that represented the frustration, instructional, and independent levels for each student. Although RN is a reading series with passages written at various grade levels, the grade level of the passage was only used to approximate reading difficulty. The specific reading difficulty of the passages was computed using the percentage of known words for each individual student. As traditional estimates of readability have not been found to predict measures of reading fluency (Ardoin, Suldo, Witt, Aldrich, & McDonald, 2005), the grade levels used by RN were not confirmed with a readability formula. Design and Procedures A multielement single-subject design was used in which reading materials at the three difficulty levels were presented three times each in a counterbalanced order, for a total of nine sessions. These sessions took place in a whole-class setting with 23 total students, all of whom were asked to read silently two randomly selected passages from the RN series and answer comprehension questions in writing over a series of nine data collection sessions. Thus, 18 total passages were read (two each time) by each student. For the three study participants, the reading difficulty of the passages was manipulated as described, and passages already read by a participant were not repeated, but other passages at the appropriately matched levels were randomly selected from within the RN series. No assistance was provided while the 162 students worked, except for clarification of instructions. Dependent Variables Time on task. Each participant's time on task was observed and recorded using momentary time sampling procedures with 10-s intervals for the duration of the time taken to complete the task. Observations of each participant ended when he or she handed in the pages, which signaled that the task was completed. The length of time taken to complete the task was not limited, and ranged from 5 to 15 min. On-task behavior was defined as actively attending to the assigned instructional material. Examples included looking at the reading material, writing, listening to instructions from the teacher, and raising a hand for assistance from the teacher (Shapiro, 2004). Off-task behaviors included talking about anything other than the assigned reading, leaving the seat for nonrelevant reasons, aimless movement of the reading passages (e.g., flipping back and forth), gazing away from the reading passages, reading something other than the assigned passages, and focusing attention to the activities of others (Shapiro, 2004). Comprehension. Comprehension questions were scored according to the RN teachers' manual. The total comprehension score for each session was recorded as the total correct out of 10 for the two passages administered. Fidelity To check that the stories administered were indeed at each students' frustration, instructional, and independent levels, each student was asked to read aloud to the examiner one randomly selected passage at each level after completing the reading task and questions. The examiner recorded the percentage of known words obtained. The results of the fidelity checks found that the percentages of known words obtained were within the expected ratios for 100% of the passages selected for Jeremy and Jessica.
The Effects of Instructional Match Results of Benjamin's fidelity checks indicated that he struggled with reading even those passages determined to be at his instructional and independent levels. Although initial CBA-ID data indicated that Level 1 stories within the RN series constituted the instructional level for Benjamin, during fidelity checks he read the Level 1 passage with only 92.6% known words. Therefore, stories at the.8 level, the lowest level within the RN series, were used to reflect the instructional level for Benjamin, and included 94% known words for Benjamin during the manipulation check. A different reading task was taken from Gickling and Armstrong (1978) to represent the independent level for Benjamin. He was given two separate worksheets with the 26 letters typed on them, was given five simple pictures (e.g., apple, dog, and so on) and asked to circle the letter that corresponded to the first letter of the word that named each picture (e.g., "a" for apple). Comprehension for this task was measured by computing the percentage of items correctly completed. Subsequent fidelity checks found that this independent task fell within the defined range (100% known). Observation Integrity All observations were conducted by two graduate students in a school psychology training program with advanced training in direct observation techniques. The first observer was the graduate student who primarily collected data for the study and received an additional 3 hr of training in CBA-ID and observational data collection. The second observer was blind to the task difficulty levels presented in each session. Interobserver agreements between two independent observers were collected for 33% of the observations at each phase and were calculated by dividing the number of intervals scored consistently by both observers by the total number of recording intervals. The interobserver agreements for on-task behavior in each of the difficulty levels ranged from 96% to 100%, with a mean of 99%. Interrater agreement was used to assess scoring accuracy of known words and comprehension questions by computing the number of words or questions scored consistently by both raters divided by the total number of words or questions. An interrater agreement of 100% was obtained for both variables. Results Percentages of time on task and task comprehension for each session are displayed in Figures 1 and 2 for each student. All three students demonstrated the highest overall ontask behavior while reading materials at the instructional level, with some overlap observed within the data. For Jessica, time ontask data at the instructional level overlapped with data at the independent level. In Jeremy's data, instructional level data overlapped some with data in both the independent and frustration conditions. For Benjamin, minimal overlap occurred between the instructional and frustration conditions, and there was no overlap between the instructional and independent conditions. For all three students, time on task appeared slightly lower at the independent level than at the frustration level, and time on task within the instructional level was higher than the independent level for Benjamin and Jeremy, with some overlap for Jessica. Comprehension scores were higher overall at the instructional and independent levels than at the frustration level for each of the three participants, with some overlap again observed within the data. Comprehension was the highest for each student at the independent leovel. Discussion Although within-subject variability in responding was observed, the results of this study suggested that students demonstrated improved time on task while completing tasks at the instructional level, compared to tasks at the frustration and independent levels. The findings in this study were consistent with but not as robust as those reported by Gickling and Armstrong (1978), in which 100% of the data points at the instructional level were nonoverlapping. 163
School Psychology Review, 2007, Volume 36, No. 1 100 95 90 85 80 75 F- 70 65 60 55. 100 95 90 85 80 75 E- r. 70 0 S 8o65 Independent Level T Frustration Level I Benjamin ] 1 2 3 4 5 6 7 8 9 Instructional Level F- 60 551 100 95 1 2 3 4 5 6 7 8 9 Instructional Level I 90 85 -~ 80 Frustration Level F- 75 o 70 65 60 55 I 1 2 3 4 5 6 7 8 9 Figure 1. Percentages of time on task at each session for each student. 164
The Effects of Instructional Match An interesting effect noted in the data was the lower variability in participants' time on task at the instructional level compared to the other levels. These data suggested that the students were more consistently on task when reading materials at the instructional level than at the independent or frustration levels. Unlike the Gickling and Armstrong study, the order of presentation of the conditions in the current study was randomized for each student, and thus all students did not receive the same condition at the same time. This was done in an attempt to eliminate the possibility of a contagion effect, in which students' off-task behavior was increased by the off-task behavior of others in the classroom, which may have also contributed to the large effects found by Gickling and Armstrong. ta 0 i 100 80 60 Instructional Le<t,. 40 20 0 80( 60 40' 20 0 10a Independent Level - Frustration Level 1 2 3 4 5. 6 7 8 9 1t 80 Instructional Level 60' 40] 20[ 0 L "- Independent Level 2 3 4 5. 6 7 Inde pendent LVem " I- lnstructional Level Frustration Level 8 9 Frustration Level 1 2 3 4 5o 6 7 8 9 Se5ssions Figure 2. Percentages of comprehension questions answered correctly in each session for each student. As teachers face classrooms of students with widely ranging skill levels, differentiating instruction to the individual needs of students is crucial for ensuring individual success. Data from the current study provide preliminary evidence that students' instructional levels be identified and targeted for reading activities in school. Although these data should inform future research rather than practice, they do suggest that matching curricular materials and student skill could improve short-term student outcomes (time on task and reading comprehension), which is also important given the close link between classroom behavior problems and academic difficulties (Maughan et al., 1996; McComas et al., 2000). These data suggest tentative implications for practice and research, but some limitations should be considered. One limitation is the relative minimal number of data points obtained for students. With only three data points at each phase, variability within the data makes it difficult to discern with confidence the true levels and trends in students' performance. Thus, more consistent results might be obtained by continuing the phases over more sessions. Moreover, unforeseen threats to internal validity occurred with these children, including variability of student responding and the need to modify the independent level stimuli for Benjamin. Moreover, the length of the passages were fairly consistent within the given grade level, but variability in passage length probably existed for each child, the effect of which was unknown. Finally, the effect of novelty of stimuli or of the university observers presenting the tasks is unknown. Thus, future researchers could take steps to reduce potential novelty effects. Future research might also examine the effects of instructional difficulty levels on group means using various groups of students such as students with disabilities, different age groups, and various skill levels. Moreover, future research could replicate the current study with measures such as rates of academic responding during whole-class instruction, and using different content areas-for instance, mathematics and spelling-and types of tasks. It is hoped this study will be the first 165
School Psychology Review, 2007, Volume 36, No. 1 of many that could lead to additional uses of CBA-ID data, reading interventions, and advanced understanding of a potentially important construct. References Ardoin, S. P., Suldo, S. M., Witt, J., Aldrich, S., & McDonald, E. (2005). Accuracy of readability estimates' predictions of CBM performance. School Psychology Quarterly, 20, 1-22. Betts, E. A. (1946). Foundations of reading instruction. New York: American Book. Bramlett, R. K., Murphy, J. J., Johnson, J., Wallingsford, L., & Hall, J. D. (2002). Contemporary practices in school psychology: A national survey of roles and referral problems. Psychology in the Schools, 39, 327-335. Bums, M. K. (2004). Using curriculum-based assessment in the consultative process: A review of three levels of research. Journal of Educational and Psychological Consultation, 15, 63-78. Bums, M. K. (in press). Reading at the instructional level with children identified as learning disabled: Potential implications for response-to-intervention. School Psychology Quarterly. Gickling, E. E., & Armstrong, D. L. (1978). Levels of instructional difficulty as related to on-task behavior, task completion, and comprehension. Journal of Learning Disabilities, 11, 559-566. Gickling, E. E., & Havertape, S. (1981). Curriculumbased assessment (CBA). Minneapolis, MN: School Psychology Inservice Training Network. Gickling, E. E., Shane, R. L., & Croskery, K. M. (1989). Developing math skills in low-achieving high school students through curriculum-based assessment. School Psychology Review, 18, 344-356. Good, R. H., & Kaminski, R. A. (Eds.). (2002). Dynamic indicators of basic early literacy skills (6th ed.). Eugene, OR: Institute for the Development of Educational Achievement. Available from http:hdibels.uoregon.edu/ Kratochwill, T. R. (1992). Single-case research design and analysis: An overview. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 1-13). Hillsdale, NJ: Lawrence Erlbaum Associates. Maughan, B., Pickles, A., Hagell, A., Rutter, M., & Yule, W. (1996). Reading problems and antisocial behavior: Developmental trends in comorbidity. Journal of Child Psychology and Psychiatry, 37, 405-418. McComas, J., Hoch, H., Paone, D., & El-Roy, D. (2000). Escape behavior during academic tasks: A preliminary analysis of idiosyncratic establishing operations. Journal of Applied Behavior Analysis, 33, 479-493. McCurdy, M., Skinner, C. H., Grantham. K., Watson. T. S., & Hindman. P. G. (2001). Increasing on-task behavior in an elementary student during mathematics seatwork by interspersing additional brief problems. School Psychology Review, 30, 23-32. Read Naturally. (2003). Read Naturally master's edition teacher's manual. St. Paul, MN: Author. Shapiro, E. S. (1992). Use of Gickling's model of curriculum-based assessment to improve reading in elementary age students. School Psychology Review, 21, 168-176. Shapiro, E. S. (2004). Academic skills problems: Direct assessment and intervention (3rd ed.). New York: Guilford Press. Shapiro, E. S., Angello, L. M., & Eckert, T. L. (2004). Has curriculum-based assessment become a staple of school psychology practice? An update and extension of knowledge, use, and attitudes from 1990 to 2000. School Psychology Review, 33, 249-257. Date Received: November 21, 2005 Date Accepted: October 26, 2006 Action Editor: Tanya Eckert E Megan A. Treptow, EdS, is a school psychologist with the St. Croix River Education District in Pine City, Minnesota. She completed her graduate training at the University of Minnesota in 2006. Current interests include reading at the instructional level, implementation of response to intervention for special education decision making, the effectiveness of emotional and behavioral support services and collaboration within school settings, and the implementation effectiveness of a school-wide problem-solving model. Matthew K. Bums, PhD, is an associate professor of educational psychology and coordinator of the school psychology program at the University of Minnesota. Current research interests include curriculum-based assessment for instructional design, the instructional level, response to intervention, and problem-solving teams. Jennifer McComas, PhD, is an associate professor in the Department of Educational Psychology at the University of Minnesota. Her current research interests include functional analysis and treatment for problem behavior and academic skill deficits, the influence of the principles of behavior on learning, and the influence of social context on severe problem behavior. 166
COPYRIGHT INFORMATION TITLE: Reading at the Frustration, Instructional, and Independent Levels: The Effects on Students Reading Comprehension and Time on Task SOURCE: Sch Psychol Rev 36 no1 Mr 2007 The magazine publisher is the copyright holder of this article and it is reproduced with permission. Further reproduction of this article in violation of the copyright is prohibited. To contact the publisher: http://www.nasponline.org/index2.html