32 Journal of Reading Behavior 1971-72 Vol. 4, No. 3, Summer AN EVALUATION OF FOUR METHODS OF READING INSTRUCTION AS A FUNCTION OF TIME OF TESTING 1 Philip K. Jensen, James W. Mills, Martin Hershkowitz* Abstract Which test offers the best estimate of the improvement of a student in reading training: a test administered immediately after a training session or a test administered at the beginning of the next training session? The problem of the best time to test students for a meaningful evaluation of gains in reading scores is closely involved in any comparison of different methods of teaching developmental reading. An analysis of 276 periods of instruction which used four different methods was run on data from a college population. The results show: (1) the timing of the evaluation test substantially affects the obtained results; and (2) the methods produce significantly different patterns across speed and comprehension. Each reading center should develop its own norms for differing methods if its reading program is to be adapted to the needs of the individual student. Today there is wide-spread interest in developmental reading programs for adults. The past few years have seen an increase in the number and popularity of commercialized reading improvement programs. Colleges and universities, which now open their doors to more than half of the high school graduates rather than to just a select few, find an increased need to offer reading instruction to their students. In the midst of this activity, it is appropriate to ask, "Which methods of teaching reading improvement are more effective than others?" While this is not a new question, it takes on added importance as the interest in reading training expands. A related question is, "What is the best way of determining a method's effectiveness?" The most common approach is to test the students in reading proficiency both before and after their exposure to training. However, little attention is given to the problem of the timing of the testing. Is it better to test the students immediately after the reading training period or should the investigator test his subjects immediately prior to the subsequent training session, thus 1 The research presented herein was conducted during the data collection phase of a study on the individualization of reading improvement instruction through the optimal combination of reading instruction methods. This study has been supported by the Vitro Corporation of America, the Drew University and currently under a grant from the ESSO Education Foundation. * Dr. Jensen is Chairman of the Psychology Department and Dr. Mills is Director of the Reading Clinic, Drew University, Madison, New Jersey. Dr. Hershkowitz is Project Manager, Education Research and Evaluation, Civil Projects Office, Vitro Laboratories, Silver Spring, Maryland.
taking into account the fact that some methods may "hold" better than others outside of the classroom? Does the timing of the testing make any difference? This paper presents the results of an investigation into both the question of the timing of the testing and the problem of the comparative effectiveness of selected methods in a reading improvement program (Mills, Jensen, and Hershkowitz, 1971). Procedure The study was conducted as part of the on-going reading development program at Drew University. During Orientation Week, all incoming freshmen took a reading test (Robinson and Hall, 1949) on the basis of which half of them were placed in the Reading Development Program. The Drew program is developmental rather than remedial since the students typically read at a high level of comprehension as well as at somewhat above average speed. Students were placed in the program in one of four classes which met weekly. Each student remained in the program until passing an exit test at a level of proficiency which excused him from the program. During each class period, students were exposed to one method for the entire period. Four methods were selected for use in this study. They were as follows: 1. Pacer: The pacer is a machine which forces the subject to read at a certain rate by passing some object over the page from top to bottom, thus progressively decreasing the portion of the page which he can see. In effect, the subject pushes himself to read faster in order to keep ahead of the object. 2. Exhortation: In utilizing this method, the instructor urges the subject to read faster, relying on this verbal exhortation to produce gains in speed. There are a variety of ways to do this; the subject can be instructed to read at a normal rate, then to push himself as fast as he can, then to read at a normal rate again, etc., with the "push" periods progressively increasing in time; or the instructor can have the subject read for five minutes, note how much was read, and then challenge him to read ten percent more in the next five minutes. 3. Question-Recall: The subject is instructed to ask himself a question which can be answered in the text, often by using the passage heading to phrase the question, and then reads further to find the answer. After completing the passage, he recalls the answer by reciting it to himself or the instructor. 4. Note-Taking: The subject prevents his mind from wandering (and thus improves retention) by taking notes on what he has read. The nature of these notes can vary from exercise to exercise. 33
34 The first two of these methods, pacer and exhortation, were designed primarily to increase reading speed, while the other two emphasized the development of comprehension skills. A brief test of speed and comprehension was administered both before (pretest) and after (posttest) each training session. All tests used in this part of the study were found in the booklet Selections for Improving Speed of (Perry and Whitlock, 1959). While these tests are of nearly equal difficulty, the distortion which might result if one test were more or less difficult than another was minimized by having half of the class take one test and half the other at the beginning of the period and then reversing this at the end of the period. The mean pretest score for the subjects was determined by averaging all test scores obtained at the beginning of every period by every subject. A similar procedure was used to obtain the average posttest score. The program was conducted for twelve weeks, with each group being exposed to each of the four methods on three separate occasions. The order of presentation of the four methods to each group was determined by the use of three randomized Latin squares. One instructor conducted all sessions of all groups. Students who, in the course of the twelve weeks, reached the level of proficiency demanded by the College were excused from the program. Attendance in the program was not mandatory. Thus the number of subjects in a given group who were present varied from period to period. The number of training sessions finally utilized in the data analysis was 276. Results An examination of Table 1 illustrates the different results which may be obtained when performance is measured at different times. The effectiveness of a given method evaluated by immediate posttest will often differ substantially from that obtained by a similar evaluation conducted at a later time. For example were the effectiveness of the note-taking method assessed by means of an immediate posttest it would appear that this method had an adverse effect on speed (Table 1, column 3). However a test administered to the same students at the beginning of their next instructional period reveals that this method actually results in a gain in reading speed (Table 1, column 5). Similar distortions are apparent in each of the other methods. These effects may be due to a variety of reasons. Students using methods designed to increase comprehension show a decline in speed during the period and an increase in speed from the end of that period to the beginning of the next. One possible explanation is that exclusive attention to comprehension disrupts reading speed to some extent, an effect which quickly dissipates with the passage of
Table 1 Reading Scores at the Beginning of Instruction, Immediately After Instruction, and at Beginning of Next Instruction 35 Beginning of Period End of Period Beginning of Next Period Method Speed Speed Speed Speed: (WPM) Pacer 1 332 64 338 58 361 57 Exhortation 1 323 61 333 62 342 62 Question-Recall 1 331 57 307 60 342 59 Note-Taking 1... 327 57 307 60 335 60 = 69 in all cases. time. Similarly students using methods emphasizing speed show some gain during the period followed by an additional gain between periods. Another explanation might be the existence of a simple fatigue effect. Test scores obtained at the end of an hour of intensive effort will be depressed by the fatigue, whether psychological or physiological, experienced by the subject. With the passage of time the subject recovers and his performance increases to an appropriate level. If this interpretation is accurate the gains between the end of the training period and the beginning of the next period should be directly proportional to both the effort demanded of the student and the effectiveness of the method in increasing speed. An analysis of Table 1 shows that the size of the recovery effect actually matches the size of the overall gain in speed; a more subjective evaluation of the pressures on the individual suggests that this is also the case for effort. In this presentation the discussion has been limited to speed scores since it is here that the effects are most dramatic. Since the pretest at the beginning of one period to the pretest at the beginning of the next period, analysis takes into account both fatigue and suppression effects. The more meaningful investigation of relative effectiveness for the four methods is based on this approach. As a preliminary step in the evaluation of the methods the four groups were checked for initial equality by running an analysis of variance on their pretests for both speed and comprehension. The respective F ratios of.194 and 1.934 fall comfortably below significance indicating that the groups did not differ in competence levels. The main analyses are shown in Table 2.
36 Table 2 Mean Changes Per Session in Measured Performance Resulting from the Four Methods of Instruction Method 1 Skills Pacer Exhortation Question-Recall NoteTaking Speed (WPM) 39.2 19.5 11.1 8.5 (%) - 7.2 1.2 1.5 2.3 1 Any two means underscored by the same line are not significantly different at the five percent level. At the five percent level the only clearly significant difference in speed gains is found between the 39.2 WPM per session resulting from pacer and the 8.5 WPM per session obtained by note-taking. However the difference between pacer and question-recall barely misses meeting this criterion, while that between pacer and exhortation is at about the ten percent level. Thus the methods differ widely in their effectiveness in producing changes in speed. On the other hand when one turns to the comprehension data a different picture emerges. Three of the methods produce highly similar results, small gains in percent of comprehension, while the fourth, pacer, produces a substantial loss. The difference of pacer from the other three methods is significant at the five percent level. The generally small gains in comprehension are probably due in part to the nature of the Drew population which enters the program with a history of solid academic, performance and initially well above average reading skills as measured by the tests used. Discussion The implications of this set of results seem straightforward. There are real differences between the various methods of instruction even under carefully matched and controlled situations. Not only do the methods differ in their effects on specific skills but also in the patterns they produce across skills. Thus, pacer produces quite large gains in speed but at the cost of a reduction in comprehension level. For a student whose initial speed is low but whose comprehension is more than satisfactory the pacer is an appropriate choice of method. If the student is average in both speed and comprehension a better choice might be exhortation, which yields only half the gains in speed but meanwhile increases comprehension, albeit slowly. Until further material based on the use of a standard set of methods across a variety of populations became available it is safest for each reading center to develop its own set of probable
gains based on the methods which it has available for use. It is also clear that any attempt to judge the results of a particular training method must be based on tests run at a later date rather than immediately after the session. Results gathered at Drew's reading clinic suggest that gains can be measured with some stability at times ranging from about one day to as long as two and a half weeks after training. An interesting area for future research is that of the optimal distribution of training periods across time as a function of the method. References MILLS, J. W., JENSEN, P. K., and HERSHKOWITZ, M. A test of a linear programming model as an optimal solution to the problem of combining methods of reading instruction, 1971 (Xerox). PERRY, W. G., Jr. and WHITLOCK, C. P. Selection for improving speed of comprehension. Cambridge, Mass.: Harvard University Press, 1959 (Revised). ROBINSON, F. P. and HALL, P. Robinson-Hall reading test. Columbus, Ohio: University Press, 1949. 37