Profilig readig comprehesio i Magere Schools: a research ad developmet collaboratio Lai¹, M.K., McNaughto², S., MacDoald², S., Hall³, A., MacDoald³, B., McKee³, D., Nicholls³, J., Reeves³, J., Swa³, J., Valgrve³, D., Weir³, P., Farry², S & Warre 4, S. Affiliatios: 1. Magere AUSAD School Improvemet Iitiative 2. Woolf Fisher Research Cetre 3. Magere schools 4. Miistry of Educatio This research ad developmet collaboratio was made possible through the partership of the affiliated groups ad govermet support. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 1
Over recet years there has bee a strog focus o uderstadig the developmetal features of early literacy ad characteristics of effective istructio at the early stages of learig to read ad write at school. I the Uited Sates, the Natioal Research Coucil released a report by a pael of experts called Prevetig Early Readig Difficulties (Sow, Burs & Griffi, 1999), ad i New Zealad a Literacy Task Force (1999) ad a Literacy Experts group (1999) reported to govermet, both focusig o istructio i the early years at school. A major goal i the New Zealad case has bee to uderstad ad reduce disparities i achievemet for Maori ad Pasifika studets, ad childre i low decile schools i the first four years of school. There has bee a umber of atioal ad local iitiatives sice 1998 (Miistry of Educatio, 22) ad there is evidece for icreased effectiveess of early literacy istructio. The Natioal Educatio Moitorig Project s (Flockto & Crooks, 21) secod cycle of assessmets of readig showed reduced umbers of childre i the lowest bads at year four i readig accuracy. A recet reormig of stadardised assessmets at year 1 (6, years) coducted i 2 suggests kowledge of letters ad souds is higher (Clay, 22). Research-based itervetios i low decile schools have demostrated teachers ca raise rates of progress sigificatly across a broad bad of early literacy measures icludig accurate decodig of text (Phillips, McNaughto & MacDoald, i press). But a major challege has bee created by these advaces. I part this is a issue of sustaiability, succictly idetified by The RAND study group, set up i 1999 by the US Departmet of Educatio s Office of Educatioal research ad Improvemet to idetify the most pressig eeds for readig research. The authors of the report say: We have made eormous progress over the last 25 years i uderstadig how to teach aspects of readig. We kow about the role of phoological awareess i crackig the alphabetic code, the value of explicit istructio i soud letter relatioships, ad the importace of readig practice i producig fluecy. The fruits of that progress will be lost uless we also atted to issues of comprehesio. Comprehesio is, after all, the poit of readig. (Sweet & Sow, 23, p xii) The secod part of the challege has bee repeatedly oted by Michael Pressley (21) as a failure to tur what we kow about comprehesio ad the teachig of comprehesio ito geerally more effective teachig i schools. The issue Pressley idetified was oe of kowledge trasfer. It is how to add the kowledge about comprehesio that has bee developed i recet research to what is practiced i schools i a way that eables schools to develop effective ad sustaiable practices. The situatio i New Zealad is critical. The sapshots provided by the Natioal Educatioal Moitorig Project show that despite the gais i decodig, there are wide ad icreasig disparities i achievemet o comprehesio tasks for Maori ad Pasifika childre ad particularly i low decile schools (Flockto & Crooks, 21). The AsTTle project collected atioal data also i across multiple dimesios of comprehesio ad Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 2
foud large differeces betwee Maori ad Pasifika childre which were stable across decile levels despite sigificat treds across deciles (Hattie, 22). The preset report is the result of a cluster of seve decile oe schools developig a partership with researchers to meet the RAND group s challege of buildig sustaiability ad Pressley s challege of research-based applicatios of best practice. They have implemeted effective istructioal programmes for begiig literacy istructio. These school-based projects have idicated what a exercise would look like to uderstad teachig ad learig eeds, ad the coditios for sustaiability (McNaughto, Phillips & MacDoald, 23; Timperley, 23). Oe is that systematic data o both learig ad teachig would eed to be collected by the schools ad aalysed by the schools. That assessmet data would eed to be broad based i order to kow about the childre s patters of stregths ad weakesses, to provide a basis for iformed decisios about teachig, ad to clarify ad test hypotheses about how to develop effective ad sustaiable practices. Previous assessmets collected idividually by schools usig their batteries of stadardised achievemet tests suggested geeralised low levels of comprehesio o these tests. Several geeral hypotheses are possible for the low levels, which ca be checked ad refied i the early stages of a research ad developmet exercise i schools. Oe is that childre s comprehesio levels are low because of low levels of accurate ad fluet decodig (Ta & Nicholso, 1997). A secod is that childre may have leared a limited set of strategies, for example they may be able to recall well but are weaker i more complex strategies for drawig ifereces, sythesisig ad evaluatio or childre may ot have bee taught well eough to cotrol ad regulate the use of strategies (Pressley, 22). Other possible cotributig reasos might be more laguage based that childre s vocabulary may be isufficiet for the texts used i classrooms tasks (Biemiller, 1999), or that they are less familiar with text geres. Well kow patters of Matthew effects may be preset i classrooms, where culturally ad liguistically diverse childre receive more fragmeted istructio focused o decodig or relatively simple forms of comprehedig, or they receive relatively less dese istructio, all of which compouds low progress (McNaughto, 22; Staovich, West, Cuigham, Cipielewski, & Siddiqui,1996). There is also a set of hypotheses aroud whether the texts, istructioal activities ad pedagogy of the classroom eable cultural ad liguistic expertise to be icorporated ito ad built o i classrooms (Lee, 2; McNaughto, 22). The sigificace of collectig ad aalysig data, rather tha makig assumptios about what childre eed (ad what istructio should look like) without careful aalysis, recetly was uderscored by Buly ad Valecia (22). Policy makers i the State of Washigto had madated programmes without actually aalyzig profiles of low progress studets idetified by test scores from fourth grade Natioal Assessmet of Educatioal Progress (NAEP) scores. The assumptio uderlyig policies ad itervetios was that poor performace reflected studets difficulties with more basic decodig abilities. Yet, there was little data about this assumptio or to kow whether Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 3
focusig o such skills would improve the comprehesio at 4 th grade. Usig a broad bad of measures Buly ad Valecia idetified five groups of low progress readers, some of whom did ideed have limited fluecy ad accuracy i decodig. However, madatig phoics istructio for all studets who fell below the proficiecy levels had missed the eeds of the majority of studets, whose decodig was strog but who struggled with comprehesio or laguage requiremets for the tests. This fidig highlights the eed for research-based applicatios of best practice based o aalyses of studet eeds. This preset study is Phase Oe of a three-year research ad developmet partership betwee schools ad researchers to raise studet achievemet i readig comprehesio through the developmet of research-based applicatios of best practice. I this phase, we examie achievemet i readig comprehesio icludig patters of stregths ad weakesses ad observe classroom practice to uderstad teachig ad learig eeds. Methods Participats The overall partership ivolves schools i the Miistry of Educatio Magere AUSAD (Aalysis ad Use of Studet Achievemet Data) school improvemet iitiative, the iitiative leaders, the Woolf Fisher research Cetre ad Miistry of Educatio represetatives. Baselie data were collected from 1975 studets i seve Magere schools, all of which were decile oe 1 schools. The studets ivolved i the aalysis were from years four to ie ad cosisted of equal proportios of males ad females (5% ad 5% respectively) from 14 ethic groups. Four mai ethic groups made up 87% of the sample. These groups were Samoa (33%), Maori (2%), Toga (19%) ad Cook Islad (15%) ethic groups. Tools Data o readig comprehesio were collected usig the revised Progressive Achievemet Tests i Readig (readig comprehesio sectio oly) (Reid & Elley, 1991) ad the Supplemetary Tests of Achievemet i Readig (Elley, 21). The tests were what schools had decided as a group to collect to measure readig comprehesio because they provided a recogized, stadardized measure of readig comprehesio which could be reliably compared across schools. 2 The revised Progressive Achievemet Tests (PAT) i Readig measure both factual ad iferetial comprehesio of prose material i years four to ie. Each prose passage 1 A school's decile idicates the extet to which the school draws its studets from low socio-ecoomic commuities. Decile 1 schools are the 1% of schools with the highest proportio of studets from low socio-ecoomic commuities. A school's decile does ot idicate the overall socio-ecoomic mix of the school. 2 At the time of this decisio-makig, ot all schools had itroduced AsTTle - Assessmet Tools for Teachig ad Learig (Miistry of Educatio, 23) - ito their schools ad/or had ot yet received sufficiet professioal developmet to use it effectively for these purposes. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 4
cosists of 1-3 words ad is followed by four or five multi-choice optios. The prose passages are arrative, expository ad descriptive, ad differet year levels complete differet combiatios of prose passages. The proportio of factual to iferetial items per passage is approximately 5% - 5% i each year level. The Supplemetary Tests of Achievemet i Readig (STAR) were desiged to supplemet the assessmets that the teachers make about studets close readig ability i years four to ie (Elley, 21). I years four to six, the test cosists of four sub-tests measurig word recogitio (decodig of familiar words through idetificatio of a word from a set of words that describe a familiar picture), setece comprehesio (complete seteces by selectig appropriate words), paragraph comprehesio (replace words which have bee deleted from the text i a Cloze format) ad vocabulary rage (fid a simile for a uderlied word). Oly the paragraph comprehesio sub-test is ot multichoice ad cosists of 2 items, 1 more tha the rest of the sub-tests. I years seve to ie, studets complete two more sub-tests i additio the four sub-tests described above. The sub-tests are the laguage of advertisig (idetify emotive words from a series of seteces) ad readig differet geres or styles of writig (select phrases i paragraphs of differet geres which best fits the purpose ad style of the writer) I years seve to ie, there are 12 items per subtest except for paragraph comprehesio which cosists of 2 items. Both tests have high reliability ad validity (Elley, 21; Reid & Elley, 1991) The correlatio betwee the two tests was.62 (P <.1) 3. I the test maual, Elley (21) reported correlatios betwee.7 ad.78 for year four to eight studets. This, as Elley suggests, idicates that the tests measure similar but ot idetical facets of readig comprehesio. Procedure Represetatives from seve schools formed a Seior Assessmet Team (SAT) as part of the Miistry of Educatio Magere AUSAD iitiative to work with the researchers, the Miistry of Educatio ad the leaders i the iitiative o developig a itervetio to raise studet achievemet. Decisio-makig o the research is collaborative ad ay ethical requiremets brokered through this team. At the begiig of the year, SAT developed a itra-school stadardized process of admiisterig the test ad moderatig the accuracy of teacher scorig. This ivolved stadardizig the week ad time (morig) of testig ad creatig a system of radomly checkig a sample of teachers markig for accuracy of scorig. Accuracy of scorig was further checked by the data etry team from Woolf Fisher Research Cetre durig data etry ad durig aalysis. The STAR ad PAT were admiistered as part of schools ormal assessmet cycle at the begiig of the school year. 3 Correlatios were oly performed o studets who had sat both tests. (=1339). Thirty two percet of studets did ot sit either the PAT or STAR test. The high proportio was due to icomplete data from oe school. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 5
Aalysis Data was aalysed i several phases ad through several forums. Area-wide data was aalysed i collaboratio with SAT i two meetigs, the aalysed by seior maagers ad seior teachers with support from the first author. Four of the seve schools with support from the first author coducted further aalysis sessios with teachers. Aalysis sessios ivolved examiig ad evaluatig both researchers ad practitioers theories of the patters of achievemet, rather tha privilegig either researchers or practitioers theories. Data were aalysed usig descriptive ad iferetial statistics to examie achievemet i readig comprehesio across the year levels, icludig cotet aalysis to determie studet stregths ad weakesses. The latter ivolved aalysig studet scores o factual ad iferetial questios i the PAT, ad aalysig subtest scores i STAR. I STAR, this also icluded qualitatively codig the types of errors studets made o the cloze passage accordig to the types of errors reported i the STAR maual (Elley, 21). Four raters were traied to code the errors. These raters subsequetly discussed how to code the errors ad collectively rated a sample of tests to determie reliability of codig. The codig was subsequetly checked ad iter observer agreemet o 1% of studets subtests (across ages) was 9.5%. Geder ad ethic breakdows are also icluded i this paper. Classroom Observatios Systematic classroom observatios were carried out by the secod author. These were desiged to provide a sample of how features of teachig ad learig might map o to the achievemet data. Our argumet was that a full problem aalysis was eeded to examie classroom istructio otherwise assumptios about what was or was ot beig taught would be uchecked. Observatios were carried out i 16 classrooms i seve schools from year four/five through to year eight, (icludig oe biligual Samoa classroom). Classroom istructio was observed for 4 6 miutes durig the usually scheduled core readig sessio withi which the teachig ad learig of readig comprehesio occurred. Class sizes geerally raged from 21 26. A combiatio of diary ad audio recordig of specific activities occurred. Discussios with teachers also provided a importat level of professioal reflectio. Specific details from these observatios are reported further i the paper whe the studet achievemet profiles are discussed. Observatios revealed that the geeral programme was similar i most classes. A whole class activity typically was followed by group work, the (2 5) groups orgaised by ability level. I several classes the whole class activity ivolved the teacher itroducig ad sharig a arrative text. The activity ofte took place over 2 miutes. For the followig (up to) 4 miutes, small group activities occurred. These icluded text study ad aalysis (such as study of plot or character i arrative texts ad extractig ad usig iformatio i iformatioal texts), specific group or paired forms of istructioal / guided readig (such as reciprocal teachig ), ad idividual or group project work (such as developig taxoomies of categories i sciece topics). Typically, the teacher Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 6
worked with two groups over this time period ad held cofereces o the ru with other groups. The geeral programme appeared to work well i terms of levels of egagemet, which were high, with routies well established ad frequet istaces of teacher-studet ad studet-studet iteractios. The geeral orgaisatio meat whole class activities occurred o 3-5 days per week ad small group work with oe or two groups ofte daily, so that each group had at least oe sessio but up to three sessios with direct teacher guidace each week. However, the variatio i frequecy of cotact with each group was quite marked betwee schools, ad i some classrooms few specific iteractios with idividual childre per sessio. Whe ot with the teacher, groups egaged i a rage of activities. Some had developed to the poit of beig able to operate just with peer guidace i reciprocal teachig. I most classrooms worksheets were used, which cotaied questios about a text ad ofte cotaied setece, word or sub word studies. Texts ad topics were well plaed to be egagig ad at appropriate text readig levels. Results The results are preseted i three sectios: The geeral profile of readig comprehesio, cotet aalysis of PAT ad STAR, ad geder ad ethic group breakdows. Geeral profile of readig comprehesio The staie 4 distributio of both tests idicate that the average studet experieced difficulty o these measures of readig comprehesio. Figure 1 shows the staie distributio i both tests across all year levels. The average studet i both tests scored i the below average (staie two ad three) bad of achievemet (PAT mea=3.1, SD=1.38; STAR mea=3.1, SD=1.5). This idicates that the average studet was just belo w the average bad (staies four to six) ad was two staies below the expected average of staie five. However, early 25% of studets were withi the average, Above average ad superior bads of achievemet (staie five to ie). 4 Staies are ormalized stadard scores havig a mea of five ad a stadard deviatio of about two (Reid & Elley, 1991). They are expressed as a scale of ie uits with a low of oe ad a high of ie. I the PAT maual, staie ie is described as superior, staie seve ad eight as above average, staie four to six as average, staie two ad three as below average ad staie oe as low (Reid & Elley, 1991, p. 23). The ie staie uits may be cosidered as ie categories of readig attaimet, makig it highly suitable for iterpretig performace o the PAT:Readig (Reid & Elley, 1991, p.23). Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 7
Figure 1 PAT ad STAR staie distributio across all year levels 35 Percetage of studets 3 25 2 15 1 5 PAT STAR 1 2 3 4 5 6 7 8 9 Staie Across the year levels, the patter was the same i both tests, with the average studet i every year level scorig at staie three 5. This is displayed graphically i Figures 2 ad 3. The rage of achievemet was large, from staie oe to ie i the PAT ad oe to eight i STAR. Figure 2 Staie Distributio for PAT i years four to eight 8 6 S t a i e 4 2 N = 244 4 298 5 318 6 443 7 471 8 Year level 5 For reasos of cofidetiality, year ie studets were ot icluded i this, or ay like, aalysis, as there is oly oe school with year ie studets. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 8
Figure 3 Staie distributio for STAR i years four to eight 1 8 6 S t a i e 4 2 N = 241 243 288 293 36 4 5 6 7 8 Year level Cotet aalysis PAT cotet aalysis Mea scores o factual ad iferetial questios i the PAT were virtually idetical across all the year levels (see Figure 4). Note that the maximum raw score for both factual items ad iferetial items was approximately 2 (Reid & Elley, 1991). This suggests that studets experieced similar difficulties i aswerig factual ad iferetial questios. There was a sigificat correlatio betwee factual ad iferetial items (r =.61, p <.1). Figure 4 Mea raw scores o factual ad iferetial questios by year level Mea raw score 7 6 5 4 3 2 1 4 5 6 7 8 Total Mea of factual questios Mea of iferetial questios Year level Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 9
This patter was costat across the year levels, suggestig that studets of differet year levels experieced similar difficulties i aswerig factual ad iferetial items. Table 1 presets the breakdow of factual ad iferetial questios for PAT readig comprehesio across the year levels. Table 1 Meas (ad stadard deviatios) of factual ad iferetial questios across the year levels Factual questios Iferetial questios Year level N 4 245 4.93 4.18 (2.8) (2.36) 5 298 5.76 4.64 (3.32) (2.76) 6 319 5.75 5.85 (3.4) (2.76) 7 39 6.14 6.31 (2.82) (2.67) 8 471 6.41 6.3 (3.1) (3.12) 9 144 6.57 5.92 (3.3) (2.51) Total 1867 5.96 5.65 (3.8) (2.9) Cotet aalysis o the STAR subtests Aalysis of the subtests of STAR revealed cosistet patters across the subtests at each year level. Figures 5 ad 6 show the average percetage obtaied i each sub test. At every year level, studets scored highest o subtest oe (Word recogitio) ad lowest o subtest three (paragraph comprehesio), idicatig that studets i all year levels experieced more success i decodig words the comprehedig a paragraph. All the sub-tests of STAR were sigificatly correlated (p <.1). A series of paired t tests betwee sub tests averaged across years revealed that at both age groupigs, the meas for sub test 1 were sigificatly higher tha the meas for the other sub tests (t values > 18.; p<.), ad sub test 3 meas were sigificatly lower tha each of the other sub tests (t values > 12., p<.). I additio, i the age group y4 6, all sub tests were sigificatly differet from each other. I the older age group, subtest 2 ad 5, 2 ad 6 ad 5 ad 6 were ot sigificatly differet (p >.5). Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 1
Figure 5 Average percetages obtaied i each subtest (STAR) for year levels four to six 1 Average percetage obtaied 8 6 4 2 Year 4 Year 5 Year 6 1 2 3 4 Subtests Figure 6 Average percetage obtaied i each subtest (STAR) for year levels seve ad eight 1 Average percetage obtaied 8 6 4 2 Year 7 Year 8 1 2 3 4 5 6 Subtests Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 11
Error aalysis of subtest 3 (paragraph comprehesio Cloze) Errors o the third sub test (paragraph comprehesio) were aalysed accordig to the codig i the STAR maual based o commo errors made by studets such as irregular plurals ad tese problems. However, early half (46%) of all possible errors across all year levels did ot fit ay of these categories. These errors appeared to be of two ew types. Oe ivolved mistakes that sometimes made sese i the pre setece cotext but resulted i osese or illogical seteces. The followig examples show studet resposes i italics ad the correct respose/s is i brackets): All they did (could) afford was a tiy room i a shoe (cottage/house/shop) i a village by a river. He grabbed fratically, ad felt his head (had(s)/figer(s)) closig aroud the brach of a tree. Suddely, roud a sharp bed i the head (road/path/track), he fell agai, missed his self (footig/step) ad pluged over the ugly (cliff/rock) face. There were a few istaces of a secod type of error, where relatively close syoyms, ot listed by the maual, were used. It browses etirely o tall braches (trees/plats) especially the foliage of mimosa ad acacia trees. Ethicity ad geder Aalyses were also coducted by ethicity ad geder. For the PAT (see Figure 7) the patter of achievemet was cosistet across the major ethic groups. Studets, o average, scored at staie three with 5% of studets scorig betwee staie two ad four. Figure 7 PAT staie distributio by ethic group 1 8 6 S t a I e 4 2 N = 46 NZ Europea 382 Maori 285 Cook Islad 635 Samoa 352 Toga Ethicity of studet I the STAR test, however, the patter of scores was slightly differet betwee NZ Europea studets ad the studets from the other ethic groups. (see Figure 8) The NZ Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 12
Europea studets scored, o average, at staie four whilst the other ethic groups scored, o average at staie three. The patter of scores o STAR ad PAT, ulike the other ethic groups, is also differet for NZ Europea studets with studets, i geeral, obtaiig higher staies i STAR. Figure 8 STAR staie distributio by year level across ethic groups 1 S t a i e 8 6 4 2 N = 28 NZ Europea 278 Maori 197 Cook Islad 468 Samoa 233 Toga Ethicity of studet Similarly, there were few geder differeces betwee males ad females i both PAT ad STAR (see Figures 9 ad 1). Both males ad females, o average, scored at staie three with 5% of studets fallig betwee staie two ad four. Figure 9 PAT staie distributio by geder 1 S t a i e 8 6 4 2 N = 952 Male 951 Female Geder Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 13
Figure 1 STAR staie distributio by geder 1 S t a i e 8 6 4 2 N = 689 Male 669 Female Geder Discussio The profile of studet s comprehesio i the seve decile 1 schools cofirmed previous descriptios of below average levels i the middle to upper primary school years (Flockto & Crooks, 21; Hattie, 22). The profile was the same across age levels ad across the two tests used. However, it is importat to ote the presece of variability withi the profile. A quarter of the studets were average or sigificatly above average i their achievemet. What ca be determied from the patters withi ad across tests? Oe hypothesis is that studets have ot developed fast ad accurate decodig skills kow to be ecessary but ot sufficiet coditio for effective comprehedig i covetioal school texts (Nicholso & Ta, 1997; Pressley, 22). The fidigs of this study suggest that widespread problems with decodig skills are ulikely to be the uderlyig reaso for the low PAT ad STAR results. At every year level, the subtest word recogitio i STAR was higher tha ay of the other subtests. O average, studets got betwee 6% ad 8% of the words correct, idicatig ability to idetify words accurately uder timed coditios. These meas are betwee 1.1 ad 2 raw scores differet from the meas reported i the maual for the atio-wide sample, idicatig that word recogitio skills are similar to the coutry as whole. (Elley, 21, states that oly a raw score differece of 3 to 4 poits ca be cosidered sigificat.) Aecdotal evidece from seior maagers further support this coclusio. Seior maagers i the schools reported large umbers of studets who score highly o decodig (as measured by ruig records), but perform poorly o the tests of comprehes io such as STAR, PAT ad comprehesio questios o the ruig records. 6 6 The disparity betwee decodig ad comprehesio scores was oe of the primary reasos seior maagers bega this project with Woolf Fisher Research Cetre o readig comprehesio. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 14
Further evidece which could support the coclusio that most studets did ot have problems readig the test was the completio rates of the PAT. The studets rates of completio i the PAT s were relatively high, so that by year eight, the rate of completio was 93%. (For example, i year four, 28.6% did ot complete the test ad i year six 11.5% did ot complete the test.) But it could also suggest iaccuracy ad high rates of guessig the questios. Ideed, aalysis of the errors o the paragraph completio sub test of the STAR which has a Cloze format, idicates that there were high rates of guessig, or at least ot checkig aswers. Two further pieces of evidece support this propositio that over use of predictig or guessig without checkig may have bee occurrig. Firstly, classroom observatio data suggested studets ofte egaged i predictig durig stadard classroom readig activities. For example, i all of the guided readig lessos observed, studets geerated may predictios about arrative or expository cotet of texts, but rarely checked ad were rarely prompted to check the accuracy of these predictios usig evidece from the text. Systematic observatios recorded promptig to check predictios oly ie times i 16 hours of observatios. Recet research evidece also has begu to describe a similar patter that at least some low achievig studets have. Dewitz ad Dewitz (23) described a small group of 5 th grade readers who were fast, efficiet decoders but had low comprehesio scores. Error aalysis revealed high rates of errors termed excessive elaboratios (i.e. guessig). A surprisig fidig was that the studets geerally were ot better at factual questios compared with iferetial questios. Previous research o family literacy practices for Samoa ad Toga childre shows may childre are experieced i recitatio of texts ad this might suggest that recall of facts might be a stregth (McNaughto, 1995). Oe possible explaatio for this relates to the patter of guessig oted above, If studets strategies were focused o predictig (guessig) ad were ot checkig aswers, low accuracy rates for both types of questios could be expected. A further possibility is that the studets do ot have the lexical rage i Eglish required for these tests. The fourth of the STAR subtests which tests vocabulary rage was low. Buly ad Valecia (22), foud i their sample of 4 th grade low progress childre i Washigto State that may of the childre for whom Eglish was a secod laguage, had difficulties with word meaigs o the Eglish tests. Aother surprisig result was the lack of geder differece. There are data showig that boys achieve less well o literacy ad laguage-related school assessmets. Recet Miistry of Educatio figures for school etry assessmets ad o up to atioal pass rates i secodary schools cotiue to show this (Miistry of Educatio, 23). However, other studies of early literacy developmet ad achievemet have failed to fid substatial sigificat differeces (Prochow, Tumer, Chapma, & Greaey, 21). The patters for boys ad girls warrat further study. But, give the variatio betwee fidigs ad the substatial overlappig distributios betwee samples of boys ad girls, Prochow, Tumer, Chapma ad Greaey s claim that the patters appear to be more related to teachig practices tha iheret predispositios of boys ad girls, seems likely. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 15
A major issue is the questio of the cultural familiarity or appropriateess of the tests. Ruig through all these results is the potetial for studets capabilities i comprehedig to have bee systematically uder-assessed because of the ature of the texts ad the tests. Luke, Woods, Lad, Bahr ad McFarlad (22) have raised this issue. The evet kowledge required as backgroud by texts, ad the procedural ad liguistic kowledge required by testig formats eed careful aalysis for bias, because backgroud kowledge is such a strog determiat of comprehesio (Pressley, 21). For example, it might be assumed that Polyesia childre would perform better o the passages i the PATs that are based o Maori myths ad legeds. This was ot the case, the lowest two mea scores o PAT passages were the Maori legeds. There are at least two reasos for this. Oe is that these are based o tribal Maori cultural kowledge, ad the majority of the childre, beig urba Pasifika ad Maori childre may ot have had wide access to or experiece with these cocepts or frameworks. Luke et. al. make the poit that cultural groups ted to be homogeized by test developers, ad there may be wide differeces i practices ad experieces betwee groups that relate to school tasks. A secod is that the structure of retold legeds might create a more difficult gere tha a stadard arrative or expositio i school texts (Graesser, McNamara & Louwerse, 23). The discussio above mirrors discussios that the cluster of schools ad the research team are ow havig aroud these results. The report we have provided here illustrates features of the partership betwee school professioals ad researchers which is based o a shared problem solvig approach usig collected ad aalysed evidece. The achievemet profiles ad the classroom observatios are cotributig to a plaed research ad developmet programme. School- wide professioal developmet across the cluster will be based o what the patters have revealed or suggested. This meas amog other thigs, plaig for ad testig out the role of vocabulary i readig across the curriculum; testig the sigificace of childre s awareess of their use of strategies, particularly checkig strategies; ad how to select ad deploy literacy activities that build o the culturally ad liguistically based expertise of the childre. What we have leared is that it is critical ot to make assumptios about what studets ca or caot do, to look closely at istructioal patters as well as achievemet patters ad to have a positio that sees the studets as full of promise; with cultural ad liguistic resources that ca be built o ad from which high levels of comprehedig ca develop. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 16
Refereces Buly, M. R. & Valecia, S. W. (22). Below the bar: Profiles of studets who fail state readig assessmets. Educatioal Evaluatio ad Policy Aalysis, 24, 3, 219-239. Dewitz, P. ad Dewitz, P. K. (23) They ca read the words but they ca t uderstad: Refiig comprehesio assessmet. The Readig Teacher, 56, 5 422 435. Miistry of Educatio (23). AsTTle. http://www.tki.org.z/r/asttle/idex_e.php Biemiller, A. (1999). Laguage ad readig success. Cambridge, MA: Brooklie Books. Dymock, S. ad Nicholso, T. (1999). Readig comprehesio: What is it? How do you teach it? Welligto: New Zealad Coucil for Educatioal Research. Elley, W. B. (21). Supplemetary Tests of Achievemet i Readig. Welligto: New Zealad Coucil of Educatioal Research. Graesser, A. C. McNamara D.S. ad Louwerse, M M. (23) What do readers eed to lear i order to process coherece relatios i arrative ad expository text. A. P. Sweet, ad C. E. Sow. (eds.). (23). Rethikig readig comprehesio. NewYork NY: Guilford Press.82-98 Flockto, L. ad Crooks, T. (21). Readig ad Speakig assessmet results. Natioal Educatio Moitorig Report 19. Welligto: NZ Miistry of Educatio. Hattie, J. (22) A Research Ageda for New Zealad Schools. Vice Chacellor s Lecture Series. Upublished mauscript. The Uiversity of Aucklad, October 3, 22 Lee, C. D. (2). Sigifyig i the zoe of proximal developmet. I C. Lee ad P. Smagorisky, P. (eds.). Vygotskia perspectives o literacy research: costructig meaig through collaborative iquiry. Cambridge: Cambridge Uiversity Press. 191-225. Literacy Task Force (1999). Report of the Literacy Task Force. Welligto: New Zealad Miistry of Educatio. Literacy Experts Group (1999). Report of the Literacy Experts Group. Welligto, New Zealad Miistry of Educatio. Luke,A. Woods, A., Lad R, Bahr, M ad McFarlad, M. (22). Accoutability: Iclusive assessmet, moitorig ad reportig. Research Report prepared for the Idigeous Educatio Cosultative Body: Brisbae. August 22. School of Educatio, Uiversity of Queeslad. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 17
McNaughto, S. (22). Meetig of Mids. Welligto, New Zealad: Learig Media Limited. McNaughto, S., Phillips, G.E., & MacDoald, S. (23). Profilig teachig ad learig eeds i begiig literacy istructio: The case of childre i low decile schools i New Zealad. Joural of Literacy Research, 35(2), 73-73. Miistry of Educatio (22). Curriculum Update. Issue 5, July. Miistry of Educatio. Aual Report 23. Welligto : New Zealad Miistry of Educatio. Phillips, G., McNaughto, S., & MacDoald, S. (i press). Maagig the Mismatch: Ehacig early literacy progress for childre with diverse laguage ad cultural idetities i maistream urba schools i New Zealad. Joural of Educatioal Psychology. Pressley, M. (22). Comprehesio strategies istructio: A tur of the cetury status report. I C. C. Block & M. Pressley (Eds.), Comprehesio Istructio: Research-based best practice (pp11-27). New York: The Guilford Press Prochow, J. E., Tumer, W. E., Chapma, J. W. ad Greaey, K. T. (21). A logitudial study of early literacy achievemet asd geder. New Zealad Joural of Educatioal Studies, 36, 2, 221-236. Reid, ad Elley, W. B. (1991) Revised Progressive Achievemet Tests: Readig Comprehesio. Welligto: New Zealad Coucil for Educatioal Research. Sow, C. E., Burs, M. S. & Griffi, P. (1998). Prevetig readig difficulties i youg childre. Washigto DC: Natioal Academy Press. Staovich, K. E., West, R. F., Cuigham, A. E., Cipielewski, J. ad Siddiqui, S. (1996). The role of iadequate prit exposure as a determiat of readig comprehesio problems. I C. Coroldi ad J. Oakhill (Eds.). Readig Comprehesio Difficulties: Processes ad itervetio. (pp. 15 32) Mahwah, NJ: Lawrece Erlbaum.. Sweet, A.P. ad Sow, C. E. (eds.). (23). Rethikig readig comprehesio. New York NY:Guilford Press. Ta, A. & Nicholso, T. (1997). Flashcards revisited: Traiig poor readers to read words faster improves their comprehesio of text. Joural of Educatioal Psychology, 89, 276-288. Timperley, H. (23). Shiftig the Focus: Achievemet iformatio for professioal learig. Welligto: NZ Miistry of Educatio. Paper preseted to the Joit AARE/ NZARE Coferece, Aucklad, November 29 December 3. 23. 18