MEMORANDUM Jauary 30, 2017 TO: FROM: SUBJECT: Lace Mester Officer, Elemetary Curriculum ad Developmet Carla Steves Assistat Superitedet, Research ad Accoutability THE EFFECT OF THE WRITER S WORKSHOP ON THE EXPOSITORY WRITING PERFORMANCE OF THE HOUSTON INDEPENDENT SCHOOL DISTRICT S FOURTH-GRADE STUDENTS, 2015 2016 Writer s workshop is a process approach to writig istructio usig a workshop model beig implemeted i the Housto Idepedet School District s kidergarte to fourth grade. The process ivolves prewritig or plaig, draftig, revisio, editig, ad evaluatio at studets pace ad, i some cases, the publicatio of writig pieces usig a variety of geres, ideas, orgaizatios, ad toes. The purpose of this evaluatio was to determie the effects of the Writer s Workshop o teachers preparatio for writig istructio, their actual writig istructio, ad o studets expository writig practices usig teacher surveys, classroom observatios, ad treatmet effects aalysis of studets writig performace. Key fidigs iclude: Accordig to 2015 2016 etrai data, betwee 48.0 ad 65.7 percet of teachers i three cohorts, who erolled i the Writer s Workshop professioal developmet (PD), completed the traiig. Fifty percet of 57 teachers who respoded to the survey idicated that they dedicated 30 miutes to daily istructioal writig. With a average ratig of 3.71 o a 5-poit scale, teacher respodets agreed or somewhat agreed that the HISD curriculum resources were appropriate ad effective for implemetig Writer s Workshop. Six of the eleve Writer s Workshop classrooms observed i April 2016 had 66.7 to 83.3 percet of the 24 compoets o the observatio schedule based o oe observer. The treatmet effect with regressio adjustmet aalysis showed that fourth-grade studets whose teachers completed the Writer s Workshop professioal developmet had a average raw score that was 0.22 items lower o the expository compoet of the 2016 STAAR writig test compared to the average raw score of studets whose teacher did ot complete the PD. The differece was statistically sigificat.
Further distributio of this report is at your discretio. Should you have ay further questios, please cotact me at 713-556-6700. Attachmet cc: Greita Latha Cidy Puryear
Housto Idepedet School District RESEARCH Educatioal Program Report The effect of the writer's workshop o the expository writig performace of the Housto IdePedet school district's fourth-grade studets, 2015-2016
2017 Board of Educatio Wada Adams Presidet Diaa Dávila First Vice Presidet Jolada Joes Secod Vice Presidet Rhoda Skiller-Joes Secretary Ae Sug Assistat Secretary Aa Eastma Mauel Rodriguez, Jr. Michael L. Luceford Holly Maria Fly Vilaseca Richard A. Carraza Superitedet of Schools Carla Steves Assistat Superitedet Departmet of Research ad Accoutability Ted D. Serrat, Ph.D. Research Specialist Veia R. Holmes, DrPh Research Maager Housto Idepedet School District Hattie Mae White Educatioal Support Ceter 4400 West 18th StreetHousto, Texas 77092-8501 www.houstoisd.org It is the policy of the Housto Idepedet School District ot to discrimiate o the basis of age, color, hadicap or disability, acestry, atioal origi, marital status, race, religio, sex, vetera status, political affi liatio, sexual orietatio, geder idetity ad/or geder expressio i its educatioal or employmet programs ad activities.
The Effect of the Writer s Workshop o the Expository Writig Performace of the Housto Idepedet School District s Fourth-Grade Studets, 2015 2016 Executive Summary The Writer s Workshop is a approach to writig istructio usig a workshop model. It is beig implemeted i kidergarte to fourth-grade i the Housto Idepedet School District. The process ivolves prewritig or plaig, draftig, revisio, editig, ad evaluatio at the studets pace ad, i some cases, publishig of writig pieces usig a variety of topics, geres, ideas, orgaizatios, ad toes. The Writer s Workshop sought to improve studets writig through teacher preparatio by accommodatig weaker studets i research-based approaches that treat writig as a process rather tha a product (Murray, November, 1972). Writer s Workshop treats studets as boa-fide authors, exposig them to the approaches authors use i their writig. Teachers were exposed to four days ad/or 12 hours of traiig i the writig process i preparatio for istructig studets usig Writer s Workshop. Writig is see as essetial to effective commuicatio, learig, ad self-expressio (Graham, Bolliger, Olso, D'Aoust, MacArthur, McCuthe & Loghouse, 2012). The purpose of this evaluatio was to determie the effects of the Writer s Workshop o teachers preparatio for writig istructio, their writig istructio, ad o studets expository writig practices ad performace. The effect of the workshop o studets writig performace was determied usig the State of Texas Assessmets of Academic Readiess (STAAR) writig scale ad raw scores ad the percetage of studets who met the state s writig stadards, a teacher survey, ad classroom observatios. Highlights Accordig to 2015 2016 etrai data, betwee 48.0 ad 65.7 percet of teachers i three cohorts, who erolled i the Writer s Workshop professioal developmet (PD), completed the traiig. Fifty percet of 56 teachers who respoded to the survey questio idicated that they dedicated 30 miutes to daily istructioal writig. The teachers who were surveyed had a overall ratig of 1.70 o a 2-poit scale with 1 beig o ad 2 beig yes for their traiig ad preparatio for implemetig Writer s Workshop. With a average ratig of 3.71 o a 5-poit scale, teacher respodets agreed or somewhat agreed that the HISD curriculum resources were appropriate ad effective for implemetig Writer s Workshop. O a 5-poit scale, teachers who were surveyed had a overall ratig of 3.76 o a 5-poit scale idicatig that they agreed or somewhat agreed that they were efficacious ad that they adhered to the writig istructioal practices. The teachers who were surveyed either somewhat agreed or agreed that their studets, 3.49 o a 5- poit scale, were able to practice writig usig the fudametal Writer s Workshop priciples ad compoets. Six of the eleve Writer s Workshop classrooms observed i April 2016 had 66.7 to 83.3 percet of the 24 compoets o the observatio schedule based o oe observer.
A comparable portio of fourth-grade studets i the group whose teachers completed the Writer s Workshop traiig (61.5%) ad whose teachers did ot complete the traiig (60.7%) met the Level II Satisfactory at the 2016 progressio stadard o the STAAR 2016 writig test. Usig a treatmet effect with regressio adjustmet aalysis, fourth-grade studets whose teachers completed the Writers Workshop PD had a average scale score o the 2016 STAAR writig test that was 104 poits lower tha the scale score of studets whose teachers did ot complete the PD. The differece was statistically sigificat. The treatmet effect with regressio adjustmet aalysis also showed that fourth-grade studets whose teachers completed the Writer s Workshop professioal developmet had a average raw score that was 0.22 items lower o the expository compoet of the 2016 STAAR writig test compared to the average raw score of studets whose teacher did ot complete the PD. The differece was statistically sigificat. Recommedatios Teacher completio of the professioal developmet desiged to ehace preparatio for effective delivery of the Writer s Workshop eeds to be ecouraged. The high cotract to participatio cost icurred i deliverig this professioal developmet ad the failure of teacher participats to cofidetly cofirm studets ability to outlie the writig process usig the Bomer s Flowchart makes the case for the ecouragemet. Teachers may require additioal resources to effectively teach writig at the elemetary school level. It may be essetial to esure that writig receives the same level of attetio at all elemetary grade levels i additio to the fourth grade where writig is tested. Future evaluatios should focus o observig the delivery of the Writer s Workshop, particularly before the testig period, ad possibly multiple times durig the school year to assess teachers actual writig istructioal practices. This would allow for a more reliable observatio of the Writer s Workshop ad a more robust assessmet of fidelity. HISD Research ad Accoutability 2
Itroductio The Housto Idepedet School District (HISD) implemeted the Writer s Workshop i kidergarte to eighth grades i all schools durig the 2015 2016 school year. The Writer s Workshop uses the process approach to writig istructio withi a workshop model. The process ivolves prewritig or plaig, draftig, revisig, editig, evaluatig at studets pace, ad i some cases the publicatio of writte pieces usig a variety of topics, geres, ideas, orgaizatio, ad toes. Studets are see as boa-fide authors ad, therefore, are exposed to the approaches authors use i their writig. I preparatio for facilitatig studets writig, teachers of writig were exposed to a four-day ad/or 12-hour traiig i the writig process or Writer s Workshop. Writig has bee described as a essetial aspect of professioal, social, commuity, ad civic egagemet because the ability to write well is fudametal to effective commuicatio, learig, ad selfexpressio (Graham, et al., 2012, p. 6). A atioal survey of elemetary school teachig istructio, grades 4 6, foud that teachers were iadequately prepared to teach writig, studets time devoted to writig was iadequate, ad that studets writigs were commoly restricted to arratives, letter writig, completig worksheets, ad respodig to materials read (Gilbert & Graham, 2010). There was little evidece that teachers used research-based approaches whe teachig writig, ad that they made few or o adaptatios or did so ifrequetly to facilitate weaker studets (Gilbert & Graham, 2010). O the 2007 Natioal Assessmet of Educatioal Progress (NAEP) writig assessmet, 67 percet of eighth-grade studets performed at or below the basic level i writig idicatig that they had attaied oly partial mastery of the writig skills eeded at this grade level (Natioal Assessmet of Educatioal Progress, Salahu-Di, Persky & Miller, 2008 cited i Gilbert & Graham, 2010). Sixty-two ad 63 percet of HISD fourth- ad seveth-grade studets, respectively met phase-i 1 Level II: Satisfactory stadard o the 2015 State of Texas Assessmets of Academic Readiess (STAAR) writig test, compared to 70 ad 72 percet respectively at the state level. STAAR writig i HISD ad the State of Texas is tested i the fourth ad seveth grades. The Writer s Workshop seeks to improve studets writig through teacher preparatio, use of a variety of geres ad topics, ad use of research-based approaches that treat writig as a process rather tha a product (Murray, 1972, p. 3). The purpose of this evaluatio, therefore, is to determie the effects of the writig workshop o teacher preparatio for teachig writig, writig istructio, ad studets expository writig practices ad performace. The effect of the workshop will be determied usig STAAR writig scores ad performaces ad Stata treatmet effects with regressio adjustmets (teffects ra), a teachers survey, ad classroom observatios. The evaluatio sought to aswer the followig four questios: 1. What were the perceptios ad experieces of teachers who participated i the Writer s Workshop? 2. What were the demographic characteristics of the Writer s Workshop fourth-grade studet sample? 3. How did fourth-grade studets whose teachers participated i the Writer s Workshop perform o the overall ad expository compoet of the 2016 STAAR writig assessmet? 4. What was the effect of the Writer s Workshop istructio o the overall ad expository writig performace of fourth-grade studets whose teachers completed the Writer s Workshop? HISD Research ad Accoutability 3
Literature Review Expository writig ivolves commuicatig iformatio about a topic usig facts ad details. It icludes specific terms ad materials that help classify ad defie the topic. This type of writig is orgaized usig structures such as examples, classificatio, cause ad effect, problem ad solutio, ad compare ad cotrast (Housto Idepedet School District, 2015; Purdue Uiversity, 2016). The arrative ad expository writig samples of sixth- to twelfth-grade studets with laguage learig disabilities were evaluated for elemets of microstructure 1 ad macrostructure 2 usig experimetatio. Results of the Wilcoxo siged raks tests idicated that the arrative gere had sigificatly higher levels of productivity ad grammatical complexity compared to the expository geres. Both geres showed impoverished text structure ad revealed strog correlatios betwee the microstructure ad macrostructure (Hall-Mills & Apel, 2013). A report o a 2008 atioal writig survey cocluded that despite cosesus o research-based approaches to writig, they have ot bee widely implemeted (Laud, 2013). I additio, most schools have isufficiet comprehesive, sustaied, ad focused systems for offerig teacher professioal developmet to support research-based writig practices (Laud, 2013). Research o the effects of the Writer s Workshop has bee mixed. A study ivolvig 90 kidergarte studets, radomly assiged to joural writig ad Writer s Workshop classrooms, revealed a sigificat differece i studets writig achievemet i favor of those who were exposed to the Writer s Workshop strategies usig a repeated measures test o a 10-stage developmetal writig rubric (Mester, 2011). Whe a Chi-square cotigecy table was used to determie the effect of the Writer s Workshop o 321 studets i achievemet o the Georgia fifth grade writig assessmet over three years, the results showed that studets iitial performace prior to the itervetio was the highest (Smithso, 2008). I additio, idepedet samples t-test of the latter two years performace revealed o sigificat scale score differece betwee the groups writig performace o the assessmet (Smithso, 2008). Smithso (2008) idetified at least three factors that may explai the uexpected fidigs of the Georgia study: (1) teacher resistace to full implemetatio of Writers Workshop due i part to iadequate traiig ad teacher preparatio, (2) chages i the test ratigs ad achievemet stadards, ad (3) lower studet ability over time. The writig performace of 18 first-grade studets was compared usig writig prompts ad Writer s Workshop strategies durig a five-week period. The result showed that the readig attitudes of both groups declied while the group exposed to the prompts demostrated a larger declie. Studets who were taught usig prompts, however, had mea scores that were higher tha those who used the Writer s Workshop strategies. Overall, studets who were better writers ejoyed the writig process more (Caroll & Feg, 2010). This study, however, did ot specifically ivestigate Writer s Workshop s effect o expository writig. The Uited States Departmet of Educatio Istitute of Educatio Scieces (IES) outlied four recommedatios for teachig elemetary school studets to be effective writers (Graham, Boliger, Olso, D Aoust, MacArthur, McCuthe & Loghouse, 2012). These were (1) provide daily time for studets to write; (2) teach studets to use the writig process ad how to write for a variety of purposes; (3) teach studets to become fluet with hadwritig, spellig, setece costructio, typig, ad word processig; ad (4) create a egaged commuity of writers. (Graham, et al., 2012). Writer s Workshop, as a approach to the writig process, satisfies the Istitute s recommedatios for teachig elemetary studets to be effective writers. Writer s Workshop s effectiveess is the focus of this evaluatio study. 1 Microstructure ivolves the text base, writer s coveyace of meaig at multiple level (word, setece), productivity, grammatical complexity ad accuracy, ad lexical diversity 2 These iclude gere-specific text structure elemets like the gist of the text, orgaizatio, cohesio, ad text structure, writer s coveyace of meaig. HISD Research ad Accoutability 4
Methods Data Collectio ad Aalysis Data collectio for the evaluatio of the Writer s Workshop icluded a olie survey, classroom observatios, ad aalysis of studets overall ad expository writig results o the 2016 STAAR fourthgrade writig assessmet. STAAR writig i elemetary school is tested oly i the fourth grade. The evaluatio beefited from several meetigs ad discussios with the elemetary curriculum officer ad maagers resposible for supervisig the implemetatio of the Writer s Workshop icludig the professioal developmet (PD) compoet. It also ivolved the review of the PD documets ad other olie documet resources related to writig ad the Writer s Workshop. A olie questioaire was admiistered to teachers who completed the Writer s Workshop PD traiig. The survey questios were developed based o the course cotet. The questioaire was vetted for validity by the elemetary curriculum officer ad maagers, which they dissemiated to icrease teachers respose rate. The survey was admiistered usig SurveyMokey TM. SurveyMokey TM is a free web-based software that admiisters, collates, ad aalyzes survey data. The survey was admiistered from May 12 through 27, 2016. SurveyMokey TM calculated a average ratig scale score (See Appedix A o p. 19 for details). The survey results costitute a major part of this research report. A sample of schools were radomly selected for classroom observatios based o their performace o the 2014 2015 STAAR writig test. Schools were raked accordig to the percet of studets who met Level II: Phase-i 1 Satisfactory stadard ad stratified ito deciles. Oe school from each decile stratum was selected by radom resultig i a total of te schools. Emails were set to the pricipals of these schools explaiig the purpose ad procedures for the visit. Six schools agreed to the visit. Classroom observatios were coducted durig regular fourth-grade writig istructio classes. Teachers were iformed of the visit beforehad. All visits were scheduled betwee April 11 ad 22, 2016 based o the Eglish laguage arts (ELA) Pacig Caledar 3 to coicide closely with expository writig istructio ad to accommodate full implemetatio of the Writer s Workshop ad STAAR testig. However, two of these visits had to be rescheduled ad were held April 27 ad May 13, 2016. A observatio schedule was developed usig iformatio from the Writer s Workshop Facilitator s Guide ad The First 25 Days of Writer s Workshop implemetatio documet. Pricipals or their represetatives were debriefed followig each school visit to discuss observatios ad possible recommedatios for improvemets i program implemetatio. The results of the school visits were used to determie the program s apparet 4 fidelity. Fidelity has bee described as the proportio of a program s compoets that was implemeted (Mowbray, Holter, Teague & Deborah, 2003, p. 316). Teachers servig studets from kidergarte through fifth grade ad who atteded the summer Writer s Workshop traiig i Jue ad July 2015 were icluded i this study. They were selected because their studets, particularly i the fourth grade, would have had a full year s exposure to the teacher s writig istructio skills ad expertise acquired durig the Writer s Workshop PD traiig. Other traiig sessios were held i September, October, ad November 2015, ad Jauary 2016 for kidergarte to secod grade ad third to fifth-grade teacher cohorts. 3 The ELA Pacig Caledar is a geeral guide prepared by HISD Curriculum ad Developmet Departmet for the scheduled delivery of key Texas Essetial Kowledge ad Skills (TEKS) ad ELA curriculum compoets. 4 I described it as apparet because I was uable to actually observe Writer s Workshop lesso delivery i a umber of cases sice observatio occurred post-assessmet whe active delivery of the writig process had eded. HISD Research ad Accoutability 5
Participatio i the Writer s Workshop PD traiig was volutary or pricipals could have recommeded teachers. Oly teachers who registered for the PD, however, were icluded i this study. This study, therefore, is restricted to teachers who either completed or were recorded as o show i the Writer s Workshop PD sessios. Teachers ad their studets who completed the PD comprised the treatmet group ad studets ad their teachers who were o show made up the o-treatmet group. Studets raw scores (umber of items correct) o the 2016 STAAR fourth-grade expository writig compoet ad their overall writig scale scores were used to aalyze studets performace ad determie program effect. Studets overall writig scale scores ad performace data were also used to determie the mea scale score differece betwee studets whose teachers completed the program ad teachers who were recorded as o show i the program. Aalyses were based o the umber of fourth-grade studets who met the 2016 STAAR Level II: Satisfactory ad Level III: Advaced writig performace at the 2016 progressio stadard. Sample The sample cosisted of studets whose teachers completed the Writer s Workshop PD traiig ad those whose teachers were recorded as o show ad whose studets scored o the fourth-grade STAAR writig test. A total of 892 teachers registered to participate i the professioal developmet program. These participats ad their writig studets costituted the frame from which the fourth-grade studet sample was selected (see Murae & Willet, 2011, p. 47). Geeralizatio of results are limited, therefore, to the broader group of fourth-grade studets whose teachers registered to participate i the program ad ot to the total populatio of fourth-grade studets (see Murae & Willet, 2011, p. 47). Forty-three fourth-grade teachers completed the PD ad 12 fourth-grade teachers were recorded as o-show, makig up the fourthgrade teacher sample. Studets were liked to their fourth-grade teachers usig the Public Educatio Iformatio Maagemet System (PEIMS) ad the STAAR assessmet result files. These files icluded demographic ad test data for each studet. A total of 2,371 fourth-grade studets comprised the sample. Of these, 1,879 studets had teachers who completed the Writer s Workshop traiig ad 492 studets had teachers who were recorded as o show i the program. Tests of ormality usig detreded ormal Q-Q ad ormal Q-Q plot of the scale score ad the raw score for the compositio (expository) compoet of the STAAR test were cofirmed. Of the 447 teachers who completed the PD, 57 (12.8%) completed the survey questioaire. As metioed earlier, 11 writig classes i six of the 10 radomly selected schools were observed. A rubric of Writer s Workshop criteria was used to coduct the evaluatio by oe observer (See Appedix B, p. 20). Data Limitatios As oted, the STAAR writig assessmet is limited to fourth grade ad, therefore, does ot cover the full spectrum of grades whose teachers participated i the Writer s Workshop. Because of the limited survey admiistratio timeframe, oly 12.8 percet of teachers respoded to the Writer s Workshop survey. Ay iferece about the survey has to be restricted to that group of teachers sice they may ot be represetative of the etire populatio of Writers Workshop participats i HISD. Classroom observatios were coducted toward the ed of the school year to miimize disruptio. Istructio had to be scheduled to accommodate the visits ad may ot have reflected the preferred atural settig for observatio. HISD Research ad Accoutability 6
Percetage WRITER S WORKSHOP REPORT, 2015 2016 Classroom observatios were coducted after the STAAR testig i April 2016 by which time most maipulatives ad Writer s Workshop artifacts were removed from the classroom to esure test security. The observatios, therefore may ot be a true reflectio of the studets etire writig experiece sice it captures a momet i time. Results of the observatios, may ot adequately reflect program fidelity. Results What were the perceptios ad experieces of teachers who participated i the Writer s Workshop? Teacher Participatio The Writer s Workshop PD was offered to kidergarte through fifth-grade teachers of writig. Summer sessios were held i Jue ad July, 2015 for these teachers. As metioed before, additioal sessios were held i September, October, ad November 2015, ad Jauary 2016 for third- to fifth- ad kidergarte to fifth-grade teachers. A total of 892 teachers erolled i the PD, volutarily. Of these, 611 were erolled i the summer s kidergarte through fifth-grade traiig; 108 teachers erolled i the third to fifth grade PD traiig, ad 173 teachers were erolled i the kidergarte to secod grade PD traiig. Of the 892 teachers who erolled i the Writer s Workshop PD, 447 teachers completed the PD, 212 were o shows, 79 did ot complete the PD, ad 154 teachers dropped out of the PD. Figure 1 shows the participatio status of teachers by the Writer s Workshop PD cohorts. Figure 1. Teachers participatio status by Writer s Workshop professioal developmet cohorts, 2015 2016 100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 =293 =71 =83 =50 =140 =22 0.0 Completed No Show Icomplete Dropped K-5th Grade (%) 48.0 22.9 10.3 18.8 3rd-5th Grade (%) 65.7 20.4 6.5 7.4 K-2d Grade (%) 48.0 28.9 5.2 17.9 =63 =7 =9 =115 =8 =31 Figure 1 shows that 65.7 percet of third to fifth grade teachers competed the Summer Writer s Workshop PD. Less tha half of the teachers who erolled i the kidergarte to fifth grade Writer s Workshop PD completed the traiig. Betwee 20.4 ad 28.9 percet of teachers erolled i the Writer s Workshop PD traiig cohorts were o shows. HISD Research ad Accoutability 7
Percetage WRITER S WORKSHOP REPORT, 2015 2016 Of the teachers who erolled i the kidergarte to fifth ad kidergarte to secod-grade Writer s Workshop PD cohort, 18.8 ad 17.9 percet, respectively, dropped out. Survey Results: a. Istructioal Cotexts for Writer s Workshop A olie survey was admiistered to teachers who completed the Writer s Workshop PD traiig. Fiftyseve prekidergarte to fifth-grade teachers (12.8% of those who completed the PD) completed the survey as show i Figure 2. Figure 2. Distributio of Writer s Workshop teacher survey respodets by grade ( = 57) 100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 =1 Pre-K =13 K =15 =15 First Grade Secod Grade =9 =9 Third Grade Fourth Grade Fifth Grade % of Respodets 1.8 22.8 26.3 26.3 15.8 15.8 7.0 =4 Note: Teachers i departmetalized schools teach writig i multiple grades. There were 57 respodets who taught 66 writig classes. Most of the survey respodets were first-grade (26.3%) ad secod-grade (26.3%) teachers. Accordig to Figure 2, 15.8 percet of respodets were third ad fourth-grade teachers. Survey respodets were asked to idetify the teachig cofiguratio i which they implemeted the Writer s Workshop. Of the 57 respodets, 50.9 percet said it was delivered i self-cotaied classes, 40.4 percet said it was delivered i departmetalized classrooms, ad 8.8 percet said they delivered the Writer s Workshop i other cofiguratios that icluded, dual laguage, team teachig, special educatio, ad itervetio settigs. Most respodets provided writig istructios i Eglish (55.4%) as opposed to Spaish (42.9%) ad Chiese Madari (1.8%). Figure 3 shows the approximate umber of miutes teachers devoted to daily writig istructios. The suggested duratio of daily writig istructio is 45 60 miutes (Bach & Supovitz, 2003). HISD Research ad Accoutability 8
Percetage WRITER S WORKSHOP REPORT, 2015 2016 Figure 3. Approximate umber of miutes dedicated to daily writig istructio by Writer s Workshop survey respodets ( = 56) 100.0 90.0 80.0 70.0 60.0 =28 50.0 40.0 =17 30.0 20.0 =11 10.0 0.0 30 mis 45 mis 60 mis % of Respodet 50.0 30.4 19.6 Most istructio i writig appeared to last about 30 miutes accordig to half of the survey respodets (50.0%). About oe-third of the survey respodets (30.4%) dedicated approximately 45 miutes to daily istructio i writig. Almost oe-fifth (19.6%) of survey respodets devoted approximately 60 miutes for daily writig istructio. b. Preparatio ad Traiig A five-item Likert scale was used to measure teachers preparatio ad traiig for deliverig Writer s Workshop writig istructios. Items were weighted ad average ratigs were tabulated. Fifty-three (93.0%) of teacher respodets respoded to the statemets show i Table 1. Table 1. Teacher preparatio ad traiig of implemetig Writer s Workshop Preparatio ad traiig No (1) Yes (2) Average Ratig Atteded four-day Writer s Workshop sessio 10 39 1.80 Atteded the 12-hour Writer s Workshop sessio 26 21 1.42 Iformatio leared i the four-day sessios prepared me for Writers Workshop. 3 48 1.94 The read-aloud books worked well with the writig process. 2 50 1.96 The First 25-days documet provided adequate guide for teachig writig. 3 50 1.94 Overall Ratig 1.70 With a overall ratig of 1.70 o a 2-poit scale with o beig 1 ad yes beig 2, teacher respodets idicated that they were adequately prepared for implemetig the Writer s Workshop i their classrooms. Overall, most respodets met the preparatio ad traiig requiremets outlied. Most teacher respodets idicated that (a) the read-aloud books used for the writig process worked well (average ratig of 1.96), (b) iformatio leared i the four-day sessio prepared them for Writer s HISD Research ad Accoutability 9
Workshop (average ratig of 1.94), ad (c) the guidace provided by the First 25-days documet appeared to have bee a adequate guide for teachig writig (average ratig of 1.94). More respodets appeared to have atteded the four days Writer s Workshop sessio tha those who atteded the 12-hour Writer s Workshop sessio. Their average ratigs was 1.80 ad 1.42, respectively, o a 2-poit scale. Some may have atteded both sessios. Teacher respodets were also asked to suggest ay additioal PD that they would like to receive i support of the Writer s Workshop implemetatio. Twety-oe respodets (36.8% of all respodets) provided suggestios. The four most frequet suggestios were: ogoig writig PD workshops throughout the year (5 respodets); strategies for implemetig Writer s Workshop i kidergarte (3 respodets); improved time maagemet (2 respodets), ad the use of writig statios (2 respodets). c. Appropriateess ad Effectiveess of HISD Curricular Resources for Writer s Workshop Respodets levels of agreemet or disagreemet with six statemets were used to measure the appropriateess ad effectiveess of related district resources for supportig the Writer s Workshop. The items were rated o a scale of 1 to 5, from strogly disagree to strogly agree. The results are summarized i Table 2. Table 2. Appropriateess ad effectiveess of HISD curricular resources for Writer s Workshop implemetatio HISR Curricular Resources Disagree Agree Structure ad framework of HISD Curriculum ad Plaig guide facilitate ease of plaig. Strogly Disagree (1) (2) Somewhat Agree (3) (4) Strogly Agree (5) Average Ratig 1 4 18 25 5 3.55 First 25-days documet supports Writer s Workshop implemetatio. 1 2 12 26 12 3.87 Sixty-miutes writig sessio provided structure to classroom writig. 2 5 15 26 4 3.48 Istructioal strategies i HISD curriculum plaig guides supported my istructioal plaig ad delivery. Istructioal strategies i HISD curriculum guides supported my istructioal plaig ad delivery. Achor charts i HISD curriculum plaig guides supported my istructioal plaig ad delivery. 1 4 8 33 5 3.73 1 4 11 32 5 3.68 1 4 3 33 12 3.96 Overall Ratig 3.71 The appropriateess ad effectiveess of HISD curricular resources i the plaig ad delivery of the Writer s Workshop had a overall ratig of 3.71 o a 5-poit scale. Most respodets agreed with the statemets i Table 2. With a average ratig of 3.96 o a 5-poit scale, most respodets agreed or strogly agreed that the achor charts appeared to have supported their istructioal plaig ad delivery. The First 25-days documet had a average ratig of 3.87 o a 5-poit scale, as a support for the implemetatio of the Writer s Workshop. Most respodets agreed with this statemet. HISD Research ad Accoutability 10
O a average ratig of 3.55 o a 5-poit scale, most respodets either agreed or somewhat agreed that the structure ad framework of HISD curriculum ad plaig guides facilitated the ease of plaig for Writer s Workshop. The lowest average ratig of 3.48 o a 5-poit scale was received o the 60-miutes writig sessios provided structure to writig i the classroom. Most respodets either agreed or somewhat agreed with this statemet. Teacher respodets were also asked to provide additioal details or iformatio that could be icluded i the curriculum plaig guides to support the Writer s Workshop implemetatio. Fourtee respodets provided details ad iformatio that icluded: likig the Writer s Workshop to exemplar lesso plas ad teachig strategies; aligig Writer s Workshop to existig district resources; icludig teachers i the preparatio of the curriculum guides; orgaizig the curriculum better to follow a logical sequece; codesig the plaig guides; stickig to the istructio time frames; ad offerig actual writig topics. d. Teacher Self-efficacy ad Writig Istructioal Practices Seve survey items were used to measure teacher self-efficacy ad writig istructioal practices i the classroom. The items measured the extet of teacher agreemet or disagreemet o a 5-poit Likert scale as show i Table 3. Table 3. Survey respodets self-efficacy ad writig istructioal practices Self-Efficacy ad Writig Istructioal Disagree Practices Compared to a year ago, I have a deep uderstadig of what it takes to be a effective writig teacher. I successfully guide my studets to idetify writig topics. I facilitate small group oral story craftig i my classroom. I use rubrics to assess my studets writig Strogly Disagree (1) (2) Somewhat Agree (3) Agree (4) Strogly Agree (5) Average Ratig 0 4 9 27 12 3.90 0 3 8 31 10 3.92 2 8 12 19 11 3.56 0 4 14 24 10 3.77 I used read aloud daily as part of my writig istructio. 0 3 13 22 14 3.90 I use 6+1Traits of Writig i my classroom daily for commo laguage ad 2 6 24 11 8 3.33 demostratio of good writig. Idetified Eglish/Spaish Laguage Arts TEKS are practiced as a commo thread to 0 3 9 28 12 3.94 my daily writig istructio. Overall Ratig 3.76 With a overall ratig of 3.76 o a 5-poit scale, most teacher respodets agreed or somewhat agreed with the statemets related to their self-efficacy ad writig istructioal practices. Idetified Eglish/Spaish Laguage Arts TEKS are practiced as a commo thread to my daily writig istructio ad I successfully guide studets to idetify writig topics received the highest average ratigs of 3.94 ad 3.92, respectively, o a 5-poit scale. Most respodets either agreed or strogly agreed with these statemets. HISD Research ad Accoutability 11
Most respodets either agreed or strogly agreed that I used read aloud daily as part of my writig istructio ad that compared to a year ago, I have a deep uderstadig of what it takes to be a effective writig teacher. Both statemets had a average ratig of 3.90 o a 5-poit scale. Respodets use of 6+1 Traits of Writig to create a commo laguage of effective writig ad to demostrate what good writig looks like daily, had the lowest ratig of 3.33 o a 5-poit scale. Most respodets somewhat agreed with this statemet. The facilitatio of small group oral story craftig had a average ratig of 3.56 o a 5-poit scale. e. Studets Writig Practices Fially, teacher respodets were asked to rate their studets writig practices usig a scale of 1 to 5 to measure their degree of agreemet or disagreemet o eight items. Fifty-oe teachers respoded to these items. The results are show i Table 4. Table 4. Teachers ratigs of their studets writig practices HISR Curricular Resources Strogly Disagree (1) Disagree (2) Somewhat Agree (3) Agree (4) Strogly Agree (5) Average Ratig My studets ca outlie the writig process usig Bomer s Flowchart. 1 17 20 10 3 2.94 My studets are able to make My Heart maps as part of the writig process. 0 7 14 18 12 3.69 My studets are able to complete their writig statio activities daily ad o time. 0 7 16 19 9 3.59 My studets are able to use Metors texts, appropriately. 0 5 24 14 7 3.46 My studets are able to draft, revise, ad edit a origial piece of writig. 0 6 14 25 6 3.61 My studets cofer ad reflect o their writig pieces. 0 6 19 20 6 3.51 My studets are able to use objects to geerate their writig. 1 6 13 23 7 3.58 My studets are able to effectively icorporate the Writer s Notebook ito the writig process. 0 7 18 19 7 3.51 Overall Ratig 3.49 Overall, teachers average ratig of studets writig practices was a 3.49 o a 5-poit Likert scale, which was the lowest of all three Writer s Workshop PD costructs measured o the 5-poit Likert scale. The average ratig for six of the eight items were above the overall ratig. Most respodets either agreed or somewhat agreed with these statemets. My studets are able to make My Heart maps as part of the writig process received the highest average ratig of 3.69 o a 5-poit scale. This was followed by My studets are able to draft, revise, ad edit origial writig pieces, which received a average ratig of 3.61 o a 5-poit scale. Most teachers agreed or somewhat agreed with both statemets. My studets ca outlie the writig process usig Bomer s Flowchart received the lowest average ratig of 2.94 o a 5-poit scale. Most teacher respodets disagreed ad somewhat agreed. This was followed by studets ability to use metor text appropriately, which received a average ratig of 3.46 o a 5-poit scale. Most teachers agreed or somewhat agreed with this statemet. HISD Research ad Accoutability 12
Teacher respodets were asked to provide additioal suggestios with regards to the Writer s Workshop. Te teachers provided suggestios that icluded: how to implemet Writer s Workshop i kidergarte, more metor text for teachers, review the 6+1 Traits of Writig ad Bomer s Flowchart. Oe respodet liked the idea of the otebook. Oe respodet highlighted the time costraits i meetig Writer s Workshop requiremets. Writer s Workshop Classroom Observatios Twety-four compoets of the Writer s Workshop were selected for observatio i 11 classrooms from five HISD schools. There was oly oe observer. Writer s Workshop apparet fidelity (AF) was measured usig the umber of compoets observed as a percetage of the 24 compoets o the observatio schedule. Appedix B (p. 20) shows the AF for each class ad across all classes observed. Six (54.5%) of the classrooms had a AF of over 50 percet. Classroom K04GS had the highest apparet fidelity of 87.5 percet, followed by F04GS with 83.3 percet, ad P04NC with 75.0 percet. Classrooms P05MR ad H04BK had the lowest apparet fidelity of 4.2 ad 8.3 percet, respectively. Accordig to Appedix B, betwee 18.2 ad 36.4 percet of the observed classrooms displayed Writer s Workshop listeig habits ad Heart Maps. Betwee 45.5 ad 63.6 percet used mii-lessos, idepedet writig, draft/writig, proof readig, metor texts, ad demostrated the liks betwee TEKS-writig lessos, ad displayed celebratio artifacts ad writig assessmet, progress chart, or published studets work. Appedix B also showed that betwee 72.7 ad 81.8 percet of classrooms visited had writig visuals, used writer s otebooks, demostrated draft/writig, had teacher cofereces, ad revised ad edited studets writig (see Appedix B, p. 20). Exit iterviews with elemetary teachers ad pricipals revealed that two of the schools also implemeted the Writers i the Schools program to supplemet their Writer s Workshop, particularly at the lower writig grades. At a cost of about $26,000, the supplemetal program focused o arrative writig sice Writer s Workshop focused o expository writig. Exemplary writig pieces from the Writers i the Schools were published ito boud books. Oe of the two schools was a literature maget. Writers i the School provided supplemetal resource materials that were used to facilitate istructio ad practice i the writig process at these schools. Data available for eight classrooms showed that where 66.7 to 87.5 percet of the Writer s Workshop prompts were observed, betwee 52.7 ad 77.8 percet of the studets met Level II: Satisfactory performace at the 2016 progressio stadard except oe classroom with a AF of 70.8 percet where 29.9 percet of studets met Level II: Satisfactory performace. Oe classroom with a AF of 8.3 percet had 58.5 percet of the studets ad aother with 16.7 percet AF had 87.9 percet of the studets who met Level II: Satisfactory performace o the STAAR fourthgrade writig test. Similar classroom performace patters were observed for the average umber of items correct o the expository compoet of the STAAR fourth-grade writig test. Cautio, however, must be exercised i the iterpretatio of the AF sice, as part of the limitatios, the observatios were coducted toward the ed of the school year ad classrooms artifacts were take dow i accordace with STAAR testig procedures. HISD Research ad Accoutability 13
Percetage WRITER S WORKSHOP REPORT, 2015 2016 The relatioship betwee Writer s Workshop AF ad the percetage of studets who met Level II: Satisfactory performace o the 2016 progressio stadard o the STAAR fourth-grade writig was weak ad egative (-0.24), the relatioship for the expository writig compoet was moderate ad egative (-0.54). What were the demographic characteristics of the Writer s Workshop fourth-grade studet sample? The proportio of studets i the treatmet ad o-treatmet groups disaggregated by key demographic ad educatioal attributes are preseted i Figure 4. As oted earlier, fourth-grade studets were liked to their teachers who erolled 5 i the Writer s Workshop usig uique idetifiers i PEIMS. These studets costituted the sample for determiig the effects of the Writers Workshop o their 2016 STAAR fourthgrade writig performace. Figure 4. The demographic ad educatioal compositio of the treatmet ad o-treatmet groups i the study sample, Fourth Grade, 2015 2016 100 90 80 70 60 50 40 30 20 10 0 Female Male No Yes No Yes No Yes No Yes No Yes Black Hispaic White Geder Eco. Disadv. At Risk Special Ed. LEP G/T Ethicity Treatmet Group (%) 50.6 49.4 14.8 85.2 44.2 55.8 95.1 4.9 65.2 34.7 80.6 19.4 29.4 64.8 3.2 No-treatmet Group (%) 53.6 46.3 17.5 82.5 37.8 62.2 94.9 5.1 59.8 40.2 84.8 15.2 17.1 77.4 4.5 District 49.2 50.8 23.5 76.5 36.0 64.0 92.7 7.3 69.8 30.2 85.1 14.9 24.5 62.1 8.4 Treatmet Group (%) No-treatmet Group (%) District Note: District data were take from PEIMS Edit + Reports Data Review, 2015 2016 Fall Collectio Proportioally, the largest differece betwee the treatmet ad o-treatmet groups was amog Black studets(11.9-percetage poits) i favor of the treatmet group ad Hispaic studets (12.6- percetage poits) i favor of the o-treatmet group. How did fourth-grade studets whose teachers participated i the Writer s Workshop perform o the overall ad expository compoet of the 2015 2016 STAAR writig assessmet? Figure 5 shows the proportio of Writer s Workshop fourth-grade studets who met Level II: Satisfactory performace ad Level III: Advaced performace o the STAAR writig test at the 2016 progressio stadard. 5 This icluded the treatmet group ad the o-treatmet group, that is, those studets whose teachers completed the PD ad those whose teachers were o show. HISD Research ad Accoutability 14
Percetage WRITER S WORKSHOP REPORT, 2015 2016 Figure 5. Fourth-grade STAAR Writig Performace for Studets i the Writer s Workshop Sample, HISD, 2015 2016 100 80 60 40 61.5 60.7 20 0 Level II: Satisfactory Treatmet Source: STAAR 3-8 Regular test, July 11, 2016 11.1 15.9 Level III: Advaced No-Treatmet A comparable portio of fourth-grade studets i the observed (61.5%) ad compariso (60.7%) study groups met Level II: Satisfactory at the 2016 progressio stadard o the STAAR 2016 Writig test. A lower proportio of fourth-grade studets whose teachers completed the Writer s Workshop (11.1%) compared to those whose teachers did ot participate i the workshop (15.9%) met Level III: Advaced performace o the 2016 STAAR writig test. What was the effect of the Writer s Workshop istructio o the overall ad expository writig performace of fourth-grade studets whose teachers completed the Writers Workshop? The effects of the Writer s Workshop o the STAAR overall ad expository writig performace of fourthgrade studets whose teacher teachers participated i the workshop are preseted i Table 5 ad Table 6. Table 5 shows the effect of the workshop o the overall writig performace usig the Stata treatmet effect with regressio adjustmet (teffects ra) commad regressed o studets ecoomic status, at-risk status, special educatio, limited Eglish proficiecy (LEP), ad gifted ad taleted (G/T) idetificatio. Table 5 used studets writig scale scores as the overall outcome ad Table 6 used the raw scores for the studets expository writig. Studets could have received a total of 8 poits o the expository writig. Table 5. Average Treatmet Effect (ATET) of the Writer s Workshop istructio o the STAAR writig performace of HISD fourth-grade studets, 2015 2016 Writig Scale Score ( = 2,371) Coefficiet Robust Stadard Error Z p>z 95% Cofidece Iterval ATET Treatmet (1 vs. 0) -104.8 28.5-3.68.000-160.6-49.0 POmea Treatmet 0 3823.9 27.6 138.5.000 3769.8 3878.1 Note: The coefficiet is the writig scale score. Liear data distributio was assumed for the treatmet; POmea is the potetial outcome mea 6 As show i Table 5, the fourth-grade studets whose teachers completed the Writer s Workshop ad who may have used the workshop s related istructioal practices for writig had a average scale- 6 The Potetial Outcome Mea (POmea) is the mea of the STAAR writig outcomes fourth-grade studets would obtai if their teachers were exposed to the Writer s Workshop PD traiig ad used the related istructioal practices (See http://www.stata.com/mauals13/teglossary.pdf). HISD Research ad Accoutability 15
score that was 104.8 poits lower tha the potetial mea scale score (POmea = 3823.9) of the fourthgrade studets whose teachers registered for but did ot atted the workshop. The differece was statistically sigificat (p>.01) Table 6. Average Treatmet Effect (ATET) of the Writer s Workshop istructio o the STAAR expository writig performace of HISD fourth-grade studets, 2015 2016 Writig Raw Score ( = 2,371) Coefficiet Robust Stadard Error Z p>z 95% Cofidece Iterval ATET Treatmet (1 vs. 0) -.22.06-3.43.001 -.34 -.09 POmea Treatmet 0 3.7.06 63.1.000 3.64 3.9 Note. The coefficiet are the raw scores o STAAT expository writig marked out of 8 poits. Poisso distributio was assumed, therefore, i the calculatio of the treatmet effect; POmea is the potetial outcome mea Table 6 shows the fourth-grade studets whose teachers completed the Writer s Workshop ad who may have used the workshop-related istructioal practices i writig had a average raw score o their STAAR expository writig that was.22 correct items lower tha the potetial average raw score (POmea = 3.7) of their fourth-grade peers whose teachers registered for but did ot participate i the workshop. Discussio The purpose of this evaluatio was to measure the impact of the Writer s Workshop o the expository writig of HISD fourth-grade studets, ad teachers perceptios ad experieces with implemetig this writig process i their HISD classrooms. Because STAAR assesses elemetary writig oly i the fourth grade, the aalysis was restricted to that grade. A olie survey usig Likert scales was used to measure four mai costructs associated with the Writer s Workshop: teacher preparatio ad traiig for implemetig the workshop, which received a average ratig of 1.7 o a 2-poit scale idicatig that most respodets agreed; the appropriateess ad effectiveess of HISD curricular resources for implemetig Writers Workshop with a average ratig of 3.71 o a 5-poit scale; teacher self-efficacy ad writig istructioal practices, which received a average ratig of 3.76 o a 5-poit scale; ad teacher ratigs of studets writig practices with a average ratig of 3.49 o a 5-poit scale. Overall, the teachers ratigs of studets writig practices had the lowest average ratig of 3.49 o a 5-poit scale. These respodets mostly agreed or somewhat agreed that their studets were able to use the key elemets of the writig process. More respodets somewhat agreed tha agreed that their studets could outlie the writig process usig Bomer s Flowchart or use metor texts, appropriately. This is sigificat sice Bomer s flow chart is a fudametal compoet of the writig process. The program is desiged to get studets to adopt the attitudes ad behaviors of authors but most teacher respodets appeared to disagree or somewhat agree that their studets could use the flow chart which outlied the writig process to be followed. This was the first year of implemetatio of the program ad as such teachers may require more time to adopt these writig practices. The Smithso (2008) study coducted i Georgia showed teacher resistace to full implemetatio of the Writer s Workshop because of iadequate traiig ad teacher preparatio. This study, however, did ot ask the teachers i observed classrooms to cofirm their attedace at the ay of the Writer s Workshop professioal developmet sessios but the aalysis attempted to lik a apparet fidelity to studets performace i writig i the classrooms observed. Aalysis of the relatioship betwee the apparet fidelity ad classroom STAAR writig performace retured a small but egative correlatio (-0.24). The correlatio betwee the apparet fidelity ad the HISD Research ad Accoutability 16
classroom performace o the expository compoet of the STAAR fourth-grade writig was moderate ad egative (-0.54). As fidelity icreased based o classroom observatios, studets performace o the STAAR declied based o the classroom observatios. Observatios of what actually happeed durig the delivery of the lessos may shed more light o the relatioship betwee fidelity ad performace usig Writer s Workshop at the fourth-grade i HISD. The observatios were coducted by oly oe observer. The differece i the percetage of fourth-grade studets whose teachers completed or were reported as o show i the workshop ad who met Level II: Satisfactory at the 2016 progressio stadard was egligible. However, 4.8 percet more studets whose teachers were reported as o show compared to the studets whose teacher completed the workshop met Level III: Advaced stadard o the fourth-grade STAAR writig test. STATA treatmet effects (teffects) with regressio adjustmet (ra) showed that studets whose teachers did ot participate i the Writer s Workshop PD traiig had a average scale score (POmea) of 3823.9 while those whose teachers participated had a average scale score that was 104.8 scale score poits lower tha the POmea (3719.1). The differece was statistically sigificat (p<.001). Similarly, studets whose teachers completed the workshop received a lower umber of items correct o the expository compoet of the STAAR writig test compared to their peers whose teachers did ot participate i the workshop. Here too, the differece was statistically sigificat (p=.001). A possible explaatio for the differece i the STAAR performace betwee the treatmet ad o-treatmet groups may be the extet to which the Writer s Workshop was implemeted ad its key protocols were adhered to. More tha half of the teacher respodets devoted 30 miutes to istructio i writig, which is 15-30 miutes less tha the recommeded istructio time for deliverig Writer s Workshop. Subsequetly, most studets may have had isufficiet time to write, which could mea they may have received iadequate feedback ad isufficiet time to demostrate improvemets i their writig. This time disparity could have bee due to the departmetalizatio of istructio i those schools which resulted i reduced teachig time to 30-miute periods. Recommedatios Teacher completio of the professioal developmet desiged to ehace preparatio for effective delivery of the Writer s Workshop eeds to be ecouraged. The high cotract to participatio cost icurred i deliverig this professioal developmet ad the failure of teacher participats to cofidetly cofirm studets ability to outlie the writig process usig the Bomer s Flowchart makes the case for the ecouragemet. Studets ad teachers may require additioal resources to effectively teach writig at the elemetary school level. It may be essetial to esure that writig receives the same level of attetio at all elemetary grade levels i additio to the fourth grade where writig is tested. Future evaluatios should focus o observig the delivery of the Writer s Workshop, particularly before the testig period, ad possibly multiple times durig the school year to assess teachers actual writig istructioal practices. This would allow for a more thorough observatio of the Writer s Workshop ad a more robust assessmet of fidelity. Refereces Bach, A. J., & Supovitz, J. A. (2003). Teacher ad coach implemetatio of Writer's Workshop i America's Choice Schools, 2001 ad 2002, CPRE Research Reports. Retrieved from http://repository@pobox.upe.edu. HISD Research ad Accoutability 17