Evaluaion Brief Ocober 2016 Office of Shared Accounabiliy Adminisraors' and Specialiss Experiences wih Supporing Implemenaion of he Monioring Insrucional Reading Levels Sraegy 2015 2016 Nyambura Susan Maina, Ph.D. Execuive Summary This brief, he second in a series of repors from he Monioring Insrucional Reading Levels (MIRL) evaluaion sudy, describes findings from spring 2016 surveys of elemenary school adminisraors and specialiss (saff developmen eachers and reading specialiss). The purpose of he surveys was o ge inpu from adminisraors and specialiss on heir experiences wih supporing he implemenaion of MIRL a heir schools in 2015 2016 and o idenify areas needing improvemen. analysis of monioring daa uncovered many insrucional needs paricularly in he areas of: wriing in response o reading, comprehension, vocabulary, fluency, accuracy, and learning English. he implemenaion of MIRL was augmened by professional developmen (PD) opporuniies provided a disric and school levels, suppor from reading specialiss and saff developmen eachers, he purposive and collaboraive planning a school and grade-levels, aligning MIRL o school improvemen goals, and he work of he classroom eachers. Key Findings Overall, survey responses from 130 adminisraors and 219 specialiss showed encouraging news following he implemenaion of MIRL in 2015 2016. The following experiences and changes were repored: processes and srucures o increase familiariy and consisen implemenaion of reading insrucion and MIRL were in place in majoriy of he schools. increases in monioring (observaion and documenaion) of insrucional reading levels during guided reading. high emphasis across schools on collaboraive planning, implemenaion of guided reading insrucion, and use of he monhly Reading Daa Collecion ool. greaer focus on meeing suden needs in reading comprehension and wriing in response o reading and as a resul, an increase in sudens criical hinking skills. changes in insrucional pracices: an increase in eachers awareness of sudens srenghs and needs, more small group insrucion, consisen use of Common Core Sae Sandards (CCSS) indicaors and increased use of monhly reading daa o guide insrucion. changes in suden performance: more visible changes in insrucional reading levels, faser progress, and seady growh in reading proficiency levels. The surveys also highlighed he following concerns: Uneven implemenaion of some componens of MIRL: sysems for using formaive assessmen daa and monioring he fideliy of implemenaion of MIRL were only parially implemened in mos of he schools. Opimal implemenaion of MIRL was hindered by he widespread percepion of MIRL as he even of monhly collecing and documening of reading daa, limied ime, insufficien insrucional resources for guided reading insrucion, and eachers varying experience wih analyzing and using formaive assessmen daa o adjus insrucion. Recommendaions Increase consisency in he undersanding of he MIRL sraegy by eachers and school leaders; clarify is raionale, and is relaionship o formaive assessmen and he Early Lieracy plan. Coninue o dispel he widespread percepion ha MIRL is an even by explaining he srong connecion of MIRL and MCPS s Prioriies for 2016-2017: Focus on Learning, Resuls, and Accounabiliy. Specifically, he expecaion o inensify focus on disric-wide daa-driven monioring and analysis of suden performance in order o increase opporuniies for sudens o learn Program Evaluaion MIRL Adminisraors and Specialiss Survey 2016
and o reduce variabiliy in oucomes across schools and classrooms wihin schools. Increase implemenaion of he MIRL pracices ha were implemened a low levels. For majoriy of schools, he processes for monhly review of formaive assessmen daa wih saff and assessing fideliy of implemenaion of MIRL were parially in place. Share bes pracices relaed o MIRL wihin and across schools: efficien and sysemaic ways o monior he reading daa, use of he noes column feaure on he monioring ool, daa analysis, and meaningfully inegraing he reading daa ino planning insrucion. Inensify PD suppor o classroom eachers in he areas of sudens greaes insrucional needs in reading: wriing in response o reading, comprehension, vocabulary, fluency, accuracy, and learning English. Coninue o offer a variey of ongoing PD sessions relaed o reading insrucion, assessmen as learning (formaive assessmen), and use of daa o guide insrucion. Updae and increase he supply of insrucional resources for guided reading insrucion. Increase clariy and undersanding of he similariies, differences, as well as he funcions of he various formaive and local reading assessmens daa: MIRL, MCPS AP-PR /mclass daa; and MAP-R, and provide examples of how each can be used o guide insrucion. 2014; MCPS had recenly insiued a variey of curriculum changes and iniiaives; and i was unclear o wha exen all schools kep up wih consisen monioring o suppor suden learning a all grade levels across he disric (MCPS, 2015b). Thus, in 2015 2016, he Monhly Reading Daa Collecion Tool was inroduced as a key componen of he MIRL sraegy, making i possible for eachers o use a common monioring ool o obain he daa necessary o suppor insrucion and make immediae adjusmens o improve reading insrucion. Evaluaion Quesions Analyses of survey daa was guided by he following evaluaion quesions: 1. To wha exen did schools esablish processes and srucures o suppor he planning, coordinaion, and implemenaion of MIRL? 2. Wha are changes in a) reading insrucion and b) sudens reading proficiency repored by school adminisraors and specialiss? 3. Wha facors faciliaed or hindered he implemenaion of MIRL? 4. Wha are he areas of insrucional needs in reading idenified by adminisraors and specialiss? 5. Wha changes would adminisraors make o srenghen he implemenaion of MIRL? 6. Wha are he professional developmen needs idenified for specialiss and eachers? Background Monioring Insrucional Reading Levels (MIRL) is a disric-wide sraegy being implemened in MCPS o improve reading insrucion and increase reading performance for K 5 sudens. The objecive of he MIRL sraegy is o increase sysemaic implemenaion of high qualiy guided reading, monhly documenaion of sudens insrucional reading levels, analysis and use of monhly reading daa for insrucional planning (Appendix A). Saring in he fall of 2015, all elemenary schools were expeced o implemen MIRL sraegy. The heory of change of MIRL is ha hrough emphasis on high qualiy guided reading and daa collecion during guided reading, and analyses of formaive reading daa, eachers can: a) diagnose a suden s immediae need and b) hen use he informaion o adjus insrucion (MCPS, 2015a). While many of he pracices of he MIRL sraegy have been par of he MCPS elemenary school curriculum for many years, he disric-wide implemenaion of MIRL in 2015 2016 was necessiaed by several facors: reading performance for K 5 sudens had declined from 2012 o Mehodology Daa collecion. Two web-based surveys one each for adminisraors and specialiss were conduced beween April 30 and May 10, 2016 using NoviSurvey Tools. The survey was developed by he Office of Shared Accounabiliy, in collaboraion wih saff from he Elemenary Inegraed Reading Curriculum (EIC), in he Office of Curriculum and Insrucional Programs (OCIP) o gaher daa regarding he implemenaion of MIRL pracices a he school level. An email conaining he survey link o he online surveys was sen o all elemenary school principals, requesing ha hey disribue he link o he saff developmen eachers (n=133) and reading specialiss (n=133) a heir schools. A he same ime, an online link o he adminisraor survey was disribued by OCIP saff a he May 5, 2016 Elemenary Principals Curriculum Updae meeing, inviing principals o paricipae in he adminisraor survey. Response rae. Nearly all adminisraors compleed he survey for a response rae of 98% (130 of 133). The Program Evaluaion 2 MIRL Adminisraor and Specialiss Survey 2016
Percne respondens Percen respondens response rae for he specialis survey was 82% (219 of 266). As inended, he respondens included principals, assisan principals, reading specialiss and saff developmen eachers. Informaion on characerisics of respondens is available in Appendix B. Response raes approximaing 60% for mos research should be he goal of researchers; for survey research inended o represen a seing adequaely, a response rae of 80% is expeced (Fowler, 2002; Nuly, 2008). These high response raes give assurance ha he samples of respondens are highly represenaive of he adminisraors and specialiss in MCPS elemenary schools. Daa analyses procedures. Descripive summary saisics were compued for he srucured iems on surveys. Informaion from he open-ended responses was reviewed, analyzed, and coded o summarize similar commens ino hemes. Resuls The findings are organized by evaluaion quesion; findings from he adminisraor s survey are repored firs followed by findings from he specialis survey. Q1. To wha Exen Did Schools Esablish Processes and Srucures o Suppor he Planning, Coordinaion, and Implemenaion of MIRL? Professional Developmen Toal hours PD for adminisraors. The majoriy of adminisraors repored ha hey received a oal of 10 25 (52%) or more hours (25%) of professional developmen (including summer) relaed o reading insrucion during he 2015 2016 school year (Figure 1). 60 50 40 30 20 10 0 19 Fewer han 10 hours 52 10-25 hours Figure 1. Toal hours of professional developmen repored by adminisraors (N=128). 21 26-50 hours 4 51-75 hours 76-100 hours The professional developmen plans for eachers and school leaders are deailed Appendix C. Toal hours PD for specialiss. The majoriy of specialiss received a oal of 10 25 hours (43%) or more han 25 hours (42%) of professional developmen (including summer) relaed o reading insrucion during he 2015 2016 school year (Figure 2). 45 40 35 30 25 20 15 10 5 0 15 Fewer han 10 hours 43 Figure 2. Toal hours of professional developmen repored by specialiss. (N=218) Professional developmen aended and faciliaed by specialiss. More han hree quarers of he specialiss who provided responses repored hey had aended hree of he four couny-wide PD sessions relaed o reading insrucion; (Table 1a). In addiion, he majoriy of specialiss (94%) repored hey had faciliaed he raining on Monioring Insrucional Reading Levels K 5 (Table 1b). Table 1a Number and Percen of Specialiss Reporing Specified Professional Developmen Sessions Aended (N=209) Aended n % Session Monioring Insrucional Reading Levels K 2 185 88.5 Small Group Reading Insrucion Grades 3-5 Tier 1 Summer Training (2014) 171 81.8 Monioring Insrucional Reading Levels K 5 162 77.5 Reflecing on Resuls and Nex Seps Module 3 52 24.9 25 11 10-25 hours 26-50 hours 51-75 hours 76-100 hours 6 Program Evaluaion 3 MIRL Adminisraor and Specialiss Survey 2016
Table 1b Number and Percen of Specialiss Reporing Specified Professional Developmen Sessions Faciliaed (N=143) Faciliaed N % Session Monioring Insrucional Reading Levels K 2 33 23.2 Small Group Reading Insrucion Grades 3 5 Tier 1 Summer Training (2014) 24 16.9 Monioring Insrucional Reading Levels K 5 134 94.4 Reflecing on Resuls and Nex Seps Module 3 42 29.6 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Saus of Implemenaion of MIRL Sage of implemenaion of MIRL repored by adminisraors. One survey iem asked principals o indicae he sage of implemenaion (in place, iniiaed or parially in place, no ye iniiaed) of specified aciviies and processes for planning, coordinaing, and assessing progress of MIRL a he school level. By spring 2016, he majoriy of adminisraors repored ha hey had pu in place mos of he specified processes relaed o collaboraive planning and professional developmen (85 89%) and processes for clarifying he vision and expecaions for reading insrucion (69 79%) (Table 2) The greaes variaion in implemenaion was repored for he processes relaed o monioring and use of daa for insrucional planning. More han 60% of he principals indicaed hey had pu in place: a) sysems for measuring and reporing reading oucomes (66%) and b) process for reviewing reading performance daa wih eachers (62%); however, he remaining 32% and 35% respecively indicaed ha hese wo processes were parially in place. Less han one half repored ha hey had pu in place: a process for a) monhly review of formaive assessmen daa wih saff (48%) and b) assessing fideliy of implemenaion of MIRL (46%); 38% and 50% respecively repored hese srucures were parially in place (Table 2). Table 2 Number and Percen of Adminisraors Reporing Saus of Implemenaion of MIRL Parially No Ye In Place in Place Iniiaed Processes/srucures n % n % n % Collaboraive Planning/professional developmen Learning progression plan for professional developmen (N=108) 96 88.9 10 9.3 2 1.9 Srucures for ongoing collaboraive curriculum sudy and planning wihin he week (N=109) 95 87.2 13 11.9 1 0.9 Maser schedules o allow eacher leaders and specialis o paricipae in collaboraive planning (N=108) 92 85.2 12 11.1 4 3.7 Clarifying vision and expecaion for reading insrucion Communicaion plans o inform saff of formaive assessmen vision, curriculum updaes, expecaions (N=106) 84 79.2 21 19.8 1 0.9 Wrien expecaions for insrucional program in reading (N=108) 78 72.2 24 22.2 6 5.6 Wrien expecaions for weekly planning (N=108) 75 69.4 25 23.1 8 7.4 Monioring and use of daa for insrucional planning Sysems for measuring and reporing reading oucomes (N=109) 71 65.7 35 32.4 2 1.9 Process for reviewing reading performance daa wih eachers (N=110) 67 62.0 38 35.2 5 4.7 Process for monhly review of formaive assessmen daa wih saff (N=108) 52 48.1 41 38.0 15 13.9 Sysems for assessing fideliy of implemenaion of Monioring Insrucional Reading Levels a school level (N=107) 49 45.8 53 49.5 5 4.7 Program Evaluaion 4 MIRL Adminisraor and Specialiss Survey 2016
Level of emphasis of specified srucures and processes across schools repored by specialiss. The specialiss were asked o rae he level of emphasis given o specified processes and aciviies across heir school. Table 3 Number and Percen of Specialiss Reporing Level of Emphasis of Specified Processes and Srucures To a To a To a Grea Moderae Small Exen Exen Exen n % n % n % Specified Aciviies Using Common Core Sae Sandards as basis for insrucional planning for all sudens (N=190) d 122 64.0 54 28.4 11 5.8 Differeniaing insrucion o mee needs of varied learners (N=188) a 120 63.8 56 29.8 11 5.9 Aligning reading insrucion wih CCSS indicaors o ensure rigor for all sudens (N=189) b 120 63.5 52 27.5 15 7.9 Implemening expecaions for balanced lieracy schedule (N=188) b 118 62.6 55 29.3 13 6.9 Emphasizing reading comprehension sraegies (N=189) 117 61.9 60 31.7 12 6.3 Building capaciy for successful collaboraive conen sudy and planning (N=188) d 105 55.9 66 35.1 13 6.9 Using consisen processes and ools o documen evidence of suden learning (N=188) b 102 54.3 71 37.8 13 6.9 Use of before, during, and afer srucure of guided reading srucure insrucion (N=186) b 97 52.2 72 38.7 15 8.1 Ariculaing he formaive assessmen vision for he school (N=186) f 86 46.2 59 31.7 32 17.2 Inerpreing formaive assessmen informaion (N=185) c 73 38.8 85 45.2 27 14.4 Selecing guided reading exs (N=188) e 64 35.2 69 37.9 49 26.9 Noe. The response No a all is no is no shown in he able. Responses for No a all are indicaed on each iem as follows: an b c d e f n=3; n=2, n=4; n=1; n=6; n=9. The majoriy of specialiss who provided responses indicaed ha heir school placed high emphasis in he following areas (Table 3): Using Common Core Sae Sandards (CCSS) as basis for insrucional planning for all sudens (64%) Differeniaing insrucion o mee needs of varied learners (64%) Aligning reading insrucion wih CCSS indicaors o ensure rigor for all sudens (64%) Implemening expecaions for balanced lieracy schedule (63%) Emphasizing reading comprehension sraegies (62%) Building capaciy for successful collaboraive conen sudy and planning (56%) Using consisen processes and ools o documen evidence of suden learning. (54%) Use of before, during, and afer srucure of guided reading srucure insrucion (52%) Less han half of he specialiss repored a high emphasis on: a) ariculaing he formaive assessmen vision for he school (46%) b) inerpreing formaive assessmen informaion (38%), and c) selecing guided reading exs (35%). Furher, 17 26% repored low emphasis of hese aciviies. Change in implemenaion of MIRL processes from 2014 2015 as repored by specialiss. When asked o indicae he exen o which hey engaged in specified aciviies o suppor he implemenaion of MIRL compared o he year before, he majoriy of he respondens indicaed increases during 2015 2016 in (Table 4): Monioring sudens' monhly reading level arges over ime (78%) Working wih school saff o define reading insrucional goals (70%) Ensuring consisen implemenaion of reading insrucion across grade levels (64%) Examining formaive assessmen reading daa (63%) Sharing and reviewing reading assessmen daa wih eachers (60%) Coaching eachers on a range of insrucional sraegies relaed o reading (56%) Leading grade level eam planning groups/meeings (51%) For hree of he specified aciviies, he proporion of specialiss reporing increases in heir involvemen in 2015 2016 was comparable o hose who repored no change: gahering resource maerials for reading insrucion (48% vs. 47%), providing ongoing follow-up and feedback o eachers (48% vs. 45%), and collaboraing wih eam leaders o provide job-embedded Program Evaluaion 5 MIRL Adminisraor and Specialiss Survey 2016
professional developmen opporuniies for eachers (48% vs. 46%). Table 4 Number and Percen of Specialiss Indicaing Change in Level of Engagemen in Specified Aciviies from 2014 2015 Grea exen Abou he same Less Exen Aciviy n % n % n % Monioring sudens' monhly reading level arges over ime (N=179) 140 78.2 32 17.9 7 3.9 Working wih school saff o define reading insrucional goals (N=179) 126 70.4 48 26.8 5 2.8 Ensuring consisen implemenaion of reading insrucion across grade levels (N=176) 112 63.6 55 31.3 9 5.1 Examining formaive assessmen reading daa (N=179) 112 62.6 57 31.8 10 5.6 Sharing and reviewing reading assessmen daa wih eachers (N=175) 105 60.0 62 35.4 8 4.6 Coaching eachers on a range of insrucional sraegies relaed o reading (N=176) 99 56.3 67 38.1 10 5.7 Leading grade level eam planning groups/meeings (N=177) 91 51.4 72 40.7 14 7.9 Gahering resource maerials for reading insrucion (N=178) 86 48.3 84 47.2 8 4.5 Providing ongoing follow-up and feedback o eachers (N=174) 84 48.3 79 45.4 11 6.3 Collaboraing wih eam leaders o provide jobembedded professional developmen opporuniies for eachers (N=178) 85 47.8 79 44.4 14 7.9 Noe. Valid N for each iem varies because of missing responses; Q2a. Wha are Changes in Reading Insrucion as Repored by School Adminisraors and Specialiss? Changes in insrucional pracices repored by adminisraors. Through responses elicied from openended iems, 84 adminisraors menioned several school level changes hey had observed in insrucional pracices a heir schools following he 2015 2016 implemenaion of MIRL (Table 5). The mos frequenly cied changes were: 1) increased collecion, ongoing examinaion, and use of monioring daa for insrucional planning (n=29, 35%), 2) increased eachers' awareness of performance levels and progress (n=24, 29%), 3) consisen collaboraive planning for guided reading insrucion during grade-level planning meeings (n=19, 23%), 4) increased emphasis and implemenaion of guided reading insrucion (n=18, 21%), and 5) increased schoolwide monioring of suden progress in reading and accounabiliy for suden learning wihin grade-levels and across he school (n=17, 20%). Table 5 Mos Frequenly Menioned Changes in Insrucional Pracices Repored by Adminisraors (N=84) Changes in insrucional pracices n % Increased collecion, examining and use of monioring daa 29 34.5 Increased eachers' awareness suden performance levels 24 28.6 Consisen and horough collaboraive planning for guided reading insrucion during grade-level planning meeings 19 22.6 Increased emphasis and implemenaion of qualiy guided reading 18 21.4 Increased monioring and accounabiliy for suden learning and achievemen in reading 17 20.2 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Changes in insrucional pracices repored by specialiss. The op changes in reading insrucion following he 2015 2016 implemenaion of MIRL menioned by he 160 specialiss who provided responses ha were similar o hose repored by adminisraors: 1) increased eachers awareness of monhly reading arges and heir sudens growh over ime (n=74, 47%) and, 2) more consisen collecion of reading daa and use of running records for insrucional decision making (n=71, 45%). In addiion, he specialiss repored insrucional changes ha were more specific o eacher s use of he monioring daa: a) increased maching of insrucion o suden needs (n=61, 39%), b) more focused planning wih use of indicaors (n=37, 24%), c) increased small group insrucion (20%), and d) increased emphasis on reading insrucion, paricularly reading comprehension sraegies (n=31, 20%) (Table 6). Program Evaluaion 6 MIRL Adminisraor and Specialiss Survey 2016
Table 6 Mos Frequenly Menioned Changes in Insrucional Pracices in 2015 2016 Repored by Specialiss (N=160) Changes n % Teachers are more aware of reading arges and sudens growh over ime 74 47.1 More consisen daa collecion/more eachers aking running records during insrucion/more use of daa running records in decision making 71 45.2 Insrucion more argeed o needs of suden because eachers are more awareness of suden s srenghs and needs 61 38.9 More focus planning; use of indicaors, and increase in collaboraive planning 37 23.6 Increase in small groups insrucion 32 20.4 Increased emphasis on schoolwide focus on reading insrucion; compression sraegies and increasing consisency 31 19.7 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Q2b. Wha are he Changes in Suden Reading Proficiency Repored by School Adminisraors and Specialiss? Changes in suden performance repored by adminisraors. In response o an open-ended iem, adminisraors menioned many changes hey had observed in sudens insrucional levels and performance in reading following he implemenaion of MIRL. By spring 2016, he majoriy of he 73 adminisraors who provided responses menioned ha hey had observed posiive changes in: sudens academic growh (n=39, 53%). The repored posiive changes included saemens like improved scores, increased comprehension, and faser pace in achieving proficiency. Abou a hird of he respondens were cauious and saed ha i was oo soon o ell if he implemenaion of MIRL could be associaed wih any significan change in suden performance (n=22, 30%). Lasly, some respondens also repored ha he monhly daa collecion and monioring of reading daa had uncovered suden needs in he areas of fluency and comprehension and ha heir eachers were increasingly adjusing insrucion o arge hese needs (n=8, 11%) Changes in suden performance repored by specialiss. More han 40% of he specialiss who provided responses menioned ha he monioring daa showed more sudens movemen and progress in reading insrucional levels, and ha some sudens had acceleraed heir progress in 2015 2016 compared o previous years. Specifically, by spring 2016, he mos frequen changes menioned by he 124 specialiss who provided responses were: They had observed greaer gains in suden fluency and reading comprehension, and ha because sudens were being challenged more, hey had grown in heir areas of need (n=55, 44%). They were no sure or hey had observed no changes ha could be aribued o MIRL (n=52, 42%). Sudens insrucional reading levels documened on he MIRL ool were more accurae and more fluid han in previous years (n=26, 21%). Q3a. Wha facors Faciliaed he Implemenaion of MIRL? Facors ha faciliaed he implemenaion of MIRL as repored by adminisraors. In response o an openended iem, he adminisraors repored ha he success of MIRL a heir school was brough abou primarily by he diligence of he specialiss; paricularly he saff developmen eachers and reading specialiss (n=32, 37%) who suppored eachers and ensured ha MIRL pracices were implemened wih fideliy (Table 7). Table 7 Facors ha Suppored Implemenaion of MIRL Repored by Adminisraors (N=89) Facors n % Specified saff members-reading specialis and Saff Developmen Teachers/some adminisraive saff a school and cenral office 32 36.8 Purposive planning and aligning MIRL o SIP/maching he planning o needs of sudens 30 34.5 Requiremen o monior MIRL/monhly deadlines/requiremen o assess 29 33.3 Professional developmen (in-school and MIRL counywide professional developmen) 25 28.7 Teachers of reading for effecive implemenaion of componens of MIRL 8 9.2 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Also noed as insrumenal o he implemenaion of MIRL were: he purposive planning and aligning of MIRL o heir school improvemen plans (SIPs) and maching insrucion o needs of sudens (n=30, 35%), adherence o he deadlines and requiremens o assess sudens and o ener daa monhly (n=29, 33%), in-school as well as couny wide PD sessions (n=25, 29%), and he work of he classroom eachers (n=8, 9%). Facors ha faciliaed he implemenaion of MIRL as repored by specialiss. The summer PD sessions, ongoing suppor a he school level by reading specialiss and saff developmen eachers, regular collaboraive planning sessions, a variey of insrucional resources such as raining modules and videos, and he srong emphasis on reading in heir school improvemen goals srenghened he implemenaion of MIRL. Among hese facors, more han one half of he specialiss who provided responses (N=139), menioned he disricwide PD summer raining and school-level PD faciliaed by Program Evaluaion 7 MIRL Adminisraor and Specialiss Survey 2016
reading specialiss a heir schools (n=71, 51%). More han one hird of he respondens menioned ha he regular collaboraive planning meeings (n=60, 43%) and he ongoing suppor and encouragemen from he reading specialiss were very helpful (n=42, 30%) (Table 8). Addiionally, a leas one fifh of he respondens menioned ha he classroom eachers follow hrough wih implemenaion of MIRL pracices (n=38, 27%), having a variey of insrucional resources (n=32, 23%), daa chas and using daa for planning insrucion (n=29, 21%) srenghened he implemenaion of MIRL. Table 8 Facors ha faciliaed Implemenaion of MIRL Repored by Specialiss (N=139) Facors n % Training provided by reading specialis in school raining and summer raining 71 51.1 Collaboraive planning regular opporuniies for collaboraing 60 43.2 Ongoing suppor from reading specialiss 42 30.2 Teachers-implemenaion of reading insrucion 38 27.3 Insrucional and oher resources (e.g., modules, subsiues for specialiss o allow ime for planning and raining), 32 23.0 Using daa for planning/daa chas 29 20.9 Consisen messages regarding expecaions for reading from Insrucional Core eams and ohers 27 19.4 Srong emphasis on reading and guided reading in SIP goals 24 17.3 Noe. The percenage of responses may exceed 100% because respondens provided more han one response Q3b. Wha Facors Hindered he Implemenaion of MIRL? Facors ha hindered implemenaion of MIRL as repored by adminisraors. When asked wheher hey encounered any challenges, limied ime was cied as a major challenge by he 87 adminisraors who provided responses (Table 9). As wih any new iniiaive, balancing ime for MIRL and oher insrucional demands in he maser schedule was he mos frequenly cied challenge for he adminisraors (n=41, 48%). Also, many adminisraors menioned he eachers percepion of MIRL as primarily an even of daa collecion and monhly monioring (n=30, 35%), which had caused push back from eachers; eachers did no readily see he value of MIRL o sudens or heir insrucion. The responding adminisraors also poined o he challenge of varying levels of eachers experience wih implemening some MIRL pracices, paricularly in he areas of daa analysis (n=29, 34%). The respondens conveyed ha analysis of MIRL daa and using hese daa o provide more focused insrucion based on suden needs was differen from using jus MAP_R or mclass daa. Some of he adminisraors also menioned ha having insufficien insrucional resources for guided reading (n=19, 22%) was a problem. Finally, 15% (n=13) repored having ha limied ime and resources o adequaely address he needs of English language learners (ELLs) and sudens receiving special educaion services (SPED) was a challenge. Table 9 Facors ha Hindered Opimal Implemenaion of MIRL Repored by Adminisraors (N=87) Facors n % Sufficien ime managing MIRL insrucional focus alongside oher insrucional demand 41 47.7 seing prioriies/ime Percepion ha MIRL is an even/eacher resisance/limied picure of MIRL 30 34.9 Varying experiences and compeency of Teachers; Implemenaion exposed gaps in capaciy/capabiliy o implemen componens 29 33.7 (e.g., daa analysis highlighing grea need for PD) Insufficien resources especially leveled books and funds for subsiues o release 19 22.1 eachers and specialiss for PD Addressing needs of ELLs and SPED wih limied resources and ime 13 15.1 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Facors ha hindered opimal implemenaion of MIRL as repored by specialiss. For he specialiss, he mos frequenly menioned barriers o opimal implemenaion of MIRL in 2015 2016 were finding ime o plan and complee raining modules (n=126, 79%) and he eachers negaive percepion of MIRL resisance o he monhly collecion and mooring of reading daa (n=121, 76%). The specialiss poined ou ha some eachers fel ha MIRL (monhly daa collecion and enry on he monhly daa collecion ool) was an add-on even and ha hey did no need MIRL o provide effecive insrucion in reading (Table 10). Aside from limied ime o plan and eacher resisance, specialiss also menioned ha ime o analyze, inerpre and use he reading daa for insrucional planning was very limied or insufficien (n=75, 47%). Because of oher professional responsibiliies required of heir posiions, many specialiss expressed he need for release and subsiue ime o cover heir oher duies while hey aended o MIRL or faciliaed raining for MIRL. Addiional issues menioned by abou a hird of he respondens included: insufficien resources for guided reading (n=58, 36%) and he need o address a variey of suden needs (he mos pressing being working in large classes), and he difficuly of daa collecion during Program Evaluaion 8 MIRL Adminisraor and Specialiss Survey 2016
Percenage of Responses guided reading in upper grades (n=35, 22%). The specialiss indicaed ha some eachers found conducing running reading records very ime consuming, especially for sudens in Grades 3 5 (Table 10). 90 80 76 85 85 82 Table 10 Facors ha Hindered Opimal Implemenaion of MIRL Repored by Specialiss (N=160) Facors n % Finding ime o plan and complee raining modules-ime for PD 126 78.8 Teachers perceived MIRL as he even of daa collecion as an add on and -no needed 121 75.6 Time o use daa; analyze and inerpre daa and monioring daa inpu 75 46.9 Insufficien resources insrucional modules and subsiue ime 58 36.3 Sudens wih many and varying needs of sudens/working in large classes, MIRL daa collecion in upper grades 35 21.9 Noe. The percenage of responses may exceed 100% because respondens provided more han one response Q4. Wha are he Areas of Insrucional Needs Idenified by Adminisraors and Specialiss? Areas of insrucional needs repored by adminisraors. When asked o choose he mos criical areas of insrucional needs of sudens in reading, he adminisraors idenified: wriing in response o reading (n=82, 78%), comprehension (n=75, 72%), learning English (n=60, 58%, vocabulary (n=50, 48%), fluency (n=25, 24%), and accuracy (n=21, 20%) (Table 11). Table 11 Number and Percen of Adminisraors Idenifying Areas of Insrucional Needs in Reading (N=104) Areas of need n % Wriing in response o reading 82 78.8 Comprehension 75 72.1 Learning English language 60 57.7 Vocabulary 50 48.1 Fluency (reading accuraely wih rae and expression) 25 24.0 Accuracy (decoding words and idenifying words) 21 20.2 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Areas of insrucional needs repored by specialiss. When daa were disaggregaed by reading specialiss and saff developmen eachers, similar proporions of saff developmen eachers and reading specialiss (85%) idenified wriing in response o reading as he greaes need (Figure 3). 70 60 50 40 30 20 10 0 32 20 39 Reading specialiss (N=72) 42 38 39 17 13 Saff developmen eachers (N=87) Accuracy (decoding words and idenifying words) Fluency (reading accuraely wih rae and expression) Vocabulary Comprehension Wriing in response o reading Learning English language Figure 3. Areas of suden needs in reading idenified by specialiss Despie he sligh difference; more han hree quarers of he saff developmen eachers (82%) and reading specialiss (76%) idenified comprehension as an area of need. Compared o reading specialiss, lower proporion of saff developmen eachers idenified accuracy (32% vs. 17%) and fluency (20% vs. 13%) as criical needs for sudens. Q5a. Wha Changes Would Adminisraors Make o Srenghen he Implemenaion of MIRL? Suggesions from adminisraors for srenghening he implemenaion of MIRL. A he conclusion of he survey, adminisraors were asked for suggesions o srenghen he implemenaion of MIRL and for making he elemenary reading program more effecive. Close o one half of he adminisraors who provided responses suggesed focusing PD raining around effecive pracices for MIRL (n=35, 48%). In ha regard, many menioned ha sharing bes pracices would build on he Program Evaluaion 9 MIRL Adminisraor and Specialiss Survey 2016
improvemens from 2015 2016 and also enable schools o coninue making progress oward full implemenaion (Table 12). The adminisraors also menioned ha communicaing consisen messages abou he vision of MIRL and expecaions for reading insrucion, including dispelling he incomplee percepion ha MIRL as an even (n=23, 32%); providing ongoing PD raining, paricularly for new eachers and increasing undersanding of sraegies for working wih ELLs and sudens reading below grade-level (n=23, 32%); as well as ensuring schools have sufficien insrucional resources for guided reading insrucion for all grade levels, (n=9, 12%) would srenghen he implemenaion of MIRL. Table 12 Suggesions for Srenghening Implemenaion of MIRL Repored by Adminisraors (N=73) Suggesions for srenghening MIRL n % Focus on raining around effecive pracices o make MIRL an esablished pracice/share bes pracices 35 47.9 Coninue consisen communicaion of expecaions for saff/ Coninue o ariculae vision of MIRL and clarify expecaions for reading insrucion 23 31.5 Coninue ongoing PD and pracice for eachers new eachers, guided reading insrucion and daa collecion, comprehension sraegies for ELLs and sruggling readers 23 31.5 Provide addiional/sufficien resources (funds for subsiue ime and insrucional resources for guided reading) 9 12.3 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Suggesions for making he elemenary reading program more effecive provided by adminisraors. When asked wha changes hey would make o he elemenary reading program o make i more effecive, nearly all he suggesions (N=46) for srenghening he elemenary reading program were similar o he suggesions for srenghening MIRL. They included provide ongoing professional developmen (n=20, 44%), clarify he scope and purpose of MIRL (n=15, 33%), and provide addiional, updaed, and sufficien insrucional resources (n=14, 30%). Suggesions from adminisraors for fuure professional developmen sessions and suppor for school saff. Furher, in heir responses o ways o srenghen MIRL and he elemenary reading program, he adminisraors also recommended several opics and aciviies for fuure PD sessions: Professional developmen sessions for classroom eachers o insrucion on wriing in response o ex o vocabulary sraegies paricularly for ESOL sudens o more focus on daa and specifically how o use he daa for effecive insrucion o how o analyze running reading records; wih guidance or follow-up insrucional sraegies for inerpreing and using he daa o how o meaningfully inegrae MIRL daa ino planning insrucion Insiue ways for sharing of posiive experiences wih oher schools regarding how o analyze and reac o he formaive assessmen daa. One suggesion was o have cluser summis for eachers o mee and look a informaion wih eachers from oher nearby schools and wih cenral office expers o delve deeply ino ways o apply he daa. Clarifying issues surrounding daa collecion wih a focus on bes pracices for a) monioring he reading daa and b) inended use of he insrucional noes column feaure on he monioring ool. Increasing consisency in he reading daa colleced wihin and across schools. Many adminisraors suggesed providing a bank of comprehension quesions ha eachers could use. Q6. Wha are he Professional Developmen Needs Idenified for Specialiss and Teachers? Recommendaion for Fuure PD sessions from Specialiss. Through an open-ended survey iem, he specialiss were asked for suggesions for fuure PD for specialiss and eachers. More han one half of he specialiss who provided responses suggesed ha any fuure PD raining: a) be offered o all specialiss, including he media specialiss and b) ha i be similar or aligned wih PD for eachers (n=40, 55%). Addiional suggesions were: sraegies for specialiss o suppor eachers and keep eachers moivaed (n=31, 43%) and sraegies for differeniaing insrucion by maching insrucion o specific srenghs and needs of heir suden (n=14, 19%) (Table 13). An addiional suggesion brough up as a means of making he reading program more effecive was o increase he focus on wriing (n=6, 12%). Program Evaluaion 10 MIRL Adminisraor and Specialiss Survey 2016
Table 13 Suggesions for Fuure Professional Developmen for Specialiss (N=73) Suggesions for fuure PD n % Align PD for specialiss wih PD for eachers 40 54.8 Ways for specialiss o suppor eachers and keep hem moivaed; coaching; sraegies for specific grade-levels, insrucional resources 31 42.5 Differeniaing insrucion wih specific focus on individual suden needs idenified hrough MIRL and oher daa 14 19.2 Noe. The percenage of responses may exceed 100% because respondens provided more han one response. Through an open-ended survey iem, 106 specialiss provided many suggesions for PD aciviies, opics, and resources for classroom eachers. These suggesions were caegorized ino four general areas (Table 14): Table 14 Ideas for Fuure Professional Developmen for Classroom Teachers (N=106) Suggesions for fuure PD n % Focus on reading comprehension sraegies 83 78.0 Problem solving/differeniaing insrucion 39 37.0 More and variey of insrucional resources 19 18.0 Training on analysis, inerpreaion, and 17 16.0 use of reading daa Noe. The percenage of responses may exceed 100% because Suggesions for PD on reading comprehension sraegies included he following: In-deph undersanding of comprehension sraegies hemselves and heir use; including guided wriing as a follow up o guided reading. Deeper undersanding of he CCSS indicaors: more models of planning for small group insrucion using he indicaors in planning meeings. Previously, he focus for reading insrucion has been whole group planning and now eachers are sruggling o flip o small group. Wriing in response o reading and a deeper undersanding of sraegies for inegraing reading/wriing/speaking/lisening. For he caegory of problem solving and differeniaing insrucion (n=39, 37%). The following were recommended: Sraegies for using daa o mach insrucion o needs of various learners srong readers, ELLS, SPED, sruggling readers; addressing unique needs and srenghs of heir sudens. Managing various readers in he class (e.g., ESOL and sudens wih disabiliies); more analysis of language-relaed reading problems, and how o scaffold insrucion for hese learners. Wih regard o more and variey of insrucional resources (n=19, 18%), specialiss recommended he following: Providing eachers wih a supply of common formaive assessmens in reading for K 5. Making available a sample guided reading lesson emplaes, resources illusraing use of comprehension sraegies o ge o indicaors and showing how MIRL daa collecion was embedded in a lesson; and more videos of wha i looks like o collec daa during regular guided reading group insrucion. Modeling how he daa colleced in MIRL influences eachers' decision making for a variey of suden needs. The las caegory included raining on analysis, inerpreaion, and use of reading daa (n=17, 16%). How o inerpre running reading records. More PD on he "now wha" and how o use MIRL daa a each level--individually, as a eam, and school wide or school leadership. A beer undersanding of how o ake anecdoal insrucional noes and how o use insrucional noes o plan for insrucion. Discussion Overall, he responses from he adminisraor and specialis surveys showed ha schools made progress in he implemenaion of mos of he pracices of he MIRL sraegy in he 2015 2016 school year. The daa revealed ha mos processes for clarifying he expecaions for reading insrucion were in place. The daa from adminisraors also revealed low or uneven implemenaion of sysems o review formaive daa and sysems for monioring he fideliy of implemenaion of MIRL a he school level. Uneven or low implemenaion was also repored by specialiss in he areas of providing ongoing follow-up and feedback o eachers and collaboraing wih eam leaders o provide ongoing job embedded PD. The responses o open-ended iems indicaed ha school adminisraors and specialiss had observed many posiive changes in insrucional pracices and in suden performance. A he same ime, adminisraors as well as specialiss repored he widespread percepion among eachers ha MIRL was an even for collecing monhly reading daa. This percepion and limied ime o complee PD or o analyze and use reading daa were repored as key challenges o he opimal implemenaion of MIRL. Program Evaluaion 11 MIRL Adminisraor and Specialiss Survey 2016
The daa revealed ha less han one half of he adminisraors repored hey had esablished sysems for review of MIRL daa wih leadership eams or wih school saff in a sysemaic fashion. Also conveyed hrough open-ended responses was ha he reading insrucional levels from MIRL daa and mclass levels didn mach or align; indicaing a need o clarify he purposes and uses of he various formaive and local reading assessmens. Because one of he major goals of MIRL o increase use of hese daa of formaive assessmen, hese findings poin o he need o increase undersanding of formaive assessmen as learning and increased use of formaive assessmen daa as inended by he eachers. As such, having he monioring daa coupled wih effors o increase undersanding of formaive assessmen will increase changes in insrucional pracices as inended hrough MIRL. Researchers sugges ha PD, wih 30 o 100 conac hours dedicaed o formaive assessmens is needed o significanly impac a) consisen of use of formaive assessmens and b) suden learning (Darling- Hammond e al, 2009; Wylie & Long, 2009). Finally, i is worh noing ha he resuls from he adminisraor and specialis surveys fully corroboraed he findings from he classroom eacher survey (Maina & Wolanin, 2016). As such, he wo sources reliably porray he experience of sakeholders wih implemenaion of MIRL during he 2015 2016 school year; and provide evidence of early impacs of MIRL on insrucional pracices and suden performance in reading. schools, he processes a) monhly review of formaive assessmen daa wih saff and b) assessing fideliy of implemenaion of MIRL were parially in place. Share bes pracices relaed o MIRL wihin and across schools: :efficien and sysemaic ways o monior he reading daa, use of he noes column feaure on he monioring ool, analysis, inerpreaion, and use of reading daa for planning insrucion. Inensify PD suppor o classroom eachers in he areas of sudens greaes insrucional needs in reading: wriing in response o reading, comprehension, vocabulary, fluency, accuracy, and learning English. Coninue o offer a variey of ongoing PD sessions relaed o reading insrucion, assessmen as learning (formaive assessmen), and use of daa o guide insrucion. Updae and increase he supply of insrucional resources for guided reading insrucion Increase clariy and undersanding of he similariies, differences, as well as he funcions of he various formaive and local reading assessmens daa: MIRL, MCPS AP-PR /mclass daa; and MAP-R, and provide examples of how each can be used o guide insrucion. Recommendaions The following recommendaions are based on adminisraors and specialiss survey responses: Increase consisency in he undersanding of he MIRL sraegy by eachers and school leaders; clarify is raionale, and is relaionship o formaive assessmen and he Early Lieracy plan. Coninue o dispel he widespread percepion ha MIRL is an even by explaining he srong connecion of MIRL and MCPS s Prioriies for 2016-2017: Focus on Learning, Resuls, and Accounabiliy. Specifically, he expecaion o inensify focus on disric-wide daa-driven monioring and analysis of suden performance in order o increase opporuniies for sudens o learn and o reduce variabiliy in oucomes across schools and classrooms wihin schools. Increase implemenaion of he MIRL pracices ha were implemened a low levels. For majoriy of Program Evaluaion 12 MIRL Adminisraor and Specialiss Survey 2016
Acknowledgemens The auhor appreciaes he valuable feedback from Dr. Shahpar Modarresi, Ms. Julie Wade, and Ms. Naalie Wolanin of he Program Evaluaion (PE) Uni; The appreciaion is exended o Ms. Laura Evans, Supervisor, Elemenary Inegraed Curriculum (EIC) Reading for he collaboraion and suppor in he planning of he MIRL sudy and developmen of he daa collecion insrumens; elemenary school principals, reading specialiss, and saff developmen eachers in he 133 schools; and Ms. Rachel Hanson Summer Inern, Applied Research Uni, for helping wih conen analysis of he open-ended iems; and Ms. Maria Allendes, of PE Uni for logisic suppor.. References Fowler, F. J. (2002). Survey Research Mehod (3rd ed.). Thousand Oaks, CA: Sage Publicaions. Darling-Hammond, L., Wei, R. C., Andree, A., Richardson, N., Orphanos, S. (2009). Professional learning in he learning profession: A Saus Repor on Teacher Developmen in he Unied Saes and Abroad. Published by he Naional Saff Developmen Council and he School Redesign Nework a Sanford Universiy. MCPS (2014a). Elemenary school reading pilo: Memo o principals. Rockville, MD. Mongomery Couny Public Schools. MCPS (2015a). 2015 2016 MCPS Early Lieracy Plan. Rockville, MD. Mongomery Couny Public Schools. MCPS (2015b). Disricwide Assessmen Sraegy: Elemenary Assessmens. Rerieved Sepember 8, 2015 from hp://www.boarddocs.com/mabe/mcpsmd/ Board.nsf/files/ 9ZX24L76095D/$file/Policy% 20IKA%20Tenaive%20Acion.pdf. and hp://www.boarddocs.com/mabe/mcpsmd/boa rd.nsf/files/9y733r706c81/$file/assessmen%20srae gy.pdf MCPS (2015c). Elemenary Principal Curriculum Updae Meeing April 30 and May 1, 2015. Rockville, MD. Mongomery Couny Public Schools. MCPS (2015d). Elemenary Principal Curriculum Updae Meeing Ocober 30 and May 1, 2015. Rockville, MD. Mongomery Couny Public Schools. MCPS (2015e). Suden Performance on he 2015 Assessmen Program in Primary Reading (Kindergaren o Grade 2). Rockville, MD. Mongomery Couny Public Schools. hp://mongomeryschoolsmd.org/deparmens/sharedacc ounabiliy/repors/2015/2015%20ap%20pr%20k_12_ %20for%20Principals_revised_Nov_30.pdf. Nuly, N, D. (2008). The adequacy of response raes o online and paper surveys: wha can be done? Assessmen & Evaluaion in Higher Educaion Vol. 33, No. 3, June 2008, 301 314. Wylie, E. C., & Lyon, C. J. (2009, April). How much is enough: Wha is needed for a Disric o ake on he Formaive Assessmen Challenge? In annual meeing of he American Educaional Research Associaion, San Diego, CA. Program Evaluaion 13 MIRL Adminisraor and Specialiss Survey 2016
This page was inenionally lef blank. Program Evaluaion 14 MIRL Adminisraor and Specialiss Survey 2016
Appendix A MIRL: Aciviies, Expeced Resuls, and Anicipaed Oucomes for 2015 2016 Needs and Issues (Raionale for MIRL) Inpus Oupus/Resuls Oucomes (Resources and Srucures Insiued) Aciviies Paricipaion Merics Expeced Shor Term Changes Expeced Lasing Changes Decreasing performance in reading (K 5) as measured by MCPS AP-PR Limied observaion, documenaion, analysis, and use of formaive daa monioring) of reading performance/of reading levels a specified inervals hroughou he year Professional developmen (PD) sessions 2015 2016 MCPS Elemenary Lieracy Plan Online Monhly reading daa collecion ool (OCTO) School level common eam planning srucures Monhly principal curriculum updaes Elemenary Lieracy Insrucional Core Team (ICT) Clarify and ariculae Elemenary Lieracy Plan and vision for reading insrucion a school level Faciliae ongoing PD o school saff reflecing on resuls and bes pracices Regularly assess and documen reading levels during guided reading Inroducion of Monhly Reading Daa Collecion Tool Discuss reading daa a regular inervals (monhly collaboraive eams and principal curriculum updaes) Ongoing sraegic use of formaive reading daa o adjus insrucion PD sessions and Modules /Topics covered during PD for eachers Number and frequency of PD sessions for school leaders % Teachers aending PD % Adminisraors aending PD Types of srucures and processes in place a school level and who is involved Exen of use of Monhly reading daa collecion ool/ periodic online repors Frequency and srucure of school level eam meeings relaed o use of formaive reading daa o plan insrucion Frequency and aendance a principal s curriculum updae meeings Increased familiariy of eachers and school leaders wih lieracy plan, daa collecion ools, and monioring of reading performance Iniiaing and formalizing processes and srucures for collecing, enering, and using reading daa Increased monioring (observaion and documenaion) of insrucional levels during guided reading Consisen use of Monhly Reading Daa Collecion Tool o documen insrucional reading levels Ongoing coordinaed analyses and use of reading daa o inform insrucional pracices and suppor suden learning Increased use of monioring daa o adjus insrucion Esablished use of monioring of insrucional levels during guided reading levels for all K 5 sudens Improved reading performance for all K 5 sudens Progress oward reducing achievemen gaps. Figure 1. Logic Model for MIRL 2015 2016 Program Evaluaion 15 MIRL Adminisraor and Specialiss Survey 2016