National Assessments of English Reading and Mathematics 2014 Performance Report Gerry Shiel, Lauren Kavanagh, David Millar Educational Research Centre
Earlier National Assessments Series of National Assessments since 1972 for various grade levels and subjects NA 09 standardization of grade levels Second and Sixth classes, for English reading and Mathematics NAIMS 10 National Assessments of English Reading and Mathematics in Irish-medium schools Evaluation of DEIS on separate track. 2
Two Reports on NA 14 January 2015 Performance Report with conclusions but no recommendations. No questionnaire data analysed. Autumn 2015 Context Report with conclusions and recommendations. To include analyses examining school, teacher, pupil and parent factors associated with performance as well as trends in non-achievement outcomes. 3
International Context Decline in performance on English reading and mathematics in PISA 2009; Improvement in PISA 2012, though some underlying issues persist. PIRLS 2011 in Fourth Grade Ireland 10 th of 45 countries; five countries, including N. Ireland, had significantly higher mean scores. 15% performed at or below low benchmark. TIMSS 2011 Ireland ranked 17 th of 48 countries, with 13 countries performing significantly higher; 9% performed at the advanced benchmark, compared with 24% in Northern Ireland, and 34% in Singapore, Korea and Hong Kong-China. 4
National Strategy Targets for 2020 Linked to National Assessments To reduce the proportion of pupils performing at or below level 1 on English reading and mathematics by 5% To increase the proportion performing at Levels 3-4 by 5% 5
Sample NA 14 Same sampling procedure as in NA 09 150 primary schools 130 vertical, 10 junior and 10 senior All selected schools participated DEIS schools selected in proportion to their representation in the population High pupil response rates 93-94%, with just 1% exempted Higher pupil response rates in NA 14 than in NA 09, where rates ranged from 91% to 93%. 6
Tests NA 14 Test frameworks same as in NA 09. Small number of items added to NA 14 tests, following pilot study in 2013, replacing released items. Multiple, rotated booklets at each class level to enable coverage of content and processes Secure tests used. 7
English Reading Second Class Mean Scale Scores English Reading NA 09 NA 14 d (effect size) Vocabulary 250 265 0.30 Comprehension 250 263 0.26 Overall Reading 250 264 0.29 8
English Reading Second Class, Performance by Proficiency Levels 50 45 40 35 30 25 20 15 10 5 0 45.5 40.0 35.0 32.8 35.0 30.0 30.0 30.0 21.6 Level 1 and Below Level 2 Levels 3 and 4 2009 2020 target 2014 9
English Reading Sixth Class, Mean Scores by Components and Overall NA 09 NA 14 d Vocabulary 250 265 0.31 Comprehension 250 261 0.21 Overall Reading 250 263 0.26 10
English Reading Sixth Class, Performance by Proficiency Levels 50 45 43.9 40 40.0 35 30 25 35.0 30.0 30.0 30.0 24.8 31.3 35.0 2009 2020 Target 20 2014 15 10 5 0 Level 1 and Below Level 2 Levels 3-4 11
Mathematics Second Class - Performance by Content Area and Overall NA 09 NA 14 d Number & Algebra 250 265 0.31 Shape & Space 250 259 0.18 Measures 250 262 0.24 Data 250 254 0.09 Overall 250 264 0.28 12
Mathematics Second Class Performance by Proficiency Level 50 47.2 45 40 40.0 35 35.0 35.0 30 25 30.0 30.0 30.0 25.6 27.3 2009 2020 target 20 2014 15 10 5 0 Level 1 and Below Level 2 Levels 3 and 4 13
Mathematics Sixth Class- Performance by Content Area and Overall NA 09 NA 14 d Number & Algebra 250 261 0.22 Shape & Space 250 263 0.27 Measures 250 259 0.17 Data 250 259 0.18 Overall 250 262 0.24 14
Mathematics Sixth Class Performance by Proficiency Levels 45 40 40.0 42.1 35 30 35.0 30.0 30.0 30.0 26.5 31.4 35.0 25 2009 20 2020 Target 2014 15 10 5 0 Level 1 and Below Level 2 Levels 3-4 15
Gender Differences in NA 14 Girls significantly higher than boys on overall reading at Second class (by 7 score points, compared with 14 points in NA 09) Girls higher than boys on overall reading at Sixth class (by 4 points), but difference not significant Boys significantly higher than girls on overall mathematics in Second class, by 5 points Boys higher than girls on overall mathematics at Sixth class (by 4 points), but difference not statistically significant. Boys significantly higher than girls on Measures, Data and Apply & Problem Solve at Second, and on Measures and Apply & Problem Solve at Sixth. 16
Disadvantage DEIS Band 1 Comparison of Mean Scores (NA 14 vs. NA 09) 14*-point increase on overall reading at Second class was statistically significant; effect size = 0.35 can be considered substantively important. 13-point increase on overall reading at Sixth class not significant; effect size = 0.29 13-point increase on maths at Second class not significant; effect size = 0.28 14-point increase on maths at Sixth class not significant; effect size = 0.29 17
Disadvantage DEIS Band 2 Comparison of Mean Scores (NA 14 vs. NA 09) 27*-point increase on overall reading at Second class was statistically significant; effect size (0.60) can be considered large. 14*-point increase on overall reading at Sixth class significant; effect size (0.29) is considered substantively important. 29*-point increase on maths at Second class significant; effect size (0.62) is large 10-point increase on maths at Sixth class not significant; effect size (0.21) is small. 18
Comparison of Mean Scores in DEIS Band 1 and 2 Schools and in All Schools (NA 14) 270 264 263 264 262 260 259 255 250 246 240 241 Band 1 Band 2 230 232 233 231 233 All Schools 220 210 2nd - Read 6th - Read 2nd - Math 6th - Math 19
Percent of Pupils at or below Level 1 in DEIS Band 1 and 2 Schools and in All Schools (NA 14) 60 50 40 43.9 47.3 38.2 51.9 49.9 41.8 30 28.3 21.6 26.5 24.8 25.6 26.5 Band 1 Band 2 All Schools 20 10 0 2nd - Read 6th - Read 2nd - Math 6th - Math 20
Conclusions Average scores are significantly higher for overall reading and mathematics at both Second and Sixth classes, representing the first time that increases in performance have been recorded in the National Assessment series since 1980. Forthcoming international assessments (TIMSS 2015, PIRLS 2016) may provide corroborating evidence. Targets for improved English reading and maths established in the National Strategy to Improve Literacy and Numeracy 2011-2020 have been reached. While performance in mathematics has improved, international assessments suggest it is from a lower base than for reading. There is further room for improvement on higherlevel mathematical processes. 21
Conclusions Care should be exercised in interpreting performance on current standardised tests used for school-based assessment, as these may not have factored in increases in performance. Gender differences on overall reading and mathematics are small, and are significant for reading (favouring girls) and mathematics (favouring boys) at Second class, but not at Sixth. Although there have been substantive improvements in both reading and mathematics in DEIS schools, large proportions of pupils in DEIS Band 1 schools continue to perform at low levels of proficiency. 22