How Big are the Gaps? Emily Knowles Education Standards Analysis and Research PLUG Conference 6th March 2012
Outline Background to the PISA study Summary of pupil attainment in England Gap between average pupil attainment in England and that in top-performing countries Socio-attainment gaps More possibilities for analysis using the matched PISA-NPD file.
Background to PISA Programme for International Student Assessment Aims to compare the abilities of 15-year old pupils across participating OECD member states and partner countries; Consists of a triennial assessment of reading, mathematics and science ability and a student questionnaire to provide contextual information; 2000 2003 2006 2009 2012 Focus: Participating countries: 43 41 57 65 68
Reading attainment in England very similar to the OECD average Distribution of pupil PISA scores in the reading strand of the PISA assessment Pupil attainment in England Average points score = 500 Standard deviation = 100 0 100 200 300 400 500 600 700 800 900 1,000 pupils' scores
And the distribution of scores is similar to those in the countries scoring significantly higher than England in the reading assessment Box plots to show the distribution of pupils' scores in the 2009 PISA reading assessment for countries where pupils performed significantly higher on average than pupils in England 700 660 620 580 540 500 460 OECD 420 average 380 340 300 260 220 180 140 100 PISA points Key Pupils making the most progress in each country Scores obtained by the middle 50% of pupils in each Pupils making the least progress in each co ntr Shanghai- China Korea Finland Hong Kong- China Singapore Japan Canada Countries whose average pupil score in the reading strand of was significantly higher than in England New Zealand Australia Belgium Netherlands Iceland Norway OECD average England Source: OECD, Database
The gap in average pupil attainment in the reading assessment between England and 13 other participating countries was statistically significant Figure 2: Gap between pupils' average scores in the reading assessment in England and the top performing countries in the reading strand, Comparison country Shanghai-China Korea Finland Hong Kong-China Singapore Canada New Zealand Japan Australia Netherlands Belgium Norway Estonia Switzerland Iceland Liechtenstein Germany Chinese Taipei Denmark Key The dark blue bars show gaps which are statistically significant. The light blue bars show gaps which are not statistically significant. 0 20 40 60 80 100 120 Difference in average reading score between comparison country and England (in PISA points) Source: OECD, Database
Rather than express the gap in terms of PISA points, we translated the gap into attainment measures we are familiar with nationally Difference in attainment...in PISA points as an Effect size in KS4 capped point scores in GCSE grades in additional % pupils achieving 5 A*-C (inc. English and maths) in years progress Shanghai-China 62 0.6 66 11 22% 1.5 Korea 45 0.5 48 8 18% 1.1 Finland 42 0.4 44 7 17% 1.0 Hong Kong-China 39 0.4 42 7 16% 0.9 Singapore 32 0.3 34 6 14% 0.7 Canada 30 0.3 32 5 13% 0.7 New Zealand 27 0.3 28 5 12% 0.6 Japan 26 0.3 27 5 11% 0.6 Australia 21 0.2 22 4 10% 0.5 Netherlands 14 0.1 15 3 7% 0.3 Belgium 12 0.1 13 2 6% 0.3 Norway 9 0.1 10 2 5% 0.2 Iceland 6 0.1 7 1 3% 0.1 1. Countries listed in bold are OECD member states Source: OECD, Database & additional DfE analysis (shaded sections)
We have looked at raw attainment gaps, how do our social attainment gaps compare? Proportion of group who achieve the target 100 90 80 70 60 50 40 National performance of pupils attaining various threshold measures at the end of KS4, by FSM status England 2009/10 Overall Non-FSM FSM 5 A*-C grades including English and maths 5 A*-C grades 5 A*-G grades Any qualification 30 20 10 Ebacc achievements Ebacc Entries 0 0 10 20 30 40 50 60 70 80 90 100 Proportion of pupils who achieve the target Source: GCSE and Equivalent Attainment by Pupil Characteristics in England, 2009/1
Measure of socio-economic background in PISA Distribution of pupils' socio-economic backgrounds across OECD countries To what extent does socio-economic background play a role in pupil attainment? socioeconomically disadvantaged background socioeconomically advantaged background Index of economic, social and cultural status (ESCS) derived from: * parents education; * parents occupation; * home possessions. -3-2 -1 0 1 2 3 Index of economic, social and cultural status (ESCS) Distribution of pupils' socio-economic backgrounds in England socioeconomically disadvantaged background socioeconomically advantaged background -3-2 -1 0 1 2 3 Index of economic, social and cultural status (ESCS) Source: OECD, Database
Association between pupil ESCS and their attainment in England Scatter plot to show the association between pupils economic, social and cultural status and their attainment in the reading assessment Key o pupil in England sample orange lines show average scores across OECD countries black line shows the socioeconomic gradient in England D Source: OECD, database
Socio-economic gradient in England Average pupil performance in reading, by national quarters of the index of economic, social and cultural status in England, 600 Average pupil attainment in the reading assessment (PISA points) 580 560 540 520 500 480 460 440 420 Most disadvantaged 25% England 400-2.5-2.0-1.5-1.0-0.5 0.0 0.5 1.0 1.5 2.0 2.5 Index of economic, social and cultural status (ESCS) Least disadvantaged 25% Key X----------------- -------------------- ------------------X First quarter, second quarter, third quarter, fourth quarter Source: OECD, database Slope Steeper than OECD average Strength of association Attainment not more strongly associated with socioeconomic background than on average across OECD
Comparison of socio-economic gradient in England with that of top-performing countries Average pupil performance on the reading scale by national quarters of economic, social and cultural status for a selection of top-performing countries, 600 Average pupil attainment in the reading assessment (PISA points) 580 560 540 520 500 480 460 440 420 Shanghai-China Hong Kong - China Singapore Korea New Zealand OECD average Belgium England Finland Canada Key X----------------- -------------------- ------------------X First quarter, second quarter, third quarter, fourth quarter 400-2.5-2.0-1.5-1.0-0.5 0.0 0.5 1.0 1.5 2.0 2.5 Economic, Social and Cultural Status Source: OECD, database Four countries scoring statistically significantly higher than England in the reading assessment are not included in the chart as the patterns in these countries were very similar to others that have been included. Namely: Australia was very similar to Belgium; Japan similar to Finland; The Netherlands very similar to the OECD average; and Norway very similar to England
Comparison of social attainment gaps between participating countries Average pupil attainment by national quarters of the PISA index of economic, social and cultural status, ordered by size of attainment gap between top and bottom quartiles Comparison country 1 Standard deviation in pupil performance (A) Performance on the reading scale, by national quarters of the PISA index of economic, social and cultural status Bottom quarter Second quarter Third quarter Top quarter Gap between average performance of pupils in top and bottom quarters Difference expressed in Standard Deviations (C - B) / A Mean score (B) S.E. Mean score S.E. Mean score S.E. Mean score (C) S.E. Difference in mean score (C-B) Peru 98 303 (3.3) 354 (3.1) 390 (3.7) 434 (7.3) 132 1.3 Hungary 90 435 (5.3) 485 (3.4) 505 (4.1) 553 (4.1) 118 1.3 Uruguay 99 374 (3.2) 407 (3.2) 438 (3.8) 489 (4.1) 116 1.2 Bulgaria 113 368 (5.8) 418 (6.6) 442 (7.7) 498 (6.2) 131 1.2 Argentina 108 345 (4.9) 377 (4.6) 410 (5.5) 468 (6.2) 123 1.1 Belgium 102 452 (3.3) 489 (3.3) 525 (2.5) 567 (2.6) 115 1.1 Turkey 82 422 (3.8) 454 (3.5) 469 (3.9) 514 (4.6) 92 1.1 Chile 83 409 (3.5) 435 (3.6) 457 (3.5) 501 (3.5) 92 1.1 Panama 99 336 (7.1) 351 (6.6) 375 (6.3) 445 (10.9) 110 1.1 United States 97 451 (3.6) 481 (3.6) 512 (3.6) 558 (4.7) 107 1.1 Luxembourg 104 411 (2.7) 460 (3.0) 497 (2.8) 526 (3.0) 115 1.1 Germany 95 445 (3.9) 494 (2.9) 515 (3.5) 550 (3.3) 104 1.1 Austria 100 421 (4.3) 457 (4.2) 482 (3.8) 525 (3.9) 105 1.0 Colombia 87 371 (4.7) 398 (4.4) 422 (3.9) 462 (4.7) 90 1.0 New Zealand 103 475 (3.9) 508 (3.1) 534 (3.3) 578 (3.6) 104 1.0 Singapore 97 477 (2.4) 513 (3.2) 541 (2.4) 575 (3.0) 98 1.0 Portugal 87 451 (4.2) 472 (3.4) 499 (3.4) 537 (3.7) 87 1.0 Switzerland 93 457 (3.9) 492 (2.7) 506 (3.0) 550 (3.7) 93 1.0 Mexico 85 386 (2.8) 413 (2.3) 434 (2.2) 469 (2.2) 84 1.0 Poland 89 461 (3.4) 488 (3.1) 507 (2.9) 550 (3.8) 88 1.0 England 95 451 (3.4) 483 (3.7) 510 (3.4) 544 (3.8) 93 1.0
Comparison of social attainment gaps between participating countries Difference in average pupil attainment at top and bottom quartile of pupil socio-economic background A comparison of the range of socio-economic background and the social attainment gap between participating ti 140 120 100 80 60 40 20 0 Small socio-economic gap Large attainment gap England Japan Norway Australia Small socio-economic gap Small attainment gap New Zealand Czech Republic Finland Singapore Netherlands Korea Canada Macao-China OECD average Iceland Bulgaria Belgium Large socio-economic gap Large attainment gap Shanghai-China Hong Kong-China Indonesia Large socio-economic gap Small attainment gap Argentina Thailand Uruguay OECD average 1.6 1.8 2.0 2.2 2.4 2.6 2.8 3.0 3.2 3.4 3.6 Difference between top and bottom quartile average pupil score on the ESCS index England Peru Colombia Panama OECD average Mexico Tunisia Countries performing significantly better than England Countries performing similarly to England Countries performing significantly worse than England Linear (OECD average) Source: OECD, datatbase
NPD matched file Positives Negatives Insight into translation between pupil attainment in PISA and attainment in national tests PISA information on socioeconomic background can be cross-referenced against national measures of socioeconomic background e.g. FSM status Further analysis into results for England not explored by OECD Based on just over 3,800 pupils in England Data ownership Fuzzy matching Data for England only, so comparisons with other countries can only be done using information published in the OECD s reports
How are the OECD s ESCS index and FSM status linked? 10% of PISA sample were eligible for Free School Meals Of these, only one in three were also in the bottom 10% of the OECD s distribution of economic, social and cultural status. OECD s measure of social background (ESCS) Pupils known to be eligible for FSM, January 2009 No Yes Total Bottom 10% 8% 29% 10% Not bottom 10% 90% 64% 88% ESCS missing 2% 7% 2% Total 100% 100% 100% Source: OECD database matched with National Pupil database 2009/10
How this compares to what we know about FSM gaps Point score measures of attainment attainment Attainment gap between pupils known to be eligible for FSM and those not eligible Non-FSM pupils PISA points FSM pupils PISA points FSM gap PISA points Effect size Attainment gap between pupils in the bottom 10% of the OECD s ESCS distribution and the other 90% Other 90% PISA points Bottom 10% PISA points PISA points ESCS split Effect size Reading 503.3 438.0 65.4 0.7 503.2 438.7 64.5 0.7 Mathematics 502.5 437.8 64.7 0.7 502.2 440.4 61.8 0.7 Science 524.7 454.7 70.1 0.7 524.6 455.8 68.8 0.7 Key Stage 4 attainment KS4 point score KS4 point score KS4 point score GCSE grades Effect size KS4 point score KS4 point score KS4 point score GCSE grades English GCSE point score 41.7 36.0 5.7 1.0 0.7 41.7 35.6 6.1 1.0 0.7 Maths GCSE point score 41.1 34.4 6.7 1.2 0.6 41.1 34.1 7.0 1.2 0.7 Key Stage 4 threshold measures % achieving % achieving %point difference Odds ratio % achieving % achieving %point difference Effect size Odds ratio 5 A*- C including English and maths 65.6% 39.5% 26.1 3.0 66.1% 34.7% 31.4 3.6 5 A* - C including English and maths, GCSEs only 61.2% 31.5% 29.7 3.5 61.6% 27.4% 34.2 4.4 Source: OECD database matched with National Pupil database 2009/10
Ideas for further analysis using the NPD matched file Further analysis of the impact of socio-economic status on attainment at school-level Comparing Key Stage 4 attainment and PISA attainment as predictors for progression to Key Stage 5 Comparison between variance in pupil attainment explained by separate components of the OECD s measure of socio-economic background and the factors identified from NPD-LSYPE analysis as having an impact on pupil attainment
Comments Questions Suggestions for further analysis? Emily.Knowles@education.gsi.gov.uk