What can PISA 2015 tell you about your school?

What can PISA 2015 tell you about your school? A report by RM Education and UCL Institute of Education 6th December 2016 1

Report contents 1. Introduction 3 2. Top facts 3 3. About the study 4 4. How should I interpret the figures in this report? 4 5. How does the reading, mathematics and science performance and attitudes of 15-yearold pupils in my school compare internationally? 7 6. What do your pupils hope to do after they finish school? 11 7. Do your pupils believe that learning about science will improve their post-school prospects? 13 8. Do your pupils enjoy science? 14 9. How interested are your pupils in different areas of science? 15 10. How confident are your pupils in science? 18 11. What activities do your pupils do before and after school? 20 12. Further information 21 2

1. Introduction In November/December 2015 your school was one of 550 schools from the United Kingdom (206 from, 109 from, 140 from and 95 from Northern Ireland) to participate in the Organisation for Economic Co-Operation and Development (OECD) Programme for International Student Assessment (PISA) 2015 study. A sample of your pupils joined over half a million 15-year-olds from more than 70 countries to sit an assessment of their science, mathematics, reading and problem-solving ability that was delivered by computer. We thank you, your teachers and your pupils for their engagement with this project last year and hope you find this report an interesting insight into the results both in relation to the performance of your pupils in the PISA study and the responses they gave to questions about their engagement and experiences of science teaching; and our national results. This report provides an overview of how the 15-year-olds in your school performed on the PISA test, as well as their responses to the PISA background questionnaire. The report does not provide any comparison of your responses to the school questionnaire to ensure the data provided is not identifiable. The accompanying Excel document provides a copy of all figures and tables contained within this report. 2. Top facts 1. PISA has been used to compare the reading, mathematics and science performance of pupils across the globe every three years since 2000. 2. 75 countries participated in PISA 2015. In, 5,600 pupils in 206 schools took part in the study, which focused on science and was administered using computers for the first time in the majority of countries. 3. Pupils in performed above the OECD average in science in 2015 and similar to the OECD average for reading and mathematics. These scores are consistent with the results from PISA 2012. For more information on the performance of across all three subject domains, the National Report can be found at www.gov.uk/government/collections/international-comparisons-of-education 4. Internationally, science performance is highest in Singapore, Estonia, and Taiwan where 15-year-olds are at least half a year ahead of their peers in, on average. For more information on international trends and the performance of other countries, the OECD international report can be found at www.oecd.org/pisa/keyfindings 5. This report provides an overview of how the pupils sampled in your school compare to the national average and other OECD countries, including their performance in the science, mathematics and reading assessments, their perceptions of their science lessons and aspirations for the future. 3

3. About the study Science was the main focus of PISA in 2015. Therefore, the two-hour test and background questionnaire that your pupils completed contained a number of questions on living systems, physical systems, earth and space, as well as a range of questions about your pupils attitudes towards science, their experience of their science lessons and whether they intend to pursue science in their future education and careers. The school questionnaire, completed by a member of staff at your school, asked about the way science is taught and organised amongst other questions about your school. As mentioned above, your school was one of 550 across the UK that participated in PISA 2015. The information we collected from your school has been combined with the information collected from the other participating schools and submitted to the OECD along with the data from the other countries who participated. Your support ensured that over 85% of randomly sampled schools in this country took part in this project, which means the results from each of the four UK countries will be included in the PISA 2015 international reports, published on 6th December 2016 and available from www.oecd.org/pisa/keyfindings. These reports also provide further information about the PISA study, along with the full results for all participating countries and economies. The National report for will also be published in December 2016, and will be downloadable from www.gov.uk/government/collections/international-comparisonsof-education. This report has been written by academics at the UCL Institute of Education who are part of the consortium who administered the study across, Northern Ireland and. The national reports provide greater insight into the key findings from the PISA 2015 study for each country, including a more indepth analysis of the distribution of our pupils results, their attitudes towards science and how this links to their performance, and the variation within and across our schools. 4. How should I interpret the figures in this report? Only a random sample of your pupils took part in the PISA study. This means their results are not necessarily typical of all of your 15-year-old pupils. However, because the test and background questionnaire is fairly comprehensive, the results in this report can provide an indication of how your pupils performance and attitudes towards science compare to their peers across the country and internationally. Table 1 provides some key contextual information about the sample of pupils selected in your school and shows how this compares to the average for all pupils selected for PISA 2015 in, and Northern Ireland. 4

Table 1. Key characteristics of the PISA sample selected in your school1 Your school Northern Ireland % Boys 50% 51% 50% 51% % Parent holds a degree 88% 42% 41% 41% % born outside UK 69% 9% 15% 5% Average pupil age 15.7 15.7 15.7 15.7 Sample size 26 5,194 2,401 3,451 All results in this report are based on a random sample of pupils in your school rather than based on all of your 15 year old pupils. As such, the figures in this report are subject to some uncertainty. This reflects the uncertainty of results stemming from the study being based on a random sample from your school, rather than a result based on all your 15-year-old pupils. To document this uncertainty, all graphs include a margin of sampling error around the percentages shown, formally known as a confidence interval. These intervals are represented by a thin black line running through the centre of each bar. So while we do not know the percentage of all 15- year-old pupils in the school who would agree/strongly agree with the statement concerned, the margin of sampling error signals that the figure shown is only an estimate (see figure 1 overleaf as an example). How does PISA measure performance? Pupils answers to the questions in their tests are marked and coded according to internationally agreed standards. Each pupil answered a slightly different set of questions to ensure that the PISA 2015 study could assess a wide variety of science, mathematics, reading and problem-solving domains whilst minimising testing time. Based on the difficulty of the questions that each pupil received and their answers, pupils are scored on the PISA scale. Across OECD countries in 2000, the PISA scale was centred around 500 and given a standard deviation of 100. To aid interpretation, in PISA 2015 a 30 test point difference is equivalent to approximately one year of schooling 2. How are pupils responses to attitudinal questions measured and presented? Following the comparisons of performance, the report provides details of your pupils responses to a selection of the background questions. This includes: What your pupils hope to do once they have finished school 1 A value of 9999 in Table 1 indicates figures have been suppressed due to the small sample size. 2 OECD. 2010:210. PISA 2009 results. Learning to learn: student engagement strategies and practices. Paris: OECD. 5

Whether pupils in your school enjoy science How interested your pupils are in different areas of science If your pupils think that learning about science in school will improve their prospects once they have left school How confident your pupils are in different areas of science The activities of your pupils before and after school The accompanying Excel workbook includes further comparisons of your pupils responses to a broader array of questions asked as part of the PISA background questionnaire. In this report we will identify where questions were asked of a similar nature, and the name of the Excel worksheet where they can be found. Most questions within the pupil background questionnaire required multiple choice answers to indicate the extent to which pupils agreed or disagreed with a statement. For example: I am interested in learning about science Strongly agree Agree Disagree Strongly Disagree All of the figures within this report refer to the percentage of pupils who either agreed or strongly agreed with each statement. How are my school s results presented in comparison to other countries? A comparison is presented for your school against the average performance score or response given by the average child from a number of other countries. A conservative way to tell if the figure for your school is significantly different to that for a given country is whether the confidence intervals overlap or not. For example, in Figure 1, the confidence interval for the example school does not overlap with the confidence interval for. This indicates that the difference is statistically significant. So what does this mean? It means that the difference observed between pupils in the example school and pupils in is unlikely to be due to the fact that samples of pupils have been selected. Rather, it indicates that there is a genuine difference between the pupils in the sample school and their peers in. The confidence interval for any country is much smaller than that for a school because the size of the interval is determined by the size of the sample. Thin black lines running through the centre of the bars refer to the confidence interval. 6

Figure 1. Example of how results will be displayed EXAMPLE Northern Ireland Korea Republic of Ireland EXAMPLE SCHOOL 0 20 40 60 80 100 Per cent who agree or strongly agree 5. How does the reading, mathematics and science performance and attitudes of 15-year-old pupils in my school compare internationally? Figures 2, 3 and 4 on the following pages compare the average (mean) performance of 15-year-olds in your school on the PISA science, mathematics and reading test to the average performance of pupils from each of the UK countries and a selection of other OECD countries. (The accompanying Excel document provides additional comparisons made to non-oecd countries in the tabs labelled SCIE_POSITION, MATHS_POSITION and READ_POSITION ) As science was the focus of PISA 2015, separate results for this subject are available for various sub-domains, including: Physical scientific systems Living scientific systems Earth and Space sciences An example question from each of these science sub-domains can be found in the final section of this report. This also provides average scores for your school in each of these three areas, and draws a comparison to national results for,, Northern Ireland and the Republic of Ireland. 7

Figure 2. How did your school perform compared to other OECD countries in the PISA science test? Mexico Turkey Chile Greece Slovak Republic Israel Iceland Hungary Italy Luxembourg Latvia Spain Czech Republic France Austria United States Norway N. Ireland Portugal Denmark Belgium Rep. of Ireland Switzerland Netherlands Germany Australia Slovenia New Zealand South Korea Finland Estonia YOUR SCHOOL 416 425 447 455 461 467 473 477 481 483 485 490 493 493 493 495 495 496 497 498 500 501 501 502 502 503 506 509 509 510 512 513 513 516 528 531 534 538 570 300 350 400 450 500 550 600 650 Average PISA score 8

Figure 3. How did your school perform compared to other OECD countries in the PISA maths test? Mexico Turkey Chile Greece United States Israel Slovak Republic Hungary Latvia Luxembourg Spain Iceland Italy Portugal Czech Republic N. Ireland France Australia New Zealand Austria Norway Rep. of Ireland Germany Belgium Slovenia Finland Denmark Netherlands Estonia Switzerland South Korea YOUR SCHOOL 408 420 423 454 470 470 475 477 478 482 486 486 488 490 491 492 492 493 493 493 494 494 495 497 502 504 504 506 507 510 511 511 512 516 520 521 524 532 605 300 350 400 450 500 550 600 650 Average score 9

Figure 4. How did your school perform compared to other OECD countries in the PISA reading test? Mexico Turkey Slovak Republic Chile Greece Hungary Israel Luxembourg Iceland Italy Austria Czech Republic Latvia Switzerland Spain United States N. Ireland Portugal Belgium France Denmark Australia Netherlands Slovenia Germany New Zealand Norway South Korea Estonia Rep. of Ireland Finland YOUR SCHOOL 423 428 453 459 467 470 477 479 481 482 485 485 487 488 492 493 496 497 497 498 499 499 500 500 500 503 503 505 506 509 509 513 516 517 519 521 526 527 561 300 350 400 450 500 550 600 650 Average score 10

6. What do your pupils hope to do after they finish school? As part of the PISA study, 15-year-olds in your school were asked about their future educational and occupational plans, including the level of education that they hope to complete. Figure 5 illustrates the per cent of pupils in your school who said that they expected to complete university, and compares this to the average across a series of comparator countries. The Excel worksheet Ed_Expect provides further details. Figure 5. The percentage of pupils who expect to complete university South Korea Rep. of Ireland Northern Ireland YOUR SCHOOL 75 59 48 39 63 46 43 35 45 42 80 0 20 40 60 80 100 Percent In 2015, the PISA questionnaire also asked pupils what occupation they would like to hold at age 30. Around 30 per cent of 15-year-olds from the UK aspired to a career in science. Figure 6 illustrates the percentage of 15-year-olds in your school who expect to work in a science-related career by age 30, and how this compares to 15- year-olds in other countries. 11

Figure 6. The percentage of pupils who expect to be working in a science career at age 30 South Korea Rep. of Ireland Northern Ireland YOUR SCHOOL 19 18 21 20 34 27 23 28 33 30 23 0 20 40 60 80 100 Percent 12

7. Do your pupils believe that learning about science will improve their post-school prospects? The PISA background questionnaire asked pupils their views on a series of statements about their instrumental motivation to learn science. In other words, do pupils in your school believe science is relevant for their future? Previous academic research has found that pupil s responses to such questions are an important predictor of their post-secondary course selections, their career choice and later academic performance 3. Five statements were included in the PISA background questionnaire concerning whether pupils believe that learning about science will improve their post-school prospects. These were: Making an effort in my school science subject(s) is worth it because this will help me in the work I want to do later on What I learn in my school science subject(s) is important for me because I need this for what I want to study later on I study school science because I know it is useful for me Studying my school science subject(s) is worthwhile for me because what I learn will improve my career prospects I will learn many things in my school science subject(s) that will help me get a job Pupil s responses to these statements can be found in Excel worksheets ST113Q01 to ST113Q04. Figure 7 on the next page illustrates the per cent of pupils who either agree or strongly agree with the last of these statements they learn things from school science that will help them get a job. 3 Eccles, J. S. (1994), Understanding Women s Educational and Occupational Choice: Applying the Eccles et al. Model of Achievement related Choices, Psychology of Women Quarterly 18, Blackwell Publishing, Oxford, pp. 585-609. 13

Figure 7. The percentage of pupils who agree or strongly agree that the things they learn in school science subjects will help them to get a job South Korea Rep. of Ireland Northern Ireland YOUR SCHOOL 64 52 58 65 74 71 69 74 75 71 65 0 20 40 60 80 100 Percent agree or strongly agree Future scientists An analysis of the PISA 2015 database suggests that there is a strong correlation between positive responses to this question, pupil s performance on the PISA science test, and whether they expect to work in a science career when they turn 30. Specifically, 15-year-olds from, and Northern Ireland who either agreed or strongly agreed that school science teaches them things that will help them get a job scored (on average) 520 points on the test. In contrast, 15-year-olds who disagreed or strongly disagreed with the statement scored (on average) 500 PISA test points. 8. Do your pupils enjoy science? Pupils in your school were asked to specify to what extent they agreed with five statements regarding their enjoyment of science: I generally have fun when I am learning science topics I like reading about science I am happy doing science problems I enjoy acquiring new knowledge in science 14

I am interested in learning about science The percentage of pupils who agree or strongly agree with each of these questions can be found in Excel worksheets ST94Q01 to ST94Q05. Figure 8 below examines answers to the fourth statement in the list above: I enjoy acquiring new knowledge in science. It illustrates the extent pupils in your school enjoy acquiring new knowledge in science, as well as how this compares to pupils across the various constituent countries within the United Kingdom, and to the responses given by pupils in other countries. Figure 8. The percentage of pupils who agree or strongly agree that they enjoy acquiring new knowledge in science South Korea Rep. of Ireland Northern Ireland YOUR SCHOOL 60 55 72 66 79 78 72 69 72 72 85 0 20 40 60 80 100 Percent who agree or strongly agree Why might pupil s responses to this question be important? The PISA 2015 study suggests that pupils from the United Kingdom who agreed or strongly agreed with the statement I enjoy acquiring new knowledge in science scored (on average) around 530 points on the PISA science test. This compares to approximately 470 points for pupils who disagreed or strongly disagreed. This difference is equivalent to around two years of schooling. 9. How interested are your pupils in different areas of science? 15

Interest in a subject can influence the intensity and continuity of student engagement in learning, thus deepening pupils understanding 4. As part of the PISA questionnaire, pupils were asked: To what extent are you interested in the following science topics? Biosphere (e.g. ecosystem services, sustainability) Motion and forces (e.g. velocity, friction, magnetic and gravitational forces) Energy and its transformation (e.g. conservation, chemical reactions) The Universe and its history How science can help us prevent disease There were five possible response options: not interested, hardly interested, interested, highly interested, I don t know what this is. The proportion of pupils in your school reporting a high or medium level of interest can be found in Figure 9 below. Figure 9. How interested are your pupils in different areas of science? How science helps prevent disease 72 The Universe and its history 73 Energy and its transformation 58 Motion and forces 65 Biosphere 46 0 20 40 60 80 100 Percent high or medium interest 4 OECD (2007:140). PISA 2006. Science competencies for tomorrow s world. Volume 1: analysis. OECD: Paris. 16

The Excel worksheets ST095Q04 to ST095Q15 compare the responses of pupils in your school to those of the average pupil across a range of countries. Figure 10 on the next page illustrates results for interest in motion and forces. Figure 10. How interested are your pupils in motion and forces? A comparison across countries South Korea Rep. of Ireland Northern Ireland YOUR SCHOOL 39 37 34 44 55 47 45 45 49 45 65 0 20 40 60 80 100 Percent high or medium interest Analysis of the PISA 2015 dataset suggests that pupils who display higher levels of interest in science also obtain higher scores on the PISA test. For example, 15-year-olds from, and Northern Ireland who reported high or medium interest in motion and forces scored, on average, almost 535 points on the 2015 PISA science test. This compares to a score of around 505 points for pupils who reported low or no interest. This difference is equivalent one year of additional schooling. It suggests that there is a correlation between pupil s interest in science and their performance on the PISA test. 17

10. How confident are your pupils in science? The PISA background questionnaire also attempted to capture pupil s confidence in science. Pupils were asked to tick one out of four boxes (I could do this easily; I could do this with a bit of effort; I would struggle to do this on my own; I couldn t do this) in response to a series of statements. These were how easily pupils felt they could: Recognise the science question that underlies a newspaper report on a health issue Explain why earthquakes occur more frequently in some areas than in others Describe the role of antibiotics in the treatment of disease Identify the science question associated with the disposal of rubbish Predict how changes to an environment will affect the survival of certain species. Interpret the scientific information provided on the labelling of food items Discuss how new evidence can lead you to change your understanding about the possibility of life on Mars Identify the better of two explanations for the formation of acid rain Confidence is a strong predictor of success Previous academic research has shown that a strong belief in one s own abilities is important for successful learning, and that pupil s responses to such statements are a strong predictor of future success 5. Indeed, Zimmerman (2002) argues that pupils who believe in their capacity to succeed in school go on to obtain higher levels of academic achievement 6. A summary of pupil s responses to these statements can be found in Excel worksheets ST129Q01 to ST129Q08. Figure 11 on the next page illustrates the per cent of pupils who agree or strongly agree with the sixth statement whether they feel confident in interpreting the scientific information provided on the labelling of food items. 5 Marsh, H.W. (1986), Verbal and Math Self-concepts: An Internal/External Frame of Reference Model, American Educational Research Journal 23.1, American Educational Research Association, Washington, D.C., pp. 129-149. 6 Zimmerman, B. (2002) Becoming a self-regulated learner: an overview. Theory into Practice 41(2): 64-70. 18

Figure 11. To what extent do 15-year-olds agree or strongly agree that they can understand the scientific information provided on the labelling of food items South Korea Rep. of Ireland Northern Ireland YOUR SCHOOL 55 50 76 64 71 61 56 66 65 64 73 0 20 40 60 80 100 Percent who think they could do this 'easily' / 'with a bit of effort' 19

11. What activities do your pupils do before and after school? What activities do 15-year-old pupils do outside of school? Do they practice a sport, play video games or work for pay? As part of the PISA background questionnaire, pupils were asked whether they did any of the following before or after their most recent school day: Eat breakfast Study for school or homework Watch TV/<DVD>/Video Read a book/newspaper/magazine Internet/Chat/Social networks (e.g. <Facebook>, <country-specific social network>) Play video games Meet friends or talk to friends on the phone Talk to your parents Work in the household or take care of other family members Work for pay Exercise or practice a sport Pupil s responses to these activities can be found in the Excel worksheets ST076Q01 to ST076Q11 (before school) and ST078Q01 to ST078Q11 (after school). Figure 12 illustrates the per cent of pupils in your school who reported practicing a sport after their most recent day at school, and how this compares to 15- year-olds across a series of comparator countries. 20

Figure 12. The per cent of 15-year-olds who exercised or practiced a sport after their most recent day at school South Korea Rep. of Ireland Northern Ireland YOUR SCHOOL 43 50 75 65 72 78 67 65 67 60 77 0 20 40 60 80 100 12. Further information Percent yes The PISA 2015 international report, published by the OECD, can be found at http://www.oecd.org/pisa/keyfindings/. This provides further information about the PISA test, along with full results for all participating countries and economies. National reports for, and Northern Ireland are published at https://survey.pisa.rmassessment.co.uk/. These provide greater insight into national results on the PISA test, and pupil s responses to the background questionnaires. 21