1 PISA 2006: Science Competencies for Tomorrow s World What PISA tells us about quality, equity and efficiency in education 28 May 2008 Dr John Cresswell Analyst, OECD Directorate for Education
2 What is PISA? A three-yearly global assessment that examines the performance of 15-year-olds in key subject areas as well as a wider range of educational outcomes, including student attitudes to learning, their beliefs about themselves, and their learning strategies collects information from students, schools, parents and systems to identify policy issues
3 PISA countries in 2009 2006 2003 2001 1998 2000
4 Making international comparisons of achievement requires decisions about... what to assess, whom to assess.
5 Deciding what to assess... looking back at what they were expected to have learned looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.
6 Reading literacy PISA assessments Using, interpreting and reflecting on written material. Mathematical literacy Recognising problems that can be solved mathematically, representing them mathematically, solving them. Scientific literacy Identifying scientific questions, recognising what counts as scientific evidence, using evidence to draw conclusions about the natural world.
7 How PISA works A strong international network of expertise among the participating countries From establishing the assessment frameworks The PISA assessments include tasks from more than 40 countries developing the instruments Cross-national and cross-cultural cultural validity to analysing and interpreting the results National, regional and international analyses and reports In-depths country peer reviews supported by a consortium of the leading research institutions ACER, CITO, ETS, NIER, WESTAT co-ordinated ordinated through the OECD in collaboration with other international organisations.
8 Development of assessments Frameworks by international experts Assessment materials submitted by countries developed by research consortium screened for cultural bias by countries by expert, international panel items with prima facie cultural bias removed at this stage translated from English & French originals trialled to check items working consistently in all countries Final tests items shown in trial to be culturally biased removed best items chosen for final tests balanced to reflect framework range of difficulties range of item types (constructed response, multiple choice)
9 Science in PISA 2006 Scientific knowledge and use of that knowledge in identifying scientific issues, explaining scientific phenomena, using scientific evidence
10 Deciding whom to assess... grade-based sample OR age-based sample For PISA, the OECD countries chose the latter, selecting 15-year year-olds in school as the population.
11 PISA sampling requirements Population: all 15-year-olds in school Minimum of 150 schools randomly selected per country (85% must participate) 35 students randomly selected per school (80% must participate), giving samples of between 3,500 and 50,000 per country Total of 400,000 students Countries doing regional comparisons have larger than average samples Italy (22000), Canada (22000), Spain (20000)
12 Key features of PISA 2006 Information collected each student 2 hours on paper-and-pencil tasks (subset of all questions) ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals questionnaire (school demography, learning environment quality)
13 Overview of Results from PISA 2006
14 PISA provides key benchmarks for the quality of education systems 14 1. Overall performance of education systems 2. Equity in the distribution of learning opportunities Measured by the impact students and schools socio-economic background has on performance not merely by the distribution of learning outcomes 3. Consistency of performance standards across schools 4. Gender differences
15 Mean science scores 300 350 400 450 500 550 600 Finland Canada Japan New Zealand Australia Netherlands Korea Slovenia Germany United Czech Republic Switzerland Austria Belgium Ireland Hungary Sweden Poland Denmark France Iceland United States Slovak Republic Spain Norway Luxembourg Italy Portugal Greece Turkey Mexico The centre The line boundaries is the indicate meanthe 95% confidence limits OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Table 2.1c
16 PISA Proficiency Levels Level 6 Level 5 Level 4 OECD Slovenia 1.3% 2.2% 7.7% 10.7% 20.3% 22.5% Science Level 6 Student can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations Level 3 Level 2 27.4% 24.0% 27.6% 23.1% Science Level 1 Student have such a limited scientific knowledge that it can only be applied to a few, familiar situations Level 1 14.1% 11.1% Below Level 1 5.2% 2.8% Below Level 1 Unable to use scientific skills in ways required by easiest PISA tasks. OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Table 2.1a
Finland Canada Japan New Zealand Australia Netherlands Korea Slovenia Germany United Kingdom Czech Republic Switzerland Austria Belgium Ireland Hungary Sweden OECD average Poland Denmark France Iceland United States Slovak Republic Spain Norway Luxembourg Italy Portugal Greece Turkey 17 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Percentage of students at each proficiency level OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Table 2.1a
18 Science in PISA 2006 The three competencies that PISA measured in science in 2006: 1. Identifying scientific issues 2. Explaining scientific phenomena 3. Using scientific evidence
19 Strengths and weaknesses of countries in science relative to their overall performance Overall science score Slovenia=519 Science competencies Identifying scientific issues Explaining phenomena scientifically Using scientific evidence Knowledge about science Science knowledge Earth and space Living systems Physical systems -35-25 -15-5 5 15 25 35 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Figure 2.13
20 Gender differences in science OECD average Slovenia Overall Girls do better Boys do better Identifying scientific issues Explaining phenomena scientifically Using scientific evidence scale Knowledge about science scale Earth and space systems scale Living systems scale Physical systems scale -35-30 -25-20 -15-10 -5 0 5 10 PISA score points 15 20 25 30 35 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Tables 2.1c, 2.2c, 2.3c, 2.4c, 2.7, 2.8, 2.9, 2.10
21 Mean reading scores Korea Finland Canada New Zealand Ireland Australia Poland Sweden Netherlands Belgium Switzerland Japan United Kingdom Germany Denmark Slovenia Austria France Iceland Norway Czech Republic Hungary Luxembourg Portugal Italy Slovak Republic Spain Greece Turkey Mexico 300 350 400 450 500 550 600 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Table 6.1c
22 Mean mathematics scores Finland Korea Netherlands Switzerland Canada Japan New Zealand Belgium Australia Denmark Czech Republic Iceland Austria Slovenia Germany Sweden Ireland France United Kingdom Poland Slovak Republic Hungary Norway Spain United States Portugal Italy Greece Turkey Mexico 300 350 400 450 500 550 600 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Table 6.2c
Germany Czech Republic Austria Hungary Netherlands Belgium Japan Italy Greece Slovak Republic Turkey Switzerland Korea Luxembourg United States Portugal Mexico United Kingdom New Zealand Australia Canada Ireland Denmark Spain Poland Sweden Norway Iceland Finland 23 140 120 100 80 60 40 20 0 Is it all innate ability? Variation in student performance Performance variation across schools OECD (2007), Learning for tomorrow s world: First results from PISA 2006, Table 4.1a
Germany Slovenia Czech Republic Austria Hungary Netherlands Belgium Japan Italy Greece Slovak Republic Turkey Switzerland Korea Luxembourg United States Portugal Mexico United Kingdom New Zealand Australia Canada Ireland Denmark Spain Poland Sweden Norway Iceland Finland 24 180 160 140 120 100 80 60 40 20 0 Variation in student performance PISA 2006: Science Competencies for Tomorrow s World, Figure 4.1a.
Germany Slovenia Czech Republic Austria Hungary Netherlands Belgium Japan Italy Greece Slovak Republic Turkey Switzerland Korea Luxembourg United States Portugal Mexico United Kingdom New Zealand Australia Canada Ireland Denmark Spain Poland Sweden Norway Iceland Finland 25 100 80 60 40 20 0-20 -40-60 -80 Variation in student performance Variation of performance within schools Variation of performance between schools PISA 2006: Science Competencies for Tomorrow s World, Figure 4.1a.
26 26 From 2000 Between to 2006 PISA 2000 Poland raised its PISA reading 2003 Poland delayed the performance by 28 points separation of students into different school types beyond the age of 15 years In 2003, performance variation among schools At the same time, had fallen from 51% between to 2000 and 2003 16% of the variation of Poland showed the student performance second-largest increase in reading (17 points) and a further 11 point increase in 2006 Most of that increase resulted from smaller proportions at the bottom level (23% in 2000, and three-quarters in vocational tracks, 17%in 2003) OECD (2007), Learning for tomorrow s world: First results from PISA 2006, Table 6.1a
High science performance 27 565 Finland Average performance of 15-year-olds in science extrapolate 545 Hong Kong-China and apply Chinese Taipei Canada Estonia Japan New Zealand Australia 525 Netherlands Liechtenstein Korea Equity Slovenia United Kingdom Germany Czech Republic Switzerland Macao-China Austria Ireland Belgium 505 Hungary Sweden France Denmark Poland Iceland Croatia United States Latvia Norway Slovak Republic,Spain, Lithuania Luxembourg Portugal 485 Russian Federation Italy Greece 465 Israel 445 16 18 countries perform below this line Low science performance 6
28 Performance and socio-economic background High Student performance Low On average, there is a 40 For Slovenia, there is 46 point For change Finland, there science is a 31 point change in science score point for change a one standard in science score for a one standard deviation score for change a one in standard social deviation change in social background deviation change in social background above the background below the OECD average OECD average Disadvantage PISA Index of social background Advantage OECD (2007), Learning for tomorrow s world: First results from PISA 2006, Table 6.1a
29 Performance and migration Native students First-generation students Second-generation students 600 550 OECD average = 500 500 450 400 350 300 PISA 2006: Science Competencies for Tomorrow s World, Figure 4.2a.
30 Students attitudes to science and their awareness of the life opportunities science may open
31 Students generally value science but report stronger belief in the technological potential of science than in its capacity to make social improvements OECD average Slovenia Science is important for helping us to understand the natural world Advances in science and technology usually improve people's living conditions Science is valuable to society Advances in science and technology usually help to improve the economy Advances in science and technology usually bring social benefits 0 25 50 75 100 % Change in science performance per unit of the index Turkey Score points 0 20 40 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Figure 3.2
32 Enjoyment of science OECD average Slovenia I enjoy acquiring new knowledge in science I generally have fun when I am learning science topics I am interested in learning about science I like reading about science I am happy doing broad science problems 0 25 50 75 100 % Change in science performance per unit of the index Turkey Score points 0 10 20 30 40 50 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Figure 3.10
33 Instrumental motivation to learn science OECD average Slovenia I study school science because I know it is useful for me Making an effort in my school science subject(s) is worth it because this will help me in the work I want to do later on Studying my school science subject(s) is worthwhile for me because what I learn will improve my career prospects I will learn many things in my school science subject(s) that will help me get a job What I learn in my school science subject(s) is important for me because I need this for what I want to study later on 0 25 50 75 100 % Change in science performance per unit of the index -20-10 0 10 20 Score points 30 40 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Figure 3.12
34 but somewhat less so when it concerns themselves OECD average Slovenia I find that science helps me to understand things around me I will use science in many ways when I am an adult Some concepts in science help me see how I relate to other people When I leave school there will be many opportunities for me to use science Science is very relevant to me 0 25 50 75 100 % Change in science performance per unit of the index Score points 0 20 40 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Figure 3.4
35 and only a minority report interest in a scientific career OECD average Slovenia I would like to work in a career involving science I would like to study science after secondary school I would like to work on science projects as an adult I would like to spend my life doing advanced science 0 25 50 75 100 % Change in science performance per unit of the index Score points -20-10 0 10 20 30 40 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Figure 3.13
36 Students expecting a science-related career and their performance in science Score 650 600 Students not expecting a science career without a parent in a science-related occupation Students not expecting a science career with at least one parent in a science-related occupation Students expecting a science career without a parent in a science-related occupation Students expecting a science career with at least one parent in a science-related occupation 550 500 450 400 350 300 250 United States Portugal Canada Mexico Iceland Italy Poland Ireland France Australia Spain Belgium Greece Norway Note: Science performance scores are only shown for groups where there are at least 3% of students. Source: OECD PISA database, Table 3.14. United Kingdom New Zealand Luxembourg Turkey Netherlands Sweden Denmark Switzerland Russian Federation Austria Slovak Republic Germany Korea Finland Czech Republic Hungary Japan
37 Science and the environment 58% of 15-year-olds report familiarity and knowledge of the increase of greenhouse gases in the atmosphere, 73% with consequences of clearing forests for other land use, 60% with issues around pollution and acid rain (large variation across countries) Awareness of environmental issues is closely linked with students science performance and with their social background Only a minority are optimistic that the issues will be successfully addressed and the better they perform in science, the less optimistic they are.
38 Some degree of pessimism among students about the future of the natural environment problems associated with the areas below will improve over the next 20 years OECD average Slovenia Energy shortages Water shortages Air pollution Nuclear waste Extinction of plants and animals Clearing of forests for other land use 0 25 50 75 100 % Change in science performance per unit of the index Score points -40-30 -20-10 0 OECD (2007), PISA 2006 Science Competencies for Tomorrow s World, Figure 3.20
39 PISA 2006 some system and school factors associated with performance 39 (across 55 countries and above and beyond students socio-economic background and other school factors) Some autonomy and accountability factors show a positive association with performance. Systems providing schools more autonomy in budgeting. The proportion of students in schools with autonomy in budgeting. 57% of students across the OECD are in such schools. Schools posting achievement data publicly. But no specification as to how this is done, e.g. linking to previous year s performance or performance of schools with similar intake.
40 PISA 2006 some system and school factors associated with performance (across 55 countries and above and beyond students socio-economic background and other school factors) However, PISA offers a note of caution on selection or grouping of students : Grouping students by ability within schools. Schools where students are sorted by ability for all subjects are associated with lower performance. Schools selecting students by academic record/achievement. These schools perform better, BUT there is no benefit to the system as a whole (a higher proportion of schools selecting students by academic record/achievement is not associated with higher performance).
41 PISA 2006 some system and school factors associated with performance (across 55 countries and above and beyond students socio-economic background and other school factors) However, PISA offers a note of caution on selection or grouping of students : Systems where students are sorted into distinct educational tracks/institutions at an early age. There is no benefit to the system as a whole (an early age of selection is not associated with higher performance). BUT there is a negative impact on equity.
42 Is performance in PISA associated with later success? Increased likelihood of postsecondary participation at age 19 associated with reading proficiency at age 15 (Canada) after accounting for school engagement, gender, mother tongue, place of residence, parental, education and family income (reference group Level 1) 20 18 16 14 12 10 8 6 4 2 0 Level 2 Level 3 Level 4 Level 5
43 Further information www.pisa.oecd.org All national and international publications The complete database email: pisa@oecd.org john.cresswell@oecd.org