PISA 2003 Brief Description of Assessment Frameworks in Mathematics, Reading, Science and Problem Solving This short document is based on the OECD's publication on the PISA 2003 assessment frameworks which may be viewed at http://www.pisa.oecd.org (OECD. Organisation for Economic Cooperation and Development. (2003). The PISA 2003 assessment framework: Mathematics, reading, science and problem solving knowledge and skills. Paris: Author.) The five sections below describe, for PISA 2003: The Mathematics framework The Reading Literacy framework The Science framework The Problem Solving framework Item Types Mathematics Mathematics in PISA 2003 is concerned with the capacity of students to analyse, reason, and communicate ideas as they formulate, solve and interpret mathematical problems in a variety of contexts. The focus is on 'real-world' problems. Mathematics in PISA 2003 is defined as an individual's capacity to identify and understand the role that mathemat ics plays in the world, to make well-founded judgements and to engage with mathematics in ways that meet the needs of that individual's life as a constructive, concerned and reflective citizen. The definition and accompanying framework are heavily influenced by the Realistic Mathematics Education movement, which stresses the importance of solving mathematical problems in real-world settings. Central to this approach is the notion of mathematising which involves the following: starting with a problem situated in reality; organising the problem according to mathematical concepts; 'trimming away the reality' through such processes as generalising and formalising the problem; solving the mathematical problem; and making sense of the mathematical solution in terms of the real situation. The mathematical framework distinguishes between three dimensions: mathematical content, mathematical competencies, and situations. Mathematical Content Content encompasses four overarching ideas: Space & Shape, Change & Relationships, Quantity, and Uncertainty. It is acknowledged that these areas are not mutually exclusive, since each one can include more than one element mathematics Change & Relationships 25.9% Uncertainty 23.5% Quantity 27.1% Space & Shape 23.5% 1
curricula, such as arithmetic, measurement, geometry, algebra, and probability and statistics. PISA 2003 assesses a broader range of mathematical content than PISA 2000, while including a subset of items that were also administered in 2000. In PISA 2003, a total of 85 items or tasks was presented to students. Of these, 23 (27.1%) are categorised as Quantity, 20 (23.5%) as Space & Shape, 22 (25.9%) as Change & Relationships, and 20 (23.5%) as Uncertainty. In PISA 2000, just two of the four overarching ideas (Space & Shape, and Change & Relationships) were assessed. Mathematical Competencies PISA identifies three broad competency clusters: the Reproduction cluster, the Connections cluster, and the Reflection cluster. These are assumed to form a hierarchy, with problems in the Reflection cluster being most difficult to solve. Each PISA mathematics item is classified according to the cluster that best describes the competencies that are required to solve the problem. 26 items (30.6%) belong to the Reproduction cluster, 40 (47.1%) to the Connections cluster, and 19 (22.4%) to the Reflection cluster. Connections 47.1% Reflection 22.4% Reproduction 30.6% Mathematical Situations The ability to use mathematics in a variety of situations is viewed as being an important part of mathematics. PISA identifies four situation-types: personal (18 or 21.2% of items), educational/occupational (21 or 24.7%), public (29 or 34.1%), and scientific (17 or 20.0%). A distinction is also made between intra-and extramathematical contexts. The former refers to questions Scientific Personal that do not move outside the 20.0% 21.2% mathematical realm, while the latter are found in Educational / questions that go beyond a Public Occupational mathematical context, and 34.1% 24.7% require students to interpret and extract relevant information. 2
Reading Literacy The reading literacy assessment in PISA 2003 does not measure whether students are technically able to read. Rather, assuming that they can, it attempts to assess the ability of students to understand and reflect on a wide range of written materials, in different situations. Reading literacy is defined as: understanding, using and reflecting on written texts, in order to achieve one's goals, to develop one's knowledge and potential, and to part icipate in society. The definition emphasises higher-order comprehension skills. Reference to participation in society emphasises the role of reading literacy in economic, political, cultural and social life. In operationalising this definition, three dimensions are identified: the content or structure of texts (continuous, such as narrative and descriptive, and non-continuous, such as tables, charts and forms); the processes that need to be performed (retrieval, interpretation, and critical reflection/evaluation); and the situation in which knowledge and skills are drawn on or applied (personal, public, occupational, and educational). This framework is the same as that used in PISA 2000. All 28 PISA 2003 reading literacy items were used in the PISA 2000 assessment. Of these, 25.0% tapped retrieval processes, 50.0% were classified as interpret, and 25.0% as reflect/evaluate. Interpret 50.0% Retrieve 25.0% Reflect/ Evaluate 25.0% 3
Science In PISA 2003, science is defined as: the capacity to use scientific knowledge, to identify questions and to draw evidencebased conclusions, in order to understand and help make decisions about the natural world and the changes made to i t through human activity. PISA science is concerned with the capacity of students to draw appropriate conclusions from evidence and information given to them, to criticise claims on the basis of the evidence, and to distinguish opinion from evidence-based statements. The PISA science framework includes three dimensions: scientific knowledge or concepts; scientific processes; and scientific areas of application (situations). Concepts are selected from the four major fields of physics, chemistry, biological science, and Earth and space science. Three processes are identified: describing, explaining and predicting scientific phenomena; understanding scientific investigation; and interpreting scientific evidence and conclusions. The three broad situations in which individuals apply scientific processes are identified as science in life and health, science in Earth and environment, and science in technology. Science was a minor domain in both PISA 2000 and PISA 2003; it will be the major domain in 2006. The PISA 2003 assessment of science used 25 items that also appeared in PISA 2000, as well as 10 new items. Of the 35 items, 12 items (34.3%) were drawn from the area of life and health; 34.3% from Earth and environment, and 11 items (31.4%) from science in technology. Science in Technology 31.4% Earth and Environment 34.3% Life and Health 34.3% 4
Problem Solving Reflecting its attempts to measure competencies that cut across a range of subject areas, PISA developed an assessment of cross-curricular problem solving for 2003. The assessment, which is not expected to be included in future PISA cycles, was a minor domain. It is generally agreed that problem solving is the ability to analyse the nature of a problem, search for, or construct a solution, and finally communicate the results. Problem solving in PISA 2003 is defined as: an individual's capacity to use cognitive processes to confront and resolve real, cross-disciplinary situations where the solution path is not immediately obvious and where the literacy domains or curricular areas that might be applicable are not within a single domain of mathematics, science or reading. The problem-solving framework is organised according to the type of problem encountered, the processes involved in solving a problem, and the situations involved. The assessment comprises 19 items which are categorised according to three problem types: decision-making (7 items, 36.8%); systems analysis and design (7 items, 36.8%), and trouble shooting (5 items, 26.3%). The six processes that are assumed to be most closely related to the three problem types encountered in PISA problem-solving items are: understanding, characterising, representing, solving, reflecting on the solution, and communicating the solution. There is no implied hierarchy, and not all six processes are necessarily involved in solving every problem. The problems in PISA are embedded in real-life situations (personal life, work and leisure, community and society). Trouble shooting 26.3% Decisionmaking 36.8% Systems analysis & Design 36.8% 5
Item Types Mathematics In PISA 2003 mathematics, 20.0% of items were simple multiple-choice, 12.9% complex multiple-choice (such as a series of yes/no questions), 42.4% short response/closed constructed response, and 24.7% open constructed response. Partial credit was available for partially correct responses for 10 of the open constructed response items. Double-digit coding, which entails scoring items for both correctness and error patterns, was used for nine of these items. Open constructed response 24.7% Short response / closed constructed response 42.4% Simple multiplechoice 20.0% Complex multiple-choice 12.9% Reading Literacy In reading literacy, 32.1% of items were simple multiple-choice, 3.6% complex multiple-choice, 28.6% short response/closed constructed response, and 35.7% open constructed response. Science In science, the item types were multiple-choice (37.1%), complex multiple-choice (20.0%), and short response/open constructed response (62.9%). Problem Solving Almost two-fifths (36.8%) of items in the problem solving framework were multiplechoice, 47.4% open constructed response, and 15.8% closed constructed response (none was short response in format). 6