Proficiency levels in assessments of Reading and Mathematics

Proficiency levels in assessments of Reading and Mathematics Descriptions of the skills of students at different levels of proficiency in the National Assessments of English Reading and Mathematics (NAERM) and the Programme for International Student Assessment (PISA) May 2017

ii

Contents Introduction to proficiency levels... 1 Reading Proficiency primary level... 2 Second class reading... 3 Sixth class reading... 4 Reading Proficiency post-primary level... 5 Post-primary reading... 6 Mathematics Proficiency primary level... 7 Second class mathematics... 8 Sixth class mathematics... 9 Mathematics Proficiency post-primary level... 10 Post-primary mathematics... 11 iii

iv

Introduction to proficiency levels This document provides information on the proficiency levels that are used to describe students performance in the National Assessments of English Reading and Mathematics (NAERM) at primary level and the Programme for International Student Assessment (PISA) at post-primary level. In both of these studies, student performance and the level of difficulty of assessment items are described on a single scale for each domain assessed (i.e., reading and mathematics), meaning that each scale can be divided into clusters of skill sets. This means that the types of tasks which students, at the different levels of performance, can consistently complete successfully, can be described. Proficiency levels are constructed in such a way that all students performing at a particular level are expected to answer at least half of the items at that level correctly (and fewer than half of the items at a higher level). Further information on the reading and mathematics performance of students at primary and post-primary level in Ireland can be found in the reports for the most recent cycles of the NAERM and PISA studies: Shiel, G., Kavanagh, L., & Millar, D. (2014). The 2014 National Assessments of English Reading and Mathematics Volume 1: Performance Report. Dublin: Educational Research Centre Available at http://www.erc.ie/wp/wp-content/uploads/2016/11/na_2014_vol1_final-updated.pdf Shiel, G., Kelleher, C., McKeown, C., & Denner, S. (2016). Future Ready? The Performance of 15-year-olds in Ireland on Science, Reading Literacy and Mathematics in PISA 2015. Dublin: Educational Research Centre Available at http://www.erc.ie/wp/wp-content/uploads/2016/12/pisa2015_futureready.pdf The proficiency levels described in these two studies have been used to inform targets for literacy and numeracy outlined in the National Strategy: Literacy and Numeracy for Learning and Life (2011-2020), which were revised and updated in a report published on 14 th March 2017 Report of the Interim Review of the National Strategy: Literacy and Numeracy for Learning and Life (2011-2020): Interim Review 2011-2016; New Targets 2017-2020 - and in the DEIS Plan 2017. The proficiency levels in this document apply to all students, whether in DEIS schools or in other schools. Percentages accompanying the tables in this document relate to students in all schools. The NAERM survey is administered to representative samples of pupils in Second and Sixth class in primary schools every five years. The most recent assessment was administered in 2014. PISA is administered to 15-year-olds in post-primary schools every three years. The most recent PISA assessment was carried out in 2015. 1

Reading Proficiency primary level Four proficiency levels are described for English reading, at both Second and Sixth class, in NAERM. Pupils performing at the highest level, Level 4, are likely to be able to complete the most complex tasks expected of their grade level, while those performing at Level 1 are expected to be able to complete only the most basic tasks. Pupils who do not reach Level 1 are not consistently able to display the reading skills assessed by the simplest items on the test. Tables 1 and 2 provide descriptions of the types of tasks that pupils performing at each of the reading proficiency levels, at Second and Sixth class respectively, are capable of doing. The percentages of pupils performing at each proficiency level in the 2009 and 2014 National Assessments are also provided. 2

Second class reading Table 1: Descriptions of the four levels of proficiency on the overall English reading scale, at Second class, and the percentages of all pupils performing at each level in the 2009 and 2014 National Assessments Level & score range 4 320+ 319 3 269 268 2 225 What pupils can typically do As well as succeeding on lower proficiency level skills, pupils at Level 4 can retrieve complex information (e.g., the information needed is located in multiple parts of the text). They can link multiple pieces of information to draw inferences. They can integrate text-wide information in order to identify the main themes in a text. As well as using discrete or explicit information, they can use the text as a whole to interpret character behaviour. As well as Level 1 and 2 skills, pupils can process texts at a whole-text level, in order to retrieve information. They can make basic-level inferences, sometimes linking one or two discrete pieces of information. They can infer word meanings if the context provides clear clues. As well as Level 1 skills, pupils can retrieve explicitly stated information where the wording of the question and the text differ. However, the information sought must be specific to a small section of text. They can make low-level inferences, including character motives, if the required information is explicitly stated in a specific section of the text. NA 09 % NA 14 % 10.0 13.5 25.0 32.0 30.0 32.8 224 1 187 Level 1 pupils show basic reading skills. They can retrieve simple, explicitly stated, pieces of information, when there is a direct match between the wording of the question and the text. They are most successful on tasks that require comprehension of smaller units of text, such as sentences. They can perform some very basic interpretation and integration of text (e.g., identifying the theme of a text, where the theme is explicitly stated in the text). 25.0 16.5 <187 <1 Pupils below proficiency Level 1 have a less than 62.5% chance of correctly answering a Level 1 question. Their reading skills are very low, relative to other 2nd class pupils. 10.0 5.1 3

Sixth class reading Table 2: Descriptions of the four levels of proficiency on the overall English reading scale, at Sixth class, and the percentages of all pupils performing at each level in the 2009 and 2014 National Assessments Level & score range 4 317+ 316 3 271 What pupils can typically do NA 09 NA 14 As well as skills exemplifying lower levels, pupils at proficiency Level 4 show advanced retrieval skills. They can find answers where the phrasing of the text and question differ considerably. They do not need to rely on explicitly stated information or connections, but can infer answers from multiple pieces of text, and use broad themes at whole-text level to infer an answer. They can evaluate the rationale behind a piece of text, even where the text covers multiple events/topics, and the overall rationale is not apparent unless analysed at a global level. As well as Levels 1 and 2 skills, pupils at Level 3 have complex retrieval skills. They can examine multiple elements of the text to locate the correct response and rule out incorrect responses. They can answer items where the phrasing in the text and question are not identical, and locate detail in dense texts such as advertisements or dictionaries. Pupils at Level 3 have more strongly established inferencing skills (e.g., they are consistently able to link two pieces of information from a text to infer the correct response). They can interpret meanings at whole-text level, and integrate this with personal knowledge or experience, in order to identify a correct response. They can use opinion and external knowledge to evaluate arguments made, the clarity of information presented, or the structure and appeal of texts. 270 Pupils at Level 2 can carry out multipart retrieval processes, such as answering questions that use a modified version of the phrasing in the text. They can also match question content with information in the stimulus text that extends beyond one or two adjacent sentences, provided that the question is an almost 2 literal match with text content. They can combine two pieces of non-adjacent information in the text to infer a response, but their skills at this level are not consistent. They demonstrate integration skills such as identifying overall themes from texts, or drawing on outside 230 knowledge. 229 1 183 <183 <1 Pupils at Level 1 can carry out basic retrieval processes and can match words and phrases in the question with the same words and phrases in the stimulus text to answer items. They can also make low-level inferences, where at least part of the information required for the answer is explicitly stated in the text, or where a discrete piece of explicitly stated text coupled with very basic external knowledge is sufficient to answer the question. Pupils at this level can also engage in some interpretation and integration of information, such as identifying an idea or theme in a section of text. They can identify the rationale behind a piece of text where it is clearly flagged (for example, in the title). Pupils below proficiency Level 1 have a less than 62.5% chance of correctly answering the easiest questions. Their reading skills are very low, relative to other 6 th class pupils. 10.0 14.3 25.0 29.6 30.0 31.3 25.0 19.8 10.0 5.0 4

Reading Proficiency post-primary level In PISA, seven levels of reading proficiency (Level 1b to Level 6) are described for 15 yearolds. 1 Students who do not demonstrate the skills required to answer the easiest PISA items are classified as performing below Level 1b. Level 6 is the highest level and students performing at this level are expected to complete the most difficult items on the PISA reading test. In PISA, Level 2 is considered to be a baseline level of proficiency needed to participate effectively and productively in society and in future learning. Students performing at levels 1a and 1b have not successfully demonstrated such skills and are considered to be able to complete only the most basic PISA items. Table 3 presents descriptions of the types of reading skills that students performing at each of the PISA proficiency levels can demonstrate, along with the percentages of all students in Ireland and on average across OECD countries performing at each level in the most recent cycle of PISA (PISA 2015). 1 15 year-olds tested in PISA can be from different year groups across Second Year, Third Year, Transition Year and Fifth Year. 5

Post-primary reading Table 3. Descriptions of the seven levels of proficiency on the PISA 2015 reading literacy scale and percentages of 15 year-olds achieving each level, in Ireland and on average across OECD countries Level & score range 6 698+ Students at this level are capable of: Making multiple inferences, comparisons and contrasts that are both detailed and precise; demonstrating a full and detailed understanding of one or more texts that may involve integrating information from more than one text; dealing with unfamiliar ideas in the presence of prominent competing information, and generating abstract categories for interpretations; hypothesising about or critically evaluating a complex text on an unfamiliar topic, taking into account multiple criteria or perspectives, and applying sophisticated understanding from beyond the text; precision of analysis and fine attention to detail that is inconspicuous in the texts is needed. 697 Locating and organising several pieces of deeply embedded information, inferring which information in the text is relevant; engaging in critical evaluation or hypothesis 5 formulation, drawing on specialised knowledge; a full and detailed understanding of a text whose content or form is unfamiliar. For all aspects of reading, tasks at this level 626 typically involve dealing with concepts that are contrary to expectations. 625 Locating and organising several pieces of embedded information; interpreting the meaning of nuances of language in a section of text by taking into account the text as a whole; understanding and applying categories in an unfamiliar context; using formal or 4 public knowledge to hypothesise about or critically evaluate a text. Readers must demonstrate an accurate understanding of long or complex texts whose content or form 553 may be unfamiliar. 552 Locating, and in some cases recognising the relationship between several pieces of information that must meet multiple conditions; integrating several parts of a text in order to identify a main idea; understanding a relationship or construing the meaning of a word or phrase; taking into account many features in comparing, contrasting or categorising often the required information is not prominent or there is much competing information; 3 or there are other text obstacles, such as ideas that are contrary to expectations or negatively worded; making connections, comparisons and explanations, or evaluating a feature of the text to demonstrate a fine understanding of the text in relation to familiar, everyday knowledge. Other tasks do not require detailed text comprehension but require 480 the reader to draw on less common knowledge. 479 Locating one or more pieces of information, which may need to be inferred and may need to meet several conditions; recognising the main idea in a text, understanding relationships, or construing meaning within a limited part of the text when the information 2 is not prominent and the reader must make low level inferences; Tasks may involve comparisons or contrasts based on a single feature in the text; making a comparison or several connections between the text and outside knowledge, by drawing on personal 407 experience and attitudes. 406 Locating one or more independent pieces of explicitly-stated information; recognising the main theme or author s purpose in a text about a familiar topic, or making a simple connection between information in the text and common, everyday knowledge; the required information in the text is prominent and there is little, if any, competing information. The reader is explicitly directed to consider relevant factors in the task and 335 in the text. 1a 334 1b 262 <262 <1b Locating a single piece of explicitly stated information in a prominent position in a short, syntactically simple text with a familiar context and text type, such as a narrative or a simple list. The text typically provides support to the reader, such as repetition of information, pictures or familiar symbols. There is minimal competing information. In tasks requiring interpretation the reader may need to make simple connections between adjacent pieces of information. There is insufficient information on which to base a description of the reading skills of these students Adapted from OECD (2016b), Figure I.4.7. Ireland % OECD % 1.3 1.1 9.4 7.2 26.4 20.5 31.8 27.9 21.0 23.2 8.3 13.6 1.7 5.2 0.2 1.3 6

Mathematics Proficiency primary level As with reading, four proficiency levels are described for mathematics at both Second and Sixth class, in NAERM. Pupils performing at the highest level, Level 4, are likely to be able to complete the most complex tasks expected of their grade level, while those performing at Level 1 are expected to be able to complete only the most basic tasks. Pupils who do not reach Level 1 are not consistently able to display the reading skills assessed by the simplest items on the test. Table 4 provides a description of the types of mathematics tasks that pupils performing at each of the proficiency levels are capable of performing at Second class, along with the percentages of pupils performing at each of these levels in NAERM 2009 and 2014. Table 5 presents the corresponding information for Sixth class. 7

Second class mathematics Table 4: Descriptions of the four levels of proficiency on the overall mathematics scale, at Second class, and the percentages of all pupils performing at each level in the 2009 and 2014 National Assessments Level & score range What pupils can typically do NA 09 % NA 14 % 4 315+ Pupils at Level 4 can calculate the cost of items which may be bought with a given sum of money, and can calculate the best estimate of the sum or difference of two two-digit numbers. They show understanding of the associative property of addition; the connection between two-step word problems and their corresponding numerical expressions; and the correct use of the symbols =, <, >. They can measure length using metres and centimetres and measure area using a non-standard unit. They can interpret information from a bar-line graph and make a calculation with it. They can solve one-step word problems involving: repeated addition; addition or subtraction of clock times; halves and quarters of metres, kg, and litres. They can solve two-step word problems involving addition and subtraction of two-digit numbers and money. 10 14.5 314 3 270 269 2 232 231 1 184 <184 <1 Pupils at Level 3 can recall the subtraction facts, add a row of three numbers with renaming within 99, and find the difference between two two-digit numbers. They can use the vocabulary of ordinal number, and convert tens and units to numbers from 10 to 199. They can extend number patterns, identify quarters of 2-D shapes, and partition a 2-D shape into two other shapes. They can use the concept of an angle as a rotation, use a calendar to read days, dates, months and seasons, and select appropriate nonstandard units for measuring capacity. They can exchange coins. They can also solve: one-step word problems involving: addition or subtraction of two-digit numbers; halves and quarter of sets of up to 20 objects; addition or subtraction of money, cm and m, kg or litres; time in hr and min on 12-hour clock. They can solve one-step and two-step word problems involving minutes, hours and days. Pupils at Level 2 can be expected to add columns of three numbers with renaming within 99. They can identify odd and even numbers. They can use the symbols +, - to complete number sentences. They can identify halves of sets with up to 20 objects. Pupils at this level can combine two 2-D shapes to make other shapes. They can identify properties of 3-D shapes and compare lengths of objects in non-standard units. Pupils at this level can convert analogue to digital time (to the half-hour), and interpret information in simple block graphs. They can solve one-step word problems involving addition or subtraction of simple whole numbers. Pupils at Level 1 can be expected to count objects in groups of threes and fives; use ordinal number; locate numbers within specified intervals up to 199; connect verbal and numerical forms of numbers, up to 199; and recall the addition facts. They can use the vocabulary of spatial relations to locate objects; identify and classify simple 2-D and 3-D shapes and list some of their properties. They can identify half of a regular 2-D shape. Pupils at this level can use the vocabulary of time to sequence events, and identify a date in a calendar. They can find the value of a group of coins. They can read a simple block graph. Pupils below proficiency level 1 have a less than 62.5% chance of correctly answering a Level 1 question. Their mathematical skills are very low, relative to other 2nd class pupils. 25 32.7 30 27.3 25 19.4 10 6.2 8

Sixth class mathematics Table 5: Descriptions of the four levels of proficiency on the overall mathematics scale, at Sixth class, and the percentages of pupils performing at each level in the 2009 and 2014 National Assessments Level & score range 4 316+ What pupils can typically do NA 09 NA 14 Pupils at Level 4 can multiply and divide decimals by decimals, and carry out simple algebraic procedures involving evaluation of linear expressions and one-step equations. They can demonstrate a high level of understanding of signed integers and number theory concepts such as prime and composite numbers. They can deduce symbolic rules for simple functions. At this level pupils can also analyse geometric shapes in detail and deduce rules about them. They can construct circles. They can plot coordinates and use scales on maps or plans to calculate distances and areas. They can solve non-routine and multi-step practical problems involving ratios, mixed numbers, percentage gain or loss, value for money comparisons, currency conversions, speed, and time zones. 315 Pupils at Level 3 can add and subtract mixed numbers and decimals. They can demonstrate understanding of decimal notation, factors and multiples, exponents, and square roots. They can connect verbal and symbolic representations of word problems. They can construct and measure angles and construct triangles and rectangles given selected sides and 3 angles. Pupils at this level can classify triangles and quadrilaterals based on angle and line properties and rules. They can identify properties of 3-D shapes. They can manipulate commonly used units of area, capacity and weight. They can read, interpret, and analyse pie-charts, multiple-bar bar-charts and trend graphs. They can estimate simple probabilities. They can solve routine and non-routine word 273 problems involving operations with fractions, decimals and percentages, length and perimeter, capacity, and time. 272 Pupils at Level 2 can multiply fractions and decimals, estimate products, calculate common factors and multiples of whole numbers, and convert fractions and decimals to percentages. They can identify prime numbers within 30 and identify rules for number patterns. They can demonstrate understanding of a 2 letter as a placeholder in algebraic expressions, and complete two-step number sentences involving addition and subtraction. Pupils at this level can construct lines and circles, estimate angles and use properties of shapes to calculate line and angle sizes. They can make logical deductions from simple data sets. They can solve multi-step word problems involving operations 230 with integers, fractions and percentages. 229 Pupils at Level 1 can add, subtract, and round whole numbers and decimals. They show understanding of whole number notation and can connect numeric and verbal representations of large numbers. Pupils at this level can classify angles and 1 identify templates of simple 3-D shapes. They can manipulate commonly used units of length. They can read and interpret, without calculation, simple frequency tables, pie-charts, bar charts and trend graphs. They can solve routine word problems 184 involving the four operations with whole numbers. <184 <1 Pupils below proficiency Level 1 have a less than 62.5% chance of correctly answering a Level 1 question. Their mathematical skills are very low, relative to other 6th class pupils and are not properly assessed by the National Assessments. 10.0 14.9 25.0 27.2 30.0 31.4 25.0 20.9 10.0 5.6 9

Mathematics Proficiency post-primary level Six levels of mathematics proficiency (levels 1 to 6) are described in PISA for 15 year-olds. 2 Students who do not demonstrate the skills required to answer the easiest PISA items are classified as performing below Level 1. The highest level, Level 6, represents the skills required to successfully complete the most complex PISA mathematics items. As with reading, Level 2 is considered to be the baseline level of mathematics proficiency required to participate fully in society and in future learning. Students performing at or below Level 1 are likely to correctly answer only the easiest items on the PISA mathematics assessment. Table 6 presents descriptions of the types of mathematics skills that 15 year-old students performing at each of the proficiency levels can demonstrate, along with the percentages of students in Ireland and on average across OECD countries performing at each level in the most recent cycle of PISA (PISA 2015). 2 15 year-olds tested in PISA can be from different year groups across Second Year, Third Year, Transition Year and Fifth Year. 10

Post-primary mathematics Table 6. Descriptions of the six levels of proficiency on the PISA 2015 mathematical literacy scale and percentages of students achieving each level, in Ireland and on average across OECD countries Level & score range 6 669+ Students at this level are capable of: Ireland OECD Conceptualising, generalising and using information based on their investigations and modelling of complex problem situations; using knowledge in relatively non-standard contexts; linking different information sources and representations and moving flexibly among them; applying their insight and understanding, along with mastery of symbolic and formal mathematical operations and relationships, to develop new approaches and strategies for addressing novel situations; reflecting on their actions and formulating and precisely communicating their actions and reflections regarding their findings, interpretations and arguments, and explaining why they were applied to the original situation. Students at this level are able to successfully complete the most difficult PISA items. 668 Developing and working with models of complex situations, including identifying constraints and specifying assumptions; selecting, 5 comparing and evaluating appropriate problem-solving strategies for dealing with complex problems related to these models; working strategically using broad, well-developed thinking and reasoning skills, appropriate linked representations, symbolic and formal 607 characterisations and insights pertaining to these situations; beginning to reflect on their work and formulating and communicating their interpretations and reasoning. 606 Working effectively with explicit models of complex, concrete situations that may involve constraints or making assumptions; selecting and 4 integrating different representations (including symbolic representations) and linking them directly to aspects of real-world situations; using their limited range of skills and reasoning with some insight in straightforward contexts; constructing and communicating explanations and arguments based on their interpretations, arguments 545 and actions. 544 Executing clearly described procedures, including those that require sequential decisions; making sufficiently sound interpretations to be a base for building a simple model or for selecting and applying simple 3 problem solving strategies; interpreting and using representations based on different information sources and reasoning directly from them; showing some ability to handle percentages, fractions and decimal numbers, and to work with proportional relationships; 482 engaging in basic interpretation and reasoning. 481 Interpreting and recognising situations in contexts that require no more than direct inference; extracting relevant information from a single 2 source and make use of a single representational mode; employing basic algorithms, formulae, procedures or conventions to solve problems involving whole numbers; making literal interpretations of the 420 results. 419 1 358 Answering questions involving familiar contexts where all relevant information is present and the questions are clearly defined; able to identify information and to carry out routine procedures according to direct instructions in explicit situations; can perform actions that are almost always obvious and follow immediately from the given stimuli. <358 Performing very direct and straightforward mathematical tasks, such as reading a single value from a well-labelled chart or table where the labels on the chart match the words in the stimulus and question, so <1 that the selection criteria are clear and the relationship between the chart and the aspects of the contexts depicted are evident; performing arithmetic calculations with whole numbers by following clear and welldefined instructions. Adapted from OECD (2016b), Figure I.5.7. 1.5 2.3 8.3 8.4 21.2 18.6 30.0 24.8 24.1 22.5 11.5 14.9 3.5 8.5 11