Pan-Canadian Assessment Program (2010) Ontario Report

Pan-Canadian Assessment Program (2010) Ontario Report November 2011

Pan-Canadian Assessment Program (2010) Ontario Report November 2011

Contents Introduction 1 Achievement Results 2 Figure 1: Mean Mathematics Scale Score by Jurisdiction 2 Figure 2: Mean Science Scale Score by Jurisdiction 3 Figure 3: Mean Reading Scale Score by Jurisdiction 3 Table 1: Mean Mathematics Score by Province English 4 Table 2: Mean Mathematics Score by Province French 4 Table 3: Mean Science Score by Province English 5 Table 4: Mean Science Score by Province French 5 Table 5: Mean Reading Score by Province English... 6 Table 6: Mean Reading Score by Province French... 6 Figure 4: Percentage of Students at Each Level of Performance in Mathematics by Jurisdiction... 7 Figure 5: Results for Mathematics Subdomains by Jurisdiction.. 8 Background. 9 The Development Process.. 9 The Reading Component 9 The Mathematics Component 10 The Science Component 10 Scoring the Student Booklets 10 Guidelines for Interpreting Results.. 11 Level 1 11 Level 2 11 Level 3 11 Level 4 11 National Assessments in Previous Years. 12 Education Quality and Accountability Office I Pan-Canadian Assessment Program (2010), Ontario Report

Introduction The Pan-Canadian Assessment Program (PCAP) is a national assessment program that measures the reading, mathematics and science achievement of Grade 8 students in the Canadian provinces and territories. It has been administered in 2007 and 2010; in 2010 all 10 provinces and Yukon (referred to as jurisdictions in this report) participated. For each administration of PCAP, one subject is designated major and the other two are minor. The major subject area has a larger number of assessment items, which enables the reporting of results for subdomains; only overall results are reported for the minor subject areas. In addition, a standard setting process is conducted to determine cut points for reporting results at four performance levels for the major subject area. In 2010, mathematics was the major subject area. Each student responded to questions in all three subject areas, but subsamples of students answered different questions. Most students were expected to complete the assessment in 90 minutes. Additional time, up to 30 minutes, was permitted for students who required it. Education Quality and Accountability Office 1 Pan-Canadian Assessment Program (2010), Ontario Report

Achievement Results Results for PCAP are reported on a scale with a mean of 500 and a standard deviation of 100. In addition, results for mathematics are reported according to four performance levels. Descriptions of the assessment instruments and the reporting categories are presented later in this report. The following are key findings: Ontario and Quebec were the only provinces with results above the in mathematics. Ontario and Alberta were the only provinces with results above the in science. Ontario was the only province with results above the in reading, and the Ontario average in reading was higher than those for all other jurisdictions. Ontario was the only province with results above the in all three subjects. In 2007, Ontario was at the in all three subjects. In mathematics, the major component, 92% of Ontario students achieved Level 2 the expected level of performance for Grade 8 students or above. See Figures 1 to 3, which show which jurisdictions are above, at and below the scale score, as well as the confidence interval for each jurisdiction. * Figure 1: Mean Mathematics Scale Score by Jurisdiction Mathematics Results by Jurisdiction Average Scale Score and Confidence Interval Above the At the Below the Quebec (515 3.9) Ontario (507 4.0) Canada (500 2.2) Alberta (495 4.0) British Columbia (481 3.6) New Brunswick (478 3.9) Saskatchewan (474 3.8) Nova Scotia (474 3.9) Newfoundland and Labrador (472 5.2) Yukon (469 7.7) Manitoba (468 4.2) Prince Edward Island (460 8.3) 440 460 480 500 520 540 95% Confidence Interval * If the confidence intervals for two jurisdictions overlap, the difference between the average scores for the jurisdictions is not statistically significant. Pan-Canadian Assessment Program (2010), Ontario Report 2 Education Quality and Accountability Office

Achievement Results, cont d Figure 2: Mean Science Scale Score by Jurisdiction Science Results by Jurisdiction Average Scale Score and Confidence Interval 440 460 480 500 520 540 Above the At the Alberta (515 3.7) Ontario (510 4.1) Canada (500 2.0) British Columbia (497 3.4) Prince Edward Island (493 10.2) Nova Scotia (489 4.0) Saskatchewan (488 4.2) Below the Newfoundland and Labrador (487 5.8) New Brunswick (487 3.9) Quebec (486 3.8) Manitoba (486 3.9) Yukon (478 7.8) 95% Confidence Interval Figure 3: Mean Reading Scale Score by Jurisdiction Reading Results by Jurisdiction Average Scale Score and Confidence Interval Above the At the Below the Ontario (515 3.9) Alberta (506 4.0) Canada (500 2.2) British Columbia (499 3.7) Saskatchewan (491 3.9) Nova Scotia (489 4.0) Newfoundland and Labrador (486 5.2) Quebec (481 3.6) Prince Edward Island (481 9.0) New Brunswick (479 3.9) Manitoba (478 3.8) Yukon (465 7.1) 440 460 480 500 520 540 95% Confidence Interval Education Quality and Accountability Office 3 Pan-Canadian Assessment Program (2010), Ontario Report

Achievement Results, cont d The results for English- and French-language students are presented in Tables 1 to 6 for a number of provinces. Table 1: Mean Mathematics Score by Province English Jurisdiction Mean score and confidence interval Above Canadian mean score ONe 507 ± 5 QCe 507 ± 7 At Canadian mean score CANe 495 ± 2 ABe 495 ± 4 Below Canadian mean score BCe 481 ± 4 SKe 474 ± 4 NSe 473 ± 4 NLe 472 ± 5 YKe 468 ± 8 MBe 467 ± 4 NBe 466 ± 5 PEe 460 ± 10 Table 2: Mean Mathematics Score by Province French Jurisdiction Mean score and confidence interval At Canadian mean score QCf 516 ± 3 CANf 515 ± 4 ONf 511 ± 4 NBf 507 ± 5 Below Canadian mean score ABf 504 ± 5 BCf 504 ± 5 NSf 503 ± 3 SKf 498 ± 7 MBf 480 ± 3 Pan-Canadian Assessment Program (2010), Ontario Report 4 Education Quality and Accountability Office

Achievement Results, cont d Table 3: Mean Science Score by Province English Jurisdiction Mean score and confidence interval Above Canadian mean score ABe 515 ± 3 At Canadian mean score ONe 510 ± 4 CANe 504 ± 3 BCe 497 ± 4 PEe 493 ± 11 Below Canadian mean score QCe 490 ± 6 NSe 489 ± 4 NBe 489 ± 5 SKe 488 ± 4 NLe 487 ± 6 MBe 486 ± 5 YKe 478 ± 9 Table 4: Mean Science Score by Province French Jurisdiction Mean score and confidence interval At Canadian mean score ABf 506 ± 6 NSf 501 ± 3 ONf 497 ± 4 Below Canadian mean score BCf 496 ± 6 CANf 487 ± 3 QCf 486 ± 3 SKf 486 ± 7 MBf 482 ± 4 NBf 482 ± 5 Education Quality and Accountability Office 5 Pan-Canadian Assessment Program (2010), Ontario Report

Achievement Results, cont d Table 5: Mean Reading Score by Province English Jurisdiction Mean score and confidence interval Above Canadian mean score ONe 517 ± 5 At Canadian mean score CANe 507 ± 2 ABe 506 ± 4 Below Canadian mean score BCe 499 ± 4 SKe 492 ± 4 QCe 492 ± 6 NSe 489 ± 4 NLe 486 ± 5 NBe 486 ± 5 PEe 482 ± 10 MBe 478 ± 4 YKe 464 ± 7 Table 6: Mean Reading Score by Province French Jurisdiction Mean score and confidence interval Above Canadian mean score ABf 490 ± 5 At Canadian mean score ONf 481 ± 4 QCf 480 ± 4 CANf 480 ± 4 NSf 475 ± 3 BCf 473 ± 5 Below Canadian mean score MBf 468 ± 4 SKf 468 ± 8 NBf 464 ± 5 Pan-Canadian Assessment Program (2010), Ontario Report 6 Education Quality and Accountability Office

Achievement Results, cont d As shown in Figure 4, 92% of Ontario students were at or above the expected level of performance in mathematics (Level 2 or above), and 50% were at Level 3 or above. These percentages were 91% and 47%, respectively, for Canada. Figure 4: Percentage of Students at Each Level of Performance in Mathematics by Jurisdiction * AB QC ON CAN SK BC NB NL NS PE YK MB 7% 50% 40% 3% 8% 38% 50% 4% 8% 43% 45% 5% 9% 45% 43% 4% 10% 55% 33% 1% 11% 50% 37% 2% 11% 52% 35% 2% 12% 52% 35% 2% 12% 53% 32% 2% 13% 57% 29% 0% 14% 53% 30% 3% 16% 50% 33% 1% 0% 20% 40% 60% 80% 100% Level 1 Level 2 Level 3 Level 4 * Percentages do not add up to 100 for some jurisdictions, due to rounding. Education Quality and Accountability Office 7 Pan-Canadian Assessment Program (2010), Ontario Report

Achievement Results, cont d There were no significant differences between the average scale scores of male and female Ontario students for mathematics. The average scores for female students were higher than those for male students in science (517 compared with 505) and reading (530 compared with 503). Results were reported for four subdomains in mathematics. Of the two provinces that had overall mean scores above the Canadian average, Ontario had an average score on patterns and relationships higher than Quebec s and a lower average score for number and operations than Quebec s. The subdomain results are presented in Figure 5. Figure 5: Results for Mathematics Subdomains by Jurisdiction 540.00 530.00 95% Confidence Interval 520.00 510.00 500.00 490.00 480.00 470.00 460.00 450.00 440.00 British Columbia Alberta Saskatchewan Manitoba Ontario Quebec New Brunswick Nova Scotia Prince Edward Island Newfoundland and Labrador Yukon Numbers and operations Geometry and measurement Patterns and relationships Data management and probability Pan-Canadian Assessment Program (2010), Ontario Report 8 Education Quality and Accountability Office

Background The Pan-Canadian Assessment Program (PCAP), the most recent initiative of the Council of Ministers of Education, Canada (CMEC), was designed to inform Canadians about how well their education systems are meeting the needs of students and society. The information gained through the program gives ministers of education a basis for examining the curriculum and other aspects of their school systems. PCAP also complements existing assessments in each province and territory. School programs differ from one part of the country to another, making comparisons of results from this assessment program a complex task. However, young Canadians in different provinces and territories learn many similar skills in reading, mathematics and science. PCAP provides data on the achievement levels attained by Grade 8 students across the country and is designed to determine whether students across Canada are reaching similar levels of performance. PCAP is scheduled to be administered every three years and was first administered in 2007. Reading, mathematics and science tests were administered for the second time in the spring of 2010 in randomly selected Grade 8 classes in randomly selected schools. These sampling procedures allow jurisdictions to interpret and attribute the performance data and the distribution of scores to their population as a whole with confidence. Approximately 32 000 students (24 000 English-language students and 8000 French-language students) across Canada wrote the tests in 2010. In Ontario, the tests were administered to 3374 students in 144 English schools and 2509 students in 142 French schools. The Development Process For each subject area, a thorough review of curricula, current assessment practices and research literature was undertaken to identify the expectations common to all jurisdictions. Frameworks were developed by representatives from English- and French-language education systems, and these were reviewed and accepted by all participating jurisdictions as the basis for developing test items. Test items were developed by bilingual teams composed of subject area educators from all jurisdictions with a subject assessment expert to supervise. Questions were developed in both official languages and cross-translated to be equivalent in meaning and difficulty. The jurisdictions reviewed and confirmed the validity of the French-English translations to ensure fair and equitable testing in both languages. Items were field tested in both languages and the final test booklets were then reviewed and approved by all jurisdictions. Both multiple-choice and open-response items for all three subject areas are included in the assessments. The Reading Component According to the curricula delivered in the provinces and territories, reading is a dynamic, interactive process whereby the reader constructs meaning from texts. The process of reading involves the interaction of reader, text, purpose and context before, during and after reading. In order to make meaning of a text, readers must make a connection between what is in the text and what they know or bring to it. Students knowledge and skills determine their degree of access to particular types and forms of texts. Knowledge of language, facility with language strategies and knowledge of the way language works in print affect the student s construction of meaning. In light of the interactive process of reader, text, purpose and context, the PCAP reading component considers the reader s engagement with text and response to it. The assessment included a range and variety of text types and forms, of varying levels of difficulty, to assess comprehension, interpretation and response to text. The texts were broadly identified as fiction and non-fiction, recognizing that texts frequently mix forms or types for a variety of purposes. The texts selected for PCAP were consistent with a broad range of student reading. Education Quality and Accountability Office 9 Pan-Canadian Assessment Program (2010), Ontario Report

The Mathematics Component The PCAP mathematics component is aligned with the curricula of the participating jurisdictions as well as the standards of the National Council of Teachers of Mathematics (NCTM). For the purposes of this assessment, mathematics was broadly defined as a conceptual tool that students can use to increase their capability to calculate, describe and solve problems. The subject area was divided into four strands (number and operations; geometry and measurement; patterns and relationships; and data management and probability) and five processes (problem solving; communication; representations; reasoning and proof; and connections). The Science Component The concept of scientific literacy is generally accepted as the overarching goal of science curricula across Canada. The PCAP science component is founded on a definition of scientific literacy that advocates the development of evolving competencies of using science-related attitudes, skills and knowledge as well as an understanding of the nature of science that enables students to conduct inquiries, solve problems and make evidence-based decisions about science-related issues. Embedded in this definition of scientific literacy is the supposition that students have knowledge of the life sciences, physical sciences (chemistry and physics) and Earth and space sciences, as well as an understanding of the nature of science as a human endeavour. The science items cover five subdomains (nature of science, nature of technology, knowledge of science, skills and attitudes) and three competencies (science inquiry, problem solving and decision making). Scoring the Student Booklets The scoring was conducted concurrently in both languages in one location over three weeks. The scoring administration team, the table leaders and the scorers themselves came from a range of jurisdictions. The process included parallel training of both table leaders and scorers in each subject area; a bilingual committee with responsibility for reviewing all instruments and anchor papers to ensure comparability at every level; twice-daily rater-reliability checks, in which all scorers assessed the same student booklet in order to track the consistency of scoring immediately and double-blind scoring, in which 300 of each of the three booklets were returned to the scoring bundles to be rescored to provide an overall reliability measure. Pan-Canadian Assessment Program (2010), Ontario Report 10 Education Quality and Accountability Office

Guidelines for Interpreting Results The results of student performance on the 2010 PCAP mathematics component are presented in this report in two ways: as average overall scores on a scale with a mean of 500 and a standard deviation of 100 and as the percentages of students achieving four performance levels. For the reading and science components, results are presented only as average scale scores. Performance levels were established only for the mathematics component, as it was the major subject area for the 2010 PCAP assessment. The four performance levels represent how performances measured up to the expected level of achievement according to two factors: cognitive demand and degree of difficulty. The cognitive demands are defined by the level of reasoning required by the student to answer an item correctly, from low demand to high demand, while the levels of difficulty are determined statistically based on the collective performance of the students on the assessment. To accomplish this, a standard-setting exercise involving a group of selected educators from each jurisdiction set the cut scores for each level using the bookmark method: that is, determining the relative difficulty of the full set of assessment items and delineating the point along a scale that defines the achievement of each level of success, thus determining the cut score. Once suitable cut scores were set, student performance within the range of cut scores could be described. These descriptors of achievement-level results indicate the degree to which performance meets expectations of what students should know and be able to do at each level. The four performance levels for PCAP cannot be compared with the four achievement levels used to report EQAO results. Level 2 was established as the expected level of performance. The four levels of performance as determined by the cut scores were described as follows: Level 1 I Scores: 357 and below Students at this level were able to solve problems that were determined to be fairly easy and at a low cognitive level. Typically, at this level, students were able to retrieve information from a graph or solve previously learned routine problems. Students at this level were able to respond correctly to items that were mostly at the recall and recognition level. Level 2 I Scores: 358 513 Students at this level were required to recall facts, definitions or terms and carry out previously learned procedures such as performing one or more operation, employing formulas, evaluating a variable expression, and retrieving information from a table or a graph and applying it to solve a problem. Typically students at this level were able to identify a number of simple geometric patterns. The requirements of the items students were able to solve were clearly defined, with no extraneous information or hidden assumptions. Students at this level were able to respond correctly to items of low or moderate cognitive demand. Level 3 I Scores: 514 668 Students at this level were able to apply what they know to new situations, identify hidden assumptions and distinguish between relevant and irrelevant information to solve a problem. They had to select appropriate procedures or strategies and sometimes had to apply skills from different domains to solve problems. They were able to represent a problem in different ways and use informal reasoning to devise a solution. Students at this level were able to respond correctly to items of moderate to high cognitive demand. Level 4 I Scores: 669 and above Students at this level were able to solve problems that require complex reasoning at the analysis and synthesis levels. Solutions clearly show a mastery of the necessary conceptual and procedural knowledge. Students were able to generalize a pattern and write the rule algebraically. They were also able to explain or justify their solutions and strategies clearly. Students at this level were able to respond correctly to items that were generally determined to be difficult and of high cognitive demand. Education Quality and Accountability Office 11 Pan-Canadian Assessment Program (2010), Ontario Report

In this assessment, the reported average scores are estimates of the achievement results students would have demonstrated had all students in the population participated in the assessment. Because an estimate based on a sample is rarely exact, it is common practice to provide a range of scores for each jurisdiction within which the actual achievement level might fall. The range of scores expressed for each average score is called a confidence interval. It represents the high- and low-end points between which the actual average should fall 95% of the time. In other words, one can be confident that the actual achievement level of all students would fall somewhere in the established range 19 times out of 20 if the assessment were repeated with different samples from the same student population. A difference between jurisdiction or group means is statistically different when there is no overlap of confidence intervals from the two jurisdictions or groups being compared. Caution is advised when comparing achievement results based on assessment instruments prepared collaboratively in both languages, despite the extensive efforts to ensure equivalence for the sake of equity and fairness for all students. Every language has unique features that are not readily equivalent. While the reading items, performance descriptors, scoring scripts and processes in English and French were equivalent, pedagogical and cultural differences related to differences in language structure and use render comparisons between languages inherently difficult. Comparisons should therefore be made with extreme caution. National Assessments in Previous Years Prior to PCAP, the last national assessment administered in Canada was a science assessment administered in 2004 as part of the School Achievement Indicators Program (SAIP), which has now been replaced by PCAP. SAIP was initiated in 1993 by the CMEC and assessed 13-year-old and 15-year-old students in reading, writing, mathematics and science. Mathematics was last assessed in 2001 and reading was last assessed in 1998. Since the SAIP assessment results were reported at five performance levels, it is not possible to draw comparisons between achievement levels on PCAP and SAIP. PCAP sets a new benchmark for comparisons in future years. In the SAIP assessment, Ontario results tended not to be significantly different from the results for Canada overall. For the 2010 PCAP assessment, the Ontario average was significantly higher than the in all three subjects. Ontario was at the in all three subjects for the 2007 PCAP assessments. The scale scores from the 2010 PCAP assessment cannot be compared with those from 2007, because the assessment was administered to Grade 8 students in 2010 and to 13-year-old students, regardless of grade, in 2007. A separate analysis of the 2007 reading results for Grade 8 students was conducted. A comparison of the results of this analysis with the 2010 PCAP results showed no change in reading from 2007 to 2010 for Ontario students. Pan-Canadian Assessment Program (2010), Ontario Report 12 Education Quality and Accountability Office

2 Carlton Street, Suite 1200, Toronto ON M5B 2M9 Telephone: 1-888-327-7377 Web site: www.eqao.com 2011 Queen s Printer for Ontario