Pan-Canadian Assessment Program (2007) Ontario Report: English-Language Students

Pan-Canadian Assessment Program (2007) Ontario Report: English-Language Students April 2008

Pan-Canadian Assessment Program (2007) Ontario Report: English-Language Students April 2008

Contents Introduction........................................................................................ 1 Achievement Results................................................................................ 1 Figure 1: Reading Results by Jurisdiction English and French Combined...................................... 1 Figure 2: Reading Results by Jurisdiction English........................................................ 2 Figure 3: Mathematics Results by Jurisdiction English and French Combined................................. 3 Figure 4: Mathematics Results by Jurisdiction English.................................................... 4 Figure 5: Science Results by Jurisdiction English and French Combined...................................... 5 Figure 6: Science Results by Jurisdiction English......................................................... 6 Table 1: Reading Results by Percentage of Students at Different Levels English................................ 7 Figure 7: Gender Differences in Student Achievement English and French Combined........................... 8 Table 2: Reading Results by Subdomain English......................................................... 8 Background......................................................................................... 9 The Development Process.....................................................................9 The Reading Assessment............................................................................ 9 The Mathematics Assessment....................................................................... 10 The Science Assessment........................................................................... 10 Scoring the Student Booklets..................................................................10 Guidelines for Interpreting Results.............................................................11 Level 1..........................................................................................11 Level 2..........................................................................................11 Level 3..........................................................................................11 National Assessments in Previous Years.........................................................12

Introduction The Pan-Canadian Assessment Program (PCAP) is a national assessment program that is administered every three years; it measures the achievement of 13-year-old students in reading, mathematics and science in the Canadian provinces and territories. In May 2007, the ten provinces and Yukon (which are referred to as jurisdictions in this report) participated. For each administration of PCAP, one subject is selected as the major subject area and the other two as minor subject areas. The major subject area has a larger number of assessment items, which enables the reporting of results by subdomain; only overall results are reported for the minor subject areas. In addition, a standard-setting process is conducted to determine cut points for reporting results at three performance levels for the major subject area. In the 2007 administration, reading was the major subject area. Achievement Results Results for PCAP are reported on a scale from 0 to 1000, with a national mean of 500 and a standard deviation of 100. In addition, results for reading are reported according to three performance levels.the results for English-language students only and for English- and French-language students combined are presented in this report, while the results for French-language students only appear in a separate report. Descriptions of the assessment instruments and the reporting categories are presented later in this report. The key findings are presented below: Among English- and French-language students, only students in Quebec outperformed students in Ontario in reading. Among English-language students, those in Ontario outperformed students in all other jurisdictions. See Figures 1 and 2, which show which jurisdictions are above, at and below the Canadian average scale score, and the confidence intervals for each jurisdiction. * Figure 1: Reading Results by Jurisdiction English and French Combined Average scale score and confidence intervals 430 450 470 490 510 530 550 Above the Canadian average Quebec 526 +_ 6 At the Canadian average Ontario Canada 500 +_ 2 502 +_ 4 Below the Canadian average British Columbia Alberta 486 +_ 4 491 +_ 4 Manitoba Saskatchewan Nova Scotia Newfoundland and Labrador New Brunswick Yukon 464 +_ 4 464 +_ 3 472 +_ 4 471 +_ 4 471 +_ 4 486 +_ 10 95% Confidence Interval Prince Edward Island 460 +_ 5 * If the confidence intervals for two jurisdictions do not overlap, there is a statistically significant difference between the average scores for the jurisdictions. 1 Pan-Canadian Assessment Program (2007), Ontario Report

Achievement Results cont d Figure 2: Reading Results by Jurisdiction English Average scale score and confidence intervals 430 450 470 490 510 530 550 Above the Canadian average Ontario 503 +_ 5 At the Canadian average Canada 492 +_ 3 Alberta 491+_ 4 British Columbia 486 +_ 5 Yukon 486 +_ 11 Below the Canadian average Quebec 479 +_ 5 95% Confidence Interval Manitoba 476 +_ 5 Saskatchewan 471 +_ 4 Nova Scotia 471 +_ 4 New Brunswick Newfoundland and Labrador Prince Edward Island 459 +_ 4 466 +_ 4 464 +_ 5 * If the confidence intervals for two jurisdictions do not overlap, there is a statistically significant difference between the average scores for the jurisdictions. 2

Achievement Results cont d Among English- and French-language students, Ontario s were among the top in all jurisdictions in Canada. Among Englishlanguage students, only those in Ontario were above the Canadian average. See Figures 3 and 4. * Figure 3: Mathematics Results by Jurisdiction English and French Combined Average scale score and confidence intervals 430 450 470 490 510 530 550 Above the Canadian average Quebec 517 +_ 7 At the Canadian average Ontario 506 +_ 6 Alberta Canada 500+_ 3 499 +_ 7 Below the Canadian average British Columbia 484 +_ 7 Manitoba Newfoundland and Labrador 479 +_ 6 478 +_ 8 95% Confidence Interval Saskatchewan New Brunswick Nova Scotia 461 +_ 6 461 +_ 5 457 +_ 6 Yukon 451 +_ 19 Prince Edward Island 450 +_ 7 * If the confidence intervals for two jurisdictions do not overlap, there is a statistically significant difference between the average scores for the jurisdictions. 3 Pan-Canadian Assessment Program (2007), Ontario Report

Achievement Results cont d Figure 4: Mathematics Results by Jurisdiction English Average scale score and confidence intervals 430 450 470 490 510 530 550 Above the Canadian average Ontario 508 +_ 7 At the Canadian average Quebec 510 +_ 10 Alberta 500 +_ 7 Canada 496 +_ 4 Below the Canadian average British Columbia 484 +_ 7 95% Confidence Interval Manitoba 479 +_ 8 Newfoundland and Labrador 478 +_ 7 New Brunswick Saskatchewan Nova Scotia Prince Edward Island Yukon 462 +_ 6 461 +_ 6 457 +_ 6 449 +_ 8 448 +_ 19 * If the confidence intervals for two jurisdictions do not overlap, there is a statistically significant difference between the average scores for the jurisdictions. 4

Achievement Results cont d Among English- and French-language students, students in Alberta and Quebec outperformed students in Ontario in science. Among English-language students, only students in Alberta outperformed students in Ontario. See Figures 5 and 6. * Figure 5: Science Results by Jurisdiction English and French Combined Average scale score and confidence intervals 430 450 470 490 510 530 550 Above the Canadian average Alberta 524 +_ 7 Quebec 511 +_ 7 At the Canadian average Canada 500 +_ 3 Ontario 499 +_ 5 Below the Canadian average British Columbia 488 +_ 6 Newfoundland and Labrador 485 +_ 8 95% Confidence Interval Nova Scotia Saskatchewan Manitoba 476 +_ 6 480 +_ 6 480 +_ 7 New Brunswick Prince Edward Island 465 +_ 5 464 +_ 8 Yukon 462 +_ 22 * If the confidence intervals for two jurisdictions do not overlap, there is a statistically significant difference between the average scores for the jurisdictions. 5 Pan-Canadian Assessment Program (2007), Ontario Report

Achievement Results cont d Figure 6: Science Results by Jurisdiction English Average scale score and confidence intervals 430 450 470 490 510 530 550 Above the Canadian average Alberta 524 +_ 6 At the Canadian average Ontario Canada 496 +_ 4 499 +_ 7 British Columbia 488 +_ 7 Newfoundland and Labrador 485 +_ 7 Below the Canadian average Saskatchewan 480 +_ 7 95% Confidence Interval Nova Scotia 479 +_ 6 Manitoba 477 +_ 8 New Brunswick Quebec Prince Edward Island 468 +_ 6 464 +_ 8 467 +_ 10 Yukon 458 +_ 21 * If the confidence intervals for two jurisdictions do not overlap, there is a statistically significant difference between the average scores for the jurisdictions. Among English- and French-language students, only Quebec had a larger percentage of students at Level 3 (34%) than Ontario (22%). As shown in Table 1 on the next page, 90% of English-language students in Ontario were at or above the expected level of performance in reading (Levels 2 and 3), and 23% were at the highest level of performance (Level 3).These percentages were 88% and 18% respectively among all English-language students in Canada. No province outperformed Ontario with respect to these percentages. 6

Achievement Results cont d Table 1: Reading Results by Percentage of Students at Different Levels English * Jurisdiction Level 1 Level 2 Level 3 Levels 2 and 3 Ontario Alberta Canada British Columbia Saskatchewan Manitoba Quebec Nova Scotia Yukon New Brunswick Newfoundland and Labrador Prince Edward Island 10 11 12 13 14 15 16 16 17 17 19 19 67 73 70 71 77 72 70 72 65 72 69 70 23 16 18 16 9 13 14 12 18 11 12 11 90 89 88 87 86 85 84 84 83 83 81 81 * The jurisdictions are listed in order of decreasing percentage of students at Levels 2 and 3. 7 Pan-Canadian Assessment Program (2007), Ontario Report

Achievement Results cont d With an average scale score in Canada of 513 in reading, girls outperformed boys (whose average scale score was 490); 91% of females in Canada performed at Level 2 or 3 in reading, while 87% of male students achieved these levels. Similar trends were observed across jurisdictions and languages. In mathematics and science, there were no significant differences between the average scale scores of male and female students. See Figure 7. Figure 7: Gender Differences in Student Achievement English and French Combined 550 Average scale score in Canada 530 510 490 513 490 501 501 502 500 Female Male 470 450 430 Reading Mathematics Science The average scale score among English-language students in Ontario was significantly higher than the Canadian average on questions on the reading test requiring interpretation and response to text, but not on questions relating to comprehension. Table 2: Reading Results by Subdomain English * Jurisdiction Comprehension Interpretation Response to Text Alberta British Columbia Canada Manitoba New Brunswick 493 ± 4 489 ± 5 492 ± 2 483 ± 5 475 ± 4 491 ± 4 486 ± 4 492 ± 3 475 ± 5 463 ± 4 493 ± 4 489 ± 4 495 ± 2 476 ± 5 467 ± 4 Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan Yukon 466 ± 4 482 ± 4 499 ± 5 473 ± 4 483 ± 6 480 ± 4 479 ± 12 469 ± 4 467 ± 4 503 ± 4 457 ± 4 477 ± 6 469 ± 4 489 ± 11 470 ± 5 470 ± 4 506 ± 4 458 ± 4 482 ± 6 471 ± 4 494 ± 11 * The jurisdictions are listed in alphabetical order. 8

Background The most recent initiative of the Council of Ministers of Education, Canada, PCAP was designed to inform Canadians on how well their education system is meeting the needs of students and society.the information gained from the assessments gives ministers of education a basis for examining the curriculum and other aspects of their school systems. PCAP also complements existing assessments in each province and territory. School programs differ from one part of the country to another. Comparing results from these varied programs is a complex task; however, young Canadians in different jurisdictions learn many similar skills in reading, mathematics and science. PCAP provides Canada-wide data on the achievement of 13-year-olds and is designed to determine whether students across Canada reach similar levels of performance at about the same age. Reading, mathematics and science tests were administered in the spring of 2007, with a random assignment of booklets to randomly selected students in a random sample of schools. These sampling procedures allow jurisdictions to interpret and attribute the performance data and the distribution of scores to their population as a whole with confidence. The assessment instruments consisted of two 90-minute booklets on reading and one 90-minute booklet that was half mathematics and half science items. Each student wrote one reading booklet or the mathematics and science booklet. Across Canada, 35 866 13-year-olds wrote the test. Most of these students were in Grade 8 or 9. About two-thirds of the students wrote the reading test and one-third wrote the mathematics and science test. In Ontario, the tests were administered to 2476 students in 154 English schools and 2132 students in 183 French schools. The Development Process For each subject area, a thorough review of curricula, current assessment practices and research literature was undertaken to identify the common expectations among all jurisdictions. Frameworks were developed by representatives from English and French education systems, and these were reviewed and accepted by all participating jurisdictions as the basis for developing test items.test items were developed by bilingual teams composed of subject-area educators from all jurisdictions with a supervising subject assessment expert. Questions were developed in both official languages and translated to be equivalent in meaning and difficulty. Jurisdictions reviewed and confirmed the validity of the French-English translations to ensure fair and equitable testing in both languages. Items were field tested in both languages by 2000 students in 100 schools across Canada. The final test booklets were then reviewed and approved by all participating jurisdictions. The Reading Assessment According to pan-canadian curricula, reading is a dynamic, interactive process whereby the reader constructs meaning from texts.the process of reading effectively involves the interaction of reader, text, purpose and context before, during and after reading. In order to make meaning of a text, readers must make a connection between what is in the text and what they know or bring to the text. Students' knowledge and skills determine their degree of access to particular types and forms of texts. Knowledge of language, facility with language strategies and knowledge of the way language works in print affect students' construction of meaning from the text. In light of the interactive process of reader, text, purpose and context, the PCAP assessment of reading considers the reader's engagement with text and response to it.the following subdomains of reading were assessed: Comprehension: Students understand the explicit and implicit information provided by the text. Interpretation: Students make meaning by analyzing and synthesizing the parts, elements or events to develop a broader perspective and/or meaning for the text. Response to text: Students engage with the text by making personal connections between aspects of the text and their own knowledge and experiences, respond emotionally to ideas or take an evaluative stance. 9 Pan-Canadian Assessment Program (2007), Ontario Report

The assessment included a range and variety of text types and forms of varying levels of difficulty.these were broadly identified as fiction and non-fiction, recognizing that texts frequently mix forms or types for a variety of purposes. Texts selected for PCAP were consistent with a broad range of student reading experiences. The reading test included multiple-choice and short constructed-response items, plus an item for which students were required to read a short text and discuss what it meant to them. The Mathematics Assessment The PCAP mathematics component is aligned with the curricula across the participating jurisdictions in Canada as well as the standards of the National Council of Teachers of Mathematics. For the purposes of this assessment, mathematics was broadly defined as the study of patterns and relationships and as a discipline involving processes, connections and conceptual understanding. The subject area was divided into four strands (numbers and operations, geometry and measurement, patterns and relationships, and data management and probability) and three processes (problem solving, communication or representations and reasoning or connections). The mathematics test included multiple-choice and short constructed-response items. The Science Assessment The concept of scientific literacy is generally accepted as the overarching goal of science curricula across Canada. The PCAP science assessment is founded on a definition of scientific literacy that asserts that students' evolving competencies of using sciencerelated attitudes, skills and knowledge as well as an understanding of the nature of science enables them to conduct inquiries, solve problems and make evidence-based decisions about science-related issues. Embedded in this definition of scientific literacy is the supposition that students have knowledge of the life sciences, physical sciences (chemistry and physics) and earth and space sciences, as well as an understanding of the nature of science as a human endeavour. The science items covered five subdomains (nature of science, nature of technology, knowledge of science, skills and attitudes) and three competencies (science inquiry, problem solving and decision making). The science test included multiple-choice and short constructed-response items. Scoring the Student Booklets The scoring was conducted in both languages concurrently in one location over three weeks. The scoring administration team, the scoring leaders and the scorers themselves came from a range of jurisdictions. The process included parallel training of both scoring leaders and scorers in each subject area; a bilingual committee responsible for reviewing all instruments and selections of anchor papers to ensure comparability at every level; twice daily rater-reliability checks in which all scorers assessed the same student booklet in order to track the consistency of scoring on an immediate basis and blind scoring in which 400 booklets were returned to the scoring bundles to be re-scored to provide an overall reliability measure for the scoring process. 10

Guidelines for Interpreting Results The results of student performance on the 2007 PCAP reading assessment are presented in this report in two ways: as average overall scores on a scale of 0 to 1000, with 500 representing the Canadian average, and as the percentages of students attaining the three performance levels. For the mathematics and science assessments, the results are presented only as average scale scores. Performance levels were established only for the reading component, as it was the major subject area for the 2007 PCAP assessment. The three performance levels represent how jurisdictional performances measured up against set expectations for achievement in reading. To accomplish this, a standard-setting exercise was conducted to set cut scores for achievement at Level 2 and Level 3. This involved examining the relative difficulty of the complete set of assessment instruments and delineating where the defining levels of success should be for each level of achievement.the suitable cut scores were then set, and the student performances within the range of cut scores were described.these descriptors indicate the degree to which student performance meets expectations of what the students should know and be able to do at each level. Level 2 was designated the acceptable level of performance for 13-year-olds.The descriptions for each performance level and the range of scale scores represented by each level are presented below: Level 1 Scores: 379 and below The student demonstrates a partial understanding of some fiction and non-fiction texts. The student understands directly stated information, relying on familiar vocabulary, concrete details, and explicit statements. In interpreting texts, the student provides a simplified or general perspective, often relying on directly stated conclusions or by connecting some aspects of the text to one another. The student demonstrates understanding of how some content and text features are used to accomplish particular purposes. In responding personally and critically to texts, the student s responses are often vague or general. For example, to demonstrate comprehension, students were asked to connect the image of a small map containing two identified islands to the content of the article. The content was about these islands. Level 2 Scores: 380 575 The student comprehends, interprets, and responds to a variety of texts in a clear and reasonable manner. The student understands both directly stated information and information implied by the text.the student connects general statements and supporting details, draws conclusions about the broader meaning and intent of the text, and interprets specific parts of the text based on inferences and figurative language. The student demonstrates knowledge of how texts are structured and organized to accomplish a variety of purposes. The student s personal and critical responses are supported with references to the text and other sources. For example, students were asked to demonstrate comprehension by connecting a character s motivation for a particular decision to a statement about some other cultural idea. Level 3 Scores: 576 and above The student comprehends, interprets, and responds to a variety of texts in a thoughtful and elaborated manner. The student understands directly stated information and information implied by the text, including implications arising from subtle aspects of style and tone. The student demonstrates insightful interpretation of the text by synthesizing several elements or thoughtful analysis of one or more significant elements, often relying on subtle relationships among elements and ideas. The student demonstrates knowledge and insight of how writers structure texts and use other elements of style to accomplish a variety of purposes. The student provides extended personal and critical responses, often including social and cultural implications or literary evaluation. For example, students were asked to demonstrate comprehension by sifting through two different biased viewpoints of a particular event presented in two different media forms (personal narrative and television reporting) to determine the first stage of the event. 11 Pan-Canadian Assessment Program (2007), Ontario Report

In this assessment, the reported average scores provide estimates of the achievement results students would have demonstrated had all students in the population participated in the assessment. Because an estimate based on a sample is rarely exact, it is common practice to provide a range of scores for each jurisdiction within which the actual achievement level might fall. This range of scores expressed for each average score is called a confidence interval. It represents the high- and low-end points between which the actual average should fall 95% of the time. In other words, one can be confident that the actual achievement level of all students would fall somewhere in the established range 19 times out of 20 if the assessment were repeated with different samples from the same student population. In this report, the terms difference and different, used in the context of average scale scores and percentages of students at performance levels, refer to a statistically significant difference. A difference is statistically different when there is no overlap of confidence intervals between measurements. Caution is advised when comparing achievement results based on assessment instruments prepared collaboratively in both languages, despite the extensive efforts to ensure equivalence for the sake of equity and fairness for all students. All languages have unique features that are not readily made equivalent. While the reading items, performance descriptors, scoring scripts and processes were equivalent in English and French, pedagogical and cultural differences related to differences in language structure and use render comparisons between languages inherently difficult, so comparisons should be made with extreme caution. National Assessments in Previous Years The last national assessment administered in Canada was a science assessment administered in 2004 as part of the School Achievement Indicators Program (SAIP), which has now been replaced by PCAP. SAIP was initiated in 1993 by the Council of Ministers of Education, Canada and assessed 13-year-old and 16-year-old students in reading, writing, mathematics and science. Mathematics was previously assessed in 2001 and reading in 1998. Since the SAIP assessment results were reported at five performance levels, it is not possible to draw comparisons between achievement levels on PCAP and SAIP. PCAP sets a new benchmark for comparisons in future years. In the SAIP assessments, Ontario English results tended not to be significantly different from the results for Canada, and the Ontario French results tended to be lower than the results for Canada. In the 2007 PCAP assessment, the Ontario English results were significantly higher than the results for Canada in reading and mathematics. The Ontario French results continue to be lower than those for Canada. 12

2 Carlton Street, Suite 1200,Toronto ON M5B 2M9 Telephone: 1-888-327-7377 Web site: www.eqao.com 2008 Queen s Printer for Ontario