Table of Contents Section A: Overview of the Grade 4 Literacy Assessment Background................................................................. 2 What was assessed?......................................................... 2 Who was assessed?.......................................................... 2 How were students assessed?................................................. 2 How are the results reported?.................................................. 2 Section B: Grade 4 Literacy Assessment Results Norm-Referenced Data: Percentage of Students at Stanine 4 and Above................. 3 Norm-Referenced Data: Mean Grade Equivalent.................................... 4 Norm-Referenced Data: Mean National Percentile Rank.............................. 5 Criterion-Referenced Results: Reading............................................ 6 Criterion-Referenced Results: Language.......................................... 7 Criterion-Referenced Results: Vocabulary........................................ 8 Constructed Response Rubric Scores Narrative Writing............................ 9 Constructed Response Rubric Scores Informational Writing.......................... 10 Summary of Results and Next Steps............................................. 11 Tables of HRSB Results............................................... 13 Section C: Appendix Statistical Analysis........................................................... 15 Halifax Regional School Board 1
Section A: Overview of the Grade 4 Literacy Assessment BACKGROUND In April 2008, the Halifax Regional School Board (HRSB) conducted a student assessment in literacy at the grade 4 level. Three thousand seven students wrote the assessment. The assessment tool, the Canadian Achievement Test.3 (CAT.3) Level 14, is an assessment tool developed externally by the Canadian Test Centre and is used in many jurisdictions across Canada to assess student achievement in literacy. Although the assessment was externally developed, the literacy tasks on the CAT.3 align with the Atlantic Provinces English Language Arts curriculum outcomes for grades 3 and 4. The data collected in this school year will form the baseline of comparison for subsequent data collection at the grade 4 level. WHAT WAS ASSESSED? The assessment tool consisted of three components: Reading and Language: Students completed a selected response section consisting of sixty-six questions. Questions were aligned to Reading and Viewing General Curriculum Outcomes (GCO s) outlined in the Atlantic Provinces English Language Arts Curriculum document. The selected response questions focused on reading comprehension and analysis of text, within a variety of genres and contexts. Vocabulary: Students completed a selected response section consisting of twenty-four questions related to word meaning, multi-meaning words and words in context. Writing: The constructed response section of the assessment required each student to complete four writing tasks. The writing tasks were scored using a scoring guide that assessed various attributes of writing, under the categories of ideas, artistry and conventions. The constructed response tasks and scoring guides were aligned to Writing General Curriculum Outcomes outlined in the Atlantic Provinces English Language Arts Curriculum document. WHO WAS ASSESSED? The CAT.3 literacy assessment was administered to all grade 4 English Program students enrolled in the regular English Language Arts program. Students in French Immersion program were not assessed, as they would only have had one year of instruction in English language arts. Students following an Individualized Program Plan (IPP) for language arts were not assessed, as they do not follow the regular grade 4 language arts program. HOW WERE STUDENTS ASSESSED? Every grade 4 language arts teacher was provided with professional development training to allow him/her to administer the assessment to his/her own students. Students were assessed over four days in April, 2008. HOW ARE THE RESULTS REPORTED? The assessment provides both norm-referenced and criterion-referenced data. Norm-referenced data, using Grade 4 Spring Norms, is presented graphically on pages 4 to 6 where the HRSB results are compared to the Canadian Norms for this assessment. Criterion referenced data is found on pages 7 to 9. Constructed response narrative and informational writing scores are reported on pages 10 and 11. A summary of the results in both written and tabular form is found on pages 13 and 14. Halifax Regional School Board 2
Section B: Grade 4 Literacy Assessment Results Halifax Regional School Board CAT-3 Level 14, Grade 4 Literacy Individual Student Assessment Norm-Referenced Data Percentage of Students at Stanine 4 and Above HRSB 2007-08 Canadian Norm % of Students 100 75 50 25 0 84 82 83 77 77 77 Reading Language Vocabulary GCO#4 Reading/Viewing, GCO#5 Interpret, Select and Combine Information, GCO#7 Critical Response to Text This graph provides information on the percentage of students scoring at Stanine 4 and above on the literacy subtests: Reading, Language, and Vocabulary. For each subtest, there are two bars. The first bar represents the HRSB data for 2007-08 and the second bar represents the Canadian norm. Stanines place scores into nine divisions. In examining the data, the higher the Stanine ranking, the better the students performance can be considered to be relative to the other same-grade students across Canada. A Stanine score of 4, 5, or 6 falls in the Average category, while Stanine scores of 7, 8 and 9 fall in the High category. Scores falling in Stanines 1, 2 and 3 would be of concern as these scores fall in the Low category. In the HRSB student sample, there is a higher percentage of students performing at Stanine 4 or above, compared to the Canadian norm of 77%. The differences between the HRSB student sample results for 2007-08 and the Canadian norm for Reading, Language and Vocabulary are statistically significant. The next steps will involve a closer analysis of the data related to students who performed below Stanine four. This analysis will help identify patterns within the group of students who performed in the bottom three Stanines, and provide information for planning and support at the Board, school and individual student levels. Halifax Regional School Board 3
Halifax Regional School Board CAT-3 Level 14, Grade 4 Literacy Individual Student Assessment Norm-Referenced Data Mean Grade Equivalent HRSB 2007-08 Canadian Norm 6 Grade Equivalent 4 2 5.6 5.2 5.4 4.7 4.7 4.7 0 Reading Language Vocabulary This graph provides information about Mean Grade Equivalent. Again, the data is shown in reference to the literacy subtests Reading, Language, and Vocabulary. For each subtest, there are two bars. The first bar represents the HRSB data for 2007-08 and the second bar represents the Canadian norm. The Mean Grade Equivalent provides a comparison of achievement by students from different grades. If performance on this assessment was given a Mean Grade Equivalent score of 4.3, it would be a typical performance for students in the third month of grade 4 on this same assessment. A score of 5.1, on the other hand, would indicate that students were doing as well on this assessment as would students in the first month of grade 5 if they completed this same assessment. It would not mean that our grade 4 students were working at a grade 5.1 level. The HRSB students performance in Reading, Language, and Vocabulary exceed the Canadian Mean Grade Equivalent of 4.7 for this assessment. Reading: Mean Grade Equivalent of 5.6; performance typical for students in the sixth month of grade 5 on this same assessment. Language: Mean Grade Equivalent of 5.2; performance typical for students in the second month of grade 5 on this same assessment. Vocabulary: Mean Grade Equivalent of 5.4; performance typical for students in the fourth month of grade 5 on this same assessment. The differences between the HRSB student sample results for 2007-08 and the Canadian norm for Reading, Language and Vocabulary are statistically significant. Halifax Regional School Board 4
Halifax Regional School Board CAT-3 Level 14, Grade 4 Literacy Individual Student Assessment Norm-Referenced Data Mean National Percentile Rank HRSB 2007-08 Canadian Norm 100 National Percentile 75 50 25 61 61 50 55 50 50 0 Reading Language Vocabulary This graph provides information about Mean National Percentile Rank compared to the Mean HRSB Percentile Rank. Again, this data is shown in reference to the three subtests. The first is Reading, followed by Language, and Vocabulary. For each subtest, there are two bars. The first bar represents the HRSB data for 2007-08 and the second bar represents the Canadian norm. The National Percentile ranking places students on a scale of 1 to 99, where 99 is high. A National Percentile score of 99 would mean that the student performed better than 99 percent of the national sample of the population at the grade level. If a school or group scored at the 67 th percentile, it would indicate that the average student in that school or group scored better than 67% of the national sample at that grade level. The differences between the HRSB student sample results for 2007-08 and the Canadian norm for Reading, Language and Vocabulary are statistically significant. Halifax Regional School Board 5
Halifax Regional School Board CAT-3 Level 14, Grade 4 Literacy Individual Student Assessment 2007-08 Criterion-Referenced Scores for Literacy - Reading Low Competent Proficient % of Students 100 75 50 25 0 73 67 24 21 3 12 75 55 59 26 16 19 18 23 17 25 9 57 77 13 9 14 68 18 9 88 3 Fiction Non-Fiction Environmental Print Words/Phrases in Context Stated Information Visual Materials Central Thought Analysis of Text Critical Assessment GCO#4 Reading/Viewing, GCO#5 Interpret, Select and Combine Information, GCO#7 Critical Response to Text This graph shows the percentage of students scoring in each of the three criterion-referenced categories: Low, Competent and Proficient. The graph shows the data for Reading in relation to processing text in a variety of genres (Fiction, Non-Fiction and Environmental Print). Criteria for student success on this assessment were established by a randomly selected group of grade 4 classroom teachers from across Canada. Looking at the actual assessment questions, these teachers established the number of correct responses required of a student in each of the nine reading processing areas assessed and thus established the categories Low, Competent, and Proficient for each area. The majority of students in the HRSB sample scored in the Competent or Proficient categories in all of the reading subtests on this assessment. Fiction: 76% scored in the Competent and Proficient category Non-Fiction: 79% scored in the Competent and Proficient category Environmental Print: 84% scored in the Competent and Proficient category Words/Phrases in Context: 74% scored in the Competent and Proficient category Stated Information: 82% scored in the Competent and Proficient category Visual Materials: 82% scored in the Competent and Proficient category Central Thought: 86% scored in the Competent and Proficient category Analysis of Text: 86% scored in the Competent and Proficient category Critical Assessment: 91% scored in the Competent and Proficient category Halifax Regional School Board 6
Halifax Regional School Board CAT-3 Level 14, Grade 4 Literacy Individual Student Assessment 2007-08 Criterion-Referenced Scores for Literacy - Language Low Competent Proficient 100 % of Students 75 50 25 0 36 57 7 53 44 3 9 65 63 Sentence Structure Writing Conventions Paragraph Structure Information Management 26 19 18 GCO#4 Reading/Viewing This graph shows the percentage of students scoring in each of the three criterion-referenced categories: Low, Competent and Proficient. The graph shows the data for Language in relation to various language structures and conventions. Criteria for student success on this assessment were established by a randomly selected group of grade 4 classroom teachers from across Canada. Looking at the actual assessment questions, these teachers established the number of correct responses required of a student in each of the four areas of language structure assessed and thus established the categories Low, Competent, and Proficient for each area. The majority of students in the HRSB sample scored in the Competent or Proficient range in all but one category for language on this assessment. Sentence Structure: 64% scored in the Competent and Proficient category Paragraph Structure: 91% scored in the Competent and Proficient category Information Management: 81% scored in the Competent and Proficient category A considerable number of students in the HRSB sample scored in the Low range in the areas of Sentence Structure and Writing Conventions. These two components of the Reading assessment, which focus on editing, comprise twelve of the sixty-six assessment items. Each of the twelve assessment items requires students to read and identify the sentence that is structurally correct or punctuated correctly (capitalization and punctuation). Sentence Structure and Writing Conventions are also considered in the constructed response scoring guide for writing, under the category Control of Conventions. It is essential to consider student performance related to Sentence Structure and Writing Conventions, in both the reading and writing components of this assessment before drawing inferences related to student achievement in these skills. Halifax Regional School Board 7
Halifax Regional School Board CAT-3 Level 14, Grade 4 Literacy Individual Student Assessment 2007-08 Criterion-Referenced Scores for Literacy - Vocabulary Low Competent Proficient 100 % of Students 75 50 25 68 58 32 71 0 18 14 14 10 Word Meaning Multi-meaning Words Words in Context (modified cloze) 16 GCO#4 Reading/Viewing This graph shows the percentage of students scoring in each of the three criterion-referenced categories: Low, Competent and Proficient. The graph shows the data for Vocabulary in relation to word comprehension in various contexts. Criteria for student success on this assessment were established by a randomly selected group of grade 4 classroom teachers from across Canada. Looking at the actual assessment questions, these teachers established the number of correct responses required of a student in each of the three vocabulary items assessed and thus established the categories Low, Competent, and Proficient for each area. The majority of students in the HRSB sample scored in the Competent or Proficient categories for vocabulary on this assessment. Word Meaning: 82% scored in the Competent and Proficient category Multi-meaning Words: 90% scored in the Competent and Proficient category Words in Context: 87% scored in the Competent and Proficient category Halifax Regional School Board 8
The Constructed Response component of the CAT.3 Literacy Assessment consisted of a series of writing tasks. The tasks required students to respond in writing to a prompt. Each student in the HRSB sample completed two narrative writing tasks and two informational writing tasks. Student achievement in writing was assessed using the CAT.3 scoring guide. The scoring guide framework clusters various attributes of writing into three categories: Content of Writing, Artistry of Communication, and Control of Conventions. The scoring guide framework also delineates achievement on a 5-point scale, with 5 being the highest score and 1 being the lowest score. Classroom teachers scored the student writing by selecting the statements in the scoring guide that best described the writing and then assigned the score (1 to 5) associated with those statements. Halifax Regional School Board CAT3 Level 14-Grade 4 Literacy Scores for Constructed Response Tasks Writing - Narrative 1 Inadequate 2 Low 3 Competent 4 Proficient 5 Excellent 75 % of Students 50 25 40 39 35 36 37 34 0 1 11 13 1 9 14 2 14 13 Artistry Content Conventions GCO#8 Writing to explore, clarify and reflect and use imagination, GCO#9 Write for a range of audiences and purposes, GCO#10 Use a range of strategies to develop effective writing and enhance clarity, precision and effectiveness This graph shows the percentage of students who, in Narrative writing, performed in each of the Scoring Guide performance levels (1 to 5) for the categories of Artistry, Content and Conventions. A score of 5 indicates excellent performance, a score of 4 indicates proficient performance, and a score of 3 indicates competent performance on the writing task. Scores of 1 and 2 indicate performance that does not meet expectation for the CAT.3 assessment. A score of 3 indicates performance that meets expectation for the CAT.3 assessment and scores of 4 and 5 indicate performance that exceeds expectation for the CAT.3 assessment. The majority of students in the HRSB sample achieved within the performance levels 3 to 5 on this assessment. Artistry of Communication: 88% of students met or exceeded expectations on this assessment Content of Writing: 89% of students met or exceeded expectations on this assessment Control of Conventions: 84% of students met or exceeded expectations on this assessment Approximately 12% of students did not meet expectations on this assessment Halifax Regional School Board 9
Halifax Regional School Board CAT3 Level 14, Grade 4 Literacy Individual Student Assessment 2007-08 Scores for Constructed Response Tasks Writing - Informational 1 Inadequate 2 Low 3 Competent 4 Proficient 5 Excellent 75 % of Students 50 25 43 40 38 31 32 35 0 2 14 2 15 2 15 10 11 Artistry Content Conventions GCO#8 Writing to explore, clarify and reflect and use imagination, GCO#9 Write for a range of audiences and purposes, GCO#10 Use a range of strategies to develop effective writing and enhance clarity, precision and effectiveness 10 This graph shows the percentage of students who, in Informational writing, performed in each of the Scoring Guide performance levels (1 to 5) for the categories of Artistry, Content and Conventions. A score of 5 indicates excellent performance, a score of 4 indicates proficient performance, and a score of 3 indicates competent performance on the writing task. Scores of 1 and 2 indicate performance that does not meet expectation for the CAT.3 assessment. A score of 3 indicates performance that meets expectation for the CAT.3 assessment and scores of 4 and 5 indicate performance that exceeds expectation for the CAT.3 assessment. The majority of students in the HRSB sample achieved within the performance levels 3 to 5 on this assessment. Artistry of Communication: 84% of students met or exceeded expectations on this assessment Content of Writing: 83% of students met or exceeded expectations on this assessment Control of Conventions: 83% of students met or exceeded expectations on this assessment Approximately 10% of students did not meet expectations on this assessment Although a score of 3 indicates performance that meets expectation for the CAT.3 assessment, a goal for the Halifax Regional School Board will be to increase the number of students achieving a score of 4 in all categories on the Scoring Guide for Writing. With continued and targeted focus on developing student writing and engaging students in daily authentic and purposeful writing experiences, through teaching structures such as the Writing Workshop, improved scores can be expected. Halifax Regional School Board 10
Summary of Results Strengths: Student performance on this literacy assessment indicates that grade 4 students in the HRSB are performing above the Canadian norm in Reading, Language and Vocabulary. 84% of students performed at or above acceptable performance levels (Stanine 4 and above) in Reading, 82% of students performed at or above acceptable performance levels (Stanine 4 and above) in Language and 83% of students performed at or above acceptable performance levels (Stanine 4 and above) in Vocabulary. Students in the HRSB performed well in Reading, where over 80% of the student sample performed in the Competent and Proficient categories of the Criterion-Referenced scores specifically: Environmental Print: reading and understanding media text, such as newspaper articles and directions (i.e. for making a craft) Stated Information: reading, understanding and locating information Visual Materials: reading and understanding graphical information, such as concept webs and picture cues. Central Thought: understanding the main idea from text Analysis of Text: making inferences and drawing conclusions from text, and identifying literary techniques and expressions Critical Assessment: distinguishing between fact/reality and opinion/fantasy in text, making predictions and transferring ideas to other situations Students in the HRSB performed well in Language, where over 80% of the student sample performed in the Competent and Proficient categories of the Criterion-Referenced scores specifically: Paragraph Structure: identifying effectively written paragraphs Information Management: understanding and identifying information sources, outlines, parts of books, classifications of reference books and techniques that enhance writing Students in the HRSB performed well in Vocabulary, where over 80% of the student sample performed in the Competent and Proficient categories of the Criterion-Referenced scores specifically: Word Meaning: identifying words that have the same or opposite meaning Multi-meaning Words: identifying one word that is missing from two sentences Words in Context (modified cloze): identifying words that are missing from a paragraph Students in the HRSB performed well in the Constructed Response tasks: Narrative Writing and Informational Writing with over 80% of the student sample scoring at performance levels 3, 4 and 5 in all three categories: Artistry of Communication: organization and words and expressions that together establish the focus and connection in the writing Content of Writing: ideas and details that together establish the purpose and context (Narrative or Informational) of the writing Control of Conventions: sentence structure and mechanics (spelling, capitalization, punctuation and paragraph marking) that together allow for clear and unambiguous communication Halifax Regional School Board 11
Areas requiring continued focus: Although encouraged by the strengths students in the HRSB have shown on this assessment, areas for continued Board focus have been identified in the following areas: In reading, the goal for the HRSB will be to increase the percentage of students performing in the Competent category for Fiction and Non-Fiction. In writing, the goal for HRSB will be to increase the percentage of students performing in the Proficient category (score of 4) for all categories on the Scoring Guide (Content, Artistry and Conventions) in both Narrative Writing and Informational Writing. Next steps for continued improvement in literacy achievement within the HRSB: Provide focused professional development opportunities and support for teachers in the areas of Reading and Writing. (i.e. Board level opportunities for professional development, on-site professional development opportunities provided through the literacy coordinator) Provide support to schools in analyzing school level CAT.3 assessment data. Schools should analyze the CAT.3 data, along with classroom, school and provincial assessment data, to inform school progress and plans for improvement.. Complete a more thorough analysis of the Board data related to the percentage of the student sample that performed: below Stanine four in the Norm-Referenced data in the Reading/Language and Vocabulary assessments. in the Low category on the Criterion-Referenced data in Reading/Language and Vocabulary assessments. This analysis will help identify patterns within this group of students and provide information for Board level planning and support to schools. The Halifax Regional School Board has made continuous improvement in student achievement in literacy and each year we report on this progress. Our focus for improvement will be to move individual students from the Low category into the Competent or Proficient category in all components of this assessment. With continued and targeted focus on reading and writing development by engaging students in daily authentic and purposeful literacy experiences and through teaching structures such as the Reading and Writing Workshop, an increase in the number of students experiencing success can be expected. Halifax Regional School Board 12
Tables of 2007-08 HRSB Grade 4 Literacy Assessment Results Table of Criterion-Referenced Literacy Scores Assessments Year % Low % Competent % Proficient Reading Fiction 2007-08 24 73 3 Non-Fiction 2007-08 21 67 12 Environmental Print 2007-08 16 75 9 Words/Phrases in Context 2007-08 26 55 19 Stated Information 2007-08 18 59 23 Visual Materials 2007-08 17 25 57 Central Thought 2007-08 13 77 9 Analysis of Text 2007-08 14 68 18 Critical Assessment 2007-08 9 88 3 Language Sentence Structure 2007-08 36 57 7 Writing Conventions 2007-08 53 44 3 Paragraph Structure 2007-08 9 65 26 Information Management 2007-08 19 63 18 Vocabulary Word Meaning 2007-08 18 68 14 Multi-meaning Words 2007-08 10 58 32 Words in Context 2007-08 14 71 16 Halifax Regional School Board 13
Table of Norm-Referenced Results Assessment Year % Students at Stanine 4 and Above Percentile Rank Mean Grade Equivalent Reading 2007-08 84 61 5.6 Language 2007-08 82 55 5.2 Vocabulary 2007-08 83 61 5.4 Table of Constructed Response Scores Criteria Artistry (Organization & Word Choice) Content (Details & Ideas) Conventions (Sentence Structure & Mechanics) Artistry (Organization & Word Choice) Content (Details & Ideas) Conventions (Sentence Structure & Mechanics) Year 1 % Inadequate Narrative Writing 2 % Low 3 % Competent 4 % Proficient 5 % Excellent 2007-08 1 11 40 35 13 2007-08 1 9 39 36 14 2007-08 2 14 37 34 13 Informational Writing 2007-08 2 14 40 32 11 2007-08 2 15 40 32 11 2007-08 2 15 38 35 10 Scoring Guide Scale: Scores 1 and 2 Does not meet expectations on this test Score 3 Meets Expectations on this test Scores 4 and 5 Exceeds expectations on this test Halifax Regional School Board 14
Section C: Appendix 2007-08 HRSB Results Compared to the National Norms This section contains a brief comparison with the national norms. The 2008 results were significantly higher than the national norms for all three tests at p <.001. Table 6 2007-08 HRSB Results Compared to National Norms 2008 HRSB CAT3 Norms Mn SD N Mn SD N Reading 530 56 2974 515 57.78 2558 Language 527 61 2974 517 61.55 2558 Vocabulary 497 68 2898 480 67.63 2531 Note: the 2008 HRSB means exceed the norms for all three tests at p <.001 Table 7 shows the percentages of the 2007-08 HRSB students who are at Stanine 4 or above for Reading, Language, or Vocabulary. These percentages were compared to the 77% of the students who were at the same level in the national sample. For all three tests, the difference favouring the Halifax students was significant (p <.001). Table 7 Percentage of 2008 HRSB Students At and Above Stanine 4 in 2008 and the National Norm % HRSB Students % National Sample Reading 84 77 Language 82 77 Vocabulary 83 77 Note: the 2008 HRSB means exceed the norms for all three tests at p <.001. Summary The analyses reported show that the 2007-08 HRSB sample scored significantly higher than the norming sample. It must be noted that with such large samples, very small differences are sometimes statistically significant. However, the consistency of this level of achievement is notable and worthy of celebration. Halifax Regional School Board 15
Technical Details on Statistical Tests The alpha level for these analyses was set at.05. No adjustment was made to adjust for family-wise error rate, partially because the tests were significant at such a high level that the adjustment would not have affected the results reported. The tests underlying the original tables 1 and 6 were grouped t-tests. The actual values of the tests and degrees of freedom are shown in Tables A1 and A2 below. Table A2 2008 HRSB Results Compared to CAT 3 Norms 2008 HRSB CAT3 Norms Mn SD N Mn SD N t df Reading 530 56 2974 515 57.78 2558 9.77 5352 Language 527 61 2974 517 61.55 2558 6.05 5530 Vocabulary 497 68 2898 480 67.63 2531 9.21 5427 Note: the 2008 HRSB means exceed the norms for all three tests at p <.001 The test used to compare the percentages at and above Stanine 4 was the Pearson chi-square test. The values are shown in Table A5. Table A5 Values of the Pearson Chi-Square Test Comparing the HRSB and National Samples at and above Stanine 4 Value df Asymp. Sig. (2- sided) Reading Pearson Chi-Square 42.56 1.000 N of Valid Cases 5440 Language Pearson Chi-Square 20.83 1.000 N of Valid Cases 5440 Vocabulary Pearson Chi-Square 30.62 1.000 N of Valid Cases 5440 Halifax Regional School Board 16