Mathematics Assessment Collaborative MAC Report on the 2012 Tests October 3, 2012 MAC & STAR technical report September 21, 2012 by Educational Data Systems 15850 CONCORD CIRCLE, SUITE A MORGAN HILL, CA 95037 WWW.EDDATA.COM (408) 776-7696
Acknowledgements There are several contributors to the Mathematics Assessment Collaborative s Final Report for 2012. The Mathematics Assessment Collaborative is a program of the Silicon Valley Mathematics Initiative supported by the Santa Clara Valley Mathematics Project, and the forty member public school districts. The Tools for Teachers section of the report was researched, analyzed and written by Linda Fisher, Mia Buljan, Kristy Leo and the MAC Senior Trainers: Mia Buljan, Kristy Leo, Melissa Adams, Carol Hatalsky, Sandy Devlin, Sally Keyes, Ford Long, Jean Short, Margie Trainer, Debbie Borda, Barbara Scott, and Jeff Trubey. Data analysis was conducted by the Educational Data Systems of Morgan Hill, California. The data analysis team headed by Sonyo Montelongo. Contributors to the Final Report The Mathematics Assessment Collaborative Linda Fisher, Director The Santa Clara Valley Mathematics Project Joanne Rossi Becker, Ph.D., Director The Silicon Valley Mathematics Initiative - David Foster, Executive Director The Mathematics Assessment Resource Service Hugh Burkhart, Ph.D., Director Data Analysis Jim Ridgway, University of Durham Educational Data Systems Caroline Fahmy, President Spanish Translation - Rosita Fabian and Donna Goldenstein Member Districts Albany USD Alvord SD (Riverside County) Antioch Unified School District Aspire School District Assumption School (San Leandro) Bayshore School District Belmont-Redwood Shores School District Berryessa School District Bolinas and Lagunitas School District Brisbane School District Buckeye SD Cambrian School District Campbell Union ESD Castro Valley USD Charter School of Morgan Hill Chicago Public Schools Creative Arts Charter (SF) CSU San Bernardino Cotati/Rohnert Park Cupertino SD Dade County Schools (GA) Del Mar USD (San Diego County) Discovery Charter School Dioceses of Santa Clara Dublin USK East Side UHSD Edmonds Community College Emery SD Etiwanda SD (San Bernardino County) Gilroy (Brownell Middle School) Fairfield-Suisun USD Fremont USD Forsyth County School (GA) Hamilton County (TN) Hayward USD Jefferson ESD Jefferson HSD Las Lomitas SD La Honda-Pescadero SD Livermore USD Los Altos SD Los Gatos SD Menlo Park SD Mountain School District Monterey Peninsula USD Moreland SD 2
Member Districts part 2 National Council of La Raza Schools New York City PS New Visions for Public Schools Oakland Unified SD Pacifica SD Pajaro Valley USD Palo Alto USD Pittsburgh USD Portola Valley SD Ravenswood City SD Riverside COE Redwood City Schools Sacramento City USD Salinas City Schools San Carlos CLC San Francisco USD SMFC (Park School) San Jose Unified SD San Leandro USD San Ramon USD Santa Clara USD Santa Cruz City Schools Saint Michael s School (Poway) Saint Patrick s School (San Jose) Saratoga Scotts Valley USD SCCOE County Court Schools Sequoia HSD SMCOE Court Schools South Cook Service District South San Francisco USD Sumter County (GA) The Nueva School Union SD University of Illinois, Chicago Valley Christian (Dublin) Valdosta City GA Walnut Creek SD Woodside SD 3
Table of Contents Summary /Overview pg. 4 MAC Examinations/ Level Descriptions pg. 6 Comparing Student Performance pg. 9 Second Grade Overall Frequency Distribution pg. 19 Demographic Data Analysis pg. 21 Analysis of Data By Task pg. 38 Third Grade Overall Frequency Distribution pg. 44 Demographic Data Analysis pg. 46 Analysis of Data By Task pg. 63 Fourth Grade Overall Frequency Distribution pg. 70 Demographic Data Analysis pg. 72 Analysis of Data by Task pg. 89 Fifth Grade Overall Frequency Distribution pg. 96 Demographic Data Analysis pg. 98 Analysis of Data by Task pg. 115 Sixth Grade Overall Frequency Distribution pg. 122 Demographic Data Analysis pg. 124 Analysis of Data by Task pg. 141 Seventh Grade Overall Frequency Distribution pg. 148 Demographic Data Analysis pg. 150 Analysis of Data by Task pg. 167 Eighth Grade Overall Frequency Distribution pg. 174 Demographic Data Analysis pg. 176 Analysis of Data By Task pg. 192 Course One Overall Frequency Distribution pg. 199 Demographic Data Analysis pg. 201 Analysis of Data By Task pg. 217 Course Two Overall Frequency Distribution pg. 224 Demographic Data Analysis pg. 226 Analysis of Data by Task pg. 240 Appendix 1 NSLP by Perform Levels pg. 248 Appendix 2 Comparison of CST to MARS pg. 251 Appendix 3 Correlation STAR to MARS pg. 254 Appendix 4 Correlations CST clusters & MAC pg. 263 Appendix 5 Status of the Data pg. 272 Appendix 6-MARS Score Compared by District pg. 273 Appendix 7-Audit Report pg. 289 4
Summary The Silicon Valley Mathematics Assessment Collaborative uses performance assessment of the kind used to assess mathematics in most countries worldwide. The assessments and their information are then used for professional development and as data for improving instruction. This year MAC has a membership base of 85members: school districts, charter networks, County Office of Educations, Counties, and Individual Schools. Over 300 teachers were trained in March to lead district scoring- sessions. This year we tested over 38,750 students from grades 2 through Course Two. The STAR results come from 29 of MAC s 37 districts. Other districts used MAC exams for professional development purposes only. This report describes test results across the collaborative as a whole. The results are based on the 2012 tests, which are the fourteenth annual cycle of tests administered by the collaborative. For each grade level the report provides a statistical analysis of the test as a whole, and the results task-by-task. The report is supplemented by the Tools for Teachers, which provides a more in-depth look at student work, student misconceptions, instructional implications, task commentary, and suggested lessons related to the task. Statistical Analyses show that scoring has been done to high standards, across MAC, at all grade levels. Local scoring is working well. Data analysis provides clear evidence that MARS tests produce information on student attainment in mathematics, which complements rather than duplicates evidence from other tests (e.g. CST, CAT-6, SAT-9). Over the fourteen years of testing, there have usually been year-on-year gains in student performance at grades 2, 3, 4 and 5. This year 2 nd grade increased from 78.9% meeting or exceeding standards to 82.5%. 3 rd grade stayed about the same at 70% meeting or exceeding standards. 4 th grade decreased to 71.5% meeting or exceeding standards to 62.2%. 5th grade decreased to 76.5% meeting or exceeding standards to 60.6%. The CCSS puts a greater emphasis on fractions for these grade levels, which is reflected in the difficulty level of the exams. 6 th grade decreased from 43.2% to 38% meeting or exceeding standards. 7 th grade increased from 23.7% to 38.7%. Eighth also showed an increase from 19.9% to 25.9% meeting or exceeding standards. Algebra decreased from 46% meeting or exceeding standards to 34.5%. Geometry has declined for the last two years. Geometry went from 78% to 54% meeting standards in 2010 and this year continued the decline to only 24.6% meeting or exceeding standards. However, the number of students taking the geometry increased by 82% with more high school participation. Districts, with support from the Collaborative, are encouraged to provide continued professional development for teachers to help use this information and data to improve mathematics education. The SVMI provided an August Coaching Institute to focus on key ideas. That work will be followed by a series of Professional Development Sessions during the school year, focusing on student thinking and misconceptions and developing better instructional strategies. The Silicon Valley Mathematics Initiative provides funding 5
for professional development and mathematics coaches for many school districts to help teachers in their classroom Overview MAC, The Silicon Valley Mathematics Assessment Collaborative, is composed of 85 members including public elementary and high school districts in San Mateo, Santa Clara, Alameda Counties, Marin, Santa Cruz, and Contra Costa Counties in association with the Santa Clara Valley Math Project. We have also added other districts throughout the state including Sonoma County, Solano, Sacramento, San Diego County, and Riverside County. The Collaborative is also working with other states, charter networks, and Universities. The purpose of the Collaborative is to produce, score and report on tests of student performance in mathematics. In addition, the Collaborative provides professional development opportunities for teachers, teacher leaders, and principals as instructional leaders, helping to build the capacity of all member districts. Many new districts throughout the state and in other states to better prepare for the CCSS and the national formative assessment tests. The assessment of students mathematics requires far more than the administration of psychometric tests. Assessment tasks exemplify the core mathematics to be attained, therefore examples are given within the report to illustrate the educational ambitions of MAC. Through the process of giving performance exams, the teaching community of the member districts is building a common vision of the mathematical standards for each grade, a clear set of expectations for good student work, and how mathematical standards and expectations change as students move through the grades. This shared vision promotes a truly standard-based mathematics curriculum. Over the course of the project the focus of professional development opportunities has changed. Current sessions spend considerable time examining student work and thinking about the implications for instruction. As a result of the Action Research team, teachers are learning a new protocol, re-engagement, which helps teachers develop lessons that use the information directly from the MARS task to further instruction through a more detailed examination of misconceptions and trying-on strategies of other students. This protocol is an important tool to further the deep mathematical understanding of students. A web-site with video and student work is being established to make this protocol more readily available to all teachers in the collaborative (www.insidemathematics.org ) This website also has videos of number talks, lesson study, and coaching. The work of the lesson study projects has also developed tools, such as number talk and pre-teaching for low students, which improve instruction. The lessons from the MARS exams are continually improving the types of offerings available in the staff development options. As the work of the collaborative grows over time, the bank of materials for classroom use, such as Problems of the Month, also increases. This year the materials and resources are divided into two locations. Tools for Teachers is available on the Noyce Foundation Website (www.noycefdn.org) Past tasks, rubrics, Spanish translations, Problems of the Month, and presentation power points are available on the SVMI website (www.svmi.org). As the tools are perfected they will be made available to all groups 6
in the collaborative. SVMI also continues to provide professional development to improve the skills and knowledge base of the math coaches. MAC coaches are working on integrating the new National Core Standards into their work by looking at some of the formative assessment lessons currently under development. These formative assessment lessons are also being field-tested in select classrooms. Many of these lessons on now available at http://map.mathshell.org 7
The MAC Examinations The purpose of the MAC project is to develop a high stakes test that districts can use for multiple measures on student performance in mathematics. While many tests are currently available to measure how students can compute under time pressure, we need a test that measures students ability to apply mathematics in problem situations and explain their thinking. These tests are designed to measure the student performance in using the key curriculum content of their grade level in tackling worthwhile tasks. The concept is for the testing process to be transparent to all the stakeholders. Teachers and students know in advance what will be covered. Likewise they are familiar with the type of rubric to be used and they have the opportunity to have a practice test before the exam, so that the format will not be new and confusing. This helps to demystify the testing process, so that it can com closer to measuring what students know and are able to do. Research has also shown that a well-designed test can help drive quality instruction. This test is designed to show how well the student understands the mathematical tools learned and how well they can be applied in problem solving situations, nonroutine tasks. An unexpected, but significant benefit is the opportunity to increase content knowledge of teachers and develop common, consistent expectations of good student performance. By having clearly defined Core Ideas for each grade level, tests matched to those Core Ideas, and a consistent criteria for measuring student performance, MAC is truly building meaningful, working mathematical standards. Designing the tests: The range, variety and balance of the tasks are critical to the success or otherwise of this endeavor. The task sets used here include the kinds of tasks that are found on the National Assessment of Education (NAEP) and on the Third International Mathematics and Science Study (TIMMS); because these assessments are constrained in a number of practical ways, MAC was able to achieve better balance. The assessment covers a familiar range of mathematical topics: Number Operations and Properties, Geometry, Measurement, Patterns and Functions, Data Analysis, Probability, Algebraic Symbols and Properties as well as Mathematical Representation, Communication, Connections, Reasoning and Proof, and Problem Solving. For MAC, teachers met to determine which core ideas comprised the mathematical foci specific to each grade level. This year there was a substantial change in CORE ideas as the collaborative embrace the Common Core State Standards and exams were designed to these new criteria. An item related to each core idea was selected to sample the overall range of student thinking. In addition the content dimension, the tasks were chosen to achieve reasonable balance in other import dimensions of performance types of task, and of practical context, problem solving process, etc. Scoring student work: Examining student responses to such tasks provides teachers with the opportunity to look at student thinking, see common misunderstandings, and make adjustments in curriculum and instruction. In the scoring and in-service process, 8
teachers can start to see the road map in mathematical instruction and how expectations and sophistication in responses is stepped up as students move up in grades. At each scoring session, teachers and students were first trained on the point scoring process and then on a rubric specific to the problem they would be scoring. After discussion of the question and its rubric, scorers were given 5 sample student responses to score. Their scores were then compared to the score determined by the senior assessor a MARS consultant or a MAC scoring facilitator. The differences between how they scored compared to the established score were then discussed, to clear up their thinking and help everyone develop the same internal standards and understandings. After this process was completed, another set of 10 Standardizing Papers were given to them for practice. If these papers were correctly scored against the established score, scorers were then allowed to start correcting student work. After they completed their first folder of student papers, they traded with another scorer for checking. Differences were discussed and settled with the help of scoring facilitators. Periodically throughout the day random papers were double scored to check for accuracy and consistency. Looking at the reliability analysis in the appendix, it appears that this training process provides us with relatively accurate and consistent scores across many different local scoring sessions. Setting the Level boundaries: In international assessment, the setting of standards through decisions on level boundaries ( cut scores ) is through informed professional judgments on the basis of three kinds of input. The following procedure, based on international practices outlined above, was used in the MAC tests Analysis of the tasks and rubrics, in relation to the Standards. For each task, an analysis of the elements of performance and the rubric suggests a typical score that represents the core mathematics of the task. This points to provisional level boundaries. Each member of the Awarding Committee (A Board of top international Educators and Educational Leaders) makes holistic judgments of a sample of student papers around provisional level boundaries. The question for each student paper is Does this student performance, viewed as a whole, meet the standard? Statistical information on the score distributions for each task, and for the test, informs these professional judgments. It includes cross-comparison data between different tasks, which links the demand levels of parallel tests, from one year to another, for example. 9
Performance Level Descriptions 4 Achieves Standards at a High Level The student performed at a high level on the tasks and consistently achieved the standards. 3 Performance at Standards The student performed at a level that met the standards. 2 Performance Below Standards The student showed some evidence of performing at the level of the standards, but overall the performance did not meet the standards. 1 Demonstrates Minimal Success The student showed minimal evidence of performing at the level of the standards. Much of the response was either fragmented or non-existent. Comparing Student Performance 10
Comparing Student Performance Data are available from 1999-2012 at grades 3,5,7, and 9. Data are available from 2000-2012 at grades 4,6,8, and 10. Data for grade two is available for 2003-2012. These data are analyzed here. Different tests are given each year in MAC, and while standards remain the same each year, no attempt is made to create parallel test forms. It is impossible, therefore, to compare performance in adjacent years by examining raw scores. Test performance is described each year in terms of levels of performance. A careful process is used to allocate points on tests to levels, and so changes from year to year can be made in terms of the numbers of students achieving different performance levels. A year-on-year comparison is not as straightforward as it might first appear. For example, if the demographic composition of samples changed dramatically between 2000 and 2001, then observed differences in students performance may be due to this factor, and not to factors such as changes in teaching approaches, increased familiarity with previously unfamiliar task types, and so on. Over the fourteen years of testing, there have been considerable changes in the sample of students tested. In particular, large numbers of students have been tested in later years. Most significant is the student population for eighth grade. Since 2005 the scores no longer include eighth graders taking algebra. The Algebra scores now contain a majority of eighth graders rather than a majority of high school students. While for the last few years geometry scores were from middle school, in 2012 the high schools were again included in the data. The analyses, which follow, should, therefore, be treated with caution. A better understanding of changes in student attainment will be gained by detailed analysis of District Reports, where the nature of any demographic changes can be assessed with some accuracy. 11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Silicon Valley MAC Test Second Grade 2012 Core Idea Task Score Measurement and Data Cake Sale This task asks students to interpret and use data from a tally chart and a bar graph with a one-to-one scale. Students who are successful are able to compare and combine values on the graph, and use words, numbers, or pictures to justify their calculations. Students should demonstrate that they understand the mathematical language of combining and comparison, and set up appropriate mathematical models based on these understandings. Number and Operations in Desert Creatures Base Ten This task asks students to demonstrate an understanding of how to combine, separate, and compare two- and three- digit values. Successful students demonstrate fluency in adding and subtracting whole numbers in context, and they are able to communicate reasoning using words, numbers, or pictures. Students should have access to a variety of strategies, including those based on place value, properties of operations, and/or the relationship between addition and subtraction. Students are asked to justify their calculations using models, such as expressions, equations, or drawings. Operations & Algebraic Maria s Fruit Stand Thinking This task asks students to look for and express regularity in repeated reasoning. Successful students make sense of the arrays of objects (fruits), and can use repeated addition or multiplication to express the total number. They can also work backward from a total number of objects to create and appropriate array model. These students also use knowledge of money relationships to assign coin values to the fruit, and find a total dollar value for a given number of fruit. These students also had an understanding of how a dollar is decomposed into quarters (two quarters are 50, two groups of 50 are $1.00) and understood how to represent that relationship in numbers, words, or drawings. Geometry The Playhouse This task asks students to identify basic geometric shapes such as triangles, circles, and quadrilaterals. Successful students can also recognize and quantify specific attributes, such as number of vertices and sides of a given polygon. Students are asked to construct viable arguments when comparing and contrasting two quadrilaterals, using specific geometric attributes as the structure of the argument. Operations & Algebraic Misha s Marbles Thinking This task asks students to demonstrate an understanding of how to interpret and extend a growing pattern of two colored marbles, using drawings, words, charts, and/or number sentences to describe the growth. Successful students represent the plus 3 growth pattern at any stage of marble pattern growth, including if a stage of marble pattern growth is skipped, and are able to work backward from the number of marbles to the stage of marble pattern growth. Students are asked to look for and express regularity in repeated reasoning, and represent it for the linear pattern. 30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
Silicon Valley Mathematics Initiative Assessment Test Third Grade 2012 Core Idea Task Score Measurement Clubhouse Plans This task asks students to use a diagram to count area and perimeter in square units. It asks students to draw a picture with a given area on a grid. The task asks students to compare areas and perimeters by calculating measures using a grid. Operations and Cookie Dough Algebraic Thinking The task asks students to equal groups to solve problems involving multiplication and division. It also asks students to calculate with money. Measurement and Data Rise and Shine! The task asks students to read and interpret data recorded in a frequency table or bar graph. It asks students to add data to the frequency table and bar graph and use the data to solve problems involving putting together, taking from, and comparing. Number and Operations Travel for Work in Base Ten The task asks students to their place value knowledge and estimation to solve problems involving addition and subtraction using data presented in a table. Operations and Worms Algebraic Thinking The task asks students to extend a geometric pattern using drawings, tables, and number patterns. 48
The maximum score available for this task is 7 points. The minimum score available for a level 3 response, meeting standards, is 3 points. Most students, 93%, were able to find the area of shape A using the grid. Many students, about 76%, were also able to draw a different shape with the same area as A. About 35%, could find the area of all the shapes to make a convincing argument and draw a shape with the same area as A. About 9% could also find the perimeters of the shapes. Almost 7% of the students scored no points on this task. All the students in the sample with this score attempted the task. 49
The maximum score available on this task is 7 points. The minimum score needed for a level 3 response, meeting standards, is 4 points. Most students, almost 90%, could find the number of tubs sold to earn $32. Almost 85% could also find the cost of 2 tubs of Oatmeal Cookie Dough. Many students could find cost of 2 Tubs of Oatmeal Cookie Dough, find the cost of 4 tubs each of Chocolate Chip and Peanut Butter Cookie Dough, and find the number of tubs for $32. Almost half, 45%, could determine the number of tubs that could be purchased with $20. 28% could explain why that was true including talking about not having enough for another complete tub. 50
The maximum score available for this task is 9 points. The minimum score needed for a level 3 response, meeting standards, is 5 points. Most students, 95%, were able to read the graph and fill in the missing value on the chart. Many students, almost 79%, were also able to fill in the graph and find the time when the least number of students woke up. More than half the students, 62%, could complete the chart and graph, find the time for least students, and use comparison subtraction. About 44% could also either find the total waking before 6:45 or find the total students represented on the graph. Almost 22% of the students could meet all the demands of the task. 51
The maximum score available on this task is 7 points. The minimum scored needed for a level 3 response, meeting standards, is 4 points. Most students, 97%, could identify most and least from a group of numbers in the hundreds. 90% could also choose the nearest place value for adding the group of numbers. Many students, about 77%, could identify most and least, and do comparison subtraction. More than half, could identify most, least, do comparison subtraction, and estimate place value for adding the set of numbers. 28.6% of the students could meet all the demands of the task including finding two days in a row that add to more than 900 miles. 52