Mathematics Performance at the TIMSS Advanced 2008 International Benchmarks

Chapter 3 Mathematics Performance at the TIMSS Advanced 2008 International Benchmarks As was described more fully in the Introduction, the TIMSS advanced mathematics achievement scale summarizes students performance on test items designed to measure breadth of content in algebra, geometry, and calculus, as well as a range of cognitive processes within the knowing, applying, and reasoning domains. To interpret the achievement results in meaningful ways, it is important to understand the relationship between scores on the scale and students success on the content of the assessment. As a way of interpreting the scaled results, three points on the scale were identified as international benchmarks and descriptions of student achievement at those benchmarks in relation to students performance on the test items were developed. The TIMSS Advanced benchmarks represent the range of performance shown by students internationally. The Advanced International Benchmark is 625, the High International Benchmark is 550, and the Intermediate International Benchmark is 475. In TIMSS at the fourth and eighth grade levels, four benchmarks were used: viz., advanced, high, intermediate, and low. The low international benchmark was not included in the TIMSS Advanced benchmarking analysis since,

90 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks in all the participating countries, this is a highly select population of students. The TIMSS & PIRLS International Study Center worked with a committee of experts 1 from several countries to conduct a detailed scale anchoring analysis to describe mathematics achievement at these benchmarks. Scale anchoring is a way of describing TIMSS Advanced 2008 performance at different points on the advanced mathematics scale in terms of the types of items students answered correctly. In addition to a data analysis component to identify items that discriminated between successive points on the scale, 2 the analysis also involved a judgmental component in which committee members examined the mathematics content and cognitive processing dimensions assessed by each item and generalized to describe students knowledge and understandings. This chapter presents the TIMSS Advanced 2008 mathematics achievement results at the international benchmarks for the participating countries. Then, benchmark by benchmark, there is a detailed description of the understanding of mathematics content and types of cognitive processing skills and strategies demonstrated by students at each of the international benchmarks, together with illustrative items. For each example item, the percent correct for each of the TIMSS Advanced 2008 participants is shown. For multiple-choice items, the correct answer is identified by a bullet,, and the percent of students in each country who chose each response choice is also given. For constructed-response items, a copy of the scoring guide showing the percent of students choosing each correct or incorrect approach is provided, along with a student response that was given full credit. 3 The items published in this report were selected from the items released for public use. 4 Every effort was made to include examples which not 1 In addition to Robert A. Garden, the TIMSS Advanced Mathematics Coordinator, and Svein Lie, the TIMSS Physics Coordinator, committee members included Carl Angell, Wolfgang Dietrich, Liv Sissel Gronmo, Torgeir Onstad, and David F. Robitaille. 2 For example, in brief, a multiple-choice item anchored at the Advanced International Benchmark if at least 65 percent of students scoring at 625 answered the item correctly and fewer than 50 percent of students scoring at the High International Benchmark (550) answered correctly, and so on, for each successively lower benchmark. Since constructed-response questions nearly eliminate guessing, the criterion for the constructed-response items was simply 50 percent at the particular benchmark. For more information, see the TIMSS Advanced 2008 Technical Report. 3 All of the constructed-response items were scored according to detailed scoring guides containing descriptions and examples of the types of responses that should receive credit. Although most constructed-response items were worth 1 point, some were worth 2 points (with 1 point awarded for partial credit). If the example item was worth 2 points, the data are for responses receiving 2 points (full credit). 4 After each TIMSS assessment, a certain proportion of the items are released into the public domain and the rest of the items are kept secure for use in measuring trends over time in subsequent assessments. In the case of TIMSS Advanced, more than one-half of the items are being released.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 91 only illustrated the particular benchmark under discussion, but also represented different item formats and content area domains. How Do Countries Compare on the TIMSS Advanced 2008 International Benchmarks of Mathematics Achievement? Exhibit 3.1 summarizes what students of advanced mathematics in the participating countries who score at the TIMSS international benchmarks typically know and can do in mathematics. The data show that that there were substantial differences in students performance across the three benchmarks. Students at the Advanced International Benchmark demonstrated their understanding of concepts, mastery of procedures, and mathematical reasoning skills in algebra, trigonometry, geometry, and differential and integral calculus to solve problems in complex contexts. Students at the High International Benchmark used their knowledge of mathematical concepts and procedures in algebra, calculus, and geometry and trigonometry to analyze and solve multistep problems set in routine and non-routine contexts. Those at the Intermediate International Benchmark demonstrated knowledge of concepts and procedures in algebra, calculus, and geometry. Exhibit 3.2 displays the percent of advanced mathematics students in each country that reached each of the three international benchmarks. The percents displayed in each row corresponding to the three international benchmarks are cumulative. Every student who scored at the Advanced Benchmark is also included in the High and Intermediate Benchmark categories. For each country, the exhibit shows the percent of advanced mathematics students who reached each international benchmark as well as the TIMSS Advanced Mathematics Coverage Index for that country (see Exhibit 1.2). In the table, the countries are listed in descending order of the percent of their students who reached the

92 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.1: TIMSS Advanced 2008 International Benchmarks Exhibit 4.1 3.1 Books TIMSS in Advanced the Home 2008 with International Trends Benchmarks of Mathematics Achievement of Mathematics Achievement Advanced International Benchmark 625 Summary Students demonstrate their understanding of concepts, mastery of procedures, and mathematical reasoning skills in algebra, trigonometry, geometry, and differential and integral calculus to solve problems in complex contexts. In algebra, students can solve word problems involving permutations and geometric sequences, and solve logarithmic equations. They demonstrate some facility with complex numbers and can find sums of infinite geometric series. In calculus, students demonstrate understanding of the concept of integration. They can integrate exponential functions, recognize the relationship between a definite integral and the area under a curve, and solve problems about areas between curves. They can identify from the graph of a function points where it is not differentiable. They can determine maxima, minima, and points of inflection of a function by analyzing the graph of its derivative or by finding the first and second derivatives. They can solve problems in kinematics, and find the maximum value of a quantity under given conditions. Students use geometric reasoning to solve problems. They can use trigonometric ratios to solve a non-routine practical problem, and demonstrate knowledge of the concepts of period and amplitude of trigonometric functions. They use vector sums and differences to express a relationship among three vectors. In the Cartesian plane, they can determine whether lines are parallel, show that the diagonals of a given quadrilateral bisect each other, and find the locus of points satisfying a given condition. High International Benchmark 550 Summary Students can use their knowledge of mathematical concepts and procedures in algebra, calculus, and geometry and trigonometry to analyze and solve multi-step problems set in routine and non-routine contexts. Students can solve algebra problems that require analysis, including problems set in a practical context and problems requiring interpretation of information related to functions and their graphs. They can determine a term in a geometric sequence, compare two simple mathematical models, solve quadratic inequalities, and analyze a proposed solution of a simple logarithmic equation. In calculus, students can analyze properties of functions and their graphs on the basis of the sign of the first and second derivatives. They can find the derivative of a function involving radicals. They can find definite and indefinite integrals of simple rational functions. In geometry, students can use basic properties of trigonometric functions to identify solutions of simple trigonometric equations and solve word problems involving angle of elevation. They can identify the equation of a line or a circle in the Cartesian plane, and use slopes of lines to solve problems. They can use properties of vectors to analyze equivalence of conditions involving the sum and difference of two vectors.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 93 Exhibit 3.1: TIMSS Advanced 2008 2008 International Benchmarks of Mathematics Achievement (Continued) of Mathematics Achievement (Continued) Intermediate International Benchmark 475 Summary Students demonstrate knowledge of concepts and procedures in algebra, calculus, and geometry to solve routine problems. Students can perform basic operations of algebra, including solving equations and inequalities, and simplifying polynomial and rational expressions. They can determine the sign of a rational function and find the function of a function in simple cases. In calculus, students show an understanding of the concepts of continuity and limit of a rational function. They can find the derivative of simple rational, exponential, and trigonometric functions. They can make connections between the graph of a function and the derivative of the function. Students use knowledge of basic properties of geometric figures and of the Cartesian plane to solve problems. They can add and subtract vectors in coordinate form. They can draw the image of a polygon under a reflection, and identify the shape traced by a line rotating in space.

94 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.2 Percent of Students Exhibit Reaching 3.2: Percent the TIMSS of Students Advanced Reaching 2008 the TIMSS Advanced 2008 International Benchmarks of International Mathematics Achievement Benchmarks of Mathematics Achievement Percent of Students Reaching the International Benchmarks Advanced Benchmark (625) High Benchmark (550) Intermediate Benchmark (475) TIMSS Advanced Mathematics Coverage Index Russian Federation 24 (2.9) 55 (3.2) 83 (2.2) 1.4% Iran, Islamic Rep. of 11 (1.8) 29 (3.0) 56 (2.8) 6.5% Lebanon 9 (1.2) 47 (1.9) 88 (1.3) 5.9% Netherlands 6 (0.8) 52 (2.8) 95 (1.1) 3.5% Italy 3 (1.0) 14 (2.0) 41 (3.0) 19.7% Slovenia 3 (0.5) 14 (1.4) 41 (2.4) 40.5% Armenia 2 (0.8) 13 (1.6) 33 (2.0) 4.3% Norway 1 (0.4) 9 (1.0) 35 (2.2) 10.9% Sweden 1 (0.4) 9 (1.2) 29 (1.9) 12.8% Philippines 1 (0.3) 4 (0.7) 13 (1.5) 0.7% Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Exhibit 3.3: Trends in in Percent of of Students Reaching the the TIMSS Advanced 2008 2008 International Benchmarks of Mathematics Achievement TIMSS Advanced Mathematics Coverage Index 2008 1995 Advanced International Benchmark (625) 2008 Percent of Students Percent of Students Reaching the International Benchmarks 1995 Percent of Students High International Benchmark (550) 2008 Percent of Students 1995 Percent of Students Intermediate International Benchmark (475) 2008 Percent of Students 1995 Percent of Students Russian Federation 1.4% 2.0% 24 (2.9) 22 (3.1) 55 (3.2) 51 (3.5) 83 (2.2) 78 (2.7) Italy 19.7% 20.2% 3 (1.0) 5 (2.2) 14 (2.0) 22 (5.0) 41 (3.0) i 59 (4.9) Slovenia 40.5% 75.4% 3 (0.5) 5 (1.3) 14 (1.4) i 23 (3.5) 41 (2.4) i 54 (4.5) Sweden 12.8% 16.2% 1 (0.4) i 6 (1.4) 9 (1.2) i 30 (3.3) 29 (1.9) i 64 (3.1) h 2008 percent significantly higher than 1995 i 2008 percent significantly lower than 1995 ( ) Standard errors appear in parentheses.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 95 Advanced Benchmark. As might be expected, given that it had the highest mathematics achievement average, the Russian Federation had the greatest percentage of students (24%) reaching the Advanced International Benchmark. Next came Iran with 11 percent, then Lebanon with 9 percent, and the Netherlands with 6 percent. It is noteworthy that relatively more students reached the Advanced Benchmark in Iran and the Lebanon than in the Netherlands, even though average achievement was higher in the Netherlands. This is a reflection of the relatively narrow range of achievement in the Netherlands, evident in Exhibit 2.1, compared to most other participating countries. A more positive consequence of the Netherlands narrow achievement range is that it had the highest percentage of students (95%) reaching the Intermediate Benchmark. The percent of students who scored at the Intermediate Benchmark ranges from a low of 13 percent in the Philippines to a high of 95 percent in the Netherlands. Results for Slovenia and Italy indicate that countries with a comparatively high TIMSS Advanced Mathematics Coverage Index are still able to obtain strong performance from many of their students. These results show that a system-wide policy of allowing a larger proportion of students to enroll in advanced courses in mathematics does not necessarily have a negative impact on overall students performance. It can provide opportunities for further study in mathematics-related specialty areas to more students. In all of these kinds of comparisons, it is important to bear in mind the potential impact of the Mathematics Coverage Index on performance levels. On the one hand, these students the very best mathematics students in their respective countries found the TIMSS advanced mathematics test to be challenging. In six countries the percent of students reaching the Advanced Benchmark was 3 percent or less. On the other hand, in six countries, more than 40 percent of students

96 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks reached at least the Intermediate Benchmark which, as shown in Exhibit 3.1, means that those students demonstrated knowledge of the concepts and procedures in algebra, calculus, and geometry assessed by TIMSS Advanced 2008. Exhibit 3.3 presents changes in the percent of students reaching the benchmarks between 1995 and 2008 for the four countries that participated in both studies. Countries are ranked in descending order of the percent of students who reached the Advanced International Benchmark. The display also shows the TIMSS Advanced Mathematics Coverage Index for each country in the 1995 and 2008 assessments. Over that period, the index declined in all four countries. The most dramatic drop in the Coverage Index occurred in Slovenia: from 75 percent coverage in 1995 to 40 percent in 2008. The results reflect the overall changes in achievement for the four countries, with all experiencing declines since 1995 except the Russian Federation, which evidenced little, if any, change (see Exhibit 2.4). No country showed a significant improvement in the percent of students reaching any of the three international benchmarks. However, there were several significant declines. Sweden experienced declines at all three benchmarks even though the population appears to have become more specialized between 1995 and 2008. Also, Slovenia, with the broadest population coverage but still greatly reduced in scope compared to 1995, had significantly fewer students reaching the High and Intermediate Benchmarks. Italy had declines at the Intermediate Benchmark.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 97 Mathematics: Achievement at the Advanced International Benchmark The TIMSS Advanced 2008 Assessment Frameworks called for an almost equal partitioning of the items to be included in the advanced mathematics assessment among the three content domains: 35 percent for algebra, 35 percent for calculus, and 30 percent for geometry. According to the framework, the algebra content domain includes much of the algebra and functions content that provides the foundation for mathematics at the college or university level. Students should be able to use properties of the real and complex number systems to solve problems set in real-world contexts or in abstract, mathematical ones. They should have facility in investigating basic characteristics of sequences and series, and skill in manipulating and using combinations and permutations. The ability to work with a variety of equations is fundamental for such students, providing a means of operating with mathematical concepts at an abstract level. The concept of function is an important unifying idea in mathematics, and students should be familiar with it. Since the calculus content of national and system-level advanced mathematics curricula varies considerably across countries, the calculus content for TIMSS Advanced Mathematics 2008 was limited to material likely to be included in final year mathematics in almost all the participating countries. The focus was on understanding limits and finding the limit of a function, differentiation and integration of a range of functions, and using these skills in solving problems. The TIMSS geometry items related to four strands or topics: Euclidean geometry (traditional or transformation), analytic geometry, trigonometry, and vectors. Euclidean geometry and analytic geometry have been important components of the secondary mathematics

98 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks curriculum for centuries, and are still widely viewed as important prerequisites for the study of mathematics at the university level. Trigonometry is part of the mathematics curriculum in all countries, but not always as part of the geometry domain. Transformation geometry and vectors are more recent additions to the mathematics curriculum in many countries, and there is considerable variation both in the amount of emphasis given to them across countries, as well as the degree of rigor with which the area is approached. The TIMSS items related to these two areas dealt with fairly elementary topics. In the algebra domain, the framework specifies that students should recognize representations of functions and be able to solve various kinds of equations, including quadratic equations. Exhibit 3.4 presents an algebra item likely to be solved correctly by students performing at the Advanced International Benchmark. In this example (Example Item 1), students were asked to find the numerical coefficients of a quadratic function having been given its graph and its x- and y-intercepts. An example of a correct solution to this constructedresponse item is shown in the exhibit. According to the information provided in Chapter 1 on the topics that were in the intended curriculum and taught to the students (Exhibit 1.13), all countries included polynomial equations and functions in their curriculum, and taught these topics (except function of a function in the Philippines) to their students. Nevertheless, students found this item difficult, and this was true for all of the items that anchored at the Advanced Benchmark. The percent of students receiving full credit ranged from a high of 64 in Lebanon to a low of 8 in Sweden. After Lebanon, the next highest result was 39 percent correct in the Russian Federation. The scoring guide for Example Item 1 shows the five correctand the four incorrect-response categories used by the item scorers as well as the non-response category. Also shown are the percents of

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 99 students in each category in each country. Category 13 refers to the use of a graphing calculator to find the coefficients of the equation. The total percent correct for a given country is the sum across the various correct-response categories. The most frequently used correct solution method for Example 1, in every country except Armenia, was using simultaneous linear equations in three variables (a, b, and c) given three pairs of values for x and f(x). The other four correct approaches were used by very few students. Non-response rates for this item ranged from a low of 10 percent in Lebanon, the country with the highest score on the item, to 55 percent in Sweden, 63 percent in Norway, and 70 percent in Armenia. The category 72 indicates that many students in some countries were able to find the value of the constant term, c, but not of a or b. Exhibit 3.5 shows an example multiple-choice item from the calculus domain that anchored at the Advanced Benchmark (Example Item 2). The item was designed to test students understanding of the definite integral, and the alternatives were chosen to reflect common errors or misconceptions. Students had to realize that the definite integral was not simply the sum of the three shaded areas, but the signed or algebraic sum, where the value of area B was negative. Not surprisingly, the incorrect response most frequently chosen in most countries was 7.6, the sum of the absolute values of the three areas identified on the graph. The percent correct in every country was rather low, and there was not as much variation in the proportion of students selecting the correct response across countries as was the case with many other items. The highest performance on this item was 46 percent in the Islamic Republic of Iran and 41 percent correct in the Russian Federation. About one-third of the students responded correctly in the Netherlands, Lebanon, and Slovenia. Understandably, the lowest

100 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.4: TIMSS Advanced 2008 2008 Advanced International Benchmark (625) of Mathematics Achievement Example Item 1 Content Domain: Algebra Description: Determines the coefficients of a quadratic function given the points of intersection between the graph and the axes Copyright protected by IEA. The answer shown is an example of a student response that was scored as correct Percent Correct Lebanon 64 (2.9) Russian Federation 39 (2.7) Slovenia 32 (2.5) Iran, Islamic Rep. of 32 (2.7) Italy 22 (2.8) Netherlands 16 (1.8) Armenia 16 (2.7) Norway 10 (1.6) Philippines 9 (1.7) Sweden 8 (1.8) This item may not be used for commercial purposes without express permission from IEA. Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 101 Exhibit 3.4: TIMSS Advanced 2008 2008 Advanced International Benchmark (625) of Mathematics Achievement Example Item 1 (Continued) Code Response Correct Student Responses 10 a = 2, b = 2, c = 4 using factorization Scoring Guide Item: MA23141 11 All values correct by solving three simultaneous equations 12 All values correct using calculator to solve simultaneous equations 13 All values correct using calculator for quadratic regression 19 All values correct by other correct method. Incorrect Student Responses 70 Calculator used but incorrect or explanation inadequate 71 All values correct but no correct method shown. 72 c = 4 with values of a and b missing or incorrect. 79 Other incorrect NR No Response Percent of Students in Each Scoring Guide Category Correct Student Responses Incorrect Student Responses 10 11 12 13 19 70 71 72 79 NR Lebanon 8 (1.7) 54 (2.9) 2 (0.7) 0 (0.2) 1 (0.4) 1 (0.6) 17 (2.1) 0 (0.0) 7 (1.6) 10 (1.7) Russian Federation 1 (0.4) 31 (2.7) 0 (0.0) 0 (0.0) 6 (1.2) 0 (0.0) 2 (0.8) 14 (1.8) 14 (1.2) 31 (2.7) Slovenia 8 (2.0) 24 (2.3) 0 (0.0) 0 (0.0) 1 (0.4) 0 (0.0) 0 (0.3) 28 (2.4) 23 (2.1) 16 (2.1) Iran, Islamic Rep. of 1 (0.4) 29 (2.6) 0 (0.0) 0 (0.0) 1 (0.5) 0 (0.0) 1 (0.5) 12 (1.6) 15 (1.9) 40 (2.9) Italy 7 (1.6) 14 (2.4) 0 (0.1) 0 (0.0) 0 (0.0) 0 (0.3) 1 (0.4) 12 (2.1) 7 (1.1) 58 (3.5) Netherlands 1 (0.6) 11 (1.8) 0 (0.0) 1 (0.7) 2 (0.7) 1 (0.6) 2 (0.7) 30 (2.2) 27 (2.6) 23 (2.1) Armenia 8 (2.4) 6 (2.1) 0 (0.0) 0 (0.0) 1 (0.7) 0 (0.0) 0 (0.0) 5 (2.0) 10 (3.0) 70 (3.2) Norway 1 (0.4) 1 (0.5) 0 (0.1) 6 (1.5) 1 (0.5) 2 (0.7) 1 (0.4) 9 (1.5) 15 (1.4) 63 (3.1) Philippines 2 (0.5) 7 (1.3) 0 (0.1) 0 (0.0) 0 (0.0) 0 (0.2) 0 (0.1) 8 (1.0) 49 (2.3) 34 (2.5) Sweden 2 (0.6) 5 (1.4) 0 (0.4) 0 (0.3) 0 (0.3) 0 (0.3) 0 (0.0) 18 (2.0) 19 (1.8) 55 (2.5) Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

102 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.5: TIMSS Advanced 2008 2008 Advanced International Benchmark (625) of Mathematics Achievement Example Item 2 Content Domain: Calculus Description: Calculates the definite integral given the graph of a function and the areas between the curve and the x-axis Copyright protected by IEA. A B Correct Response Percent of Students C D NR* Iran, Islamic Rep. of 3 (0.5) 46 (3.1) 6 (1.1) 12 (1.6) 32 (2.5) Russian Federation 5 (0.8) 41 (3.3) 14 (1.4) 29 (2.2) 11 (1.3) Netherlands 4 (1.1) 36 (2.6) 13 (1.5) 30 (2.8) 18 (2.3) Lebanon 3 (0.6) 35 (2.7) 7 (1.3) 36 (2.1) 19 (2.0) Slovenia 3 (0.7) 32 (2.7) 15 (1.7) 28 (3.5) 21 (2.1) Italy 5 (1.2) 26 (2.8) 14 (2.0) 20 (2.3) 34 (3.2) Sweden 11 (1.1) 26 (1.7) 21 (1.8) 20 (1.9) 21 (2.1) Norway 4 (1.1) 23 (1.9) 19 (2.4) 36 (2.3) 18 (1.6) Philippines 12 (1.6) 23 (1.8) 24 (1.5) 35 (1.8) 6 (0.9) Armenia 7 (1.9) 18 (3.2) 14 (2.9) 9 (2.3) 53 (3.7) Percent Correct Iran, Islamic Rep. of 46 (3.1) Russian Federation 41 (3.3) Netherlands 36 (2.6) Lebanon 35 (2.7) Slovenia 32 (2.7) Italy 26 (2.8) Sweden 26 (1.7) Norway 23 (1.9) Philippines 23 (1.8) Armenia 18 (3.2) This item may not be used for commercial purposes without express permission from IEA. * No Response Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 103 performance (18%) was in Armenia where this topic is not included in the advanced curriculum. Non-response rates for this item ranged from a low of 6 percent in the Philippines to a high of 53 percent in Armenia. The third example of an item that anchored at the Advanced Benchmark comes from the geometry domain and is shown in Exhibit 3.6. Example Item 3 required students to solve a multi-step word problem involving trigonometric ratios to identify the length of a side of a regular polygon inscribed in a circle. All participants included trigonometry in their intended curriculum, and teachers reported teaching these topics to nearly all students in their advanced mathematics classes (94 100%). The problem was posed in a situation that was practical, yet novel for most students. The best performance on this item was in the Russian Federation where 40 percent of students selected the correct response. In 6 of the 10 countries, the average percent correct was at the chance level, 25 percent, or lower. One method of solving this problem would be to drop a perpendicular bisector from the center of the circle to the base of the triangle formed by a pair of adjacent radii and one of the windows. The perpendicular divides the triangle into two right triangles, and the length of the base of each of those triangles is r sin 9. A second method would involve the use of the sine law. Non-response rates for this item were quite low in most countries, and response C was the most common incorrect response in all countries except the Islamic Republic of Iran. All three alternatives attracted significant numbers of students in all countries.

104 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.6: TIMSS Advanced 2008 2008 Advanced International Benchmark (625) of Mathematics Achievement Example Item 3 Content Domain: Geometry Description: Solves a multi-step word problem involving trignometric ratios to identify the length of a side of a regular polygon inscribed in a circle Copyright protected by IEA. A B Correct Response Percent of Students C D NR* Russian Federation 10 (1.3) 40 (2.4) 25 (1.8) 22 (1.7) 3 (0.6) Netherlands 8 (1.4) 36 (2.7) 32 (2.4) 22 (2.2) 2 (0.8) Iran, Islamic Rep. of 11 (1.5) 28 (2.4) 15 (1.9) 22 (2.1) 24 (1.9) Slovenia 10 (1.1) 26 (2.0) 40 (2.1) 20 (2.0) 4 (1.1) Lebanon 11 (1.6) 25 (2.5) 29 (2.4) 22 (2.6) 13 (1.8) Italy 12 (2.0) 22 (2.5) 28 (2.5) 21 (1.8) 16 (3.0) Sweden 10 (1.4) 22 (1.7) 42 (2.2) 22 (1.7) 4 (0.8) Philippines 21 (1.5) 21 (1.4) 36 (1.5) 21 (1.4) 1 (0.3) Armenia 9 (2.8) 20 (3.1) 26 (3.3) 18 (2.4) 27 (2.8) Norway 13 (1.5) 18 (1.8) 42 (1.9) 22 (1.9) 4 (1.0) Percent Correct Russian Federation 40 (2.4) Netherlands 36 (2.7) Iran, Islamic Rep. of 28 (2.4) Slovenia 26 (2.0) Lebanon 25 (2.5) Italy 22 (2.5) Sweden 22 (1.7) Philippines 21 (1.4) Armenia 20 (3.1) Norway 18 (1.8) This item may not be used for commercial purposes without express permission from IEA. * No Response Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 105 Mathematics: Achievement at the High International Benchmark Exhibit 3.7 shows a multiple-choice item from the algebra domain that anchored at the High International Benchmark. Example Item 4 required students to identify which of four given graphs represented the relationship between the volume of a sphere and its diameter. Performance on this item was best in the Netherlands, where 60 percent of students recognized that the correct response was the only one showing that the volume of a sphere increases monotonically without an upper bound in a non-linear fashion as its diameter increases. In more than half of the countries, the percent of students responding correctly was below 40 percent. The three alternatives all attracted significant numbers of students, and the non-response rates were quite low: 7 percent or less in 9 countries and 13 percent in Armenia. Example Item 5, shown in Exhibit 3.8, is from the calculus domain and also anchored at the High International Benchmark. This constructed-response item showed students the graph of a trigonometric function and asked why the slopes of the tangent to the graph at two given points were equal. In order to answer the item correctly, students had to know that the slope of the tangent to a curve is given by the first derivative of the function. Then they had to calculate the derivative of the given function,, and know the values of sin π and sin 2π. It is not possible to tell from the incorrect response categories for this item what specific kinds of errors students made most frequently. Students from the Netherlands had the best result on this item (53% correct, and only 3% non-response), but there was a considerable range across countries and the percent correct in six countries was less than 25. Referencing Exhibit 1.14 from Chapter 1, it can be seen that although all participants included derivatives in the intended

106 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.7: TIMSS Advanced 2008 2008 High High International Benchmark (550) of Mathematics Achievement Example Item 4 Content Domain: Algebra Description: Identifies the graph that represents the relationship between the volume of a sphere and its diameter Copyright protected by IEA. A Correct Response Percent of Students B C D NR* Netherlands 60 (2.8) 21 (1.8) 10 (1.6) 9 (1.5) 0 (0.0) Russian Federation 49 (2.7) 9 (1.6) 15 (2.4) 25 (1.8) 1 (0.4) Iran, Islamic Rep. of 47 (2.9) 10 (1.6) 19 (2.0) 17 (2.1) 7 (1.3) Sweden 42 (2.9) 27 (2.7) 9 (1.2) 21 (1.7) 2 (0.6) Italy 38 (2.9) 17 (2.0) 10 (2.1) 30 (2.3) 5 (1.2) Norway 37 (2.3) 23 (2.0) 16 (1.9) 23 (1.7) 1 (0.4) Philippines 34 (2.0) 21 (1.4) 11 (1.2) 33 (1.8) 1 (0.3) Armenia 31 (3.6) 29 (3.6) 14 (2.9) 13 (2.3) 13 (1.7) Lebanon 30 (2.2) 31 (2.5) 13 (1.9) 19 (2.1) 7 (1.3) Slovenia 29 (2.3) 29 (2.3) 7 (1.6) 34 (2.0) 1 (0.5) Percent Correct Netherlands 60 (2.8) Russian Federation 49 (2.7) Iran, Islamic Rep. of 47 (2.9) Sweden 42 (2.9) Italy 38 (2.9) Norway 37 (2.3) Philippines 34 (2.0) Armenia 31 (3.6) Lebanon 30 (2.2) Slovenia 29 (2.3) This item may not be used for commercial purposes without express permission from IEA. * No Response Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 107 curriculum, this topic was not always covered in the implemented curriculum, with about 81 percent of the students in Lebanon taught the topic, about two-thirds in Armenia and Slovenia, and about half in the Philippines. Non-response rates varied widely across countries, and in Italy and Armenia more than 60 percent of students failed to provide an answer to this item. The third example of an item that anchored at the High Benchmark, Example Item 6, is from the geometry domain and is shown in Exhibit 3.9. To solve this multiple-choice item, students had to be familiar with some basic properties of the slopes of lines. Again, students from the Netherlands had the best performance on this item with 75 percent responding correctly. For 6 of the 10 countries, the percentage responding correctly was above 50 percent. Responses C and D were the most frequently chosen incorrect responses.

108 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.8: TIMSS Advanced 2008 High International Benchmark (550) of Mathematics Achievement Example Item 5 Content Domain: Calculus Description: Justifies a statement about slopes at two points on the graph of a trigonometric function Copyright protected by IEA. The answer shown is an example of a student response that was scored as correct Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). Percent Correct Netherlands 53 (2.7) Lebanon 48 (2.7) Iran, Islamic Rep. of 45 (2.8) Russian Federation 39 (3.3) Sweden 22 (2.5) Italy 19 (2.7) Armenia 18 (2.7) Slovenia 10 (1.5) Norway 9 (1.2) Philippines 2 (1.0) This item may not be used for commercial purposes without express ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent. permission from IEA.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 109 Exhibit 3.8: TIMSS Advanced 2008 High International Benchmark (550) of Mathematics Achievement Example Item 5 (Continued) Code Response Correct Student Responses Scoring Guide Item: MA23198 10 Differentiates or uses the cosine function to show gradient the same at x = and x = 2 11 Correct answer using calculator Incorrect Student Responses 70 Calculator used answer incorrect or explanation inadequate 71 Differentiates correctly explanation inadequate 79 Other incorrect NR No Response Percent of Students in Each Scoring Guide Category Correct Student Responses Incorrect Student Responses 10 11 70 71 79 NR Netherlands 52 (2.9) 0 (0.5) 0 (0.0) 3 (0.9) 41 (2.8) 3 (0.8) Lebanon 48 (2.7) 0 (0.0) 3 (0.7) 0 (0.0) 33 (2.4) 16 (2.4) Iran, Islamic Rep. of 45 (2.8) 0 (0.0) 1 (0.6) 1 (0.4) 38 (2.6) 15 (1.7) Russian Federation 39 (3.3) 0 (0.0) 0 (0.0) 2 (0.5) 37 (2.1) 22 (2.3) Sweden 21 (2.5) 0 (0.1) 0 (0.4) 4 (0.6) 56 (2.3) 19 (1.9) Italy 18 (2.8) 1 (0.0) 0 (0.0) 2 (0.8) 11 (1.5) 69 (3.2) Armenia 18 (2.7) 0 (0.0) 0 (0.0) 1 (0.0) 20 (3.0) 61 (3.9) Slovenia 10 (1.5) 0 (0.0) 0 (0.0) 2 (0.5) 64 (2.4) 24 (2.5) Norway 9 (1.2) 0 (0.0) 0 (0.0) 1 (0.4) 61 (2.2) 30 (2.5) Philippines 2 (1.0) 0 (0.0) 0 (0.0) 0 (0.0) 71 (1.8) 27 (1.6) Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

110 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.9: TIMSS Advanced 2008 2008 High High International Benchmark (550) of Mathematics Achievement Example Item 6 Content Domain: Geometry Description: Finds the sum of the slopes of the three sides of an equilateral triangle with one side along the x-axis Copyright protected by IEA. A Correct Response Percent of Students B C D NR* Netherlands 75 (1.5) 1 (0.5) 6 (0.9) 10 (1.1) 4 (0.6) Iran, Islamic Rep. of 61 (2.3) 2 (0.5) 6 (0.9) 9 (1.0) 4 (0.7) Lebanon 54 (2.0) 3 (0.5) 9 (1.1) 17 (1.5) 7 (0.9) Slovenia 53 (2.0) 3 (0.6) 18 (1.4) 14 (1.6) 5 (0.7) Russian Federation 52 (2.5) 3 (0.6) 11 (1.0) 22 (1.5) 6 (0.9) Norway 51 (2.1) 1 (0.4) 18 (1.7) 14 (1.1) 6 (0.9) Sweden 45 (1.8) 6 (0.8) 21 (1.5) 13 (1.3) 7 (0.7) Italy 42 (2.3) 3 (0.5) 10 (1.0) 15 (1.5) 6 (0.7) Armenia 33 (2.2) 6 (1.2) 11 (1.5) 19 (2.6) 9 (1.9) Philippines 29 (1.7) 4 (0.4) 21 (1.2) 26 (1.2) 19 (1.3) Percent Correct Netherlands 75 (1.5) Iran, Islamic Rep. of 61 (2.3) Lebanon 54 (2.0) Slovenia 53 (2.0) Russian Federation 52 (2.5) Norway 51 (2.1) Sweden 45 (1.8) Italy 42 (2.3) Armenia 33 (2.2) Philippines 29 (1.7) This item may not be used for commercial purposes without express permission from IEA. * No Response Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 111 Mathematics: Achievement at the Intermediate International Benchmark Example Item 7, shown in Exhibit 3.10, is taken from the algebra domain. This constructed-response item required students to solve an inequality involving a rational expression in one variable. All countries included inequalities in their curricula, and teachers reported that nearly all students had been taught this topic (96 100%). In the Russian Federation, 80 percent of students responded correctly. In half the countries, the percent of students providing correct responses was greater than 50. Students were not required to show their work, and it is not possible to tell from the scoring guide how students attempted to solve the inequality. The calculus item shown in Exhibit 3.11 is a constructed-response item requiring students to find the derivative of a rational function (Example Item 8). To find this derivative, students had to know and be able to apply the quotient rule. Students in several countries performed very well on this item, with the best performance being registered in Lebanon with 91 percent of students obtaining full credit for the item. Approximately three fourths of the Iranian and Russian students as well as two thirds of the Slovenian students also received full credit. On the other hand, students in Norway, the Philippines, and Sweden found the item much more difficult. The most frequent incorrect response in several countries was based on an attempt to use the quotient rule for differentation, but doing so incorrectly. Example Item 9, a multiple-choice item shown in Exhibit 3.12, is taken from the geometry domain. One way to solve this problem is to visualize or draw a right triangle, and recall that the vertices of a right triangle can be inscribed in a circle with the hypotenuse, being the diameter of the circumcircle. This means that T, the mid-point of

112 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.10: TIMSS Advanced 2008 Intermediate International Benchmark (475) of Mathematics Achievement Example Item 7 Content Domain: Algebra Description: Solves a rational inequality with linear numerator and denominator Copyright protected by IEA. The answer shown is an example of a student response that was scored as correct Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). Percent Correct Russian Federation 80 (1.8) Armenia 74 (2.6) Italy 60 (3.7) Iran, Islamic Rep. of 54 (2.5) Lebanon 51 (2.4) Netherlands 47 (2.4) Sweden 30 (2.4) Slovenia 26 (2.6) Norway 16 (1.7) Philippines 15 (1.7) This item may not be used for commercial purposes without express ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent. permission from IEA.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 113 Exhibit 3.10: TIMSS Advanced 2008 Intermediate International Benchmark (475) of Mathematics Achievement Example Item 7 (Continued) Code Response Correct Student Response 10 x > 2 Incorrect Student Responses 79 Incorrect NR No Response Scoring Guide Item: MA23135 Percent of Students in Each Scoring Guide Category Correct Student Response Incorrect Student Responses 10 79 NR Russian Federation 80 (1.8) 19 (1.7) 1 (0.4) Armenia 74 (2.6) 21 (2.1) 4 (1.3) Italy 60 (3.7) 34 (3.3) 7 (1.4) Iran, Islamic Rep. of 54 (2.5) 42 (2.5) 4 (0.9) Lebanon 51 (2.4) 46 (2.3) 3 (1.0) Netherlands 47 (2.4) 48 (2.5) 5 (1.2) Sweden 30 (2.4) 60 (2.2) 10 (1.4) Slovenia 26 (2.6) 71 (2.7) 3 (1.1) Norway 16 (1.7) 64 (2.1) 20 (2.0) Philippines 15 (1.7) 78 (1.6) 8 (0.9) Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

114 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.11: TIMSS Advanced 2008 Intermediate International Benchmark (475) of Mathematics Achievement Example Item 8 Content Domain: Calculus Description: Differentiates a rational function where the numerator and denominator are both linear Copyright protected by IEA. The answer shown is an example of a student response that was scored as correct Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). Percent Correct Lebanon 91 (1.6) Iran, Islamic Rep. of 79 (2.2) Russian Federation 75 (2.4) Slovenia 67 (2.1) Italy 60 (3.4) Armenia 56 (3.6) Netherlands 48 (2.9) Norway 29 (2.1) Philippines 21 (2.1) Sweden 20 (1.8) This item may not be used for commercial purposes without express ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent. permission from IEA.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 115 Exhibit 3.11: TIMSS Advanced 2008 Intermediate International Benchmark (475) of Mathematics Achievement Example Item 8 (Continued) Code 10 Response Correct Student Responses f ( x) 5 ( x 1) = 2 using 2 11 Correct expression using calculator Incorrect Student Responses Scoring Guide Item: MA23159 u ( u v uv ) = or, ( uv) = u v + uv v v 70 Calculator used answer incorrect or explanation inadequate 71 Correct answer no working shown 72 Using quotient rule but no correct expression 73 Using product rule but no correct expression 79 Other incorrect NR No Response Note: Students were instructed that if they used a calculator they were to explain how and for what it was used. Percent of Students in Each Scoring Guide Category Correct Student Responses Incorrect Student Responses 10 11 70 71 72 73 79 NR Lebanon 91 (1.6) 0 (0.0) 0 (0.2) 4 (1.1) 0 (0.0) 0 (0.0) 4 (1.0) 1 (0.6) Iran, Islamic Rep. of 79 (2.2) 0 (0.0) 0 (0.0) 0 (0.1) 10 (1.2) 0 (0.0) 9 (1.7) 2 (0.9) Russian Federation 75 (2.4) 0 (0.0) 0 (0.0) 0 (0.0) 8 (1.7) 0 (0.0) 14 (2.1) 3 (0.6) Slovenia 67 (2.1) 0 (0.0) 0 (0.0) 0 (0.0) 10 (1.3) 0 (0.0) 21 (1.7) 3 (0.8) Italy 60 (3.4) 0 (0.1) 0 (0.0) 0 (0.0) 11 (1.6) 0 (0.0) 17 (2.7) 13 (2.1) Armenia 55 (3.6) 2 (1.0) 1 (0.6) 0 (0.0) 2 (1.1) 0 (0.0) 25 (3.3) 15 (2.0) Netherlands 48 (2.9) 0 (0.0) 0 (0.0) 0 (0.0) 40 (2.9) 4 (1.1) 7 (1.3) 1 (0.4) Norway 29 (2.2) 0 (0.3) 0 (0.2) 0 (0.0) 33 (3.0) 0 (0.0) 30 (2.9) 8 (1.4) Philippines 21 (2.1) 0 (0.0) 0 (0.0) 0 (0.0) 10 (1.2) 0 (0.0) 57 (2.3) 12 (1.6) Sweden 19 (1.7) 0 (0.2) 0 (0.0) 0 (0.0) 19 (2.0) 3 (0.9) 48 (2.3) 10 (1.4) Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

116 chapter 3: mathematics performance at the timss advanced 2008 international benchmarks Exhibit 3.12: TIMSS Advanced 2008 Intermediate International Benchmark (475) of Mathematics Achievement Example Item 9 Content Domain: Geometry Description: Uses properties of an isosceles right triangle to determine the length of a given median Copyright protected by IEA. Percent of Students A B C D Correct Response Lebanon 2 (0.5) 3 (0.7) 3 (0.7) 90 (1.4) 2 (0.5) Russian Federation 4 (0.7) 3 (0.5) 5 (0.7) 87 (1.3) 1 (0.3) Netherlands 4 (0.8) 4 (0.8) 10 (1.2) 79 (1.7) 4 (0.7) Iran, Islamic Rep. of 5 (0.8) 5 (0.7) 7 (0.9) 74 (1.8) 10 (1.1) Italy 7 (1.0) 11 (1.5) 8 (1.1) 65 (2.2) 9 (1.4) Slovenia 10 (1.2) 11 (1.3) 13 (1.0) 63 (2.0) 4 (0.8) Armenia 8 (1.5) 9 (1.4) 13 (1.8) 60 (2.5) 10 (1.3) Norway 10 (0.8) 14 (1.4) 18 (1.2) 49 (1.8) 9 (0.9) Philippines 19 (1.1) 17 (1.2) 17 (1.1) 47 (1.8) 1 (0.2) Sweden 11 (1.0) 15 (1.3) 25 (1.1) 41 (1.2) 8 (1.0) NR* Percent Correct Lebanon 90 (1.4) Russian Federation 87 (1.3) Netherlands 79 (1.7) Iran, Islamic Rep. of 74 (1.8) Italy 65 (2.2) Slovenia 63 (2.0) Armenia 60 (2.5) Norway 49 (1.8) Philippines 47 (1.8) Sweden 41 (1.2) This item may not be used for commercial purposes without express permission from IEA. * No Response Met guidelines for sample participation rates only after replacement schools were included (see Appendix A). ( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

chapter 3: mathematics performance at the timss advanced 2008 international benchmarks 117 the hypotenuse, QR, is the center of the circumscribed circle and that, since PT and QT are radii of that circle, they must be of equal length. The percent of students choosing the correct response to this item was at least 60 in 7 of the 10 participating countries, and in no country was the percent correct less than 40. The best results were in Lebanon (90%) and the Russian Federation (87%), and approximately three-fourths of the Dutch and Iranian students answered correctly. Non-response rates were quite low, and incorrect responses were distributed across the three alternatives.