SCANTRON Corp. Research Briefs Research Department Examination of Growth Trajectories at the State Level in Reading and Mathematics Performance Series Test Scores ember 2004 Introduction One of the several advantages of tracking student test scores on a longitudinal scale is the ability to model and observe trends in the growth trajectories. A growth trajectory is simply the shape of the curve formed when fitting consecutive test results for individual students (or group of students) over time. This study is the first in a series of efforts to understand and model the growth patterns present in the Performance Series customer base across grade levels for an entire academic year. Growth trajectories were estimated in Mathematics and Reading for the 2003-2004 academic year. In addition, state level growth trajectories were calculated and compared for observable differences. Summary and Results In this first growth modeling effort, we focused our attention on the two subject areas with the most users in grades 2 through 8: Mathematics and Reading. We restricted our data to two specific testing times (ember 2003 and May 2004) for the 2003-2004 academic year. These two testing times were selected because they represent the peak testing periods in the data. These two points also allow us to measure the greatest growth within a grade level. Table 1 list the test counts by state and subject area. State Math Read State Math Read GA 1 43,965 40,075 LA 903 873 MI 1 32,483 34,033 NM 456 458 IL 1 24,738 25,503 WA 331 487 CA 1 22,775 21,778 NY 541 134 AR 1 11,615 9,978 MD 179 487 IN 1 10,936 9,545 NC 133 468 SD 1 9,471 6,969 AZ 248 262 PA 1 7,684 7,758 WI 133 178 KY 1 5,572 5,793 ME 122 123 TX 1 2,553 1,991 TN 125 86 FL 1,644 2,481 ID 66 59 AL 2,036 2,024 KS 101 0 OK 1,854 1,733 MO 94 5 NJ 2,631 716 VA 35 48 MN 2,237 1,039 MS 35 31 SC 940 1,075 AK 0 25 CO 932 906 OH 8 17 Table 1: Test counts by state. 1 indicates the top 10 states in the restricted sample. Our data set consisted of roughly 170,000 scores in each subject area for students in over 30 states across the country. For each grade level and subject area, we calculated descriptive statistics at both testing times. Table 1 and Table 2 display the summary statistics for the entire data set at both testing time points. Since the distribution of tests was not uniform over states, we focused our comparisons on the top 10 states in the sample. Descriptive statistics were also calculated at the state level for the top 10 states in Copyright 2004 SCANTRON Corporation. All rights reserved. 1 of 6
our sample. These top 10 states were: Georgia, Michigan, Illinois, California, Arkansas, Indiana, South Dakota, Pennsylvania, Kentucky, and Texas. Mathematics Reading Grade N Mean Std. I-Q Range Level 25 50 75 2003 2 8,355 2007 162.9 1884 2010 2125 3 7,931 2130 162.3 2028 2148 2246 4 13,252 2288 157.7 2307 2393 5 8,877 2370 172.0 2280 2382 2479 6 15,314 2502 188.0 2390 2511 2631 7 15,177 2589 206.7 2464 2727 8 7,699 2654 224.3 2517 2657 2805 May 2004 2 10,308 2171 159 2075 2193 2286 3 13,059 2290 170 2189 2311 2409 4 17,217 2410 173 2309 2422 2526 5 12,696 2501 193 2390 2513 2626 6 15,691 2593 213 2474 2737 7 16,149 2660 226 2523 2665 2812 8 9,031 2686 243 2529 2685 2856 Table 2: Descriptive statistics for the entire sample at the two testing times of ember 2003 and May 2004 in mathematics. Grade N Mean Std. I-Q Range 25 50 75 2003 2 6,898 1939 326.6 1673 1927 2214 3 7,810 2142 335.5 1905 2176 2383 4 11,724 2377 331.9 2189 2410 2606 5 9,098 2507 327.2 2322 2550 2736 6 15,220 2664 310.7 2489 2703 2874 7 15,398 2768 305.1 2593 2796 2987 8 7,570 2815 300.8 2649 2843 3026 May 2004 2 8,800 2179 335 1942 2218 2441 3 12,552 2338 329 2141 2396 2583 4 15,568 2525 321 2353 2591 2753 5 11,456 2628 309 2473 2701 2841 6 15,922 2754 305 2611 2813 2956 7 15,327 2835 307 2694 2888 3043 8 7,669 2851 303 2712 2897 3052 Table 3: Descriptive statistics for the entire sample at the two testing times of ember 2003 and May 2004 in reading. Using these statistics, we calculated growth trajectories for the entire data set (referred to as the National growth trajectory) using the 25 th, 50 th, and 75 th percentile score at each time period and grade level as our anchor points. There are a variety of ways to smooth out the data, but we opted to use a logistic equation to fit the growth trajectories to the inter-quartile range of the data points. Growth trajectories were calculated for the entire data set (referred to as the National growth trajectory) as well as for the top 10 states in the sample. The following graphs are the estimated growth trajectories for Georgia, Michigan, Illinois, and California; plotted along with the National growth trajectories. In each plot, the horizontal axis measures the academic year, beginning in ember and ending in May, from grade 2 through grade 8. The vertical axis measures the Performance Series at each point in the academic year. In mathematics, we can observe that the state level growth trajectories all vary. The Georgia growth trajectories are higher than the National growth trajectories. This means that the students in Georgia are performing higher in mathematics than our National sample in mathematics. Both Michigan and California growth trajectories are lower than the National growth trajectories, with Michigan being the lowest and California mirroring the National trajectory. This means that students in Michigan and California are scoring lower than the National sample in mathematics. The Illinois growth trajectories are lower for grades 2 through 4 and then they cross the National growth trajectories and become higher in grades 5 through 8. This means that the students in grades 2 through 4 in Illinois score lower than the National sample in mathematics, while the students in grade 5 through 8 score higher than the National sample in mathematics. Similar differences are observed in reading scores. As in mathematics, students in the state of Georgia appear to be scoring higher in reading than all other students in our National Sample. Michigan and California growth trajectories in reading are also lower that the National growth trajectories. In this subject California students are performing lower in reading than the National sample. Once again as in mathematics, the reading growth trajectory in Illinois is lower in grades 2 through 4 and slightly higher in grades 5 through 8. Copyright 2004 SCANTRON Corporation. All rights reserved. 2 of 6
Mathematics Growth Trajectories - Georgia Mathematics Growth Trajectories - Michigan Mathematics Growth Trajectories - Illinois Mathematics Growth Trajectories - California Mathematics growth trajectories for Georgia, Michigan, Illinois, and California. National growth trajectories (50 th, 25 th, 75 th ), State Level growth trajectories (50 th --, 25 th --, 75 th --), State Level observed percentiles points (50 th, 25 th, 75 th ). Copyright 2004 SCANTRON Corporation. All rights reserved. 3 of 6
Reading Growth Trajectories - Georgia Reading Growth Trajectories - Michigan Reading Growth Trajectories - Illinois Reading Growth Trajectories - California Reading growth trajectories for Georgia, Michigan, Illinois, and California. National growth trajectories (50 th, 25 th, 75 th ), State Level growth trajectories (50 th --, 25 th --, 75 th --), State Level observed percentiles points (50 th, 25 th, 75 th ). Copyright 2004 SCANTRON Corporation. All rights reserved. 4 of 6
In order to identify the magnitude of these observable differences in State Level growth trajectories to the National growth trajectories, we decided to calculate a metric to help establish the difference. A statistic was calculated to measure the average distance of the State level growth trajectory to the National growth trajectory. We calculated the Root Mean Squared Distance from each State Level growth trajectory g( t ) to the National growth trajectory G() t. We only used the median growth curve for this calculation and ignored the 25 th and 75 th percentile curves. The Root Mean Squared Distance is given by 1 G t g t dt n RMSD = ( () ()) 2 This statistic measures the average distance in scaled units the State Level growth trajectory is deviating from the National Growth Trajectory. Table 4 summarizes the RMSD statistic for each state and each subject area. State Mathematics Reading RMSD RMSD GA 49.79+ 53.66+ MI 78.24 72.06 IL 31.94* 43.12* CA 19.43 136.65 AR 8.30+ 34.50+ IN 47.26+ 60.72+ SD 34.03* 23.24+ PA 36.34+ 81.88+ KY 42.22+ 60.60* TX 7.18* 56.49* Table 4: Root Mean Squared Difference of the State Level growth trajectories from the National growth trajectories. State Level trajectories that are larger/smaller/same as than the National growth trajectories are labeled by a positive sign (+)/negative sign ( )/an asterisk (*). The RMSD statistic measures the distance, but does not indicate the relative location of the State Level growth trajectory in comparison to the National growth trajectory. We identify State Level growth trajectories that are larger than the National growth trajectory with a positive sign (+) next to the RMSD statistic. State Level growth trajectories that are lower that the National growth trajectory are identified with a negative sign ( ) next to the RMSD statistic. States that had little observable differences from the National growth trajectory or those that crossed the National growth trajectory at some point in the academic year are marked with an asterisk (*). Given this statistic we see that Georgia, Indiana, Kentucky, Pennsylvania, and Arkansas growth trajectories in mathematics are all higher than the National growth trajectory (in that particular order, Georgia showing the largest difference and Kentucky showing the smallest difference). The states with the lower growth trajectories in mathematics are California and Michigan, with Michigan showing the lowest performance. The states of Illinois, South Dakota, and Texas did not show observable differences in the growth trajectory or crossed the National growth trajectory in mathematics. In the reading subject test, the states with the largest growth trajectories, in order of largest to smallest difference are, Pennsylvania, Indiana, Georgia, Arkansas, and South Dakota. The two states with the lowest reading growth trajectories are the same as in mathematics: Michigan and California. California shows the most Copyright 2004 SCANTRON Corporation. All rights reserved. 5 of 6
Root Mean Squared Difference in reading at over 130 scaled units. Illinois, Kentucky, and Texas are the states were no difference is observed or where the growth trajectory crosses the National reading growth trajectory at some point in the academic year. Conclusions This first attempt at modeling the growth trajectories in mathematics and reading using Performance Series as the growth measure illustrated the differences that are present in student performance across states. In some cases (as in California reading and Michigan mathematics), the growth curves are very different from other states. Even though these State Level comparisons are not based on random state level population samples, we feel that the difference in State Level performance is still relevant and informative for Performance Series users in interpreting their test score changes over time. For more information concerning this report, contact the Scantron Research Department at: Scantron Corporation 34 Parker, Irvine, CA 92618-1604 Toll Free: (800) 722-6876 www.scantron.com Copyright 2004 SCANTRON Corporation. All rights reserved. 6 of 6