Los Angeles Unified School District Today s Learners, Tomorrow s Leaders Making Meaning of Your Teacher Academic Growth over Time (AGT) Report This Focus Guide will serve four purposes: 1. To be used in connection with face-to-face professional learning for Teachers conducted by Building Leaders 2. To serve as an ongoing reference toolkit 3. To provide support for principals and district leaders to guide educators in the understanding of AGT reports 4. To guide teachers in a line of inquiry and root cause analysis for school improvement The components of this report provide Academic Growth over Time (AGT) information. The results reported here measure your impact on the academic growth of students in your classroom for English Language Arts (ELA) and Math. We will begin by looking at the report key. This key is located on the first page of your school AGT Report. It will be helpful to review as we begin to make meaning of the reports. LAUSD has chosen to use a five color system. Based on statistics, these colors indicate whether or not student growth was significantly far above, above, below, far below or not different than the predicted AGT estimate. Keep in mind that the predicted AGT estimate and the district average are one in the same. 4.8 Blue - Far Above Predicted AGT: AGT Estimate is significantly more than 4. 4.1 Green - Above Predicted AGT: AGT Estimate is significantly above the District Average (3). 2.7 Gray - Within the range of Predicted AGT: AGT Estimate is not significantly different from the District Average (3). 2.2 Yellow - Below Predicted AGT: AGT Estimate is significantly below the District Average (3). 1.3 Red - Far Below Predicted AGT: AGT Estimate is significantly less than 2.
We will work through the anatomy of each portion of these reports. Take a few minutes to examine the table on the left. The results in this table provide AGT estimates for English Language Arts (ELA) and Math. These results represent your work with all of your students. As you can see in the chart below, results are provided for both the past academic year and for an average of the last three years. The yellow arrow is pointing to the district average. The district average is the prediction customized for each teacher s group of students. AGT is reported on a 1-5 scale, with 3 as the district average. The AGT estimate range is unbounded, but almost all scores fall between 1 and 5. For visual purposes, the 1-5 range is shown. Scores outside of this range are placed on the edge, with their correct score. The 1-5 scale is used to facilitate comparisons of AGT across grades, subjects and years. To help give context tot he 1-5 scale, the distribution of teacher effects follows the normal bell curve below. Note: The actual distribution of teacher-level AGT is slightly compressed towards the middle due to a statistical procedure that optimally adjusts for imprecision. For more information, see the technical manual section on shrinkage. The predictions for each group of students is based on the actual performance of real students in LAUSD. These predictions are based on the average performance of students like them during the same time period. Students like them is defined as the variables listed: Prior CST ELA scores Prior CST Math scores Grade level Gender Race/Ethnicity Low income status IEP status Mobility Normal, Bell-shaped Curve 2.27% 13.59% 34.13% 34.13% 13.59% 2.27% 1 2 3 4 5 2
Now we will explore the meaning of the information in this table. The YELLOW arrow is pointing to the number of students included in this report. You will notice that logically, there are more students included for the three year averages. The GREEN arrow is pointing to an English Language Arts AGT point estimate of 4.0 in the green bubble. In this example, 4.0 and the green color indicate that with statistical significance these students are growing above their predicted estimate. This means that when compared to other students like them in the district, these students are performing above the district average. The BLUE arrow is pointing to a three year average Math AGT point estimate of 3.4. The bubble is grey indicating that over the past three years in math, students in this school statistically are not growing in a manner that is significantly different than the district average. This means that when compared to other students in the district, these students are more than likely performing within the district average or as predicted. The BLACK arrow is pointing to the confidence interval. The confidence interval is a measurement of error. It takes into consideration several factors such as consistency of student academic growth, number of students and the inherent error in tests. In this example, the AGT point estimate is 3.6. The bubble is green, indicating that with statistical significance these students have grown above their predicted estimate. The most likely AGT estimate is 3.6, but it could be within a range of 3.1 to 4.1. +3.6 1 2 3 4 5 Confidence Interval +2.9 to +4.3 Let s take a closer look at another example of a confidence interval: When considering confidence intervals, the most likely value is at the center. However, in this example, the true value could range anywhere from 2.9 to 4.3. It is less likely that the true value is at the extremes of the confidence interval. It is critically important to keep confidence intervals in mind when using data for decision making. Alright, numbers are great, but the real question is, so what? What does this mean in terms of helping students learn and continuously improving educator practice? 3
Now go back to the original report and begin breaking this down from interpretation to application. Q: What does this report say about the results of your work on academic growth with ELA students? A: This report indicates that for the 2009 2010 school year, students grew statistically and significantly more than predicted when compared to other district ELA students like them in grade three. This year mirrors what has occurred over the past three years in ELA. The pattern is fairly consistent over the years. Keep in mind that there are statistical growth expectations and teacher classroom expectations there is always room for students to grow. Q: What does this report say about the results of your work on academic growth with Math students? A: This report indicates that for the 2009 2010 school year, student growth was within the district average when compared to other Math students in grade 3. The 2009 2010 school year also mirrors what has occurred over the past three years in Math. The pattern is fairly consistent over the years. Although growth has occurred in both ELA and Math, when compared to the district, math instruction could be an area of opportunity for this teacher. In the 2009 2010 year, he or she has facilitated significantly more growth with students in ELA. There is not that much of a distinction between reading and math in the three year average. The confidence intervals are very similar. 4
Now You Try! Examine your own teacher report, or a sample report. What information for ELA and Math does the first table tell you? Discuss with your colleagues. Summarize your findings: 5
We will now examine the teacher-level results with specific groups of students. The table below provides us with more information than the overall teacher report. It reflects the results of this teacher s work with specific groups of students and can help us look at student growth in a more granular way. For these teacher reports, only the three year averages are used. In some cases, AGT estimates are provided for specific groups of students. For this example, we will examine AGT by Prior Student Achievement Level, English Language Learners (ELL) status and Special Education (SPED). For each student group, the AGT estimate compares the actual achievement of students in your class to the predicted achievement of other students like them in the district. The information in this table is about English Language Arts students in your class. You will have a similar report for Math. All Proficient and Advanced students in this school are being compared to other Proficient and Advanced students like them in the district. Likewise, Basic category students are only being compared to other students like them in the district. The same is true for those students in the Below Basic and Far Below Basic category. Q: What is the impact of your work on the learning of students by prior achievement levels? A: There are a few considerations that are important in understanding this report. In the ELA section by prior achievement level, students are placed in one of three performance categories based on their pre-test achievement level. The three classifications that are used in these reports have been collapsed from the five classifications of the CST achievement levels. In this classroom in 2009 2010, students in the highest achievement category are performing somewhere within the range of their predicted AGT. It is a large confidence interval that barely spans across the district average. The students in the Below Basic performance category have grown more than one standard deviation above what was predicted. The students in the Below Basic performance category have, in essence, more than one standard deviation above what was predicted. In considering goals for school improvement, this educator is impacting high levels of growth with lower-achieving students. This is a great place to be when considering closing the achievement gap. A concern might be the average amount of growth with higher-achieving students. Is the curriculum rigorous enough to allow these students to grow even more significantly? 6
Q: What is the impact of your work on the learning of English Language Learners (ELL)? A: The ELL students in this class are growing within the range of their predicted AGT when compared to other students like them in LAUSD. Non-ELL students are growing statistically and significantly above what was predicted. Non-ELL students in this class are statistically and significantly growing more than other non-ell students like them in the district, NOT necessarily more than ELL students. Remember, that each group of students is being compared to other students with the same characteristics in the district. Q: What is the impact of your work on the learning of Special Education Students (SPED)? A: The special education students in this classroom are statistically significantly making below expected growth when compared to other students like them. This could be problematic for these students if we expect them to make Adequate Yearly Progress (AYP). The non-sped students are statistically and significantly above expected growth when compared to other students like them. Again, we are not saying that non-sped are growing more than SPED students, just that when compared to other non-sped students, this educator is more effective with this group than with the SPED students. Now You Try! Examine your own teacher report for prior achievement levels, ELL and SPED students or a sample report. What information for ELA and Math does this table tell you? Discuss with your colleagues. Summarize your findings: 7
The next table shows teacher-level results by gender. The information in this table is about English Language Arts students in this classroom. You will have a similar report for Math. Q: What is the impact of your work on the learning of males and females? A: During the 2009 2010 school year, both male and female students statistically significantly made growth above what was predicted. It would be interesting to now look at the math AGT estimates for this classroom to see if the same is true. Now You Try! Examine your own teacher report for Gender, or a sample report. What information for ELA and Math does this table tell you? Discuss with your colleagues. Summarize your findings: 8
This table shows teacher-level results by race. The information in this table is about English Language Arts students in your classroom. You will have a similar report for Math. Three Year Average In examining this report, there are a few things to consider. The bright green arrows are pointing to the column indicating the number of students for the three year average. Q: What is the impact of this teacher on the growth of students by race in 2009 2010? A: In this classroom, there are insufficient numbers of African American and Asian students to produce an AGT estimate. The Latino students have made growth which is statistically significant. Their growth was far above what was predicted (4.6 in the blue bubble) when compared to other Latino students like them. This is cause for celebration and an opportunity to uncover the reasons for the success. The White students in this school fell far below the range of their predicted AGT and significantly different than the district average. This is an area of challenge for this educator. Note: Since 75% of the student population in LAUSD is Latino, they become the standard comparison group. Therefore, if a particular school has a small or non-existent number of Latino students, they will not receive a differential effect table for race. 9
Now You Try! Examine your own school report for Race or a sample report. What information for ELA and Math does this table tell you? Discuss with your colleagues. Summarize your findings: 10
Now, let s take an overall reflective look at your report. What surprised you? Why? What affirmed what you already knew? Why? What is your area of strength? How might you best leverage your strength with your colleagues? What is an area of challenge or opportunity? In your opinion, are your results (average, above or below) reflective of the curriculum, your alignment to the curriculum, your instruction, the amount of rigor in your teaching, your assessment practices, your content knowledge, a combination of these or something else? Please be honest and honorable in your reflection. The LAUSD Teaching and Learning Framework may be a resource to help answer this question. What professional learning or support would you like to help in your areas of opportunities or challenges? 11
It is always an important step to examine and celebrate strengths. We can learn from our strengths and even apply strategies to our areas of challenge. So let s set a few focused goals for strengths. What is your personal strength-based goal? Strategy: 1. 2. 3. Example of a Strength-Based Goal Based on my teacher report, I am going to analyze what I do with my lower-achieving students to determine if I can apply strategies to other groups of students. Strategy: 1. Apply successful strategies to more advanced students or other subjects 2. Collaborate and leverage what is working with other teachers 3. Continue the momentum of growth with these students so that they continue to catch up It is equally as important to examine challenges and/or opportunities. In a culture of continuous improvement, they cannot be ignored. So let s set a few focused goals for challenges and/or opportunities. What is your personal challenge-based goal? Strategy: 1. 2. 3. Example of a Challenge-Based Goal Based on my teacher report, I am going to work with my higher-achieving students so that they are able to make more growth. Strategy: 1. Align my teaching with the level of rigor that content standards demand 2. Scaffold my instruction so that students are challenged beyond mastery of the standard 3. Collaborate with other teachers in my school who are facilitating greater gains with high-achieving students 12
Battelle for Kids is partnering with the Los Angeles Unified School District to expand the use of the Academic Growth over Time (AGT) Reports and other educational-improvement strategies to increase student progress and achievement. Visit www.battelleforkids.org to learn more about Battelle for Kids.