Re-examining Prediction of Freshman Grade-Point Average in the CUNY system Daniel Koretz Harvard Graduate School of Education CUNY Graduate Center March 26, 2015
Thanks To CUNY and staff for providing CUNY data and assistance especially David Crook Colin Chellman Zun Tang (2)
Three research topics 1. How do Regents scores compare to SAT scores as predictors of freshman GPA (FGPA) in Senior and Comprehensive colleges? Potential for different score inflation from coaching 2. How do the does the prediction of FGPA differ within and between schools? 3. How are the benefits distributed? Which types of students and schools win and lose? (3)
Status of work Study 1: working paper completed and available Study 2: results presented today are preliminary; we expect a working paper by summer Study 3: just started; we expect to complete a working paper by fall Conducted for 2010 and 2011 cohorts; 2010 reported here (4)
Background Validation studies typically predict FGPA from high school grade point average (HSGPA) and SAT (or ACT) scores Use a single-level student regression model Effects of differences among schools confounded with differences among students within schools Problematic for example, between-school differences in grading standards Done campus-by-campus because of betweencampus differences in grading standards (5)
Study 1: comparing Regents to SAT scores Used a traditional single-level model: regressed FGPA on HSGPA, scores, and the combination of the two All predictors were standardized for comparability Analyzed SAT and Regents (Math A and ELA) separately, and SAT and Regents together Initially conducted analysis separately by campus Results differed among campuses But unnecessary for overall results in the CUNY data Final analyses were using data pooled across campuses (6)
Primary conclusions HSGPA predicts substantially better than either set of scores Larger difference than in national studies may reflect more refined CUNY HSGPA measure Adding scores to HSGPA improves aggregate prediction slightly Regents and SAT scores provide similar aggregate prediction Subject-specific and composite scores provide similar aggregate prediction Adding a second test has only trivial effects on aggregate prediction (7)
Simple correlations, CUNY and CEEB CUNY CEEB High School GPA 0.50 0.36 SAT Total Score 0.37 0.32 SAT Math 0.35 0.26 SAT Critical Reading 0.31 0.29 Regents Math 0.36 Regents English 0.35 CEEB correlations from Kobrin, J. L. et al. (2008), Validity of the SAT for Predicting First-Year College Grade-Point Average. NY: CEEB, RR 2008-5. (8)
OLS regressions, composite scores SAT Regents HSGPA 0.42*** 0.39*** SAT Total 0.19*** Regents Total 0.19*** R 2 0.28 0.27 (9)
OLS regressions, subject scores SAT Regents HSGPA 0.42*** 0.39*** SAT Math 0.10*** SAT Critical Reading 0.12*** Regents Math 0.11*** Regents ELA 0.13*** R 2 0.28 0.28 (10)
Mean FGPA by Mean HSGPA (11)
Study 2: predictive relationships withinand between schools Issue: prediction among students within high schools may differ from predictions between schools Hypothesis: because of between-high-school differences in grading standards: Predictive power of HSGPA will be weaker between schools than within Predictive power of test scores will be comparable or greater between schools than within (12)
Modeling approach Two-level random-coefficients, fixed-slopes models, school-mean-centered, with aggregates entered as predictors at the school level Student level: estimates within-school relationships, pooled across high schools School level: estimates relationships between school means on predictors and school mean FGPA (13)
Example of two-level model YY iiii = ββ 0jj + ββ 10 GG iiii + ββ 20 SS iiii + εε iiii ββ 0jj = γγ 00 + γγ 01 GG jj + γγ 02 SS jj + uu jj Let : G = grades S = scores i index individuals j index schools (14)
Results: single composite variables Included only one of the three predictors: SAT composite, Regents composite, or HSGPA All three variables showed stronger prediction between schools than within That is, a 1-unit difference between two students in one school had a smaller effect on FGPA than a 1-unit difference in school means (15)
Two-level regressions, single predictor HSGPA SAT Regents Student-Level HS GPA 0.39*** SAT Total 0.27*** Regents Total 0.32*** School-Level Average HS GPA 0.49*** Average SAT Total 0.36*** Average Regents Total 0.39*** (16)
Results: HSGPA and composite scores together Similar results for SAT and Regents As predicted, HSGPA predicts less strongly between schools than within Composite scores predict much more strongly between schools than within (17)
Two-level regressions, composite scores SAT Regents Student-Level HS GPA 0.37*** 0.35*** SAT Total 0.05*** Regents Total 0.09*** School-Level Average HS GPA 0.24*** 0.20*** Average SAT Total 0.27*** Average Regents Total 0.30*** (18)
Two-level regressions, subject scores Again, similar results for both tests Again, HSGPA predicts less well between schools than within Pattern shown by composite scores is more a result of math: Verbal scores predict similarly or better between schools than within Math scores predict only between schools (19)
Two-level regressions, subject scores SAT Regents Student-Level HS GPA 0.38*** 0.36*** SAT Math -0.01 SAT Verbal 0.07*** Regents Math 0.00 Regents English 0.09*** School-Level Average HS GPA 0.24*** 0.19*** Average SAT Math 0.18*** Average SAT Verbal 0.12*** Average Regents Math 0.25*** Average Regents English 0.09** (20)
Study 3 Results of the first two studies show that different prediction models will rank students and schools differently but don t identify winners and losers Study 3 will examine which types of students win and lose with different prediction approaches (21)
Implications Understanding the predictive value of scores and grades requires contrasting prediction within- and between schools Need to distinguish two questions: Aggregate strength of prediction Who benefits and loses from different sets of predictors Value of scores may be less the improvement in aggregate prediction than leveling the playing field between high schools (22)
Future directions Examine who wins and loses (Study 3) Results of studies 1 and 2 show that the choice of predictors should matter Replicate for new Common Core tests. May differ because of: Different content and difficulty Initially, less opportunity for score inflation Improve analytical methods to address problematic distributions of key variables (23)
Supplementary slides 24
Campus-specific results Baruch Brooklyn City Hunter John Jay Queens York Medgar Staten NYCCT Evers Island HSGPA 0.490*** 0.258*** 0.371*** 0.454*** 0.265*** 0.328*** 0.337*** 0.411*** 0.312*** 0.395*** SAT Math -0.012-0.021 0.017 0.028 0.077* 0.036 0.075 0.1 0.090** -0.02 SAT CR 0.124*** 0.117** 0.06 0.079** 0.113*** 0.102* 0.039 0.011 0.091*** 0.128*** Regents Math 0.127*** 0.037-0.037 0.058 0.027-0.001-0.002-0.025 0.089** 0.137*** Regents ELA -0.002 0.096* 0.126*** 0.108*** 0.097** 0.097* 0.111** 0.067 0.027 0.055 R 2 0.31 0.14 0.18 0.28 0.15 0.19 0.15 0.21 0.19 0.28 N 817 704 953 1,128 1,243 766 878 514 1,749 1,315 (25)
Example of problematic distribution: FGPA by HSGPA (26)
Student-level correlations among predictors HSGPA SAT Regents HSGPA 1 SAT total 0.43 1 Regents total 0.56 0.74 1 (27)