Test Optional or Not Test Optional, That s a Research Question? Steve Kappler Vice President ACT 1
Test Optional Motives Outline Added Value of Test Scores Benefits of Statewide Testing Holistic Model of Student Success Subgroup Differences and Test Bias 2
TEST OPTIONAL MOTIVES SECTION I 3
Test Optional Motives The stated goal of most test optional institutions is to increase the diversity of the admitted class. Research suggests: That the diversity of the student body among testoptional institutions, pre and post adoption is largely unaffected (Belasco, Rosinger, & Hearn, 2014). Indicates that institutions that have implemented testoptional policies have benefited from an increase in perceived selectivity and status. 4
Uneducated Guesses: On the Value of Entrance Exams Students from the Bowdin class of 1999, who did not submit SAT scores performed about 120 points lower than those students who did submit them. Moreover, their FYGPA was substantially lower, as would have been predicted by their lower SAT scores (Wainer, 2011). 5
ADDED VALUE OF TEST SCORES SECTION II 6
Incremental Validity of Test Scores: Six year Cumulative GPA of 3.00 or Higher Among students with the exact same HSGPA, students with higher ACT composite scores are more likely to earn a cumulative GPA of 3.00 or higher than students with lower ACT composite scores (Radunzel & Noble, 2012). For example among students with a 3.0 HSGPA, students with an ACT composite score of 20 have roughly.40 probability of earning a cumulative GPA of 3.00 or higher as compared to a nearly.70 probability for students with an ACT composite score of 35. 7
Incremental Validity of Test Scores: Six year Bachelor s Degree Completion Rates Among students with the exact same HSGPA, students with higher ACT composite scores are more likely to earn a bachelor s degree in six years than students with lower ACT composite scores (Radunzel & Noble, 2012). For example among students with a 3.0 HSGPA, students with an ACT composite score of 20 have roughly.35 probability of earning of a bachelor s degree in six years as compared to about.40 probability for students with an ACT composite score of 30. 8
Discrepant Performance: Definition and Frequency Some students have significantly higher HSGPAs than ACT Composite scores and vice versa. Students were divided into three groups based on the difference between their standardized HSGPA and ACT Composite score Based on 2015 ACT tested graduating cohort 13% had significantly higher ACT composite scores 74% had non discrepant performance 13% had significantly higher HSGPA 9
Discrepant Performance: Student Characteristic of 2015 ACT tested Students by Discrepant Category Student Characteristics Higher ACT Non discrepant Higher HSGPA Number of Students 212,476 1,211,589 208,520 ACT Composite 24.5 21.5 16.7 Academic Preparation (mean) HSGPA 2.66 3.27 3.60 Female 37.7% 55.3% 63.5% Male 61.7% 44.2% 35.8% Gender Missing 0.6% 0.5% 0.6% No College 17.1% 18.6% 28.0% Some College 21.4% 24.8% 27.6% Bachelor's Degree 23.6% 25.8% 19.8% Graduate Degree 25.1% 19.8% 11.0% Parental Education Level Missing 12.8% 11.0% 13.6% Less than $36,000 21.0% 23.6% 35.3% $36,000 to $60,00 15.0% 16.7% 17.9% $60,000 to 100,000 16.9% 19.0% 15.1% Over $100,000 22.5% 19.7% 10.4% Income Missing 24.6% 20.9% 21.3% Racial/ Ethnic Background Black/African American 9.3% 12.5% 19.9% American Indian/Alaska Native 0.6% 0.7% 1.2% White 59.4% 58.5% 43.1% Hispanic/Latino 12.7% 14.7% 22.3% Asian 6.3% 4.4% 5.0% Native Hawaiian/Other Pacific Islander 0.3% 0.3% 0.4% Two or more races 4.7% 4.0% 3.5% Prefer not to respond 6.7% 4.9% 4.7% Females, students from lower SES backgrounds, and minority students comprise a larger percentage of the higher HSGPA group as compared to the 10 other two groups.
Discrepant Performance: Differential Validity Regression of ACT and HSGPA on FYGPA by Discrepant Group Model Higher HSGPA Non discrepant Higher ACT HSGPA 0.35 0.51 0.49 ACT 0.37 0.48 0.46 HSGPA & ACT 0.38 0.52 0.49 For the higher HSGPA group, ACT scores were more predictive of FYGPA than HSGPA. Presumably students who have significantly higher HSGPAs as compared to their ACT scores would be most likely to withhold their test scores when applying to a test optional school; however, the results indicate that their tests scores are more strongly related to FYGPA than HSGPA. For the higher ACT and non discrepant groups, HSGPA was more predictive of FYGPA than ACT scores. For all groups, ACT added to the prediction of FYGPA above and beyond HSGPA only 11
Discrepant Performance: Differential Prediction Mean Residual by Discrepant Group Model Higher HSGPA Non discrepant Higher ACT HSGPA 0.31 0.02 0.20 ACT 0.37 0.02 0.60 HSGPA & ACT 0.05 0.01 0.05 For each regression model, a student s predicted FYGPA was compared to their observed FYGPA. FYGPA predicted > FYGPA earned over prediction (negative residual values) FYGPA predicted < FYGPA earned under prediction (positive residual values) All three models accurately predicted FYGPA for the non discrepant group. For the Higher HSGPA group, the HSGPA only model over predicted their FYGPA while the ACT model under predicted their FYGPA; the HSGPA & ACT model most accurately predicted their performance. If test optional institutions rely solely on HSGPA for the higher HSGPA group, this group will not perform as well on campus as expected. For the Higher ACT group, the HSGPA only model under predicted their FYGPA while the ACT model over predicted their FYGPA; the HSGPA & ACT model most accurately predicted their performance. 12
BENEFITS OF STATEWIDE TESTING SECTION III 13
Benefits of Statewide Testing: Increased Access to Underserved Students The states that have adopted the ACT have observed decreased gaps in ACT participation rates across sociodemographic subgroups (Allen, 2015). After ACT adoption, greater representation among the ACTtested cohorts is observed for male, African American, American Indian, and Hispanic students. Students with lower family income and parents who did not attend college are also more likely to be represented after statewide adoption. 14
Benefits of Statewide Testing: Increased Enrollment in Higher Education 75 70 Pre adoption Post adoption 65 60 55 50 IL Nation 45 40 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014
HOLISTIC MODEL OF STUDENT SUCCESS 16
Holistic Model of Education and Work Success Core Academic Skills Cross Cutting Capabilities Education and Work Success Behavioral Skills Education & Career Navigation See Mattern, Burrus, Camara et al. (2014) for more information.
Broad Domains of the Holistic Framework Core Academic Skills: Domain specific knowledge and skills that are necessary to perform essential tasks in the core academic content areas of English language arts, mathematics, and science. Cross Cutting Capabilities: General knowledge and skills that are necessary to perform essential tasks across content areas. Behavioral Skills: Interpersonal, self regulatory, and taskrelated behaviors that are important for adaptation to and successful performance in educational and workplace settings. Education & Career Navigation: Personal characteristics, processes, and knowledge that influence individuals as they progress through their educational and career paths. 18
Core Academic Skills English Language Arts Mathematics Science Cross Cutting Capabilities Technology and Information Literacy Collaborative Problem Solving Studying and Learning Thinking and Metacognition Behavioral Skills Acting Honestly Maintaining Composure Socializing with Others Getting Along Well with Others Sustaining Effort Keeping an Open Mind Education & Career Navigation Self Knowledge Environmental Factors Integration Managing Career and Education Actions 19
Holistic Model of Student Success: Timely Degree Attainment Navigation Behavior Core Academic Environmental 20
Holistic Model of Student Success: The Role of Behaviors on College Completion Percent 50 46 40 35 32 30 25 25 20 19 15 18 12 10 0 0 Benchmarks 1 or 2 Benchmarks 3 or 4 Benchmarks Number of ACT Benchmarks Met Low ACT Engage College Scores Moderate ACT Engage College Scores High ACT Engage College Scores Students with higher ACT Engage College scores (based on the mean percentile scores of the ACT Engage College scales Academic Discipline, Commitment to College, and Social Connection) attained a postsecondary degree within four years of entering college at higher rates than students with lower ACT Engage College scores (Moore, Casillas, & Way, 2015). These differences were larger for students at higher achievement levels. For students meeting three or four Benchmarks, those with high ACT Engage College scores attained a timely postsecondary degree at nearly twice the rate as those with low ACT Engage College scores (46% vs. 25%). 21
Holistic Model of Student Success: The Role of Interest Major Fit on Timely Degree Completion Similar to the major persistence findings, interest major fit provides added value in predicting timely degree attainment. Specifically, students with a good interestmajor fit were more likely to graduate in four years than students with poor fit, even after taking into account ACT performance.
SUBGROUPS DIFFERENCES & TEST BIAS SECTION V 23
Achievement Gaps: A problem with the test or a larger issue? African American students perform lower than the total group across a variety of educational indicators (Kobrin, Sathy, & Shaw, 2007). 24
Achievement Gaps: A problem with the test or a larger issue? Hispanic students perform lower than the total group across a variety of educational indicators (Kobrin, Sathy, & Shaw, 2007). 25
Achievement Gaps: A problem with the test or a larger issue? Low income students perform lower than the total group across a variety of educational indicators (Kobrin, Sathy, & Shaw, 2007). 26
Achievement Gaps: A problem with the test or a larger issue? African American and Hispanic students have lower degree attainment rates as compared to White or Asian students. 27
Achievement Gap: Is ACT simply a measure of a student s wealth? ACT and HSGPA are much stronger predictors of first year GPA in college and retention as compared to SES. In fact, the correlations were twice as large for ACT and HSGPA (.51 and.58) as compared to SES (.24) for predicting first year GPA (Westrick, Le, Robbins, Radunzel & Schmidt, 2015). 28
Achievement Gap: Is ACT simply a measure of a student s wealth? Based on the data from the Westrick et al. study (2015), path models were run to examine the relative contributions of pre college predictors on firstyear cumulative GPA and second year retention. Even after controlling for SES, both ACT and HSGPA remain strong predictors of first year GPA. The effects of ACT and HSGPA on college retention are indirect through their influence on FYGPA. 29
Achievement Gaps: What factors are related to ACT performance Socioeconomic status and other demographic characteristics (including parent s education level, race/ethnicity, and gender) accounted for a small percentage of the variance in ACT scores (4% or below), after high school coursework and grades, school characteristics, and other noncognitive student characteristics were taken into account (McNeish, Radunzel, and Sanchez, 2015). 30
Differential Prediction If the ACT is an underestimate of underserved students potential, then they should perform better in college than what their scores would predict. Research has consistently found that test scores over predict underserved students college performance. That is, underserved students do worse in college than what their ACT scores predict. 31
Differential Prediction: Over prediction of Underserved Students Probability of Earning a FYGPA Among students with the same ACT composite score, African American and Hispanic students are less likely to earn a FYGPA of 2.5 or higher (and 3.0 or higher) than White students (Sanchez, 2013). 32
Differential Prediction: Over prediction of Underserved Students Probability of Earning a bachelor s degree Among students with the same ACT composite score, minority students are less likely to earn a bachelor s degree within six years than White students (Radunzel & Noble, 2013). 33
Differential Prediction: Over prediction of Underserved Students Probability of Earning a bachelor s degree Among students with the same ACT composite score, low income students are less likely to earn a bachelor s degree within six years than high income students (Radunzel & Noble, 2013). 34
Three Final Thoughts 1. Multiple Dimensions of College & Career Readiness is the real answer 2. Intent and results from Test Optional Movement 3. Is Test Optional really a movement? 35
3 Is Test Optional really a movement? Selective colleges means nonopen enrollment institutions What is the impact to students of not testing? IPEDS Data