A Decision Tree Analysis of the Transfer Student Emma Gunu, MS Research Analyst Robert M Roe, PhD Executive Director of Institutional Research and

Transcription

1 A Decision Tree Analysis of the Transfer Student Emma Gunu, MS Research Analyst Robert M Roe, PhD Executive Director of Institutional Research and Planning

2 Overview Motivation for Analyses Analyses and Results Descriptive Logistic Regression Decision Tree Analysis 1 st Year Retention 6 year Graduation Rate Impact of 2year Degree on Performance at CMU Conclusion 2

3 Motivation CMU typically enrolls between 1400 and 1500 transfer students each year Constitutes nearly 25% of all new students In the past used all new transfers as cohort ignoring number of transfer credits Given the limited resources of the Office of Student Success, can we identify specific groups for outreach (intervention)? 3

4 Entry Credentials by Class 60% 50% 3-year ( ) mean %count by Class of Entry 80 Mean Transfer hours by Class of Entry ( ) 40% 30% 20% 10% 0% 21.2% 50.3% Freshman(<26hrs) Sophomore (27-56hrs) 25.9% Junior(57-86hrs) Freshman Sophomore Junior 4.00 Mean Transfer GPA Fresh Soph Junior 4

5 First Year Performance First Term GPA by Entry Level Fresh Soph Junior 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% % Persisting into 2nd Year by Class of Entry 70.5% 77.5% 84.3% Fresh Soph Junior 5

6 LOESS of Persistence by Transfer Hours

7 Graduation by Level of Entry Last 3-year mean % Graduating in Year by Class of Entry 90% 80% 70% 60% 50% 40% Freshman Sophomore Junior 30% 20% 10% 0% 1st Year 2nd year 3rd Year 4th Year 5th Year 6th Year 7

8 Impact of 2 Year Degree At CC Next we looked at the impact of obtaining a 2 year degree on persistence, graduation, and GPA 8

9 Impact of 2 year Degree 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 85.7% 74.6% 67.2% 43.8% 75.8% 58.8% Pers2 Grad4 Grad6 Degree No Degree 9

10 Graduation in 4yrs No Associate Degree Associate Degree Graduated within 4 Years No Yes 1006 (56.2%) 190 (32.8%) 784 (43.8%) 389 (67.2%) χ 2 = , p <.001 However, persistence and graduation are confounded with number of transfer hours. 10

11 Logistic Regression- Graduation in 4yrs B S.E. df Sig. Exp(B) TRANHRS Prior_2yr_Degree Constant a. Variable(s) entered on step 1: TRANHRS, Prior_2yr_Degree. 11

12 Results The impact of a 2 year degree on persistence to second year and graduation in 4 or 6 years was significant beyond the impact of number of transfer hours 12

13 Best Way to Classify Students Who are at Risk? Performance varies by transfer hours but what about other factors? Could use logistic regression to determine which factors are predictive However, this is not useful in determining which groups of students are at risk 13

14 Factors That Likely Impact Persistence Variables Full or part time student Transfer GPA Total transfer hours Transfer College Year Low Income Status First Generation Status 14

15 Logistic - Stepwise Variables in the Equation B S.E. Wald df Sig. Exp(B) Step 1 a TRANGPA Constant Step 2 b TRANGPA TRANHRS Constant Step 3 c FULLPART TRANGPA TRANHRS Constant Step 4 d LOWINC FULLPART TRANGPA TRANHRS Constant a. Variable(s) entered on step 1: TRANGPA. b. Variable(s) entered on step 2: TRANHRS. c. Variable(s) entered on step 3: FULLPART. d. Variable(s) entered on step 4: LOWINC. 15

16 Decision Trees for Outreach 16

17 Decision Tree Models Several types of decision tree models Here we chose the Chi-square Automatic Interaction Detection (CHAID) Model over Classification and Regression Trees (CRT) With CRT, GPA might split several times to be refined enough to be predictive 17

18 Decision Tree Models These models can be used simply to classify a set of data (e.g. what is the best way to classify our transfer students in terms of retention factors) Or can be used for prediction (e.g. can we flag new transfer students who are at risk (or not at risk) for persistence?) 18

19 CHAID The procedure creates tree-based models that determine how variables best combine to explain the outcome in a given dependent variable Dependent variable is binary response Retained vs Not Predictor variables are any combination of variable types (continuous or categorical) 19

20 Method Start by selecting a subset of data for training Use model to predict a new set of data Here we chose a subset of 70% of the data and fit to the remaining 30% Check misclassification rate and standard error for predictability 20

21 CHAID Analysis For First Year Retention Start by classifying to determine if even possible If yes, build prediction model Input variables Full or part time student Transfer GPA Total transfer hours Transfer College Year Low Income Status First Generation Status All were Selected by Model 21

22 Persist to 2 nd Year <=2.410 ( ] ( ] ( ] >

23 23

24 24

25 Which Nodes are Important? Gains for Nodes Training Node Index % % % % % % % % % % % % % % % % % % % % % Node 5: Transfer GPA > % Persist to 2 nd year Node 6: Transfer GPA <= 2.41 Transfer Hrs <=23, 54.8% Persist to 2 nd year 25

26 Model Evaluation Classification Predicted Percent Sample Not Persist2 Persist2 Correct Training Not Persist % Persist % Overall Percentage 0.0% 100.0% 78.3% Test Not Persist % Persist % Overall Percentage 0.0% 100.0% 77.3% Risk Sample Estimate Std. Error Training Test Growing Method: CHAID Dependent Variable: Persisted to 2nd Year 26

27 Summary of Persistence Decision Tree Transfer GPA is most predictive Other predictive factors vary by transfer GPA For clarity and ease of understanding for those who use the information, we created the following tables: 27

28 Retention at Risk Table Transfer GPA Transfer Credits > % 10.5% Low income in this range at higher risk. 3.3% 17.4% 18.3% Low income in this range at higher risk. 4.5% 1.5% 5.0% First Generation in this range at higher risk. 3.5%

29 Decision Tree Analysis For 6 Year Input variables Graduation Rate Full or part time student Transfer GPA Total transfer hours Transfer College Year Low Income Status, First Generation Status All variables except full vs part time were selected by model 29

30 6 Year Graduation <= <=

31 Summary of 6 Year Graduation Decision Tree For low transfer GPA (<=2.3), 1 st generation status at greater risk For second lowest GPA ( ], transfer credit hours and 2 vs 4 year important For middle GPA ( ], 2 vs 4 year and transfer credit hours important For highest GPA (>3.29) transfer hours, 2 vs 4 year, and low income important 31

32 Transfer Credits > Grad at Risk Table Transfer GPA % CC in this range at higher risk. 7.9% CC transfer in this range at higher risk. 10.0% First Generation in this range at higher risk. 32

33 Summary The more refined assessment of transfer students revealed some interesting findings Using decision tree analyses on transfer data, we are able to identify very specific groups at risk (and groups likely to succeed) We have presented these results to the retention subcommittee, strategic enrollment group, and to the vice president of enrollment services Questions? 33