Dell student retention model for Harper College. White paper

Transcription

1 Dell student retention model for Harper College White paper

2 TABLE OF CONTENTS Abstract... 4 Introduction... 5 Literature review... 5 High Level 2008 Harper College cohort review... 6 Ethnicity of the 2008 Harper College cohort... 6 Employment status of the 2008 Harper College cohort... 6 Enrollment reason of the 2008 Harper College cohort... 7 First generation and first time in college of the 2008 Harper College cohort... 7 Remedial student population of the 2008 Harper College cohort... 8 Completion of 15 hours, 30 hours and credential of the 2008 Harper College cohort... 9 Completion of 15 hours, 30 hours and credential completion with comparison of high school GPA of the 2008 Harper College cohort... 9 Completion of 15 hours, 30 hours and credential completion with comparison of tutoring visits by semester of the 2008 Harper College cohort Dell student retention model for the 2008 Harper College cohort Student retention results of 2008 Harper Cohort 15 hour credits in one year Student retention results of 2008 cohort 30 hour credits in two years Student retention results of 2008 cohort credential Back testing of the Dell student retention model Back testing the hour risk score on the 2006 and 2007 cohort Back testing the hour risk score on the 2006 and 2007 cohort cohort graphs cohort graphs Tracking student GPA through matriculation Heat maps Zip code by English exam scores Zip code by math exam scores Conclusions hour success score boundaries hour success score boundaries Degree completion success score boundaries Next steps Longitudinal model of student matriculation Literature review Zip code analysis by high school Correlation between compass scores and college success scores Student Retention Model for Harper College Dell Inc., 2014

3 TABLE OF TABLES Table 1- Ethnicity of the 2008 Harper Cohort...6 Table 2- Employment status of the 2008 Harper College cohort...6 Table 3- Enrollment reason of the 2008 Harper College cohort...7 Table 4- First generation and first time in college tables of the 2008 Harper College cohort...7 Table 5- Remedial math student population of the 2008 Harper College cohort...8 Table 6- Remedial English student population of the 2008 Harper College cohort...8 Table 7- Remedial reading student population of the 2008 Harper College cohort...8 Table 8- Certification awards for the 2008 Harper College cohort...9 Table 9- Completion of 15 hours, by enrollment reason and high school GPA...9 Table 10- Completion of 30 hours, by enrollment reason and high school GPA...10 Table 11- Credential completion, by enrollment reason and high school GPA...10 Table 12- Completion of 15 hours, by employment status and average tutoring center visits per semester...11 Table 13- Completion of 30 hours, by employment status and average tutoring center visits per semester...11 Table 14- Credential completion, by employment status and average tutoring center visits per semester...11 Table 15- Comparison between predicted risk score and completion percentage...12 Table 16- Overall results of risk model- 15 hour credits in one year...13 Table 17- English placement scores compared to predicted risk scores, actual completion of 15 hours, and the model coefficient...14 Table 18- Comparing 30 hour risk score and completion percentage...14 Table 19- Overall results of risk model- 30 hour credits in two years...15 Table 20- Math exam placement scores risk scores at 30 hours compared to actuals...16 Table 21- Comparing credential risk score and completion percentage...16 Table 22- Students who completed 15 hours in one year vs. students who completed certification...17 Table 23- Overall results of credential risk score for students who did not complete 15 hours in one year...18 Table 24- Comparing credential risk score and completion percentage for students who did not complete 15 hours...18 Table 25- Overall results of credential risk score for students who did 15 hours in one year...19 Table 26- Comparing credential risk score and completion percentage for students who did complete 15 hours...19 Table 27- Comparing credential risk score and completion percentage for students by enrollment reason for students who did complete 15 hours...20 Table 28- Back testing of the 2008 cohort on the 2006 and 2007 model...21 Table 29- Academic preparedness and cohort changes by HS GPA 2006, 2007, Table 30- Students who tested at least math one level below cohort and cohort changes by HS GPA 2006, 2007, Table 31- Students who tested at least English one level below cohort and cohort changes by HS GPA 2006, 2007, Table 32- Student enrollment was between the cohorts...23 Table 33- Predicted success score vs. actual success score 2008 model on cohort...24 Table 34- High school GPA Table 35- High school GPA Table 36- High school GPA Table 37- Rates of 15 hour success score by zip code...32 Table hour credit comparison by close geographical comparison...33 Table 39- Rates of 30 hour success score by zip code...34 Table 40- Rates English exam scores by zip code...35 Table 41- Rates math exam scores by zip code...36 Table 42- Population statistics for 15 hour success score and completion...37 Table hour success score population graph...38 Table 44- Population statistics for 30 hour success score and completion...39 Table hour success score population graph...39 Table 46- Population statistics for credential success score and completion...40 Table 47- Credential success score population graph...41 Table 48- Credential success score population graph Student Retention Model for Harper College Dell Inc., 2014

4 Abstract Within the foundational academic work, the Dell student retention model seeks to help colleges and universities retain students by quickly identifying students at risk through a structured information technology solution, and delivering the information to administrators, instructors, counselors and students in an easy-to-access model. We developed a model that looks at AP Hours, high school GPA, Harper math placement score, Harper English placement score, enrollment reason and employment status, developing scores for 15 hours in one year, 30 hours in two years, and the probability of receiving credentials. We achieved outstanding results from the modeling activities. In our sample, we looked at almost 3000 students entering Harper College; they had a 37% chance of reaching 15 hours, 31% of reaching 30 hours, 20% chance of completing a credential program. From our modeling, we found that students enter Harper with a high school GPA of 2.56, and average one tutoring session in the first year. If students would visit the tutoring center just one more time a semester, Harper would see a 3.4% (2.4% to 5.6% 95% CI) increase in students hitting 15 hours in one year. Similarly, students would have a 3.7% (2.6% to 4.8% 95% CI) increased chance of hitting 30 hours in two years. Harper would also see a 4.7% (3.1% to 7.1% 95% CI) increase in credentials. A 3.4% increase in 15 hour completion, combined with a 3.7% increase in 30 hour completion, and a 4.7% in certification is about 357 extra students over 2 years that otherwise would have not completed the target. We figure an average of 26 hours a student (11 fall, 10 spring, 4 summer) a semester. This equates to about 9,282 hours, with a range between 6,355 and 13,740, at a 95% confidence interval. We also noticed that increase in the Harper first year GPA impacted completion of the 30 hour mark. On average, the Harper Cumulative first year GPA was 2.12 GPA. A ¼ point change in first year GPA brought about an increase of 1.2% (1.0% to 1.4% 95% CI) in hitting the 30 hour mark. A ½ point increase in Harper average first year GPA corresponds to a 2.2% (1.0% to 1.5% 95% CI) increase in hitting the 30 hour mark. A 2.2% in 30 hour completion is about 66 extra students over the final year that otherwise would have failed to complete the 30 hour mark. The lift associated with the GPA effect includes consideration that students at Harper College average 26 hours a student (11 fall, 10 spring, 4 summer) a semester. This equates to about 1,716 hours, with a range between 1,370 and 2,016, at a 95% Confidence Interval. We imply from these results that the breadth of tutoring content and hours spent when a student visits a tutoring center will greatly improve student success at Harper. We also suspect from the data that there are interactions between the variables in the model. For example, a low GPA might motivate students to visit the tutoring center and/or visiting the tutoring center may be driving a student s higher GPA. 4 Student Retention Model for Harper College Dell Inc., 2014

5 Introduction The ultimate goal for Harper College is to increase student retention and maximize the number of students who complete the 30 hour credential program within an open-admissions environment where students are not entering the program with adequate preparation. To this end, Harper seeks to: Identify high-risk students Determine how visits to the tutoring center may impact GPA Utilize findings to implement programs that increase completion rates Dell Business Intelligence and Analytics Service entered into a proof of concept with Harper College to see if Dell could pull disparate data sources together, provide a reporting and analytic framework for the Achieve the Dream students for the 2008 cohort, and produce insightful and actionable insights that would help Harper achieve its retention goals. We received data for 2907 Students for the 2008 cohort, 2946 students for the 2007 cohort and 2846 students for the 2006 cohort. Dell loaded the data for all the students, including student demographics, activities, grades and courses. From this effort, we produced several analytical models using R as the analytic platform and the Microsoft Software Stack, including SQL Server and SharePoint reporting services. Literature review We have based the Dell Student Retention Model on the academic work of Vincent Tinto. In the seminal work by Tinto in 1975, he introduced the interactionalist model for student retention. In this model, he expanded upon the work of Feldman and Newcomb and was further expanded upon by Alexander Astin and William Spady in the late 1970s and 80s. This spurred an increased academic awareness and structure to student retention, especially at the universities. In these studies, educators were made increasingly aware of the impact of demographic information (race, sex, marital status), along with academic preparedness (high school GPA, standardized tests) and student involvement in campus life. However, most of the academic research has looked at the matriculation of students at four-year universities, with little work being done at both community and junior colleges, as reported by Gloria Crisp and Liliana Mina. In their literature review, they acknowledge the huge growth in community colleges in the US since World War II. The environment that a community college occupies has many challenges, mostly due to diversity of the student population: Serving students who seek a 2 year pre-university degree Pre-professional degree-seeking students Dual enrollment of high school students GED-seeking students, ESL Students Along with the diversity of academically prepared students, community colleges also have to maintain an open-door admissions policy. The central problem in today s community college is how to maintain the rigor of post-secondary education, while providing access to students who may not be academically prepared, all in the era of decreased funding and increased government accountability iv. 5 Student Retention Model for Harper College Dell Inc., 2014

6 High level 2008 Harper College cohort review In the student records as provided by Harper College, we have 2907 students. The gender split was 1497 female students (51.5%), 1409 male students (48.5%), with 1 not reporting. The ethnicity of the 2008 cohort was: Ethnicity Number of Students Percentage of Population American Indian/Alaskan Native % Asian % Black/African American % Pacific Islander 2 0.1% White % Hispanic % Non-resident alien % Other % Unknown % % Table 1 - Ethnicity of the 2008 Harper cohort Given the school s location, the school s demographics are in-between Chicago demographics and Illinois statewide demographics, with the interesting higher percentage of Asian students than either Chicago or Illinois demographics, and lower percentages for Black/ African American students at Harper. Employment status of the 2008 Harper College cohort With employment status, the largest percentage of students work between 15 and less than 32 hours (28.8%), while the unemployed account for 26.4%, and full time students are at 20.1%. Employment Status Number of Students Percentage of Population Employed full- time % Homemaker % Employed part- time 15 hours or less per week % Employed part- time over 15 hours per week % Unemployed % Other % % Table 2 - Employment status of the 2008 Harper College cohort 6 Student Retention Model for Harper College Dell Inc., 2014

7 Enrollment reason of the 2008 Harper College cohort Over half the students at Harper are planning to transfer to a four-year college or university (51.3%), while job preparation (19.8%) and exploring courses (17.0%) make up the remaining student population goals. Enrollment Reason Number of Students Percentage of Population Prepare for a job immediately after attending Harper College % Improve skills needed in present job % Explore courses to decide on a career % Coursework to transfer to a four-year college or university % Prepare for GED test or improve basic academic skills (includes ESL) % Personal interest-self-development- Non career oriented % Already accepted a four-year college or university % Unknown 7 0.2% % Table 3- Enrollment reason of the 2008 Harper College cohort First generation and first time in college of the 2008 Harper College cohort Most of the students at Harper are both first generation students and first time in college (39.6%), followed closely by first-time college students (25.8%). The remaining students, who have previously attended college, make up 30.8% of the 2008 cohort. First Time in College First Generation Students Yes No Unknown Yes No Table 4- First generation and first time in college tables of the 2008 Harper College cohort 7 Student Retention Model for Harper College Dell Inc., 2014

8 Remedial student population of the 2008 Harper College cohort Within the 2008 Harper cohort, 88.2% of all students have tested for one of the following remedial classes - math, English or reading. The following tables show the descriptive breakout of all 2907 students in the Cohort. Remedial Math Number of Students Percentage of Population Does Not Apply % Missing % No % Tested 1 level below college level % Tested 2 levels below college level % Tested 3 or more levels below college level % % Table 5- Remedial math student population of the 2008 Harper College cohort We notice that 39.9% of all students are in need of remedial math, testing at least 1 level below college level. Remedial English Number of Students Percentage of Population Does Not Apply % Missing % No % Tested 1 level below college level % Tested 2 levels below college level % All others % % Table 6- Remedial English student population of the 2008 Harper College cohort We notice that 26.5% of all students are in need of remedial English, testing at least 1 level below college level. Remedial Reading Number of Students Percentage of Population Does Not Apply % Missing % No % Tested 1 level below college level % Tested 2 levels below college level % % Table 7- Remedial reading student population of the 2008 Harper College cohort We notice that 21.1% of all students are in need of remedial English, testing at least 1 level below college level. We also notice that there is significant collinearity between remedial English and remedial reading scores. 8 Student Retention Model for Harper College Dell Inc., 2014

9 Completion of 15 hours, 30 hours and credential of the 2008 Harper College cohort From the dataset, we also have 1076 (37.0%) of students completing 15 hours in the first year, 909 (31.3%) students completing 30 hours in the first two years, and 570 (19.6%) students completing a credential program. Percentage of Population Credential Awards Number of Students Completes Credential Associate in Arts % Associate in Applied Science % Associate in Fine Arts 1 0.2% Associate in General Studies 2 0.4% Associate in Science % Credential % Credential 30 hours or more % Credential less than 30 hours % % Table 8- Certification awards for the 2008 Harper College cohort The majority of Harper Students (56.0%) who complete a program at Harper receive an Associate s degree, with the next most popular receiving a credential of less than 30 hours. Completion of 15 hours, 30 hours and credential completion with comparison of high school GPA of the 2008 Harper College cohort When we look at students who were successful in reaching 15 hours in the first semester, and had a high school GPA reported >0 (1829 Students), and corrected the 5 point scale to a 4 point scale, here are the results by enrollment type: High School GPA for 15 Hours in First Year 15 Hours in First Year Enrollment Reason No Yes No Yes Prepare for a job Improve skills needed in present job Explore courses to decide on a career Coursework to transfer to a four-year college Prepare for GED test / improve basic academic skills Personal interest-self-development Already accepted at a four-year college or university Total 1829 Average GPA 2.82 Table 9- Completion of 15 hours, by enrollment reason and high school GPA 9 Student Retention Model for Harper College Dell Inc., 2014

10 This is the first of the interesting results in that there appears to be a differentiation between students who are successful in completing 15 hours in the first year and those who are not; the successful students have a grade point average in high school of 3.06, while the students who appear to struggle have a grade point average of 2.62, adjusting for a 5 point scale when necessary. High School GPA for 30 Hours in First Two Years 30 Hours in First Two Years Enrollment Reason No Yes No Yes Prepare for a job Improve skills needed in present job Explore courses to decide on a career Coursework to transfer to a four-year college Prepare for GED test / improve basic academic skills Personal interest-self-development Already accepted a four-year college or university Total 1829 Average GPA 2.82 Table 10- Completion of 30 hours, by enrollment reason and high school GPA We see the same trend in completing 30 hours in the first two years. There appear to be differentiations between students who are successful in completing 30 hours in the first two years and those who are not; the successful students have a grade point average in high school of 3.09, while the students who appear to struggle have a grade point average of 2.65, adjusting for a 5 point scale when necessary. High School GPA for Degree in First Year Credential Seeking Students Enrollment Reason No Yes No Yes Prepare for a job immediately after attending Harper College Improve skills needed in present job Explore courses to decide on a career Coursework to transfer to a four-year college or university Prepare for GED test or improve basic academic skills Personal interest-self-development Already accepted a four-year college or university Total 1829 Average GPA 2.82 Table 11- Credential completion, by enrollment reason and high school GPA We see the same trend in credential completion. There appears to be a differentiation between students who are successful in certification hours and those who are not; the successful students have a grade point average in high school of 3.12, while the students who appear to struggle have a grade point average of 2.74, adjusting for a 5 point scale when necessary. 10 Student Retention Model for Harper College Dell Inc., 2014

11 Completion of 15 hours, 30 hours and credential completion with comparison of tutoring visits by semester of the 2008 Harper College cohort Another interesting point is that the tutoring visits per semester had a positive effect on the outcome, when looking at the cross-tabs of employment status, by completion and tutoring visits in the time period under consideration: 15 Hours Tutoring Visits for 15 Hours in the First Year in the First Year Employment Status No Yes No Yes Employed full-time Homemaker Employed part-time 15 hours or less per week Employed part-time over 15 hours per week Unemployed Other Total 2907 Average Tutoring.089 Table 12- Completion of 15 hours, by employment status and average tutoring center visits per semester 30 Hours Tutoring Visits for 30 Hours in the First Two Years in the First Two Years Enrollment Status No Yes No Yes Employed full-time Homemaker Employed part-time 15 hours or less per week Employed part-time over 15 hours per week Unemployed Other Total 2907 Average Tutoring.089 Table 13- Completion of 30 hours, by employment status and average tutoring center visits per semester Credential Tutoring Visits for Completion Credential Completion Enrollment Status No Yes No Yes Employed full-time Homemaker Employed part-time 15 hours or less per week Employed part-time over 15 hours per week Unemployed Other Total 2907 Average Tutoring.089 Table 14- Credential completion, by employment status and average tutoring center visits per semester 11 Student Retention Model for Harper College Dell Inc., 2014

12 Dell student retention model for the 2008 Harper College cohort Within the foundational academic work provided by Tinto and others, the Dell student retention model seeks to help colleges and universities retain students, by quickly identifying students at risk, through a structured information technology solution, delivering the information to administrators, instructors, counselors and students in an easy to access model. Based upon our data assessment and sample size, we used the same methodologies proposed in academic papers creating a general logistic model, with a binomial transfer function, combining classification variables with continuous variables to quickly identify Harper students at risk. The overarching model has the following variables selected: AP_HOURS_FLAG- Whether or not a student took AP classes in high school Avg_Tutoring_Per_Semester - Defined as the average tutoring visits a student had registered for during the time period in question CORRECTED_HS_GPA- High school GPA, adjusted for students who have a 5 point GPA, scaled down to a 4 point GPA MATH_PLACEMENT_EXAM_SCORE- the Harper College placement scores for math ENGLISH_PLACEMENT_EXAM_SCORE- the Harper College placement scores for English. We noticed high collinearity between the English placement score and the reading placement score. We choose English, as it encompasses both the ability to write and to read. ENROLL_REASON- The enrollment reason, as reported by Harper College EMPLOYMENT_STATUS- The employment status, as reported by Harper College Overall, this is a very robust model, able to handle both the 15 hour and 30 hour probabilities, at well above a 95% level of confidence. At the credential level, we find the model breaks down at the corrected high school GPA and employment status; both will be covered in the discussion at that section. Student retention results of 2008 Harper cohort 15 hour credits in one year For the overarching model accuracy, from a table perspective, we will compare the predicted 15 hour risk score with the actual completion percentage, by the forecasted risk score: Predicted Predicted Number Number of Students Success Score Risk Score Actual of Students in Risk Score Group Table 15- Comparison between predicted risk score and completion percentage 12 Student Retention Model for Harper College Dell Inc., 2014

13 From this analysis, we expect almost a 1:1 ratio of predicted risk to actual. The results of the model are a slope of and a R2= Visually, it looks like: Hour in One Year Compared to Actual Results y = x R 2 = Actual Predicted Linear (Actual) Figure 1-15 hour in one year predicted risk score to completion percentage The overall model statistics are as follows: 15 hour Credit Df Deviance Pr(>Chi) AP_HOURS_EARNED < 2.2e-16 *** EMPLOYMENT_STATUS < 2.2e-16 *** ENGLISH_PLACEMENT_EXAM_SCORE < 2.2e-16 *** MATH_PLACEMENT_EXAM_SCORE E-12 *** ENROLL_REASON E-06 *** HIGH_SCHOOL_GPA_ E-06 *** Avg_Tutoring_Per_Semester E-15 *** Table 16- Overall results of risk model - 15 hour credits in one year Overall, the 2008 Harper cohort had a 37.01% chance of reaching 15 hours in one year. The resulting model predicted a 36.97% change of reaching 15 hours, resulting in a 0.04% difference between the model and actual results. Reporting on the 15 hour in one year risk score, we notice that the English placement score, the high school GPA and the average tutoring per semester variables are the most significant predictors of student risk. The high school GPA and average tutoring make sense: A higher high school GPA is indicative of students who are well prepared for the academic rigors of college work. Similarly, students who are engaged in their academic success will have a tendency to use all the resources available to be successful in school. The interesting results we find are how the English placement scores were the most predictive of student risk. 13 Student Retention Model for Harper College Dell Inc., 2014

14 The breakdown of the English placement scores is as follows: Predicted Success Average 15 Hours Count of Model Scores (1- Risk Score) in One Year Students Coefficient Does not apply Missing Student tested at college level Student tested 1 level below college level Student tested 2 levels below college level Student tested 3 levels below college level Total Table 17- English placement scores compared to predicted risk scores, actual Completion of 15 hours, and the Model Coefficient We notice immediately that the students who tested at college level were 55.7% of the population, but the predicted success score (1- risk score) matches the actual average of the students making 15 hours in one semester, along with matching the sign and strength of the model coefficient. An interesting result is for students who tested below 3 grade levels for the English placement exam. On average, the students were on par to hit 15 hours, slightly below the students who tested one grade level below. The outlier is that the cohort of students who tested 2 grade levels below were the lowest performing students, sandwiched between the second and third best performing group. Student retention results of 2008 cohort 30 hour credits in two years For the overarching model accuracy, from a table perspective, we will compare the predicted 30 hour risk score with the actual completion percentage, by the forecasted risk score: Predicted Predicted Number Number of Students Success Score Risk Score Actual of Students in Risk Score Group Table 18- Comparing 30 hour risk score and completion percentage 14 Student Retention Model for Harper College Dell Inc., 2014

15 From this analysis, we expect almost a 1:1 ratio of predicted risk to actual. The results of the model are a.99:1 and a R2= Visually, it looks like: Hour in Two Years Compared to Actual Results R 2 = Actual Predicted Linear (Actual) Figure 2-30 hour in one year predicted risk score to completion percentage When looking at the 30 credit hour in two year model, we notice that we have similar results to the 15 credit hour in one year model. The significant change is that the math placement score now becomes more relevant to the high school GPA, and is now in the third predictive power for success/risk scores. Df Deviance Pr(>Chi) AP_HOURS_EARNED < 2.2e-16 *** EMPLOYMENT_STATUS < 2.2e-16 *** ENGLISH_PLACEMENT_EXAM_SCORE < 2.2e-16 *** MATH_PLACEMENT_EXAM_SCORE < 2.2e-16 *** ENROLL_REASON *** HIGH_SCHOOL_GPA_ ** Avg_Tutoring_Per_Semester < 2.2e-16 *** Table 19- Overall results of risk model - 30 hour credits in two years 15 Student Retention Model for Harper College Dell Inc., 2014

16 An interesting result is that the students at Harper in math placement have no statistically significant difference in achieving hours from students with missing test scores or students that test 3 levels below college level. Overall, the 2008 Harper cohort had a 37.0% chance of reaching 30 hours in two years. The resulting model predicted a 31.30% chance of reaching 30 hours, resulting in a -5.7% difference between the model and actual results. Reporting on the 30 hour risk score, we notice that the English placement score, math placement score, and the average tutoring per semester variables are the most significant predictors of student risk. The high school GPA, as a student matriculates, will have less of an effect than in the 15 hours retention score. This topic will be covered in the section: Tracking students GPA through matriculation. Predicted Success Average 15 Hours Count of Model Scores (1- Risk Score) in One Year Students Coefficient Does not apply Missing Student tested at college level Student tested 1 level below college level Student tested 2 levels below college level Student tested 3 levels below college level Total Table 17- Math exam placement scores risk scores at 30 hours compared to actuals We notice immediately that the students who tested at college level were 53.7% of the population, but the predicted success score matches the actual average of the students making 30 hours in two years, along with matching the sign and strength of the model coefficient. An interesting result is for students who tested below 1 grade level for the math placement exam. On average, the students were on par to hit 30 hours, slightly below the students who are at grade level or below. The interesting result was that the students who had missing/ did not apply math placement scores were less likely to hit the mark of 30 hours in two years. Student retention results of 2008 cohort credential When we applied the standard Dell Student Retention Score to the certification program, with no adjustments, the model failed to produce satisfactory results. We noticed immediately that this model, which looks at the entire 2008 cohort, fails to predict degree completion. This is to be expected due to only 570 students completing a degree at Harper, which is 19.7% of the population. When we did a deeper dive into the population, we noticed certain populations of students that were credential-seeking students. The results are as follows. Predicted Actual Number Number of Students Risk Score Completion of Students in Risk Score Group Table 21- Comparing credential risk score and completion percentage 16 Student Retention Model for Harper College Dell Inc., 2014

17 Credential Compared to Actual Results Actual Results R 2 = Actual Predicted Linear (Actual) Student Population Scores Figure 3 - Overarching credential compared to actual results While the table tells the story, the graph is a more visual way to identify why the overarching model should not be applied to the credential and competition. First, we notice that as the predicted success score rises, the prediction falls apart based upon the lack of population. In our deeper dive into the data, we realized that there was a correlation between the students who completed 30 hours in two years and a credential and those that did not make the 30 hours in two years. Credential Completed 15 Hours No Yes No Yes Table 22 - Students who completed 15 hours in one year vs. students who completed certification 17 Student Retention Model for Harper College Dell Inc., 2014

18 Realizing this central population fact, we proceeded with the analysis that within the 2008 cohort, the 38.48% of the students who completed 15 hours in one year were more likely to go on to get a credential at Harper. For the students who failed to complete 15 hours, here are the model results: Df Deviance Pr(>Chi) AP_HOURS_EARNED EMPLOYMENT_STATUS * ENGLISH_PLACEMENT_EXAM_SCORE MATH_PLACEMENT_EXAM_SCORE ENROLL_REASON *** HIGH_SCHOOL_GPA_ Avg_Tutoring_Per_Semester *** Table 23 - Overall results of credential risk score for students who did not complete 15 hours in one year Notice that the driving factors within this population are the enroll reason, average tutoring per semester and employment status; the other factors are included in the model for completeness, but are not significant at the 95% level of confidence. For the students who did not completed 15 hours, here are the model results: Predicted Predicted Number of Success Score Risk Score Actual Students Table 24 - Comparing credential risk score and completion percentage for students who did not complete 15 hours 18 Student Retention Model for Harper College Dell Inc., 2014

19 2008 Cohort-Credential Target for Students Who Did Not Complete 15 Hours R 2 = Actual Results Student Score - Actual Model Prediction Linear (Student Score-Actual) Student Population Scores Figure 4 - Comparing credential risk score and completion percentage for students who did not complete 15 hours Df Deviance Pr(>Chi) AP_HOURS_EARNED EMPLOYMENT_STATUS ENGLISH_PLACEMENT_EXAM_SCORE MATH_PLACEMENT_EXAM_SCORE ENROLL_REASON ** HIGH_SCHOOL_GPA_ Avg_Tutoring_Per_Semester Table 25 - Overall results of credential risk score for students who did 15 hours in one year Notice that the driving factor within this population is only enroll reason; the other factors are included in the model for completeness, but are not significant at the 95% level of confidence. Predicted Predicted Number of Success Score Risk Score Actual Students Table 26 - Comparing credential risk score and completion percentage for students who did complete 15 hours 19 Student Retention Model for Harper College Dell Inc., 2014

20 Below is the graphical representation of the data. We notice a tight fight between the.75 and.45 risk scores, with skewing at the tail ends, due to sample size Cohort-Credential Target for Students Who Did Complete 15 Hours R 2 = Actual Results Student Score - Actual Model Prediction Linear (Student Score-Actual) Student Population Scores Figure 5 - Comparing credential risks score and completion percentage for students who did complete 15 hours Average Number of Success Degree Students who Number of Enrollment Reason Score Flag Completed Students Prepare for a job immediately after attending Harper College Improve skills needed in present job Explore courses to decide on a career Coursework to transfer to a four-year college Prepare for GED test or improve basic academic skills Personal interest-self-development Already accepted a four-year college or university Table 27 - Comparing credential risk score and completion percentage for students by enrollment reason for students who did complete 15 hours 20 Student Retention Model for Harper College Dell Inc., 2014

21 We notice that students who are preparing to transfer to a four-year college, who reach the 15 hour mark, have both a predicted and an actual success score of 34.7%, or about a 1/3 chance to achieve certification. We also notice students who are either preparing for a job or exploring classes have a respective 50.7% and 45.4% chance, or about a 50% chance of achieving certification. The overall results show the ability to predict student success before the student enters Harper, at both the 15 hour in one year and 30 hour in two year mark, at the 95% level of significance. We have also uncovered both the significance and accuracy of Harper s testing program for English and math. With the Harper institutional testing procedures, we find that the existing process clearly indicates student success at Harper College. We also notice the importance of tutoring and high school GPA. Within the tutoring center, we notice that historical educational disadvantages can be overcome. Further study and detail around the various tutoring centers will only enhance and differentiate the impact on Harper student success. Potential future studies could include a longitudinal study of a student s courses, the effect of student behavior and interactions, and the incorporation of survey measures into the Harper/ Dell student retention model. Back testing of the Dell student retention model Back testing is a technique for an assessment of how statistical analysis will generalize to an independent data set, which the model was not developed on. The goal is to estimate how accurately a predictive model will perform on live data. Back testing is important in guarding against testing hypotheses suggested by the data, Type III errors, especially where further samples are impossible to collect. In our case, there would be a suspicion of just throwing all the variables into the model, selecting the best subset, and using that as the final result. The best way to back test the model, is to craft the model, based upon the 2008 cohort, and run the model with the exact 2008 coefficients, against the Harper College cohort 2006 and 2007, noting any discrepancies and strange results. The power of this type of model is that when we have the essential characteristics of a student first entering Harper, we can run the 2008 model on any student to accurately gauge their risk score. Back testing the hour risk score on the 2006 and 2007 cohort When we back test the hour risk score model on the 2006 and 2007 cohorts, without updating the coefficients for the model, we have the following results: Predicted Predicted Actual Predicted Number Number of Students Success Score Risk Score Results Risk Score of Students in Risk Score Group Table 28 - Back testing of the 2008 cohort on the 2006 and 2007 model 21 Student Retention Model for Harper College Dell Inc., 2014

22 We notice this is a similar graph of the 2008 model, applied to the 2006 cohort and 2007 cohort. In this graph we notice similar results, with the slope of the line parallel to the predicted, and with an R2 in line with predicted values. We also notice that the line is consistently below the risk score prediction, implying that the 2008 cohort model consistently overestimated the actual risk score for the 2006 and 2007 cohorts. The strength of the model is showing that the academic preparedness for the 2006 and 2007 cohorts was stronger than in the 2008 cohort. Number of Students in Cohort Percentage of Students in Cohort High School GPA % 2.4% 2.4% % 8.1% 7.4% % 16.0% 14.9% % 16.1% 16.6% % 9.7% 10.3% % 2.5% 3.0% % 0.3% 0.2% % 45.0% 45.3% Table 29- Academic preparedness and cohort changes by HS GPA 2006, 2007, 2008 We notice that the 2008 cohort was on par for the students above a 3.01 GPA, but the students who were in the GPA group were underrepresented in the 2008 cohort, while the students under 2.0 GPA were on par to be overrepresented. These cohort changes are to be expected and easily overcome through use of the tutoring center and focus on the first semester Harper GPA. Looking into the math placement exam, students who tested at least one level below grade, by high school GPA, we notice the exact same trend. Number of Students in Cohort Percentage of Students in Cohort High School GPA % 1.6% 1.1% % 6.3% 5.4% % 17.8% 16.5% % 21.8% 22.9% % 15.4% 16.6% % 5.0% 5.2% % 0.5% 0.3% % 31.6% 32.0% Table 30- Students who tested at least math one level below cohort and cohort changes by HS GPA 2006, 2007, Student Retention Model for Harper College Dell Inc., 2014

23 Again, we are on par for the GPA above 3.01, underrepresented in the GPA and overrepresented for students below a 2.0 GPA. Similar results are given for the English scores, as shown in the table below. Number of Students in Cohort Percentage of Students in Cohort High School GPA % 1.8% 0.4% % 5.4% 4.4% % 16.5% 15.2% % 23.6% 23.0% % 14.4% 17.1% % 6.6% 5.1% % 0.8% 0.1% % 30.8% 34.8% Table 31- Students who tested at least English one Level below cohort and cohort changes by HS GPA 2006, 2007, 2008 Again, we are on par for the GPA above 3.01, underrepresented in the GPA and overrepresented for students below a 2.0 GPA. Similar results are given for the English scores, as shown in the table below. Even within cohort demographics, the student enrollment was different between the cohorts. Number of Students in Cohort Percentage of Students in Cohort Enrollment Reason Prepare for a job immediately after attending Harper College % 20.8% 19.8% Improve skills needed in present job % 5.0% 5.1% Explore courses to decide on a career % 17.0% 17.0% Coursework to transfer to a four-year college or university % 49.8% 51.3% Prepare for GED test or improve basic academic skills % 2.1% 2.2% Personal interest-self-development % 2.1% 2.1% Already accepted by a four-year college or university % 3.2% 2.3% Other % 0.1% 0.2% Table 32- Student enrollment was between the cohorts 23 Student Retention Model for Harper College Dell Inc., 2014

24 The cohort for 2008 has a lower percentage of job preparation and a higher percentage of students who were planning on transferring course work to a four-year institution. Back testing the hour risk score on the 2006 and 2007 cohort When looking at the hour risk score on the 2007 and 2007 cohort, we notice that the model underperforms on back testing, consistent with what we see in the hour cohort model. For the 30 hour model, one of the most important variables in the matriculation studies is not high school performance, but first year performance at Harper College. We expect that moving forward, from an analytical perspective, a longitudinal model will perform better than a static model. Predicted Predicted Actual Predicted Number Number of Students Success Score Risk Score Results Risk Score of Students in Risk Score Group Table 33 - Predicted success score vs. actual success score 2008 model on cohort Hour in Two Years compared to Actual Results Model on Data Actual Results y = x R 2 = Student Population Scores Student Score - Actual Model Prediction Linear (Student Score - Actual) Figure 7 - Remedial reading student population of the 2008 Harper College cohort 24 Student Retention Model for Harper College Dell Inc., 2014

25 What is significant is that the 2006 and 2007 cohort under predicts, especially at the higher risk factors. Lower success scores broke from the expected at below the.60 bin and below. The implication is that the static risk score model is accurate for the high performing student, but when the risk score drops below.60, it is at a level where the interventions, counseling and tutoring sessions, may be used to help the Harper students succeed on Certification Actual Results R 2 = Student Score - Actual Model Prediction Linear (Model Prediction) Student Population Scores Figure 8 - Credential awards for the 2008 Harper College cohort 2006 cohort graphs For completeness, we have also run the Dell student retention model, updated for the 2006 cohorts. We find that the model coefficients slightly change, as seen in the back testing example, but are not out of 95% level change for any coefficients. Visually, it graphs as follows Cohort Model 15 Hour in One Year Actual Results Student Score - Actual Model Prediction Linear (Student Score - Actual) R 2 = Student Population Scores Figure cohort model for 15 hours in one year 25 Student Retention Model for Harper College Dell Inc., 2014

26 We notice a tight fit for these models, especially with the lines concurrently on top of each other. The cohort 2006 model is where we see some significant differences between the 2008 model on the 30 hour in two-year model. We notice that this cohort significantly over performed in risk score, by an average of.25 points better than the predicted model Cohort Model 30 Hours in Two Years Actual Results Student Score - Actual Model Prediction Linear (Student Score - Actual) R 2 = Student Population Scores Figure cohort model for 30 hours in two years There may be a couple of significant factors here. First, it will be noted that the 30 hour model can be shown to have better predictive power when you include Harper GPA in the dataset. Second, in late 2007 the financial crisis hit the overall economy. This potentially would improve a student s chance of reaching 30 hours in two years, outside of the predicted model. This is potentially confirmed by running the 2008 data on the 2006 model, noticing that the risk score decreases, increasing the student success rate Cohort Certification Model R 2 = Predicted Actual Linear (Actual) Figure cohort certification model We notice that the 2006 model on the 2006 data performs on the line, until we get to students with a higher risk score. The kick up should be ignored due to sample size. 26 Student Retention Model for Harper College Dell Inc., 2014

27 2007 cohort graphs For completeness, we have also run the Dell student retention model, updated for the 2007 cohorts. We find that the model coefficients slightly change, as seen in the back testing example, but are not out of 95% level change for any coefficients. Visually, it graphs as follows Cohort Model 15 Hours in One Year Actual Results Student Score - Actual Model Prediction Linear (Student Score - Actual) R 2 = Student Population Scores Figure cohort model for 15 hours in one year The cohort 2007 model, on the following page, is where we see some significant differences between the 2008 model on the 30 Hour in two-year model. We notice that this cohort significantly over performed in risk score, by an average of.15 points better than the predicted model Cohort Model 30 Hours in Two Years R 2 = Risk Score Predicted Linear (Risk Score) Figure cohort model 30 hours in two years 27 Student Retention Model for Harper College Dell Inc., 2014

28 Similar to the 2006 model, continuing economic uncertainty may lie at the heart of the disparity. First, it will be noted that the 30 hour model can be shown to have better predictive power when you include Harper GPA in the dataset. Second, in late 2007 the financial crisis hit the overall economy. This potentially would improve a student s chance of reaching 30 hours in two years, outside of the predicted model. This is potentially confirmed by running the 2008 data on the 2006 model, noticing that the risk score decreases, increasing the student success rate Cohort Certification Model R 2 = Actual Results Student Score - Actual Model Prediction Linear (Model Prediction) Student Population Scores Figure cohort credential model We notice that the 2007 model on the 2007 data performs on the line, until we get to students with a higher risk score. The kick up should be ignored due to sample size. 28 Student Retention Model for Harper College Dell Inc., 2014

29 Tracking student GPA through matriculation The interesting result we found for Harper was the significance of the cumulative first year GPA on students continuing their education and reaching the 30 hours in two-year mark. We noticed almost immediately that the high school GPA was significant for first year success, but the model started to become less predictive for the 30 hour score. When back testing the model on the 2006 cohort and 2007 cohort, we also noticed that both the 2008 back test, and the model specification updates for 2006 and 2007 produced results that implied that there was a difference between the 15 hour threshold and the 30 hour threshold. We also noticed that outside of the tutoring effect, the Harper GPA should be targeted for deeper research. When comparing the Dell student retention model without Harper GPA to the Dell student retention model with Harper GPA, we notice that they are very similar at the 50,000-foot level, as shown below Cohort Comparison Between First Year Harper GPA and High School GPA Actual Results No College GPA College GPA Predicted R 2 = Student Population Scores Figure cohort comparison between first year Harper GPA and high school GPA However, when you start to break the cohort down into high school GPA groups by Harper GPA groups, we start to notice that success is a combination of both Harper GPA and tutoring. For example, here are the students who entered Harper with a high school GPA between 3.0 and 4.0, corrected for a 5-point high school GPA. We notice immediately that even well prepared students who perform poorly in the first year, as indicated by Harper GPA, struggle to complete 30 hours in two year. For example, from the chart below, we notice that a student in this cohort, who has a GPA at Harper of 2.0, only has a 35% chance of making it through the second year. But if a student reached a 2.5 GPA, they are above 52% likely to reach 30 hours in two years. We also notice that students above a Harper 3.0 GPA consistently visit the tutoring center at least once a semester. We also notice that students who are struggling, with a Harper GPA of <1, use the tutoring center at the same rate as the 3.0 Harper students. 29 Student Retention Model for Harper College Dell Inc., 2014

30 30 Hour Success 30 Hour Success with Average 15 Hour Success w/o GPA Harper 2008 GPA Tutoring Harper GPA Table 34 - High school GPA We notice that the average prepared student, who performs poorly in the first year, as indicated by Harper GPA, struggles to complete 30 hours in two years. For example, from the chart below, we notice that a student in this cohort, who has a GPA at Harper of 2.0, only has a 33% chance of making it through the second year. This is on par with a student who came into Harper with a greater than 3.0 GPA. However, if a student reached a 2.5 GPA, they are above 47% likely to reach 30 hours in two years. Again, this score is on par with the well-prepared students. We also notice that students above a Harper 2.0 GPA consistently visit the tutoring center at least once a semester. We also notice that students who are struggling, with a Harper GPA of <1, use the tutoring center at the same rate as the 3.0 Harper students. 30 Hour Success 30 Hour Success with Average 15 Hour Success w/o GPA Harper 2008 GPA Tutoring Harper GPA Table 35 - High school GPA Student Retention Model for Harper College Dell Inc., 2014

31 For students who are not well prepared, we find that the first year cumulative Harper GPA is lower than the previous Cohorts. Even a student who has a 4.0 GPA at the end of the year only has a 55% chance of reaching 30 hours in 2 years, on par with the average students ( ). The interesting result for tutoring is that the students who have a 2.0 GPA at Harper are more likely to use the tutoring center resources. 30 Hour Success 30 Hour Success with Average 15 Hour Success w/o GPA Harper 2008 GPA Tutoring Harper GPA Table 36 - High school GPA The bottom line of the research clearly shows that first year GPA is far more indicative of 30 hour success at Harper than using only the high school GPA. We imply from these results that a deeper understanding of when a student visits a tutoring center, and the content covered, will greatly improve student success at Harper. The modeling type required is called a longitudinal model. This allows us to look at a student via a time series, modeling the interactions between GPA, tutoring, and Harper success. We also suspect from the data that there are interactions between the variables in the model. For example, does the tutoring center drive a higher GPA, and thereby increase success? Or does a low GPA drive students going to the tutoring center? These are questions that would be interesting to look at in the future. 31 Student Retention Model for Harper College Dell Inc., 2014

32 Heat Maps One of the interesting results we also found was the relationship between certain Harper student zip codes and math and English Placement scores. We looked at the following 5 digit zip codes: 60169, 60193, 60074, 60004, 60056, 60007, 60067, 60090, 60008, 60010, 60194, 60133, 60005, 60173, 60192, 60089, 60016, 60070, 60195, 60107, 60018, 60110, 60047, The selection criterion was to look at zip codes with at least 30 students in the 5 digits, to have an adequate sample for each of the zip codes. The total size of the population was 2560 students (88.1%), with 347 students being from outside the above zip codes (11.9%). The first heat map chart of the zip codes is for the 15 hour completion rate: By city, here are the following results: Figure 16 - Rates of 15 hour success score by zip code Min 1st Quartile Median Mean 3rd Quartile. Max ZIP_CODE Success Score Arlington Heights Streamwood Elk Grove Village Schaumburg Barrington Hoffman Est Des Plaines Wheeling Arlington Heights Mount Prospect Carpentersville Bartlett Kildeer Prospect Heights Hoffman Est Des Plaines Hoffman Est Buffalo Grove Schaumburg Hoffman Est Rolling Meadows Echo Lake Hoffman Est Keeneyville Table 37 - Rates of 15 hour success score by zip code 32 Student Retention Model for Harper College Dell Inc., 2014

33 We notice that within the Hoffman Estates zip codes (60067, 60169, 60173, 60192, and 60195) there is a range of average probabilities of success, ranging from (0.39 to 0.33). The inference to be drawn is that certain zip codes have a higher degree of success, even in a short geographical distance. This can be dependent on student demographics, academic preparedness, or motivation of the student. We believe that for college success, identifying students earlier and helping them with the transition between high school and college will make a positive impact on Harper s long-term student retention. City Zip Code 15 Hour Credit Number of Students Hoffman Est Hoffman Est. Total Arlington Heights Arlington Heights Total Des Plaines Des Plaines Total Schaumburg Schaumburg Total Table hour credit comparison by close geographical comparison Even within close geographical distances, we notice that Hoffman Estates and Arlington Heights have significant differences. Both zip codes have some of the highest average success scores, and also some of the lowest for the 15 hour success scores. 33 Student Retention Model for Harper College Dell Inc., 2014

34 Figure 17 - Rates of 30 hour success score by zip code Min 1st Quartile Median Mean 3rd Quartile. Max ZIP_CODE Success Score Barrington Buffalo Grove Elk Grove Village Hoffman Est Bartlett Keeneyville Arlington Heights Schaumburg Prospect Heights Schaumburg Rolling Meadows Hoffman Est Des Plaines Streamwood Hoffman Est Hoffman Est Kildeer Arlington Heights Mount Prospect Wheeling Hoffman Est Echo Lake Carpentersville Des Plaines Table 39 - Rates of 30 hour success score by zip code Visually, we notice that for the 30 hour risk score, there is significant deviation from the 15 hour score, with Barrington and Buffalo Grove significantly outperforming their peers. We also notice that Kildeer had the lowest 15 hour completion rate, and moved into the middle quartiles. We also notice that the Hoffman Estates zip codes are now all performing in the middle. 34 Student Retention Model for Harper College Dell Inc., 2014

35 Zip code by English exam score Visually, we notice some similarities between the well performing zip codes, and the 15 hour and 30 hour risk scores. Looking at the top tier of zip codes, all of them have a better than average success score. However, even with well performing scores in the English exam Scores, we notice that Carpentersville (60110) and Echo Lake (60047) perform better than their peers in English, yet still produce below average 15 Hour and 30 hour success scores. We also notice that Streamwood outperforms on the 15 hour and 30 hour success scores, but is just average on English exam scores. Figure 18 - Rates English exam scores by zip code Min 1st Quartile Median Mean 3rd Quartile. Max Zip Code Ave_score Barrington Echo Lake Prospect Heights Elk Grove Village Arlington Heights Carpentersville Mount Prospect Schaumburg Buffalo Grove Hoffman Est Arlington Heights Hoffman Est Hoffman Est Rolling Meadows Hoffman Est Kildeer Bartlett Hoffman Est Schaumburg Keeneyville Des Plaines Des Plaines Wheeling Streamwood Table 40 - Rates English exam scores by zip code 35 Student Retention Model for Harper College Dell Inc., 2014

36 Zip code by math exam scores Visually, we notice some similarities between the well performing zip codes and the 15 hour and 30 hour risk scores. Looking at the top tier of zip codes, all of them have a better than average success score. However, even with well performing scores in the English exam Scores, we notice that Barrington (60010) performs with their peers in math, yet still produces above average 15 hour and 30 hour success scores. We also notice that Wheeling outperforms on the math exam scores, but underperforms on 15 hour and 30 hour success scores. 1st 3rd Min Quartile Median Mean Quartile. Max Figure 19 - Rates math exam scores by zip code Conclusions Ave_score Hoffman Est Des Plaines Buffalo Grove Wheeling Prospect Heights Mount Prospect Arlington Heights Schaumburg Hoffman Est Barrington Arlington Heights Elk Grove Village Kildeer Rolling Meadows Bartlett Schaumburg Hoffman Est Hoffman Est Des Plaines Streamwood Echo Lake Hoffman Est Carpentersville Keeneyville Table 41 - Rates math exam scores by zip code The key to the Dell student retention model is to quickly identify students at risk. The students we need to identify are students who are on the bubble of potentially failing, the students who can be successful. 36 Student Retention Model for Harper College Dell Inc., 2014