Dropout Prediction Model. Executive Summary

Transcription

1 Charlotte-Mecklenburg Schools Research Brief 2008, Charlotte-Mecklenburg Schools Report from the August Dropout Prediction Model Executive Summary Student decisions to drop out have future implications not only for themselves but for society as a whole. Dropouts often face increased risks of unemployment, criminal activity, and health problems. Associated with these increased risks are well-documented public economic costs. Thus, preventing student dropout is important not only for the future of students, but for the strength of society as a whole. To provide assistance to School Social Work staff, the (CRE) constructed a dropout prediction model based on school year data. Dropouts were readily identifiable in data files from this school year, allowing student information from previous school years to be associated with future decisions to dropout. A number of demographic, behavioral, and academic measures were included in a logistic regression model to generate predicted probabilities of dropout for each student. The predicted dropout indicator generated by the model can be compared with the known dropout indicator, yielding a measure of classification accuracy. The model was able to accurately identify 81% of all dropouts while simultaneously identifying 81% of all non-dropouts. The predicated probability of dropout was used to create a Predictor Index, a numeric value used to designate a student s level of risk of dropping out. School rosters were shared with the Office of School Social Work that contained pertinent student information along with the Predictor Index for each student. Counselors will utilize these rosters to identify a smaller group of students they can provide targeted assistance to in hopes of preventing a decision to drop-out. Office of Accountability 1

2 Office of Accountability 2 Introduction A facet of student achievement often overshadowed by standardized test scores is the student dropout rate. As the United States economy becomes increasingly dependent on a highly skilled and educated workforce, the student dropout phenomenon will have more drastic and farther reaching effects on society (Dorn, 1993). Rumberger (1987) noted that dropouts experience higher levels of unemployment, are more likely to have health problems, engage in criminal activities, become dependent on government programs, and can expect to earn less than high school graduates. A summary table on the National Dropout Prevention Center s (2004) website showed a 31% decrease in the average hourly wage (adjusted for inflation) of dropouts between 1973 and McDill, Natrielo, and Pallas (1986) estimated the lifetime earnings losses for 516,000 sophomore dropouts in 1980 at approximately $55 billion, after adjusting the estimated lifetime earnings downward by half to adjust for biases related to differences in ability. Costs associated with student dropout are not only felt by the dropouts themselves. Catterall (1987) estimated the total loss in lifetime tax revenue associated with dropouts at around $70 billion for one cohort of 8 th grade students in the U.S. LeCompte and Dworkin (1991) claimed that between 1986 and 1991, New York City spent close to $40 million annually on dropout prevention programs and Rumberger (1995) cited increases in the amount of money spent on dropout prevention, job training, and welfare programs. These programs and forms of assistance impose economic stressors on other members of society at large. In addition, increasing dropout rates do not bode well for the advancement of a highly-educated workforce, thereby reducing the future labor productivity potential. Given the long-term negative effects of dropping out for both the student and the community at large, Charlotte-Mecklenburg Schools (CMS) has instituted a case management model focused on dropout prevention. To maximize the efficiency of this case management model, those students most likely to dropout need to be identified before dropout actually occurs. Some scholars subscribe to the theory that dropout is a culmination in the process of disengagement from school (Newmann, et al., 1992; Wehlage, et al., 1989), related to school, home and societal factors. Despite the research-based evidence identifying factors such as family socioeconomic status (SES) and family structure (single-parent family, etc.), the prediction models generated here are based on information readily available to CMS staff. A review of the model-building activities, as well as the results of model-efficacy tests, will be presented in the Methods and Results sections that follow. Data Methods To test the efficacy of a statistical prediction model, the true outcome to be modeled must be known a priori. Once the model has been generated, predicted outcomes can be compared to true outcomes to determine the prediction accuracy of the model. A data file of all students was constructed at the end of the school year, containing student demographic and achievement information, as well as attendance, retention, mobility, and withdrawal code information. Because students dropout throughout the entire year, academic achievement and attendance measures were identified from prior years for use in predicting dropout during the school year.

3 Basic student demographic information included gender, race or ethnicity, Free or Reduced Lunch status (FRL), Exceptional Child status (EC), Limited English Proficiency status (LEP) and whether or not the child was gifted. The various categories within FRL, LEP and EC were all re-coded as dichotomous variables to ease parameter interpretation. For example, students with no EC status were coded as a zero, while all students with any EC status (other than gifted) were coded as one. Total absences were calculated from the school year data, along with the number of days served in Out-of-school Suspension and In-school Suspension in Finally, the number of times previously dropped out, the number of times previously retained, and the number of schools enrolled-in (mobility) were tracked back through each student s academic history. For the high school students in the data file, performance on their Algebra I and English I End of Course (EOC) assessments were obtained. In addition, students were tracked longitudinally back through time to identify the last year they were in 8 th grade and had valid EOG scores. In all instances, assessment achievement levels were used in the predictor models to facilitate interpretation of coefficients. Model The dropout prediction model takes the form of the following mathematical formula: Office of Accountability 3 1 p = 1+ exp( α β x β x... β x) 1 2 j In words, this formula states that the probability of dropping out (p) is equal to one divided by one plus the exponential function of a negative intercept (α) minus the effect of a predictor(s) (β 1 3 ) multiplied by the value of that predictor(s) (x) for any given student. The result of this formula is a value for each student between zero and one indicating the likelihood of that student dropping out. This value can then be calculated for future students to target those most likely to drop out of school. These values were calculated for each student in the cohort file, resulting in a predicted value of whether the student dropped out or remained in school. The efficacy of the model was tested by randomly splitting the cohort file into two separate files; one for generating the prediction model and the second for determining how well the prediction model identifies students likely to drop out. Comparing predicted values generated by the prediction model to the actual value known to be true, we can determine the percentage of cases for which the model predicted dropout accurately. Statistically significant parameter estimates were retained for use in identifying future potential dropouts. These estimates were entered into a logistic regression formula and applied to the second data file containing students from the school year. A predicted probability of dropout was generated for each student in the data file, and a subsequent receiver operating

4 characteristic (ROC) curve was plotted. To maximize the predictive utility of the model, the probability associated with the smallest difference between sensitivity (identifying actual dropouts as dropouts) and specificity (identifying non-dropouts as non-dropouts) measures associated with the ROC curve was identified as the optimal cut-point (Hosmer & Lemeshow, 2000). This cut-point allowed us to classify students into one of two categories: low-risk or high-risk students. The prediction model and established cut-points will then be applied to a data file for incoming students to provide the School Social Work office with a list of hi-risk students. Model Development Results Table 1 below shows the parameter estimates and odds ratios for the statistically significant explanatory variables utilized in the dropout prediction model. Initial models entering all available predictor variables were generated (see Appendix A for saturated model results). Table 1. Parameter estimates and odds ratios associated with dropout probability Parameter Estimate Odds Ratio Constant Grade 9 Grade Grade Grade Exceptional Child Black Overage Unexcused Absences Excused Absences ISS Days OSS Days Mobility Prior Dropout Algebra I Level 1 Algebra I Level Algebra I Level Algebra I Level English I Level 1 English I Level English I Level English I Level Office of Accountability 4

5 The estimates presented in Table 1 provide the reader with an indication of each explanatory variable s influence on the probability of dropout. Generally, a negative estimate suggests that an increase in the value of the predictor is associated with a decline in the probability of dropout. Conversely, a positive estimate suggests an increase in the value of the predictor is associated with an increase in the probability of dropout. Odds ratios are easier for interpretation purposes, as they represent the predicted odds for students with a particular characteristic, controlling for all other characteristics in the model. For example, Prior Dropout has an odds ratio of 2.398, meaning the odds of dropout for students that dropped out in prior years are 2.4 times the odds for a student that never dropped out previously. In other words, the odds of dropping out for these students are 140% higher than for students never having previously dropped out. Conversely, the English I Level 4 odds ratio of.4057 suggests that students attaining a level 4 on the English I EOC assessment are less likely to dropout, or the odds are 59% ( ) lower. We applied the significant coefficients in logistic model form to the data in the second half of the file containing student records. Resulting predicted probabilities of dropouts were compared to the actual dropout indicator variable to generate the ROC curve. An ROC curve depicts the model s ability to correctly classify students into two groups as the discrimination threshold changes. Sensitivity is the model s ability to accurately predict dropouts as dropouts, while specificity is the ability to accurately predict non-dropouts as nondropouts. By plotting the sensitivity and specificity against each other, the intersection of the two curves easily identifies the optimal cut-point. ROC analysis output allowed us to calculate the difference between the two measures in absolute value form, choosing the minimal difference as the optimal cut point. A cut point of.038 accurately identified 81% of dropouts and 81% of non-dropouts as non-dropouts. Office of Accountability 5

6 Sensitivity-Specificity Probability Specificity Sensitivity Figure 1. Sensitivity-specificity plot for determining classification cut-point. Office of Accountability 6

7 Model Application Applying the coefficients (labeled estimates in Table 1 above) to demographic and academic school year values with the formula presented earlier yielded a probability of dropout for each expected student. In previous years, a value labeled the Predictor Index was provided to schools as a numeric way to represent the level of risk for students. To remain consistent, predicted probabilities for students were converted to logits, which perform similar to a z-score with a mean of zero. Figure 2 below displays the association between probability and logits. Note that a student with a probability of.5 has a corresponding logit value of zero. Thus, students more than 50% likely to dropout have a positive logit value. Additionally, a student with a logit of 2.5 is more at-risk of dropout than a student with a logit of Probability Logit Figure 2. Probability by logit plot for high school dropout prediction. Microsoft Excel rosters were created for each school that contained pertinent information and their associated risk. The School Social Work office will share these rosters with school counselors prior to the start of the school year. Counselors will use these rosters and the information contained within them to focus efforts on those students most in need in the hopes of keeping students engaged in school. Conclusion Office of Accountability 7

8 The brief review of the literature presented in the introduction outlines the eventual consequences that dropouts face. In addition, student decisions to dropout ultimately impact society in both financial and social ways. Thus, the NCLB legislation includes graduation and dropout rates as measures to be included in the Adequate Yearly Progress (AYP) of schools. In an attempt to prevent students from dropping out, CMS school counselors provide assistance to those students deemed to be at risk of dropping out. The production of a dropout prediction model allows for Prevention & Intervention staff to combine their intuitive knowledge of individual students with statistical prediction based on raw data. The model presented herein was generated based on historical data for CMS high school students, as the greatest rate of incidence occurs at the high school level. A number of indicators, including student demographic variables, behavioral and academic measures were found to be associated with student decisions to dropout. The wide array of variables associated with dropout provides some insight into understanding why students decide to dropout, and also allows for a more accurate overall model. Though the model was also applied to at the elementary and middle school level, the accuracy with which the model predicts dropout at this level is suspect at best. Future research into the reasons and indicators associated with the small population of elementary and middle school dropouts is warranted. Better understanding of the early indicators associated with dropout would allow for intervention at an early age, inhibiting disengagement from school and an eventual decision to drop out. Prepared by Jason Schoeneberger, Director of Research & Evaluation References Catterall, J. (1987). On the social costs of dropping out. High School Journal, Dorn, S. (1993). Origins of the Dropout Problem. History of Education Quarterly, 33, Hosmer, D. & Lemeshow, S. (2000). Applied Logistic Regression. New York, NY: John Wiley & Sons, Inc. LeCompte, M. & Dworkin, G. (1991). Giving Up On School Student Dropouts and Teacher Burnouts. Newbury Park, CA: Corwin Press. McDill, E., Natriello, G, & Pallas, A. (1986). A population at risk: Potential consequences of tougher school standards for student dropouts. American Journal of Education, 94, Newmann, F., Wehlage, G, & Lamborn, S. (1992). The significance and sources of student engagement. In Newmann, F. (Ed.), Student Engagement and Achievement in American Secondary Schools. New York, NY: Teachers College Press. No Child Left Behind Act of 2001, Pub. L. No , 2, 115 Stat (2002). Office of Accountability 8

9 Rumberger, R. (1987). High school dropouts: A review of issues and evidence. Review of Educational Research, 57, Rumberger, R. (1995). Dropping out of middle school: a multilevel analysis of students and schools. American Educational Research Journal, 32, 3, Wehlage, G. & Rutter, R., Smith, G., Lesko, N., & Fernandez, R. (1989). Reducing the Risk: Schools as Communities of Support. New York, NY: Falmer. Office of Accountability 9

10 Appendix A Appendix A. Saturated model parameter estimates and odds ratios associated with dropout probability Parameter Estimate p-value Odds Ratio Constant Grade Grade Grade Grade FRL LEP EC Gender Asian Black Hispanic Native American Multi-racial Gifted CIS Served Overage Prior Dropout Grade 8 Math Level Grade 8 Math Level Grade 8 Math Level Grade 8 Math Level Grade 8 Read Level Grade 8 Read Level Grade 8 Read Level Grade 8 Read Level Algebra I Level Algebra I Level Algebra I Level Algebra I Level English I Level English I Level English I Level English I Level Unexcused Absences Excused Absences ISS Days OSS Days Mobility Office of Accountability 10