INITIAL QUANTITATIVE RESULTS FOR ILLINOIS COLLEGE AND CAREER READINESS (CCR) ACADEMIC INTERVENTIONS (1-11) June, 1 Matthew A. Linick George C. Reese Jason L. Taylor Debra D. Bragg Lorenzo D. Baber OFFICE OF COMMUNITY COLLEGE RESEARCH AND LEADERSHIP Department of Educational Organization and Leadership College of Education University of Illinois of Urbana-Champaign 19 Children s Research Center 1 Gerty Drive Champaign, IL 618
The Office of Community College Research and Leadership (OCCRL) was established in 1989 at the University of Illinois at Urbana-Champaign. Our primary mission is to use research and evaluation methods to improve policies and programs to enhance community college education and transition to college for diverse learners in Illinois and the United States. Projects of this office are supported by the Illinois Community College Board (ICCB) and the Illinois State Board of Education (ISBE), along with other state, federal, and private and not-for-profit organizations. The contents of our publications do not necessarily represent the positions or policies of our sponsors or the University of Illinois. Comments or inquiries about our publications are welcome and should be directed to OCCRL@illinois.edu. This document can be found on the web at: http://occrl.illinois.edu. This publication was prepared pursuant to a grant from the Illinois Community College Board and printed June, 1 by the Authority of the State of Illinois (ICCB Grant Agreement Number 1CCR1). Recommended Citation: Linick, M.A., Reese, G.C., Taylor, J.L., Bragg, D. D., & Baber, L. D. (1, June). Results for Illinois college and career readiness (CCR) academic interventions (11-1). Champaign, IL: Office of Community College Research and Leadership, University of Illinois at Urbana-Champaign. Copyright 1 Board of Trustees, University of Illinois Office of Community College Research and Leadership ii
TABLE OF CONTENTS INTRODUCTION... 1 Participation and Completion Criteria for and Limitations in Assessing Success... Reducing-Remediation Criteria for and Limitations in Assessing Student Success... Definition of Terms... 3 THE STUDENTS OF THE COLLEGE AND CAREER READINESS ACT... COLLEGE OF LAKE COUNTY (CLC)... 8 JOHN A. LOGAN COLLEGE (JALC)... 1 KANKAKEE COMMUNITY COLLEGE (KCC)... 14 MORAINE VALLEY COMMUNITY COLLEGE (MVCC)... 17 SOUTH SUBURBAN COLLEGE (SSC)... 3 SOUTHWESTERN ILLINOIS COLLEGE (SWIC)... DISCUSSION... 8 REFERENCES... 9 APPENDIX A: CCR LEGISLATION 3 LIST OF TABLES Table1: CCR Students Percentages... 6 Table : CCR Students Raw Numbers... 7 LIST OF FIGURES Figure 1: CLC Completion by Intervention... 8 Figure : CLC Academic Progress... 9 Figure 3: JALC Completion by Intervention... 1 Figure 4: JALC Academic Progress in MAT... 11 Figure : JALC Academic Progress in Mat 6... 11 Figure 6: JALC Academic Progress in ENG... 1 Figure 7: JALC Academic Progress in ENG 3... 1 Figure 8: KCC Completion by Intervention... 13 Figure 9: KCC Academic Progress in Math Instructional Support... 14 Figure 1: KCC Academic Progress in Summer Bridge Math... 1 Figure 11: KCC Academic Progress in Summer Bridge English... 16 Figure 1: MVCC Completion by Intervention... 17 Figure 13: MVCC Academic Progress in College Prep Institute for Juniors... 18 Figure 14: MVCC Academic Progress in College Prep Institute for Seniors... 18 Figure 1: MVCC Academic Progress in College Prep Institute for Graduated Seniors... 19 Figure 16: Shawnee Completion by Intervention... Figure 17: Shawnee Academic Progress in MAT 4... 1 Figure 18: Shawnee Academic Progress in Basic Writing I... 1 Figure 19: Shawnee Academic Progress in Basic Writing II... Figure : SCC Completion by Intervention... 3 Figure 1: SCC Academic Progress in Algebra 9 and Algebra 1... 4 Figure : SWIC Completion by Intervention... Figure 3: SWIC Academic Progress in MAT 94... 6 Figure 4: SWIC Academic Progress in MAT 97... 6 Figure : SWIC Academic Progress in ENG 9... 7 Office of Community College Research and Leadership iii
INTRODUCTION In the spring of 11, OCCRL requested that the seven community colleges participating in Illinois College and Career Readiness (CCR) Pilot grant participate in quantitative data collection on the academic (mathematics and English) interventions. The mechanism for gathering these data included a set of online questionnaires tailored to program- and studentlevel information available from each site. In this report, we focus on the academic interventions, and it complements a report submitted earlier in FY1 that presents results from the evaluation of local CCR implementation (Bragg, Baber, Cullen, Reese, & Linick, 11). These interventions included mathematics and English courses taught at the community college or high school, workshops or summer bridge programs with an academic component and assessment. To measure outcomes from these academic interventions we examined course participation, course completion, and pre-test and post-test performance where it was available. A major goal of CCR legislation is to find ways to reduce remediation needed by students entering the community college (see Appendix A). As evaluators of the pilot programs we are especially interested in seeing a change in academic placement in students after participating in the CCR intervention. The academic interventions are only one part of Illinois CCR pilot program; sites might also have CCR courses or workshops, faculty alignment meetings or other professional development activities, and/or special support sessions with guidance counselors. However, since academic performance is a key indicator of college and career success, we made a special effort to gather data on these student outcomes. We present here a preliminary report of results using quantitative data gathered on the academic interventions implemented during the 1-11 academic year. In alphabetical order, the sites are College of Lake County, John A. Logan College, Kankakee Community College, Moraine Valley Community College, Shawnee Community College, South Suburban Community College, and Southwestern Illinois College. As each of the participating CCR pilot sites approach their own program differently, utilizing varied instructional techniques and curriculums, we believe it is best to present reports for each site individually. Our purpose is to share this information with the goal of inspiring conversation and an exchange of knowledge among CCR sites, in order to develop a CCR model that offers the most promising practices of all sites and succeeds in accomplishing the goals of CCR legislation. The primary evaluation questions we are seeking to answer with this quantitative report are: 1. Are CCR sites successful in recruiting and retaining students through completion of the academic intervention?. Are CCR sites successful in reducing remediation needs, based on placement cut-scores, for students that participate in CCR interventions? The measures we use for this analysis include participation numbers, completion numbers, completion rates, average completion rates, overall completion rates, raw test score changes, and placement level changes. Our criteria for assessing program success are directly related to our evaluation questions and concern program recruitment and completion, as well as successfully reducing remediation needs of participating students. Office of Community College Research and Leadership 1
Participation and Completion Criteria for and Limitations in Assessing Success While we did not set specific benchmarks for program recruitment or completion, it is important for us to acknowledge that some sites face hurdles to program recruitment not experienced by other sites; such limitations include transportation, population density (e.g., urban vs. rural), or intervention location. Though these contextual issues are important, they are not addressed in depth in this report because they are discussed in prior evaluation reports. For example, information on these considerations is discussed in the Year Four Summary Report available at http://occrl.illinois.edu/publication/997. Whereas evaluating program success on participation and completion alone, more advances need to be made to improve these measures across all of the sites, including documenting successful practices supported by these data. Promising recruitment practices offered by John A Logan College (JALC) or retention strategies used by Southwestern Illinois College (SWIC) represent but two examples of lessons that could benefit other sites. Some other CCR sites have shown other positive outcomes that may be instructive to the other sites. Reducing Remediation Criteria for and Limitations in Assessing Student Success Similar to participation and completion measures, neither the site personnel nor the evaluators set benchmarks for deeming a program successful in terms of reducing the remediation needs of participating students. Rather, the evaluation sought to paint a picture that included: the number of students enrolled in remedial interventions, the number of students that completed the interventions with some indicator of success, the number of students that showed improvement on a post-intervention assessment. First, the CCR sites rely on pre-existing college placement tests used by the community colleges, and these placement tests have limitations when used as to measure improvement from pre- to post-test. The COMPASS, ACCUPLACER, and ASSET tests that the seven CCR pilot sites use were not designed to function as pre- to post-test measures, but these assessments are dictated by the CCR legislation and preferred by the local community colleges in order to avoid giving students an additional battery of exams. Until a more consistent instrument is selected or developed for CCR, we use these scores to answer questions as best we can about student gains of the academic content. Other limitations include our lack of ability to match pre-test and post-test measures for some students, due to missing data and the effects of test overload for Junior and Senior high school students who had already taken multiple standardized tests. It is unclear whether students participating in the academic interventions understood the value of the post-test and therefore lacked incentive to perform to the best of their ability when the given test was given at the end of the CCR program. If this was the case, the post-test was not likely to give an accurate estimate of the growth that students experienced in academic competence from the time they began to the time they completed the CCR program. Finally, several students participated in more than one intervention, and it was not possible to partition the effects of each intervention by the students test score changes from pre-test to post- Office of Community College Research and Leadership
test. Therefore, the graphs presented in this report reflect changes in raw test scores and placement level from pre-test to post-test by intervention for all students who participated in the intervention. In other words, when a student participated in more than one intervention, that student s raw test score change is included in the graph for each intervention in which that student participated. Definition of Terms Academic interventions or interventions: The academic interventions, including programs, workshops, and other strategies, supported by the various sites as part of CCR. To qualify as an intervention for this quantitative evaluation the program must include extended contact with the student, focus on math or English, and have a pre/post-test measure. Intervention description: A brief description of the intervention(s) discussed in the analysis. Some sites include more than one academic intervention. Number of students participating: The number of students participating in each intervention, with the total being the number of discrete, individual students participating in an intervention. Number of students completing at least one intervention: Of the total number of students, this measure estimates the number of students who completed at least one intervention. Note: Completion does not imply successful completion or passing grades in the academic interventions which is locally determined. It simply means the site identified the student as someone who was present at the conclusion of the intervention. Average completion rate of interventions: This figure takes the completion rate of each intervention and averages it across the CCR interventions for that site, if a site had more than one academic intervention. Overall rate of students completing at least one intervention: This estimate divides the total number of discrete participating students by the total number of students completing at least one intervention. Each student is counted only once in this measure. Placement instruments: This measure refers to the assessment used by the site to determine a student s academic placement. Examples are COMPASS, ACCUPLACER, ASSET and locally developed assessments. Often, these instruments are used as a post-test as well. They give us some estimate of the reduction of remediation needs, and the limitation of these instruments is presented elsewhere in this report. Criteria for success in interventions: This measure refers to the criteria established by the CCR sites for indicated if a student completed an intervention successfully. Number of interventions completed with a passing grade: This measure refers to the total number of interventions (could be multiple per student) that were completed during the CCR program in which students received a passing grade. The definition of a passing grade is locally determined. Office of Community College Research and Leadership 3
Raw test score: This measure refers to the specific numerical scores received by a student on the pre- and/or post-test exam. This is a student level variable. These raw scores are not reported here, but were used to compute placement levels and the raw test score changes from pre-test to post-test gains for individual students. Raw test score changes: This measure refers to the changes in a student s raw test score from the pre-test to the post-test and is calculated by subtracting the student s pre-test score from the student s post-test score. This is a student level variable and not a measurement of gain, but of change. These changes are not reported here, but were used to calculate the number of students whose raw test score changed from pre-test to post-test. Number of students whose raw test score changed from pre-test to post-test: This measure refers to the total number of students whose post-test raw score increased, decreased, or did not change in relation to the pre-test score. This is a program level variable, and is calculated by counting students with positive, negative, and zero raw test score changes. This variable is depicted in the graphs of academic progress attached to the following site reports. Placement level: This measure refers to the level of coursework at which a student is placed based on the student s raw test score on the placement instrument or instruments used by the community college. For example, student s raw score on the COMPASS may place them at the pre-algebra level. A higher score would place them at the basic algebra level. The lower the level of placement the more remediation is needed. This is a student level variable. This variable is not reported here, but used to calculate a student s placement level change. To calculate the value of this variable, local cut scores were applied to a student s raw test score to determine the class in which the student was placed. Classes were assigned ordinal, numerical values ranging from -. NOTE: the placement levels are locally determined. There is no uniform college placement test, cut-off score, or course-level placement across the seven community colleges involved in the CCR pilot program. Placement level change: This measure refers to the changes in a student s placement level from the pre-test to the post-test and is calculated by subtracting the student s pre-test placement level from the student s post-test placement level. This is a student level variable and not a measurement of gain, but of change. These changes are not reported here, but were used to calculate the number of students whose placement level changed from pre-test to post-test. Number of students whose placement level changed from pre-test to post-test: This measure refers to the total number of students whose post-test placement level increased, decreased, or remained the same in relation to the student s pre-test placement level. This is a program level variable, and is calculated by counting students with positive, negative, and zero placement level changes. This variable is depicted in the graphs of academic progress attached to the following site reports. Office of Community College Research and Leadership 4
THE STUDENTS OF THE COLLEGE AND CAREER READINESS ACT Tables 1 and demonstrate the diversity of students that participate in CCR interventions across the participating sites. While all sites are charged with helping students that are not academically prepared for college-level coursework, there is a great diversity in the number of students served and the background of the students. For example, College of Lake County (8.7%) and South Suburban College (86.4%) provided CCR services for a much greater proportion of traditionally underrepresented students than other sites such as Kankakee Community College (11.6%). Other factors to consider are the differences in participating students concerning English Language Learner (ELL) status, highest enrolled math class, and ACT/PLAN score. Understanding the differences in the students participating at the different sites is an important first step to understanding the differences in student outcomes between the sites. A concern of the evaluation is the number of students with unreported demographic and academic histories. Increasing the accuracy of this data collection will improve the accuracy of future evaluation results. Office of Community College Research and Leadership
Table 1: CCR Student Characteristics Percentages by Site Site CLC JALC KCC MVCC SCC SSC SWIC Total Total 49 177 146 8 3 8 614 Latino 8.7%.3% 4.1%.4% 3.8%.% 6.1% 1.7% Asian 4.1% 6.8%.7% 1.% 1.9% 4.%.%.9% Race ESL Status Grade Level Highest Enrolled* Math Class** ACT COMP RANGE*** Gender Native American.%.6%.7% 1.%.%.%.%.% Black 6.1% 6.6% 7.% 3.% 13.% 86.4% 8.% 1.% White 4.1% 6.% 87.% 1.8% 79.% 9.1% 4.1% 7.7% Other.%.6%.%.%.%.%.%.% Unreported.% 6.8%.%.% 1.9%.%.7% 4.9% ESL Student 67.3% 1.1% 1.4% 8.%.%.%.4% 7.% Non-ESL 3.7% 98.9% 98.6% 91.8% 1.% 1.% 97.6% 9.% Junior 18.4% 13.% 87.% 8.%.8% 1.%.% 3.% Senior 79.6%.3% 1.3% 71.8% 77.4%.% 1.% 3.3% Gen. Math.% 1.1%.%.4%.% 4.9%.%.3% Geometry 1.% 9.% 39.7% 7.1%.6%.%.4% 8.1% Alg. 1.%.3% 8.9% 3.9% 9.4%.%.% 18.9% Alg. 1.% 11.9% 3.1%.%.8% 9.1%.4% 16.8% Adv/Coll Algebra 4.% 1.3% 6.% 1.6%.%.%.% 9.3% Calculus.%.6%.%.%.%.% 1.%.7% Pre-Calc 1.%.6% 9.6%.%.% 9.1% 4.9% 3.1% Stats.%.%.%.% 1.9% 4.%.4%.7% Trig 8.%.6%.% 3.%.%.7% 7.3%.% Unreported 3.6% 3.7%.% 3.% 4.3% 13.6% 79.3% 34.7% <=1 18.4% 14.1% 19.9% 1.3% 4.% 7.3% 11.% 16.9% 16-18 3.6% 13.% 6.% 48.% 8.3% 4.%.% 6.1% 19-1 1.%.6% 4.% 3.% 9.4% 13.6% 1.9% 14.8% -4.% 4.% 17.8%.4% 1.9% 9.1% 7.3% 7.3% +.%.3% 3.4%.%.% 4.%.% 1.6% Unreported 4.8% 6.% 8.9% 1.6% 3.8%.% 43.9% 33.% Male 8.6% 44.1% 4.4% 34.1% 3.8% 36.4% 6.8% 37.3% Female 71.4% 6.% 9.6% 6.9% 64.% 63.6% 73.% 6.9% Note: *Some sites provided highest enrolled math class, and some provided highest completed math class (KCC). This portion of the table refers to the most complete information provided by each site in reference to student math class. ** The names of courses offered at high schools differed, for cross-referencing similarly named math classes were combined. Algebra 1 and basic algebra are combined, intermediate algebra and algebra are combined, etc. ***Some sites used scores from earlier tests in ACT series such as the PLAN. Office of Community College Research and Leadership 6
Table : CCR Student Characteristics Raw Numbers by Site Site CLC JALC KCC MVCC SCC SSC SWIC Total Total 49 177 146 8 3 8 614 Latino 4 4 6 19 78 Asian 1 1 1 1 1 18 Race ESL Status Grade Level Highest Enrolled* Math Class** ACT COMP RANGE*** Gender Native American 1 1 1 3 Black 3 47 11 7 19 3 13 White 1 17 44 4 37 34 Other 1 1 Unreported 1 1 17 3 ESL Student 33 7 46 Non-ESL 16 17 144 78 3 8 68 Junior 9 3 17 4 11 16 Senior 39 89 1 61 41 8 37 Gen. Math 1 9 14 Geometry 4 13 8 Alg. 1 16 8 3 1 116 Alg. 6 1 44 17 11 13 Adv/Coll Algebra 1 7 9 9 7 Calculus 1 1 4 Pre-Calc 4 1 3 6 19 Stats 1 14 4 3 Trig 1 1 1 3 Unreported 1 9 8 3 4 3 6 13 <=1 9 9 13 13 6 9 14 16-18 1 3 38 41 1 1 18 16 19-1 1 3 3 13 91-4 8 6 1 6 4 + 4 1 1 Unreported 17 13 9 19 36 4 Male 14 78 9 9 19 8 9 Female 3 1 87 6 34 14 6 386 Note: *Some sites provided highest enrolled math class, and some provided highest completed math class (KCC). This portion of the table refers to the most complete information provided by each site in reference to student math class. ** The names of courses offered at high schools differed, for cross-referencing similarly named math classes were combined. Algebra 1 and basic algebra are combined, intermediate algebra and algebra are combined, etc. ***Some sites used scores from earlier tests in ACT series such as the PLAN. Office of Community College Research and Leadership 7
COLLEGE OF LAKE COUNTY (CLC) Interventions: English 19 (spring and summer interventions) Intervention descriptions: 16-week spring semester course or 8-week summer semester course offered to high school seniors, taught by CLC faculty with experience in developmental instruction. Number of students participating: Total number of students participating: 49 English 19 Spring: 6 English 19 Summer: 43 Number of students completing the intervention: 48 Overall rate of students completing at least one intervention: 98.% Placement Instruments: ACCUPLACER Criteria for success in interventions: Passing grade in class. Number of interventions completed with passing grade: 48 4 4 3 3 1 1 4 Completing WITH Passing Grades Completing WITHOUT Passing Grades Not Completing 6 ENG 19 Spring 1 ENG 19 Summer Figure 1: CLC Completion by Intervention Office of Community College Research and Leadership 8
4 4 3 3 1 1 14 1 9 Number of students whose ACCUPLACER SENTENCE SKILLS raw test score changed from pre-test to post-test Figure : CLC Academic Progress 9 1 14 Number of students whose placement level changed from pretest to post-test INCREASED NO CHANGE DECREASED Discussion Recruitment and Retention Results: Nearly all students (98 percent) participating in the CCR intervention completed the ENG 19 course, demonstrating a very high completion rate. Reducing Remediation Results: Of the 48 completers of the academic intervention, 3 percent (14 of 44) had a higher post-test than pre-test score; remediation was reduced by at least one placement level for 9 of these students. Despite these positive results, 14 students dropped at least one level of remediation, which suggests a net loss in CLC s efforts to reduce remediation for the overall CCR group. Office of Community College Research and Leadership 9
JOHN A. LOGAN COLLEGE (JALC) Interventions: Math, Math 6, English, English 3 Intervention description: Developmental mathematics and English courses completed at a high school site. Number of students participating: Total number of students participating: 177 Math : 118 Math 6: 11 English : 77 English 3: 79 Number of students completing at least one intervention: 169 (of 177) Overall rate of students completing at least one intervention: 9.% Average completion rate of each intervention: 88.% Placement Instruments: Not applicable Assessment tool and criterion for academic progress: Passing grade in class. Number of interventions completed: 338 14 1 1 8 6 94 98 Completing WITH Passing Grades Not Completing 4 73 73 4 17 4 6 MAT MAT6 ENG ENG3 Figure 3: JALC Completion by Intervention. Office of Community College Research and Leadership 1
6 4 3 46 INCREASED NO CHANGE DECREASED 1 Discovery raw test score changes from pre-test to post-test Figure 4: JALC Academic Progress in MAT 19 1 1 My Foundations Lab raw test score changes from pre-test to post-test 7 6 4 3 6 INCREASED NO CHANGE DECREASED 1 3 6 Discovery raw test score changes from pre-test to post-test Figure : JALC Academic Progress in MAT 6 11 1 My Foundations Lab raw test score changes from pre-test to post-test Office of Community College Research and Leadership 11
4 4 3 3 1 1 34 INCREASED NO CHANGE DECREASED 4 3 Discovery raw test score changes from pre-test to post-test 11 My Foundations Lab raw test score changes from pre-test to post-test Figure 6: JALC Academic Progress in ENG 4 4 3 3 1 3 INCREASED NO CHANGE DECREASED 1 Discovery raw test score changes from pre-test to post-test 13 My Foundations Lab raw test score changes from pre-test to post-test Figure 7: JALC Academic Progress in ENG 3 Office of Community College Research and Leadership 1
Discussion Recruitment and Retention Results: JALC recruited and retained large numbers of students to and through its CCR interventions. A total of 177 students participated in the CCR program that included four academic interventions (equating to four developmental courses), and 96% of these students completed at least one intervention. At least out of 3 CCR students participated in more than one intervention. About 6% of the total group (n 16) took both math and English interventions, and this group included students who completed all four interventions, 4 who completed three interventions, and 7 students who completed two interventions. Of this last group, participated in two math interventions, 6 participated in two English interventions, and 46 completed one each of math and English. Reducing Remediation Results: The vast majority of JALC s students placed higher on the posttest than on the pre-test. Since the pre-tests/post-tests administered were not designed to attribute placement, it is not possible to determine whether JALC s interventions actually reduced remediation needs in the strictest sense. However, as almost all of the students that took a pre-test and a post-test demonstrated improvement, it is important to acknowledge JALC s success in improving student scores. Further study of student performance and analysis would be needed, though, to determine whether student success is attributable to JALC or a student s high school. Office of Community College Research and Leadership 13
KANKAKEE COMMUNITY COLLEGE (KCC) Interventions: Math Instructional Support, Summer Bridge Math, Summer Bridge English Intervention description: Year-long math instructional support uses online programs (MyMathXL, ALEKS, or Carnegie Learning). Summer Bridge programs are developmental courses. Number of students participating: Total number of students participating: 146 Math Instructional Support: 14 Summer Bridge Math: 16 Summer Bridge English: 9 Number of students completing at least one intervention: 14 Overall rate of students completing at least one intervention: 71.% Average completion rate of interventions: 78.3% Placement Instruments: COMPASS Criteria for success in interventions: Passing grade in bridge classes. Number of interventions completed with passing grade: 16 14 1 1 8 6 4 88 36 Completing WITH Passing Grades Completing WITHOUT Passing Grades Not Completing Math Instructional Support 1 8 4 1 Summer Bridge Math Figure 8: KCC Completion by Intervention Summer Bridge English Office of Community College Research and Leadership 14
1 1 8 6 37 6 4 3 47 43 3 INCREASED NO CHANGE DECREASED Number of students whose raw test score changed from pre-test to posttest Number of students whose placement level changed from pretest to post-test Figure 9: KCC Academic Progress in Math Instructional Support 1 9 8 7 6 4 3 1 7 Number of students whose raw test score changed from pre-test to posttest Figure 1: KCC Academic Progress in Summer Bridge Math 3 1 Number of students whose placement level changed from pretest to post-test INCREASED NO CHANGE DECREASED Office of Community College Research and Leadership 1
7 6 4 3 6 6 1 1 1 1 1 Number of students whose raw test score changed from pre-test to post-test in Reading Number of students whose placement level changed from pre-test to post-test in Reading Number of students whose raw test score changed from pre-test to post-test in Writing Number of students whose placement level changed from pre-test to post-test in Writing INCREASED NO CHANGE DECREASED Figure 11: KCC Academic Progress in Summer Bridge English Discussion Recruitment and Retention Results: The instructional intervention with the most participants at Kankakee is the online instructional support done in cooperation with the feeder high schools wherein students do not receive a grade. Students who completed the intervention were determined to have completed the intervention with a passing grade. The summer bridges had a much smaller number of participants, but grades were assigned to students in that intervention. Reducing Remediation Results: KCC results showed an increase in the total number of students with higher placement levels in all three interventions. In the mathematic instructional support, KCC was very successful in reducing remediation, with 37 students placing higher and 3 placing lower into the mathematics curriculum. Though the total number of participants was small (n=9) in the Summer Bridge Math program, the majority of students placed into a higher math course. Only 9 students participated in the Summer Bridge English intervention, with taking pre- and post-tests in reading and 6 taking pre- and post-tests in writing. 1 student did not take a pre-test and post-test. All students that took the pre- and post-test in writing placed at a higher level on the post-test than on the pre-test, though the performance on the reading test was split with one student placing at a lower level on the post-test and one student placing at a higher level. Office of Community College Research and Leadership 16
MORAINE VALLEY COMMUNITY COLLEGE (MVCC) Interventions: College Prep Institutes for High School Juniors, Seniors, and Just-Graduated Seniors. Intervention descriptions: Developmental courses. The Institute for Just-graduated Seniors takes place in the summer. Number of students participating: Total: 8 College Prep Institute JR: 7 College Prep Institute SR: 4 College Prep Institute Just-Graduated SR: 41 Number of students completing at least one intervention: 77 Overall rate of students completing at least one intervention: 9.6% Average completion rate of interventions: 86.% Placement Instruments: COMPASS Criteria for success in interventions: Passing grade in institute Total number of interventions completed with a passing grade: 68 4 4 3 3 1 1 1 6 College Prep Institute - JR 3 3 Figure 1: MVCC Completion by Intervention 1 College Prep Institute - SR 31 7 College Prep Institute - Graduated SR Completing WITH Passing Grades Completing WITHOUT Passing Grades Not Completing Office of Community College Research and Leadership 17
16 14 3 1 1 9 8 6 4 6 9 INCREASED NO CHANGE DECREASED 3 Number students whose raw test score changed from pre-test to posttest Number of students whose placement level changed from pretest to post-test Figure 13: Academic Progress in College Prep Institute for Juniors 1 1 8 4 6 4 6 7 INCREASED NO CHANGE DECREASED Number students whose raw test score changed from pre-test to posttest Figure 14: Academic Progress in College Prep Institute for Seniors 1 Number of students whose placement level changed from pretest to post-test Office of Community College Research and Leadership 18
4 3 INCREASED NO CHANGE DECREASED 1 1 Number students whose raw test score changed from pre-test to posttest Figure 1: MVCC Academic Progress in College Prep Institute for Graduated Seniors 1 Number of students whose placement level changed from pre-test to posttest Discussion Most MVCC students completed the interventions with passing grades. As with other institutions that used the COMPASS as a post-test, we see a larger number of students who increased a level than decreased a level, but both of these changes were small. Six students increased a level, while four students decreased a level, leaving most students who had pre- and post-test scores (7 of 78) without a change in their test score placement. However, with so few students taking the pre- and post-test examinations, it is unclear what these results mean for the CCR program. The reasons for these missing data are unknown, but hopefully it will be feasible to address these serious gaps in data reporting for the quantitative evaluation in the 11-1 academic year. Office of Community College Research and Leadership 19
SHAWNEE COMMUNITY COLLEGE (SHAWNEE) Interventions: Math 4, Basic Writing I, Basic Writing II Intervention description: Developmental mathematics and writing courses Number of students participating: Total: 3 Math 4: 6 Basic Writing I: 7 Basic Writing II: Total number of students completing at least one intervention: 31 Overall rate of students to complete at least one intervention: 69.8% Average completion rate of interventions:.% Placement Instruments: ASSET and COMPASS Criteria for success in interventions: Passing grade in class Number of interventions completed with a passing grade: 34 3 1 1 13 13 18 Completing WITH Passing Grades Completing WITHOUT Passing Grades Not Completing 3 7 4 MAT 4 Basic Writing I Basic Writing II Figure 16: Shawnee Completion by Intervention Office of Community College Research and Leadership
14 1 1 8 6 4 6 INCREASED NO CHANGE DECREASED 3 Number of students whose raw test score changed from pre-test to posttest Figure 17: Shawnee Academic Progress in MAT 4 Number of students whose placement level changed from pretest to post-test 6 4 3 4 4 1 Number of students whose raw test score changed from pre-test to post-test in Reading Number of students whose placement level changed from pre-test to post-test in Reading 1 1 Number of students whose raw test scores changed from pre-test to post-test in Writing Figure 18: Shawnee Academic Progress in Basic Writing I Number of students whose placement level changed from pre-test to post-test in Writing INCREASED NO CHANGE DECREASED Office of Community College Research and Leadership 1
18 16 14 1 1 8 6 4 7 3 1 Number of students whose raw test score changed from pre-test to post-test in Reading Number of students whose placement level changed from pre-test to post-test in Reading 1 3 Number of students whose raw test score changed from pre-test to post-test in Writing 7 9 Number of students whose placement level changed from pre-test to post-test in Writing INCREASED NO CHANGE DECREASED Figure 19: Shawnee Academic Progress in Basic Writing II Discussion Recruitment and Retention Results: Shawnee had 3 participating students in the interventions. In Basic Writing I and in the math intervention, less than half the students finished, but most of the students (7%) participating in Basic Writing II completed the intervention. Reducing Remediation Results: With respect to academic gains in math, students advanced in placement while declined. With respect to academic gains in reading and writing, results for Shawnee were more favorable than for math. Shawnee successfully reduced remediation in both reading and writing in both Basic Writing I and Basic Writing II interventions. Office of Community College Research and Leadership
SOUTH SUBURBAN COLLEGE (SSC) Interventions: Algebra 9, Algebra 1, Writers Workshop Intervention description: Developmental mathematics courses and writing workshops (Writers Workshop a two-day intervention) Number of students participating: Total number of students in full intervention: Math 9: 14 Algebra 1: 8 Writers Workshop: 4 (not full intervention) Number of students completing at least one full-scale intervention: 1 Average completion rate of full interventions: 69.7% Overall rate of students completing at least one intervention: 68.% Placement Instruments: South Suburban administered pre-test and post-test measures using My Foundations Lab computer programs; this program is not designed for placement decisions Criteria for success in interventions: Passing grade in course Number of interventions completed with a passing grade: 31 3 1 16 Completing WITH Passing Grades Completing WITHOUT Passing Grades 1 9 Not Completing 6 8 Algebra 9 Algebra 1 Writers Workshop Figure : SSC Completion by Intervention Office of Community College Research and Leadership 3
1 1 8 6 4 1 INCREASED NO CHANGE DECREASED 7 Number of students whose raw test score changed from pre-test to posttest in Algebra 9 Number of students whose raw test score changed from pre-test to posttest in Algebra 1 Figure 1: SSC Academic Progress in Algebra 9 and Algebra 1 Discussion Recruitment and Retention Results: Twenty-two students participated in summer mathematics course interventions, and 4 students participated in the Writer s Workshops. These workshops lasted two days and did not include a pre-test and post-test examination. In order to eventually understand, quantitatively, if SSC s CCR program is successful, it would be helpful for all participating students to be assessed in the future. Office of Community College Research and Leadership 4
SOUTHWESTERN ILLINOIS COLLEGE (SWIC) Interventions: Math 94, Math 97, English 9 Intervention description: Developmental mathematics and English courses Number of students participating: Total: 8 Math 94: 47 Math 97: 31 English 9: 11 Number of students completing at least one intervention: 64 Overall rate of students completing at least one intervention: 78.% Average completion rate of interventions: 7.4% Placement Instruments: COMPASS Criteria for success in interventions: Passing grade in course Number of interventions completed with a passing grade: 8 4 4 3 3 7 Completing WITH Passing Grades 1 8 3 Completing WITHOUT Passing Grades Not Completing 1 1 8 4 1 4 MAT 94 MAT 97 ENG 9 Figure : SWIC Completion by Intervention Office of Community College Research and Leadership
4 3 3 1 1 33 1 Number of students whose raw test score changed from pre-test to post-test on Departmental Final 4 Figure 3: SWIC Academic Progress in MAT 94 4 Number of students whose raw test score changed from pre-test to post-test on COMPASS 17 14 Number of students whose placement level changed from pre-test to post-test INCREASED NO CHANGE DECREASED 3 11 1 1 7 4 14 INCREASED NO CHANGE DECREASED Number of students whose raw test score changed from pre-test to post-test on Departmental Final 3 Number of students whose raw test score changed from pre-test to post-test on COMPASS Figure 4: SWIC Academic Progress in MAT 97 Number of students whose placement level changed from pre-test to post-test Office of Community College Research and Leadership 6
9 8 7 6 4 3 1 3 INCREASED NO CHANGE DECREASED Number of students whose raw test score changed from pre-test to posttest in Reading Figure : SWIC Academic Progress in ENG 9 Discussion Recruitment and Retention Results: SWIC attracted 8 students to the academic interventions associated with its CCR program, retaining 78% through completion of at least one academic intervention. Reducing Remediation Results: Student performances on post-tests were positive for students in MAT 94 and MAT 97, though negative for students in the ENG 9 intervention. Performance on the departmental final in mathematics was overwhelmingly positive, with only one student s performance decreasing from pre-test to post-test. The number of students whose placement level changed was also predominantly positive, 9 increasing and decreasing in MAT 94 and 11 increasing and decreasing in MAT 97. Office of Community College Research and Leadership 7
DISCUSSION This report is a summary of the quantitative data collected on the academic interventions offered by the seven CCR sites for year four (1-11) of the CCR Act Evaluation. It is the hope of the authors that this report serves as a mechanism to facilitate conversations between and within the CCR sites and the state s Illinois Community College Board (ICCB) as to how to approach the challenges of implementing and evaluating CCR programs. This evaluation report and all previous reports pertaining to the CCR evaluation document the diversity of approaches that the seven different sites take to implementing their CCR programs, contributing to differences in student recruitment, retention and remedial readiness across the programs. With respect to recruitment and retention in the academic interventions, some CCR sites demonstrated a greater capacity for engaging students to participate in CCR interventions than other sites. For example, JALC (177 students) and KCC (149 students) both induced a large number of students to participate in their academic interventions. Both of these sites offered academic interventions during the school year, and they offered the interventions in partner high schools. This reduced the need for students to negotiate travel logistics and granted them easier access to the academic reforms. Though not definitive, these results suggest higher levels of participation may be linked to offering CCR interventions in high schools rather than on community college campuses. At the other end of the enrollment spectrum, sites that offered their academic intervention mostly or only during the summer and only on the community college campus had the fewest number of students participating in the academic interventions in all pilot sites. A challenging aspect of this evaluation is the lack of standardization in CCR program implementation. Comparing student participation and completion across the seven sites is difficult to accomplish due to regional and programmatic differences between sites. Also, while five sites used a placement exam to measure gains from pre-test to post-test (COMPASS and ACCUPLACER); two sites did not use a placement exams which made cross-site comparisons difficult. Whereas most sites used the COMPASS exam, the content of this exam is determined at the campus level, and it differs from site to site. To measure academic gains, including reduction in remediation, a uniform assessment needs to be implemented by all the CCR sites but a common assessment instrument is not required to implement CCR. Despite these limitation, this section attempts to focus on the success that some sites had in engaging, retaining, and advancing students into credit bearing college coursework. With the exception of SWIC that had 78% of students completing at least one academic intervention, the sites fell into two categories on completion. In the first category, the three sites of MVCC (9.6%), JALC (9.%), and CLC (98.%) had nearly perfect rates of students completing at least one academic intervention, and in the second category, SSC (68.%), Shawnee (69.8%), and KCC (69.8%) retained about two of every three students through to completion of the academic intervention(s). Besides the differences in placement exams (mentioned above), factors that account for the differences between sites include the content and rigor of instruction, the characteristics of students participating in the programs, or something else. Despite difficulties in making comparisons across the sites, discussions among the site personnel on how best to retain students through course completion and the various strategies used to recruit and retain students, especially at sites with near perfect completion rates, would seem to be valuable. Variation is also evident in results pertaining to the academic progress of Office of Community College Research and Leadership 8
students in the different sites. For the purposes of this report, successful academic progress refers to either an increase in individual student score from pre-test to post-test or a gain in placement level from pre-test to post-test. Besides difficulties with cross-site comparisons, it is possible that students experienced test fatigue which affected their motivation to perform on the post-test. These concerns aside, some sites demonstrated more success in reducing remediation needs for participating students than other sites. For example, SWIC s MATH 94 course revealed 17 of 33 students placed into a higher math course on the post-test, and 11 of 7 students in SWIC s MATH 97 placed into a higher math course on the post-test exam. In these courses, very few students taking MATH 94 ( of 33) and MATH 97 ( of 7) placed into a lower math class on the post-test exam. In contrast, only 9 of 44 students placed into a higher English class on the post-test exam in CLC s English 19 class, while 14 of 44 students placed into a lower English class on the post-test exam. Despite the challenges to gathering meaningful results on academic achievement, some patterns emerged. At all of the CCR sites, most students placed into the same level course on the pre-test and post-test, and at every site a modest percentage of students placed into a higher level or lower level course on the post-test exam. When looking at the raw scores on placement exams at KCC, MVCC, Shawnee, and SWIC, more students scored higher on the post-test than on the pretest, suggesting the students demonstrated some academic progress but often less progress than necessary to reduce remediation. Whereas the post-test scores of some sites (for example, SSC and JALC) were higher on the post-test exam than on the pre-test, missing data and other concerns with data quality raised questions about whether these differences are functions of the CCR programs, the assessments, or some other dimension of performance that was not measured. This quantitative evaluation report represents a big step forward in understanding the challenges and successes that the seven sites experienced in implementing academic interventions funded by Illinois CCR Act. By expanding understanding of the CCR programs, how students participate and perform, and how the programs can improve to help the state make additional strides to improve college and career readiness. REFERENCES Bragg, D., Baber, L., Cullen, D., Reese, G., & Linick, M. (11). Evaluation of Illinois college and career readiness (CCR) pilot projects year four (1-11): Summary of results. Champaign, IL: Office of Community College Research and Leadership, University of Illinois at Urbana-Champaign. Retrieved from http://occrl.illinois.edu/files/projects/ccr/report/ccr_year_four_report.pdf Office of Community College Research and Leadership 9
APPENDIX A: CCR LEGISLATION Illinois Compiled Statutes: 11 ILCS 8/- This is the way to refer to the current legislation that is in effect. The Public Acts are how a law is identified after it has been a bill (HB followed by number for House Bill; SB followed by number for Senate Bill) and signed into law (PA followed by number), but then it usually will go into the ILCS, Illinois Compiled Statutes. Now that this law is no longer a bill (so not appropriate to refer to as SB or HB ) and is part of the statute, it should be referred to by its ILCS number rather than a Public Act or PA The College and Career Readiness Pilot Program statute, with the text having incorporated the initial legislation and following amendments, can be found at: http://www.ilga.gov/legislation/ilcs/fulltext.asp?docname=118k- The history of the legislation is: Public Acts: P.A. 9-694, eff. 11--7; 9-876, eff. 8-1-8; 96-13, eff. 7-6-1. Full text of the law Full text, from Illinois Complied Statutes, of the legislation authorizing the College and Career Readiness Pilot Program. (11 ILCS 8/-) Sec. -. College and Career Readiness Pilot Program. (a) The General Assembly finds that there is a direct and significant link between academic preparation of students and success in postsecondary education and careers. Many students enter college unprepared for the academic rigors of college and require noncredit remedial courses to attain skills and knowledge needed for regular, credit coursework. Remediation lengthens time to degree, imposes additional costs on students and colleges, and uses student financial aid for courses that will not count toward a degree. All students entering college take a college entrance exam or a placement test. These tests can be used to assist high school students to identify areas for improvement and help to close skill gaps during students' senior year. College and career readiness reduces the need for remediation, lowers educational costs, shortens time to degree, and increases the overall success rate of Illinois college students. (b) Subject to appropriation, the State Board shall create a pilot project, to be known as the College and Career Readiness Pilot Program. Subject to appropriation, on July 1, 1, the State Board shall extend the current program for an additional 3 Office of Community College Research and Leadership 3
years and include an additional 7 sites (or as many as are allowed by available funding), as evidenced by the effectiveness of the current program. If in any of these 3 additional years, money is not appropriated for the program, then the State Board shall extend the program for an additional year. The goals of the program are as follows: (1) To diagnose college readiness by developing a system that aligns ACT scores or college placement examinations to specific community college courses in developmental and freshman curriculums. () To reduce remediation by decreasing the need for remedial coursework in mathematics, reading, and writing at the college level through (i) increasing the number of students enrolled in a college-prep core curriculum, (ii) assisting students in improving college readiness skills, and (iii) increasing successful student transitions into postsecondary education. (3) To align high school and college curriculums. (4) To provide resources and academic support to students to enrich the junior and senior year of high school through remedial or advanced coursework and other interventions. () To develop an appropriate evaluation process to measure the effectiveness of readiness intervention strategies. (c) The first year of the program extended under this Section by this amendatory Act of the 96th General Assembly shall begin with the high school class of 11 and the high school class of 1 (or such later classes if money is not appropriated for the program in a given fiscal year). (1) In addition to the community colleges participating in the program before July 1, 1, the State Board shall select 7 additional community colleges (or as many as are allowable by available funding) to participate in the program based on all of the following: (A) The percentage of students in developmental coursework. (B) Demographics of student enrollment, including socioeconomic status, race and ethnicity, and enrollments of first-generation college students. (C) Geographic diversity. (D) The ability of the community college to partner with local high schools to develop college and career readiness strategies and college readiness teams. () Each participating community college shall establish an agreement with a high school or schools to do all of the following: (A) Create a data-sharing agreement. Office of Community College Research and Leadership 31