COLLEGE OF THE SEQUOIAS ENGLISH ASSESSMENT & PLACEMENT PROGRAM EVALUATION CONSEQUENTIAL VALIDITY AND DIPROPORTIONATE IMPACT (ACCUPLACER CPT READING) (ACCUPLACER CPT SENTENCE SKILLS) Greg Keen Assessment and Articulation Coordinator Ryan Barry-Souza Research Analyst September 30, 2015
Brief background: College of the Sequoias implemented the ACCUPLACER Reading and Sentence Skills tests for English Placement in March 2002. Retroactive testing and development of a multiple criteria placement model occurred between August 2001 and January 2002. Initial cut-scores were intentionally set relatively low and a first year follow-up validation study was conducted between August 2002 and January 2003. Cut-scores were subsequently raised to improve the utility of the placement rules. Between August 2004 and January 2005 the college participated in a statewide validation research project to maintain ACCUPLACER on the California Community College Chancellor s Office list of approved assessment instruments. This research effort included 23 California Community Colleges and utilized consequential validity methods to evaluate the utility of the instruments. Consequential validity requires both faculty and students to be polled regarding their judgment of readiness for the courses in which the students were placed. The results of this research were very favorable for both COS and the consortium at large. The college conducted another consequential validation study in 2008 with the intent to determine the appropriateness of implementing mandatory placement for ENGL 251 (one level below freshman composition). The results were again positive and mandatory placement for ENGL 251 was implemented FALL 2009. The college attempts to adhere to the Chancellor s Office six year review cycle and the English Assessment & Placement process was last reviewed in 2008. This review was scheduled for FALL 2014; however, due to conflicting priorities this review was postponed for one semester. Going Forward: The California Community College Student Success and Support Program (SSSP) is in the final stages of the development of a Common Assessment for placement in math, English, and ESL courses. The Common Assessment will be piloted at 10 CCCs in 2015-16 and it will be gradually rolled out to all CCC over the following few years. Colleges that use a test as a component of their assessment process will be required to use the Common Assessment in order to continue to qualify for SSSP funding. If the Common Assessment becomes a reality before 2018, no additional validation research will be necessary to maintain our current assessment instruments. The college instead will begin the process of reviewing multiple measures and cut-scores for the new common assessments and placement schemes for the math, English, and ESL curricula. Test Instruments, Multiple Measures, & Placement Rules: All students included in this study completed the ACCUPLACER Reading and Sentence Skills tests. Each student also completed a questionnaire including self-reported high school GPA, and grade in last English course. Mandatory placement with standardized override procedures is
enforced for both ENGL 1 and ENGL 251 (i.e., enrollment in ENGL 1 or ENGL 251 requires eligibility determined by the placement procedures, standardized override criteria, or completion of prerequisite courses). The placement procedures including multiple measures and placement rules are described below. ACCUPLACER READING & SENTENCE SKILLS TESTS: Writing course placements are determined by either a high ACCUPLACER CPT Sentence Skills score or by the following English Composite Score: CPTCOMP = CPT Reading Score + CPT Sentence Structure (SS) Score + 5 points if grade in last English class is an A or 3 points if grade in last English class is a B. CPT SS 095-120 ENGL 1 or CPTCOMP 165-245 CPT SS 066-094 ENGL 251 or CPTCOMP 126-164 CPT SS 050-065 ENGL 360 or CPTCOMP 080-125 CPT SS 000-049 See Counselor or take ESL Test or CPTCOMP 000-079 Consequential Validity: During the seventh and eighth week of the Spring 2015 semester satisfaction with course placement data was collected for both students and faculty in all sections of ENGL 1 (College Reading and Composition), 251 (Introduction to Academic Reading and Writing), and 360 (Reading and Writing Skills). An internally designed placement survey form was utilized to administer the survey to both the students and the faculty. The overall response rate was 45%. Data were closely scrutinized to ensure only students who placed in the courses through the multiple criteria placement model including the ACCUPLACER Reading and Sentence Skills test scores were included in the study. The summary of consequential validation evidence is presented below in Table 1. The perception of the English faculty and their students indicates the multiple criteria placement model including ACCUPLACER Reading and Sentence Skills test scores appropriately placed between 78% and 92% of students into the three levels of writing courses. These results exceed the 75% standard; therefore indicate an appropriate level of consequential validation.
Student Reponses Table 1: Student and Faculty Perception of Correct Placement by Course ENGL 1 ENGL 251 ENGL 360 N = 161 N = 178 N = 84 JUST RIGHT 92% 88% 92% TOO DIFFICULT 5% 4% 2% TOO EASY 3% 8% 6% Faculty Reponses N = 161 N = 178 N = 84 JUST RIGHT 90% 78% 92% TOO DIFFICULT 9% 16% 6% TOO EASY 1% 6% 2% The percentage of both students and faculty indicating appropriate placement is very high for both ENGL 1 and ENGL 360. The percentage of students indicating appropriate placement in ENGL 251 is also high; however, the percentage of faculty indicating appropriate placement in ENGL 251 is considerably lower but meets the minimum 75% criteria necessary to establish consequential validity. Disproportionate Impact: Disproportionate impact evaluates the rate of placement of impacted groups into specific courses. When courses represent a sequence, then the placement rate evaluation into the upper level courses is most appropriate because typically most students want to be in the upper level courses. This study was conducted to answer the following question: Does the rate of placement in ENGL 1, and ENGL 251, based on the ACCUPLACER Reading and Sentence Skills tests in conjunction with multiple measures differ significantly for different groups of students? The effects of student gender, ethnicity, and age on course placement was investigated. The majority (or dominant) placement rates formed the benchmarks for this study and include: 1) males for gender, 2) whites for ethnicity, and 3) young (19 and under) for age.
This study included all students who completed the ACCUPLACER Reading and Sentence Skills tests between September 2012 and August 2015. A total of 17,091 students were assessed for initial placement into ENGL 1, ENGL 251, or ENGL 360 during this timeframe. The analysis begins by determining the percent of students in an impacted group (gender, ethnicity, age) recommended into each course over three year period. Evaluation is accomplished by dividing the minority percent placement rate (black, Hispanic, female, etc.) by the majority percent placement rate (white, male, etc.). If a ratio is less than 80% then there is evidence of disproportionate impact. For example, if 40 of 100 white students place into ENGL 1 and only 30 of 100 Hispanic students place into ENGL 1, then the placement rates are 40% and 30%. Dividing 30 by 40 gives a placement ratio of 75% which is below 80%, and would provide evidence that disproportionate impact has occurred. A. Gender: No disproportionate impact was identified on students gender. Table 2: DI Evaluation for Gender and ENGL 1 ENGL 1 Gender # Tested # Eligible Placement Rate Placement Ratio Females 8915 3246 36.4% 36.4/36.7 = 99% Males 8045 2956 36.7% Benchmark Rate Table 3: DI Evaluation for Gender and ENGL 251 ENGL 251 Gender # Tested # Eligible Placement Rate Placement Ratio Females 8915 3757 42.1% 42.1/38.7 = 1.08% Males 8045 3112 38.7% Benchmark Rate
B. Ethnicity: As illustrated in Table 4, it appears there is disproportionate impact for all ethnic groups for placement into ENGL 1. Table 5 clearly indicates that disproportionate impact does not exist for any ethnic group for placement into ENGL 251. Table 4: DI Evaluation for Ethnicity and ENGL 1 ENGL 1 Ethnicity # Tested # Eligible Placement Rate Placement Ratio American Indian/Alaskan Native 203 72 35.5% 35.5/53.2 = 66.7% Asian 449 167 37.2% 37.2/53.2 = 69.9% Black (Non-Hispanic) 728 198 27.2% 27.2/53.2 = 51.1% White (Non-Hispanic) (25% of students 4,189 2229 53.2% Dominate tested) Benchmark Rate Hispanic (60% of students tested) 10,277 3079 30.0% 30/53.2 = 56.4 % Other/Unknown 1,008 407 40.4% 40.4/53.2 = 75.9% Table 5: DI Evaluation for Ethnicity and ENGL 251 ENGL 251 Ethnicity # Tested # Eligible Placement Rate Placement Ratio American Indian/Alaskan Native 203 44.3% 44.3% 44.3/32.3 = 1.37% Asian 449 37.4% 37.4% 37.4/32.3 = 1.16% Black (Non-Hispanic) 728 294 40.4% 40.4/32.3 = 1.25% White (Non-Hispanic) 4,189 1353 32.3% Dominate Benchmark Rate Hispanic 10,277 4561 44.4% 44.4/32.3 = 1.37% Other/Unknown 1,008 383 38.0% 38.0/32.3 = 1.18%
The disproportionate impact, as calculated per the Chancellor s Office model, observed in all ethnic groups for ENGL 1 is the result of the relative high placement rate for whites verses all other groups. The only way to resolve this inequity is to raise the placement rate for ENGL 1 for all groups other than the benchmark group (whites). Raising the placement rate for all groups, including the benchmark group, may benefit all students in general but would result in continued disproportionate impact. This issue needs to be addressed; however, with the inevitable implementation of the statewide Common Assessment in the near future the college may want to postpone further inquiry at this time. All ethnic groups were represented in the Consequential Validation research conducted Spring Semester 2015; however, due to relatively small sample sizes the students were grouped into 1) White Non-Hispanic, 2) Hispanic, and 3) Others. As illustrated in Table 6, students in the three ethnic groups reported between 88% and 94% placement in ENGL 1 as Just Right. Faculty reported between 87% and 96% placement in ENGL 1 as Just Right. In conclusion, while placement data indicates potential disproportionate impact for all ethnic groups for ENGL 1, this detailed analysis of faculty and student opinions about course placement revealed substantial support that the process places students at levels deemed appropriate by both faculty and students. Table 6: Student & Faculty Perceptions of ENGL 1 Placement by Ethnicity Students Faculty Ethnicity Responses Total N Percent Total N Percent White Non- Hispanic Total 50 50 Just Right 44 88% 48 96% Too Difficult 3 6% 1 2% Too Easy 3 6% 1 2% Hispanic Total 86 86 Just Right 79 92% 75 87% Too Difficult 5 6% 11 13% Too Easy 2 2% All Others Total 32 32 Just Right 30 94% 29 90.6% Too Difficult 1 3% 3 9.4% Too Easy 1 3%
Age: No disproportionate impact was identified on students age for ENGL 1. And, the only age category that revealed a placement ratio lower than 80% for ENGL 251 was the 50 and older group. Since this age group, along with all other age groups revealed a placement ratio higher than 80% for ENGL 1 we can conclude that no disproportionate impact was identified for age in the English course sequence. Table 6: DI Evaluation for Age and ENGL 1 ENGL 1 Ethnicity # Tested # Eligible Placement Rate Placement Ratio 19 years & Younger 12,365 4856 39.3% Dominate Benchmark Rate 20-24 years 2252 792 35.2% 35.2/39.3 = 90.0% 25-29 years 1003 372 37.1% 37.1/39.3 = 94.4% 30-39 years 895 315 35.2% 35.2/39.3 = 90.0% 40-49 years 386 126 32.6 32.6/39.3 = 83.0% 50+ years 179 63 35.2 32.2/39.3 = 90.0% Table 6: DI Evaluation for Age and ENGL 251 ENGL 251 Ethnicity # Tested # Eligible Placement Rate Placement Ratio 19 years & Younger 12,365 5346 43.2% Dominate Benchmark Rate 20-24 years 2252 883 39.2% 39.2/43.2 = 90.7% 25-29 years 1003 374 37.3% 37.3/43.2 = 86.3% 30-39 years 895 329 36.8 36.8/43.2 = 85.8% 40-49 years 386 137 35.5 35.5/43.2 = 82.2% 50+ years 179 48 26.8 26.8/43.2 = 62.0%