CALIFORNIA CHEMISTRY DIAGNOSTIC TEST

Transcription

1 SANTA MONICA COLLEGE CALIFORNIA CHEMISTRY DIAGNOSTIC TEST Request for Approval of Renewal Hannah Alford, Director, Institutional Research Spring 2012

2 Introduction The current report provides empirical evidence to support the renewal of the California Chemistry Diagnostic Test (CCDT) as a valid instrument to challenge the course prerequisite for Chemistry 11 (General Chemistry 1) at Santa Monica College. Based on the guidelines governing the full approval process set by the California Community College Chancellor s Office (CCCCO), the current report document the evidence supporting the following types of validity/evidence: Chapter 1: Content Validity Content validity measures the extent to which the CCDT items are related to the curricula of the courses into which students are placed by the test (Chemistry 10 and Chemistry 11). The findings of the study demonstrate that CCDT is valid in terms of content. Chapter 2: Cut Score Validity Cut score validation measure the extent to which the established cutoff score of the test for a pass is appropriate. Data measuring criterion-related and consequential-related validity suggest that the cutoff score of 22 is too low. With the current cutoff score, 75% of students are placed into Chemistry 11. Results from an assessment activity measuring faculty judgment on the CCDT items and a department-wide discussion were used to establish a new cut-score. The findings of the study recommend raising the current cutoff score to 26. Chapter 3: Disproportionate Impact Disproportionate impact evaluates whether the CCDT has adverse impact on different groups by gender, ethnicity/race, and age. The findings of the study reveal that the test may have a disproportionate impact on Black and Hispanic students. Furthermore, raising the cut-score to 26 will disproportionately impact female, Black, and Hispanic students. The report discusses steps the college will take to address the adverse impact of the CCDT on these groups. Because the CCDT has been approved by the CCCCO as a commercially available second-party published test, the three studies (content validity, cut-score validity, and disproportionate impact) were conducted locally. Test bias and reliability was not assessed in the local study. The report begins by providing a demographic profile of Santa Monica College, overview of use of the CCDT at Santa Monica College, and description of the chemistry course sequence. The procedures and findings of the validation assessments are discussed by type. The report concludes with a signed CCCCO Request for Approval form. 2

3 Demographic Profile of Santa Monica College Established in 1929, Santa Monica College serves a diverse population of over 50,000 students annually, including a large population of international students and historically underrepresented groups. The college has one of the strongest reputations for transfer in California and transfers the highest number of students to the University of California, University of Southern California, and other four-year institutions. While a majority of students indicate a career or degree educational goal (84.7%), many are not prepared for the rigors of college-level work. Approximately one-third of students attend full-time. Table 1 describes the demographic and background characteristics of credit students enrolled at the college in fall Table 1. Demographic Profile Fall 2011 (N = 29,977) Characteristic % Gender Ethnicity/Race* Age Group Residence Status Female 54.0 Male 46.0 Asian/Pacific Islander 18.5 Black 9.8 Hispanic 33.7 Multi-Ethnicities 3.5 White and younger to to Over California 83.5 Out of State 5.9 International 10.7 Unit Load Part-time (0.5 to 11.5 units) 62.1 Full-time (12+ units)

4 Overview of the Use of CCDT at Santa Monica College The California Chemistry Diagnostic Test (CCDT) is a timed, paper-pencil challenge exam assessing students skills and knowledge in introductory chemistry concepts. Students who wish to enroll directly into Chemistry 11 (General Chemistry 1) and bypass Chemistry 10 (Introductory General Chemistry) take the CCDT to challenge the Chemistry 10 prerequisite. The test consists of 44 question items related to high school chemistry and high school mathematics. Topics and skills tested include: Math: solving simple algebraic equations, use of proportions and percentages, solving word problems; Chemistry: elements and their properties, chemical formulas, concentrations of solutions, gases, oxidation numbers, redox reactions, and the concepts of acids and bases. Students taking the CCDT are allowed 50 minutes to complete the test. The number of students taking the CCDT to challenge the course prerequisite for Chemistry 11 has been relatively small compared to the total course enrollment in the courses (average 770 course enrollments annually between 2007 and 2010 in Chemistry 11). On average, SMC administers nearly 202 CDDT tests for 193 unique students in a calendar year. Students are able to retest if they have not enrolled in Chemistry 10 or Chemistry 11 in the past three years (see Table 2). Table 2. CCDT Testing Events and Unique Students, Average/Yr Testing events Students The current cutoff score for passing the test is 22 points (of maximum 44). Approximately 75% of testing events received passing scores on the CCDT between calendar years ; students with passing scores were directly placed into Chemistry 11. Table 3 presents descriptive data for CCDT scores. Students with non-passing scores had an average score of 17, approximately 5 points below the cutoff. Students with passing scores averaged a score of 31, approximately 9 points above the cutoff. The mode, or most frequently reported score, of 21 is only 1 point below the cutoff; this finding suggests that a lot of students are at the cusp of passing the CCDT. Table 3. CCDT Descriptive Statistics Testing Events by Outcome, Outcome N Percent Mean Median Mode SD Non-passing score Pass score (22 points+) Total

5 Chemistry Course Sequence A comprehensive chemistry curriculum is offered at Santa Monica College, and includes introductory general chemistry (Chemistry 10), general chemistry (Chemistry 11 & 12), organic chemistry (Chemistry 21, 22, & 24), and biochemistry (Chemistry 31). With the exception of Chemistry 10, all other courses have validated prerequisites in place. Students may gain access to each of these courses by meeting the appropriate prerequisite with coursework completed at SMC or at another college. Students seeking entrance directly into the first semester of general chemistry (Chemistry 11) must demonstrate the knowledge and skills necessary for success in the course by obtaining a passing score on the CCDT. Students who do not achieve the minimum passing score on the challenge exam are directed to enroll in Chemistry 10. In addition, students are able to enroll in Chemistry 10 without taking the CCDT. 5

6 Chapter 1: Content Validity In order to determine whether the content of the CCDT adequately align with the prerequisite skills/knowledge necessary for success in Chemistry 11 and the exit skills/knowledge of successful Chemistry 10 students, two studies were conducted. Seven faculty members responsible for teaching the courses for which the CCDT is used to place students (Chemistry 10, Chemistry 11) participated in the content validation activities during the spring 2010 term. The group of faculty included four full-time and three adjunct instructors (three women, four men; four White, two Black, one Asian faculty). Match between CCDT Topics and CHEM10 Exit/CHEM 11 Entry Skills The first content validation study asked instructors to rate, using a binary response of yes or no, whether there is a match between the knowledge and skills purported to be measured by the CCDT (as defined in the manual) and the prerequisite knowledge and skills for Chemistry 11 and exit knowledge and skills for Chemistry 10, the prerequisite course for Chemistry 11. Instructors completed two rating sheets, one for each course. Instructors were asked to engage in the rating activity independently; however, a group discussion around the findings occurred after the independent rating activity. In preparation for their review, instructors were provided with a numbered list of entry and exit skills for both courses, an alpha numbered list of topics purported to be tested in the CCDT, and a rating sheet. Alignment between CCDT measured skills and exit/entry skills in Chemistry 10/11 were assessed in response to the following question: The column values in the table represent the knowledge and skills measured by the CCDT. The row values represent the knowledge and skills for Chemistry 10/Chemistry 11 (extracted from the official SMC course outline of record.) Please rate the extent to which you believe the knowledge and skills measured by the CCDT match the Chemistry 10/Chemistry 11 course exit/entry skills. Mark an X in cells when there is a match between the knowledge and skills measured by CCDT (column) and the Chemistry exit/entry skill (row). Leave cells blanks if you believe there is no match between the knowledge and skills measured by the CCDT (column) and the course exit/entry skills (row). The topics purported to be tested in the CCDT are listed below: A. Compounds and elements (compounds and chemical formulas) B. States of matter (gases, solids, liquids) C. Reactions of matter (calculations with chemical formulas and equations) D. Structure of Matter (atomic nature of matter, atomic structure, molecular and ionic geometry, chemical bonding) E. Periodic properties (periodic table introduction and trends) F. Solutions (concentrations and stoichiometry) G. Dynamics (qualitative kinetics and thermodynamics) H. Lab skills I. Mathematical skills 6

7 The Chemistry 10 and Chemistry 11 entry/exit skills as defined by the official course outline of records is described in Appendix A. Chemistry 10 has 21 course exit skills. Chemistry 11 has 27 course entry skills. Chemistry 10 Results The following table describes the percentages of faculty raters (out of seven) who identified a match between the CCDT skills (column) and the Chemistry 10 exits skills (row). The asterisk (*) identifies ratings with a majority consensus (more than 50%) of faculty raters identifying a match between Chemistry 10 exit skill and CCDT topic. The column to the far right describes, by Chemistry 10 exit skill, the CCDT skills/topic addressing the exit skill (only the ratings with >50% of faculty agreement response of yes ). Table 4. Content Analysis Findings Percentage of Faculty Raters Who Found Match between CCDT Topic and Chemistry 10 Exit Skills CHEM 10 Exit Skills CCDT Skills Measured (Topic) A B C D E F G H I Exit Skills Measured by CCDT Topic 1 100%* 29% 29% 14% 0% 29% 0% 71%* 0% A, H 2 86%* 0% 0% 29% 0% 0% 0% 29% 0% A 3 0% 0% 71%* 0% 0% 71%* 14% 14% 71%* C, F, I 4 29% 43% 86%* 0% 0% 71%* 0% 0% 71%* C, F, I 5 0% 0% 43% 0% 0% 29% 0% 43% 86%* I 6 14% 100%* 14% 0% 14% 0% 0% 0% 14% B 7 0% 0% 29% 0% 14% 0% 0% 57%* 86%* H, I 8 14% 14% 0% 71%* 86%* 0% 0% 0% 0% D, E 9 0% 0% 0% 100%* 57%* 0% 0% 0% 14% D, E 10 14% 0% 0% 100%* 29% 0% 0% 0% 14% D 11 29% 0% 0% 14% 0% 71% 14% 14% 57%* I 12 0% 14% 0% 0% 0% 0% 100%* 29% 29% G 13 14% 14% 14% 14% 14% 14% 14% 29% 14% None 14 14% 14% 100%* 0% 0% 71% 0% 14% 43% C 15 0% 14% 14% 0% 0% 0% 0% 86%* 14% H 16 0% 0% 0% 0% 0% 0% 0% 86%* 14% H 17 29% 0% 0% 0% 0% 14% 0% 86%* 14% H 18 0% 0% 0% 0% 0% 0% 0% 86%* 14% H 19 0% 0% 0% 0% 0% 0% 0% 86%* 14% H 20 0% 0% 0% 0% 0% 0% 0% 86%* 14% H 21 0% 0% 14% 0% 0% 0% 0% 86%* 86%* H, I The findings suggest that all but 1 of the 21 Chemistry 10 exit skills is measured by the CCDT topics described in the test manual. In addition, all knowledge/skills measured by the CCDT aligned with at least one Chemistry 10 exit skill. Chemistry 10 exit skill #13, Demonstrate a basic understanding of the scientific method was not found to match any of the knowledge and skills measured by the CCDT. However, during the discussion of the content validation, faculty members concluded that the scientific method was taught and embedded in all Chemistry courses, therefore, there was sufficient evidence to suggest that there is a content match between the exit skills in Chemistry 10 and the CCDT knowledge/skills, even without the CCDT directly measuring Chemistry 10 exit skill #13. 7

8 Chemistry 11 Results Table 5 describes the percentages of faculty raters (out of seven) who identified a match between the CCDT skills (column) and the Chemistry 11 entry skills (row). The asterisk (*) identifies ratings with a majority consensus (more than 50%) of faculty raters identifying a match between Chemistry 11 prerequisite skill and CCDT topic. The column to the far right describes, by Chemistry 11 entry skill, the CCDT skills/topic addressing the prerequisite skill (only the ratings with >50% of faculty agreement response of yes ). Table 5. Content Analysis Findings Percentage of Faculty Raters Who Found Match between CCDT Topic and Chemistry 11 Entry Skills CHEM 11 Entry Skills CCDT Skills Measured (Topic) A B C D E F G H I Entry Skills Measured by CCDT Topic 1 0% 0% 14% 0% 0% 43% 29% 14% 100%* I 2 0% 0% 0% 0% 0% 29% 43% 14% 100%* I 3 0% 0% 0% 0% 0% 0% 0% 14% 100%* I 4 0% 0% 0% 0% 0% 43% 14% 14% 100%* I 5 0% 0% 14% 0% 0% 14% 14% 14% 100%* I 6 0% 0% 0% 0% 0% 0% 14% 14% 100%* I 7 0% 0% 14% 0% 0% 14% 29% 0% 100%* I 8 0% 14% 0% 0% 14% 0% 14% 14% 100%* I 9 0% 0% 0% 0% 0% 0% 0% 14% 100%* I 10 0% 14% 0% 0% 14% 0% 29% 14% 86%* I 11 0% 0% 0% 0% 0% 0% 0% 14% 100%* I 12 0% 0% 0% 0% 0% 0% 0% 29% 100%* I 13 0% 0% 0% 0% 0% 14% 0% 14% 100%* I 14 14% 0% 14% 0% 0% 29% 0% 14% 100%* I 15 86%* 0% 57%* 0% 14% 14% 0% 57%* 0% A, C, H 16 71%* 0% 14% 14% 0% 0% 0% 43% 0% A 17 14% 14% 71%* 0% 0% 71%* 29% 29% 100%* C, F, I 18 29% 57%* 100%* 0% 0% 43% 0% 14% 43% B, C 19 0% 0% 43% 0% 0% 43% 14% 43% 86%* I 20 0% 0% 14% 0% 0% 0% 0% 43% 100%* I 21 0% 14% 0% 100%* 14% 0% 0% 0% 0% D 22 29% 29% 29% 29% 29% 29% 29% 57%* 29% H 23 14% 14% 86%* 0% 0% 57%* 14% 14% 71%* C, F, I 24 0% 0% 0% 0% 0% 0% 0% 100%* 29% H 25 0% 0% 0% 0% 0% 0% 0% 100%* 0% H 26 14% 0% 0% 0% 0% 0% 0% 100%* 0% H 27 0% 0% 0% 0% 0% 0% 0% 100%* 0% H There was agreement among raters that all of the 27 Chemistry 11 entry skills matched the knowledge/skills measured by the CCDT. All knowledge/skills measured by the CCDT aligned with at least one Chemistry 11 entry skill except: E. Period properties (periodic table introduction and trends) G. Dynamics (qualitative kinetics and thermodynamics) The findings of the first content validation study suggest that the CCDT measures more than the prerequisite skills for Chemistry 11, however, measures all of the exit skills for Chemistry 10. 8

9 Alignment between CCDT Items and Success in Chemistry 10/11 The second content validation study asked instructors to rate the level of importance of each of the 44 items on the CCDT for successful completion of Chemistry 10 or successful acquisition of skills taught in Chemistry 11. Instructors completed two rating sheets, one for each course. Instructors were asked to engage in the rating activity independently; however, a group discussion around the findings occurred after the independent rating activity. In preparation for their review, instructors were provided with a numbered list of entry and exit skills for both courses, a copy of the entire 44-item CCDT instrument, and a rating sheet. Items were rated in response to the following question: To what extent are the academic skills or knowledge measured by the CCDT items important for successful completion of Chemistry 10 (grade C or better)/acquisition of the skills taught in Chemistry 11. A Likert-like scale was used to rate each item: Chemistry 10 Results 5 = Critical 4 = Important 3 = Moderately important 2 = Slightly important 1 = Not relevant at all The mean rating for the entire test was 4.60 (SD = 0.61), suggesting that, overall, the CCDT measured skills and knowledge that were important for successful completion of Chemistry 10. No items on the CCDT were rated an average score below 3 or moderately important. Seven of the items had an average score of 5 or critically important. A majority of the items (35 items) had an average score between 4.0 and 4.9, a rating of important. The remaining two items had an average score between 3.0 and 3.9 which indicate a rating of moderately important for successful completion of Chemistry 10. Table 6 provides descriptive statistics for each item in descending mean order. 9

10 Table 6. CCDT Item-by-Item Analysis for Chemistry 10 Item N Mean SD Min Max Item N Mean SD Min Max

11 Chemistry 11 Results The mean rating for the entire test was 4.81 (SD = 0.42), suggesting that, overall, the CCDT measured skills and knowledge that were important for successful acquisition of skills and knowledge taught in Chemistry 11. No items on the CCDT were rated an average score below 4 or important. Eighteen of the items had an average score of 5 or critically important which indicate that the skills being measured by these items are critical for learning the skills the Chemistry 11. The remaining 26 items had an average score between 4.0 and 4.9, a rating of important. Table 7 provides descriptive statistics for each item in descending mean order. Table 7. CCDT Item-by-Item Analysis for Chemistry 11 Item N Mean SD Min Max Item N Mean SD Min Max Summary of Content Validity Based on the two content validation studies, the skills and knowledge being measured by the CCDT have been deemed to be relevant to the curricula in Chemistry 10 (exit skills) and Chemistry 11 (prerequisite or entry skills). The only problem related to content alignment is related to one Chemistry 10 exit skill; while a student completing Chemistry 10 are expected to understand the scientific method based on the course outline of record, it is not an entry skill for Chemistry

12 Chapter 2: Cut-Score Validity Two studies, a study of criterion-related validity and a study of consequential-related validity, were conducted to evaluate the validity of the current cut score of 22 points (out of maximum 44). Based on the findings of these studies which failed to validate the cut score, a third study assessing the faculty rating of the cut score, was conducted to establish a new cut score. A total of 770 students completed the CCDT between the calendar years 2007 and 2010, however, only 542 or 70.4% enrolled in the appropriate Chemistry course based on their placement score. Table 8 provides descriptive statistics of CCDT scores of students who subsequently enrolled in a chemistry course upon completion of the test. Among students who completed the CCDT more than once during the observed period, the highest score obtained before enrollment in the first chemistry course was recorded. Only the first enrollment in the course was recorded. Based on a cutoff score of 22, approximately 82.3% of students received passing scores on the CCDT between calendar years and were directly placed into Chemistry 11. Students with non-passing scores had an average score of 17, approximately 5 points below the cutoff. Students with passing scores had an average score of 31, approximately 9 points above the cutoff. Table 8. CCDT Descriptive Statistics Testing Events by Outcome, Test Outcome N Percent Mean Median Mode SD Non-passing score Pass score (22 points+) Total Chemistry 10 students were not included in the criterion-related and consequential-related validity analyses as the CCDT functions as a challenge exam and not a placement exam. Therefore, the CCDT only places students in Chemistry 11 and not Chemistry 10. Criterion-Related Validity The CCDT is used to challenge the prerequisite course, Chemistry 10, in order to place directly into Chemistry 11. Therefore, the CCDT s purpose is to predict success in Chemistry 11. Criterion-related validity for the CCDT is defined as the ability of the test to predict Chemistry 11 course performance. Criterion-related validity was assessed by measuring the strength of the relationship between CCDT score and three different criterion measures of student performance related to achievement: a. Course outcome (successful, non-successful); b. Instructor ratings based on evidence observed over the first few weeks of the course of each student s ability to learn the course content or likelihood of success; and c. Mid-term grades over course content. The threshold for an acceptable criterion-related validity coefficient as defined by the California Community Colleges Chancellor s Office is

13 Course Outcome A binary logistic regression analysis was used to determine the predictive value of CCDT score on the outcome variable, success in Chemistry 11. The independent variable was the CCDT score. The dependent variable was a dichotomous variable of Chemistry 11 course performance: success (grades of A, B, C, or Credit) and nonsuccess (grades of D, F, Incomplete, No-credit, and Withdrawal). Successful grades were coded with a 1 and non-success grades were coded with a 0. The sample of students included students who completed the CCDT and earned a score of 22 or higher in calendar years 2007, 2008, 2009, and 2010 and subsequently enrolled in Chemistry 11 by fall term Only the attempt earning the highest score prior to enrolling in a chemistry course was recorded for students who took the CCDT more than once. In addition, only the first attempt in Chemistry 11 was recorded for each student. Students who earned scores below 22 were excluded from the analyses as they were directed to enroll in Chemistry 10. Table 9 presents the results of the beta coefficient, standard errors associated with the coefficient, and the unstandardized odds ratio for the logistic regression model predicting Chemistry 11 course outcome. Overall, the full model accurately predicted approximately 69.1% of the cases; however, the accuracy rate of 69.1% is only slightly better than the rate of accuracy achievable by chance alone (67.9%). The -2 Log Likelihood for the model was with chi-square value of 52.1 and p-value of.001 was statistically significant at the.05 significance level which indicates that, overall, the full model successfully predicts Chemistry 11 course outcome. The data reveal that one point increase on the CCDT increased the odds of earning a successful Chemistry 11 grade by 14.8%. Overall, the analysis reveals that CCDT performance positively predicts Chemistry 11 course outcome; therefore, the CCDT shows acceptable criterion-related validity on this measure. Table 9. Odds Ratios for CCDT Score on Chemistry 11 Course Outcome (N = 446) Variable B S. E. Exp(B) CCDT Score * Note. *p <.001 In addition to the logistic regression, an independent t-test was conducted to assess whether there is a statistically significant difference in CCDT scores by Chemistry 11 course success outcome. On average, the successful course completion rate of students who were placed directly into Chemistry 11 between 2007 and 2010 was 67.9%. An independent t-test analysis revealed that there was a significant difference in the mean CCDT scores for successful Chemistry 11 students (M = 32.00, SD = 5.89) and non-successful Chemistry 11 students (M = 27.86, SD = 4.79), t(444) = 7.34, p =.000 (see Table 10). The data reveal that, on average, students who were unsuccessful in their first enrollment in Chemistry 11 earned 28 points on the CCDT, 6 points higher than the cutoff score of 22. This finding suggests that the cutoff score of 22 may be too low. 13

14 Table 10. Chemistry 11 Course Outcome Descriptive Statistics, Chemistry 11 N Percent Mean Median Mode SD Outcome Success (C or better) Non-success (D, F, I, NC, W) Total Instructor Ratings Instructor ratings of student preparedness were assessed by administering an instructor survey in Chemistry 11 classes in the fifth weeks of spring 2010 and fall Instructors were asked to rate the extent to which they believed each student in their Chemistry 11 class was able to learn the course content based on their performance in the first weeks of the course. Instructions provided to faculty were: For each student in your roster, please rate the student s level of preparedness for coursework in Chemistry 11 by checking the appropriate box. Base your rating on student s demonstrated skills, understandings, and ability to comprehend the course material as shown in homework, quizzes, tests, class discussion, and other assignments. Disregard factors such as attendance and promptness in submitting homework/assignments when making your judgments. Faculty assessed each student in their roster, including students who did not complete the CCDT, in order to control for potential rater bias. Therefore, faculty members did not know which students were placed into Chemistry 11 by the CCDT or completed the course prerequisite, Chemistry 10. A Likert-like scale was used to rate each student: 5 = Extremely well prepared 4 = Well prepared 3 = Adequately prepared 2 = Marginally prepared 1 = Unprepared A total of 23 class sections of Chemistry 11 were offered in spring and fall 2010 terms combined. Instructors for 16 of the 23 sections responded to the survey. Among the 637 unique students enrolled in Chemistry 11 in spring and fall 2010, only 127 or 19.9% of students had been placed into the course by the CCDT. Instructor ratings data was collected for 66 of the 127 students. A Pearson product-moment correlation coefficient was computed to assess the relationship between CCDT scores and instructor ratings of students preparedness for Chemistry 11. There was a positive but small correlation between the two variables, r =.284, p =.021, and the correlation was statistically significant at the.05 significance level. However, the correlation coefficient does not meet the minimum threshold set by the CCCCO of.35. The instructor ratings as a criterion variable do not provide sufficient evidence for the criterion-related validity of the CCDT. 14

15 The following table presents descriptive information related to both the CCDT scores and instructor ratings for the sample. On average, Chemistry 11 students who were placed into the course by the CCDT had a score of (SD = 5.61), nearly 9 points above the cutoff score of 22, and a faculty rating of 3.3 (SD = 1.13) which indicate that students were adequately prepared. Table 11. Descriptive Statistics, CCDT Score/Instructor Rating Variable N Mean SD Median CCDT Score Instructor Rating In addition, a simple linear regression was conducted to test the predictive validity of the CCDT score on instructor ratings of student preparedness for Chemistry 11. The analysis found that CCDT score significantly predicted instructor ratings, F(1, 65) = 5.614, p =.021, however, CCDT score only explained for 8.1% of the variance in instructor rating. Similar to the correlation analysis, the regression analysis does not provide especially strong evidence for the criterion-related validity of the CCDT. Mid-term Scores In addition to Chemistry 11 course outcome and instructor ratings of student s ability to learn the course content, mid-term course performance were used as a criterion variable. Instructors teaching Chemistry 11 classes in spring 2010 and fall 2010 were asked to provide midterm grades for each enrolled in their sections. Instructions provided to faculty were: For each student in your roster, please record the total number of points earned in the course as of the mid-point of the semester (date was provided for each term) and the total points each student can possibly earn as of the mid-point of the semester. The mid-term grades of students included points earned on quizzes, tests, and lab assignments. Faculty provided mid-term scores for each student on their roster, including students who did not complete the CCDT, in order to control for potential rater bias. Therefore, faculty members did not know which students were placed into Chemistry 11 by the CCDT or completed the course prerequisite, Chemistry 10. In order to account for varying total points assigned by instructor, midterm grades were calculated by dividing the number of points earned by the total points possible and multiplying the dividend by 100 (for a potential range of 0 to 100). A total of 23 class sections of Chemistry 11 were offered in spring and fall 2010 terms combined. Instructors for 16 of the 23 sections responded to the survey. Among the 637 unique students enrolled in Chemistry 11 in spring and fall 2010, only 127 or 19.9% of students had been placed into the course by the CCDT. Instructor ratings data was collected for 55 of the 127 students. A Pearson product-moment correlation coefficient was computed to assess the relationship between CCDT scores and Chemistry 11 mid-term grades. There was a positive but small correlation between the two variables, r =.243, p =.033, and the correlation was statistically significant at the.05 significance level. However, the correlation coefficient does not meet the minimum threshold set by the CCCCO of.35. Mid-term grades as a criterion variable do not provide sufficient evidence for the criterionrelated validity of the CCDT. 15

16 The following table presents descriptive information related to both the CCDT scores and midterm grades for the sample. On average, Chemistry 11 students who were placed into the course by the CCDT had a score of (SD = 5.69), over 9 points above the cutoff score of 22, and a mid-term grade of (SD = 16.79) which indicates a grade of C. The high standard deviation value of mid-term grades suggest variability in mid-term grades. Table 12. Descriptive Statistics, CCDT Score/Mid-term Grades Variable N Mean SD Median CCDT Score Mid-term Grades In addition, a simple linear regression was conducted to test the predictive validity of the CCDT score on mid-term grades. The analysis found that CCDT score did not significantly predict mid-term grades, F(1, 57) = 3.523, p =.066. Consequential-Related Validity Consequential-related validity was measured by administering a student and faculty survey. Student surveys were sent directly to the sample of interest: students enrolled in Chemistry 11 in spring/fall term 2010 for the first time and earned a score of 22 or higher on the CCDT prior to enrolling the Chemistry 11. A total of 127 students were sent an invitation to participate in the survey by in the fifth week of the term. A total of 46 students (response rate of 36.2%) responded to the survey. Students were asked to rate the extent to which they perceived that accuracy or appropriateness of the CCDT in placing them into Chemistry 11. The response choices were: I should have enrolled in a higher (harder) course. This course is too easy. I belong in this course the difficult level of the course matches my chemistry knowledge/skills. I should have enrolled in a lower (easier) course. This course is too hard. In addition, Chemistry 11 instructors were asked to complete a survey on the appropriateness of students placements. Instructions provided to faculty were: For each student in your roster, please determine whether you believe the students placement in Chemistry 11 is appropriate/correct or whether you believe the student should be enrolled in a lower-level chemistry course. Base your rating on students demonstrated skills, understandings, and ability to comprehend course materials as shown in homework, quizzes, tests, class discussions, and other assignments. Disregard factors such as attendance and promptness in submitting homework/assignments when making your judgments. The response choices were: The student should be enrolled in a lower-level course The student is enrolled in the appropriate/correct course Faculty provided consequential rating for each student on their roster, including students who did not complete the CCDT, in order to control for potential rater bias. Therefore, faculty members 16

17 did not know which students were placed into Chemistry 11 by the CCDT or completed the course prerequisite, Chemistry 10. A total of 23 class sections of Chemistry 11 were offered in spring and fall 2010 terms combined. Instructors for 16 of the 23 sections responded to the survey. Among the 637 unique students enrolled in Chemistry 11 in spring and fall 2010, only 127 or 19.9% of students had been placed into the course by the CCDT. Faculty rating of appropriateness of student placement data was collected for 64 of the 127 students. The Chancellor s Office recommends a satisfaction response rate of 75% or higher; therefore, 75% or more students and faculty are expected to report that the student was placed in the appropriate/correct or harder course in order to provide evidence for consequential-related validity. Table 13 provides the findings of both the student and faculty surveys. Table 13. Faculty and Student Perception of CCDT Placement Variable N Student Should be in Lower Course Student in Appropriate Course/Should be in Higher Course Student Rating 46 6 (13.0%) 40 (87.0%) Faculty Rating (21.9%) 50 (78.1%) The data indicate that approximately 78% and 87% of faculty and students, respectively, judged their placement into Chemistry 11 to be appropriate based on the students chemistry skills. The results meet the threshold of 75% recommended by the Chancellor s Office. Therefore, the evidence provides support for the adequacy of the cut-score. However, the data should be interpreted with caution as the small sample size may not be generalizable to the population. Overall, the criterion-related and consequential-related validity data findings do not provide strong evidence for the current cut score of 22 points. While the data suggest that there is a statistically significant correlation between students CCDT score and faculty rating of student preparedness and midterm grades, the correlation coefficient is small. In addition, the data reveal that students who are unsuccessful in Chemistry 11 earned 28 points on the CCDT, 6 points higher than the cutoff score of 22 which suggests that the cutoff score may be too low. A followup study related to faculty rating of the CCDT items was conducted to address the weaknesses found in the criterion-related validity data and to further evaluate the cut score. 17

18 Cut-Score Determination A new cut-score was established by administering a survey of faculty assessment of CCDT items (modified Angoff procedure). Seven faculty members responsible for teaching Chemistry 11 participated in the cut score determination activity during the spring 2010 term. The group of faculty included four full-time and three adjunct instructors (three women, four men; four White, two Black, one Asian faculty). Instructors were asked to judge each item on the CCDT based on the following prompt: In your opinion, how many students out of 100 who possess the minimum skills/knowledge required to successfully complete Chemistry 11 (with a grade of C or better) would answer the following CCDT items correctly. Instructors worked independently on the task, however, once a discussion of the ratings as a group occurred after the rating activity. After the discussion, instructors were able to adjust their scores, if needed. The cut-score was calculated by summing each rater s final probabilities divided by 100. The average score across participants was calculated. Table 14 reports the probability of minimally qualified students answering CCDT items correctly by rater. Table 14. Cut-score Findings by Rater Rater 1 Rater 2 Rater 3 Rater 4 Rater 5 Rater 6 Rater 7 Cut-Score Average = 34.1 Based on the rating activity, the cut score of 34 out of 44 items was used to start the cut-score discussion. The following table provides two data points. The first set of columns describes the CCDT pass rates by CCDT score. The rates were determined by dividing the number of testing events with the specified and higher scores by the total number of testing events. For example, 605 of the testing events earned a score of 22 or higher for a pass rate of 75%. The data was based on the 807 testing events in calendar years The second set of columns describes the Chemistry 11 course success rates of students by CCDT score earned (based on the 446 students who earned a 22 or higher score between calendar years ). For example, the Chemistry 11 success rate for students who earned a 22 on the CCDT was 41.9%. 18

19 Table 15. CCDT Pass Rates and Chemistry 11 Success Rate by CCDT Score, Hypothetical Cutoff Score CCDT Pass Events CCDT Pass Rates (out of Chemistry 11 Passing Chemistry 11 Chemistry 11 Success Rates (Cumulative) 807) Grades Enrollment % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % Total % The cut score determination findings and the pass rates/chemistry 11 success rates data were discussed at a departmental Flex Day in fall of The entire chemistry faculty group, including both part-time and full-time instructors, was present for the discussion. Based on the current 22 cutoff score on the CCDT, approximately 75% of testing events pass the test and allow students to directly enroll in Chemistry 11. However, students who earned a 22 on the CCDT were successful in Chemistry 11 only 41.9% of the time. A cutoff score of 22 produces an overall Chemistry 11 course success rate of 67.9% (303 successful grades divided by 446 enrollments). The chemistry faculty members reported being unsatisfied with the cutoff score of 22 as it allows a large majority of test takers to bypass Chemistry 10 and directly enroll in Chemistry 11 which makes the implementation of a challenge exam less meaningful (begs the question, what s the point of having a challenge exam if everybody is getting in? ). In addition, faculty were unsatisfied with the low Chemistry 11 success rates of students earning a 22, 23, 24, and 25 score on the CCDT. 19

20 Based on the 34 cutoff score determined by the faculty assessment activity, approximately 24% of testing events would pass the test and allow students to directly enroll in Chemistry 11. About 8 in 10 students who earned a 34 score on the CCDT successfully completed Chemistry 11. A cutoff score of 34 produces an overall Chemistry 11 course success rate of 86.3% (calculated by dividing the total number of students with a CCDT score of 34+ who successfully completed Chemistry 11, 126, by the total number of students with a CCDT score of 34+ who enrolled in Chemistry 11, 146). In the discussion, faculty members reported they were satisfied with the high course success rate but pointed out that the cutoff score of 34 would deny access to Chemistry 11 for a large proportion of students. Based on the findings and faculty discussion of the findings, the cut-score was adjusted to 26. A cutoff score of 26, faculty determined, was the point at which overall Chemistry course success rates would increase by at least 5% while still maintaining access to a majority of the students. A cutoff score of 26 would produce a 58% pass rate on the CCDT. The Chemistry 11 course success rate was 69.7% for students with a CCDT score of 26. A cutoff score of 26 produces an overall Chemistry 11 course success rate of 73.7% (calculated by dividing the total number of students with a CCDT score of 26+ who successfully completed Chemistry 11, 255, by the total number of students with a CCDT score of 26+ who enrolled in Chemistry 11, 346). Raising the cutoff score to 26 from 22 would increase the overall course success by 5.8% and still maintain a pass rate over 50%. Summary of Cut-Score Validity Based on the findings of the criterion-related and consequential-related validity studies, the current cutoff score of 22 for passing the CCDT was determined to be too low. The results of a cut-score determination study and faculty discussion of the results led to a recommendation of adjusting the cutoff score to

21 Chapter 3: Disproportionate Impact In accordance with Title 5 and Matriculation regulations, a disproportionate impact study was conducted in spring of 2011 to assess the rate of placement for impacted groups (gender, ethnicity, and age) into Chemistry 11. In order to evaluate the extent of disproportionate impact, the 80% guideline established by the EEOC s Uniform Guidelines for Selection Procedures was utilized as recommended by the California Community Colleges Chancellor s Office. According to this guideline, if a minority group placement rate is lower than 80% of the majority placement rate, there is sufficient evidence of disproportionate impact. When this is the case, Title 5, Section delineates the proper procedure to remedy this situation. To assess disproportionate impact, student characteristics were grouped into two groups: minority group and majority group. The minority group consisted of students comprising the smallest groups (in numbers). The majority group consisted of students comprising the largest group (in numbers). Two sets of disproportionately impact analyses were conducted. The first evaluated disproportionate impact of the current cutoff score of 22. The second study evaluated the disproportionate impact of the recommended adjusted cutoff score of 26. CCDT pass rates were calculated by dividing the total number of who passed the CCDT by the total number of students completed the CCDT in calendar years Only students initial testing events were included in the analyses. Demographic data for 706 students who completed the CCDT were available. Disproportionate Impact when Cutoff Score is 22 Gender A total of 395 of students who completed the CCDT test in calendar years were male (55.9%), therefore, men were the majority group and women were the minority group. Table 16 describes the placement rates into Chemistry 11 by gender group. Male students passed at an 83% rate; 80% of 83% is 66.4%. The minority group, female students, passed at a 72% rate, higher than 80% of placement rate for the majority group. Therefore, the data shows no indication of disproportionate impact of the CCDT on gender groups when the cutoff score is 22. Table 16. CCDT Pass Rates by Gender, Gender Group Count of Pass Rate Students Female (minority) % Male (majority) % 80% of Placement Rate for Majority 80% of 83% = 66.4% 21

22 Ethnicity/Race Asian/Pacific Islander students comprise the largest ethnic group of students who completed the CCDT between 2007 and 2010 (41.2%); therefore, Asian/Pacific Islander group was used as the majority group. The minority groups were White (27.8%), Hispanic (14.0%), and Black (3.8%). The Native American/Alaskan Native, Other, Multi-Ethnic, and Unreported groups were not included in the analyses because of the small sample size in each group. The placement rate for the majority group, Asian/Pacific Islander was 85.6%; 80% of 85.6% is 68.5%. Table 17. CCDT Pass Rates by Ethnicity/Race, Gender Group Count of Pass Rate Students Asian/Pacific Islander (majority) % Black % Hispanic % White % 80% of Placement Rate for Majority 80% of 85.6% = 68.5% The data reveal that both the Black and Hispanic student groups had placement rates lower than the threshold (80% of the placement rate for majority group), 59.3% and 62.6%, respectively. There is evidence suggesting that there may be disproportionate impact for Black and Hispanic students. However, when using the White group as the majority group, which is traditionally defined as the majority group, the data show that there is no disproportionate impact on the Hispanic student group (the threshold for White majority is 62.1% or 80% of 77.6%). The data may be impacted by the small numbers of African American and Hispanic students taking the CCDT. Age Group A large majority of students taking the CCDT are between the ages of 18 and 24 (82.9%); this group was identified as the majority group. Table 18 describes the placement rates into Chemistry 11 by age group. The pass rate for 18 to 24 year olds was 79.1%; 80% of 79.1% is 63.3% which was the criteria to assess disproportionate impact for the minority group. The minority group, 25 or older students, passed at a 74% rate, higher than 80% of placement rate for the majority group. Therefore, the data shows no indication of disproportionate impact of the CCDT on age groups when the cutoff score is 22. Table 18. CCDT Pass Rates by Age Group, Age Group Count of Pass Rate Students 18 to 24 (majority) % 25+ (minority) % 80% of Placement Rate for Majority 80% of 79.1% = 63.3% 22

23 Disproportionate Impact when Cutoff Score is 26 Gender The majority group, male students, placed into Chemistry 11 at a rate of 68.9%; therefore, the 80% threshold is 55.1%. The minority group, female students, passed at a rate of 49.2%, lower than the 80% threshold. Therefore, the data suggests there may be a disproportionate impact of the CCDT on gender groups when the cutoff score is 26. Table 19. CCDT Pass Rates by Gender (Cut-score of 26), Gender Group Count of Pass Rate Students Female (minority) % Male (majority) % 80% of Placement Rate for Majority 80% of 68.9% = 55.12% Ethnicity/Race The placement rate for the majority group, Asian/Pacific Islander students, was 69.1%; therefore, the 80% threshold is 55.3%. White students placed into Chemistry 11 at a rate of 60.2%, higher than the 80% of majority threshold. Therefore, there is no disproportionate impact of the test on White students. The data suggests that there is disproportionate impact for both Black and Hispanic students who placed at a lower rate (33.3% and 40.4%, respectively) than the 80% threshold. When using the White group as a majority group, as recommended by the Chancellor s Office, the 80% threshold criteria is 48.2%. Even with the definition of majority group as White, there evidence suggests there may be disproportionate impact of the test on both Black and Hispanic students when the cutoff score of 26 is used. However, the data should be interpreted with caution as the sample size in both the Black and Hispanic groups is small. Table 20. CCDT Pass Rates by Ethnicity/Race (Cut-score of 26), Gender Group Count of Pass Rate Students Asian/Pacific Islander (majority) % Black % Hispanic % White % 80% of Placement Rate for Majority 80% of 69.1% = 55.3% Age Group The majority group, students between 18 and 24 years of age, placed into Chemistry 11 at a rate of 61.2%; therefore, the 80% threshold is 49%. The minority group, students 25 years of age and older, passed the test at a rate of 55.4%, higher than the 80% threshold. Therefore, the data shows no indication of disproportionate impact of the CCDT on age groups when the cutoff score is

24 Table 21. CCDT Pass Rates by Age Group (Cut-score of 26), Age Group Count of Pass Rate Students 18 to 24 (majority) % 25+ (minority) % 80% of Placement Rate for Majority 80% of 61.2% = 49.0% Summary of Disproportionate Impact The data suggests that disproportionate impact on CCDT placement exits for impacted groups. Specifically, there is a disproportionate impact for Black students when the cutoff score is 22. In addition, there is a disproportionate impact for female, Black, and Hispanic students when the cutoff score is adjusted to 26. In light of these findings, the chemistry faculty will convene a taskforce to assess the full impact of these findings and provide suggestions for improvement when appropriate. Furthermore, the college will continue to monitor CCDT placement outcomes for impacted groups on a three-year cycle. In the circumstance where the trend for the impacted group continues, the college will solicit assistance from the California Community Colleges Chancellor s Office to develop a plan of action as stipulated in Title 5. 24