Accuplacer Implementation Report Submitted by: Randy Brown, Ph..D. Director Office of Institutional Research Gavilan Collegee May 01
Introduction New student matriculation is an important factor in students progressing and completing their academic programs. For some time, Gavilan College staff, faculty, and students have expressed discontent with the assessment and placement system. For example, students are currently restricted to a paper and pencil test that did not have branching capability. In Fall 10, an Task Force was established. This group reviewed different assessment models and tools. In Spring 11, the group decided upon the AccuPlacer system due to its flexibility, widespread use, and accessibility features. After training provided by AccuPlacer staff and an implementation consultant, the Task Force, in Fall 11, established screening questions and set cut scores. This report documents this process and its findings. Gavilan curriculum Gavilan College has a range of Math and English courses that focus on the development of math and reading and writing skills. Below are the courses by different levels. Table 1: Gavilan English Curriculum Levels LEVEL COURSE Transfer Level ENGL 1A, 1B, 1C, B, C, E, F, J, 4A, 4B, 5A, 5B, 9A, 9B,9C, 1A, 1B, 1C, 1D One level below Writing Reading Two levels below Writing Reading Three levels below Writing Reading ENGL 50 ENGL 60 ENGL 440 ENGL 40 n/a ENGL 439 Table : Gavilan Math Curriculum Levels LEVEL COURSE Transfer Level Math 1A, 1B, 1C,, C, 5, 6, 7, 8A, 8B, 1, 14 One level below MATH 33, 33A, 33B, 01A, 01B Two levels below MATH 05, 05A, 05B, 08 Three levels below MATH 40 Four levels below MATH 400, 404A G, 414 Method In order to establish a new assessment and placement system, two sub committees of the Task Forces were established. The sub committees, one math, and another reading and writing, were charged with determining the screening criteria, placement scores, and multiple measure protocol. Gavilan College Office of Institutional Research 1
The committees were composed of content experts from each discipline along with the Director of Institutional research. See Appendix A for a list of committee members experience. Math screening Since the Math levels are assessed by three different assessments instruments (College Level Math, Elementary, and Arithmetic), the Math subcommittee was tasked with determining what instrument an assessment participantt would begin with. After examining the content of each test, the Math judges agreed on screening criteria, which used past Math course success. The screening question indentified was: What is the highest level of Math you completed? What was your grade in that course? If students responded with or below and reported receiving a grade of C or below, he/she would be directed to the Arithmetic tests (See Diagram 1 below). If he or she reports to have taken and received a B or above, he or she is directed to the Elementary. Students, who report that they took an course or above and report earning a C or below, are directed to the Elementary,while those who report receiving an B or above are directed to take the College Level Math assessment. Diagram 1: screening process. or higher Lower than B or above C or below B or above C or below Collegee Level Math Asssessment Elementary Elementary Arithmetic Arithmetic Placement Score In preparation for the process, the Director of Intuitional Research collected examples of placement cut scoress from a total of 10 other colleges. The examples were presents as a foundation for the cut score assessment work. The AccuPlacer system uses branching technology thatt adjusts the assessment instrumentt based on a user s performance. Consequently, the company does not make a written version of each assessment instrument. As a result, the sub committee had to determine an Gavilan College Office of Institutional Research
alternative approach to the Angoff method typically used to set placement cut scores. After some research, it was decided that an alternative borderline method be used which employs a panel of experts answering collectively as a hypothetical borderline student. For example, the panel was instructed to complete each assessment item for a student who is barely able to succeed in a one level below Intermediate class. So, the response for each item was decided upon by a majority of the expert panel. The resulting score was established as a cut score. Diagram 4 details the cut scores for placement in the various levels of Math. One cut score was adjusted slightly (decreasing 5 points) as a result of its comparison to the 10 other schools and input from an assessment consultant. For a full sheet of the entire process, seee Appendix B. Diagram : Arithmetic score flow chart Arithmetic Score: 1 54 Score: 55 79 Score: 80 10 Arithmetic Pre Elementary Placement or (Student Choice) Diagram 3: Elementary score flow chart. Elementary Score: 1 49 Score: 50 79 Score: 80 89 Score: 90+ Arithmetic Elemtary Intermediate College Level Math /Intermediate Placement (Student Choice) Gavilan College Office of Institutional Research 3
Diagram 4: College Level Math score flow chart. College Level Math Score: 1 65 Score: 66 80 Score: 81 103 Score: 104 10 Elementary Transfer level (e.g. Statistics) Pre calculus Calculus/Analytic Geometry Multiple measures The multiple measures adjustments were agreed upon by representatives of the sub down and input was sought from a assessment consultant. Students scores were supplemented based on committee. The previous multiple measure adjustment were reviewed and paired their responses to an item on their level of Math completion, and their grade on their most recent course (See below). Table 3: Math multiple measures Math history. What is the highest level math classs you completed? Response Geometry II Trigonometry Pre calculus Calculus Point 4 4 % of score 3.3 3.3 3.3 Gavilan College Office of Institutional Research 4
What grade did you receive in the last Math class? Response Point % of score A B What is your attitude about Math? Response Point % of score I dislike math and don t try hard 0 N/A I dislike math, but I try hard I dislike math, but I do what is necessary to pass B I like math and but don t like working hard to succeed I like math and don t mind working hard to succeed 4 3.3 How often did you study Math when you last took a course? Response Point % of score No hours per week 0 N/A 1 hours per week 3 hours or more per week 4 3.3 When you had difficulties in Math Response Point % of score I don t ask for help 0 N/A I ask the teacher for help I seek help from friends or classmates I study with friends or in a study group Gavilan College Office of Institutional Research 5
English Placement Score The English assessment is broken into two separate skill areas: Writing and Reading. Two English sub committees, with representatives with expertise in the two skill areas, developed assessment placement scores. In preparation for the process, the Director of Intuitional Research collected examples of placement cut scores from a total of 5 other colleges. The examples were presented as a foundation for the cut score assessment work. The AccuPlacer system uses branching technology thatt adjusts the assessment instrumentt based on a user s performance. Consequently, the company does not make a written version of each assessment instrument. As a result, the sub committee had to determine an alternative approach to the Angoff method typically used to set placement cut scores. After some research, it was decided that an alternative borderline method be used which employs a panel of experts answering collectively as a borderline student. For example, the panel was instructed to complete each assessment item for a student who is barely able to succeed in a one level below transferrable English class. So, the response for each item was decided upon by a majority of the expert panel. The resulting score was presented as a cut score for a transferable English course. In addition to this method, a borderline approach was used to confirm these scores. This approach asked instructors to select a sample of students who weree consideredd borderline placements at a particular level. These students were then asked to complete the assessment. These students scores were then averaged for a comparison placement score. These two scores were considered side by side for English Reading and Writing. and a final score was established. Diagrams 5 6 detail the cut scores for placement Diagram 5: Placement score flow chart English Reading. English Reading Score: 0 36 Score: 0 36 Score: 0 36 Score: 37 49 Score: 50 84 Score: >84 & >84 on writing ENGL 411 ENGL 40 ESL MM ENGL 40 ENGL60 ENGL1A ESL assessment Gavilan College Office of Institutional Research 6
Diagram 6: Placement score flow chart English writing. English Writing Score:0 36 Score: 0 36 Score: 0 36 Score: 37 49 Score: 50 84 Score: >84 & > 84 on reading ENGL 411 ENGL 440 ESL MMM ENG 440 ENGL50 ENGL1A ESL assessment Multiple measures The multiple measures adjustments were agreed upon by representatives of the sub as to committee. Additionally, an assessment consultant provided additional suggestions multiple measures. These in turn were reviewed and approved by the committee. Students scores were supplemented based on their responses to an item on their reading habits, level of English completion and their grade on their most recent course (Seee below). Additionally, ESL assessment recommendations are made through the application of ESL multiple measure items (See below). What was your grade in the last English course you took? Response A B <= C Point % of score 4 3.3 0 N/ /A When you read Response I usually understand the material the first time I read it I sometimes have to read the material several times before I understand it I usually have to read the material several times before I understand it Point 4 0 0 % of score 3.3 N/A N/A Gavilan College Office of Institutional Research 7
Did you complete and pass an advanced placement (AP) English course? Response Point % of score Yes 4 3.3 No 0 N/A ESL Multiple Measures ESL Q1: What language did you learn to speak first? a) English only b) English and another language c) Another language +1 ESL Q: What language do you know best? a) English only b) English and another language about the same c) Another language ESL +1 ESL Q: What language are you comfortable using in a conversation? a) English only b) English and another language about the same c) Another language ESL +1 Gavilan College Office of Institutional Research 8
Appendix A: Credential of Sub-committee members. Math Expert Panel Marla Dresch, M.A. Hope Jukl, M.S. Jennifer Nari, M.A. Elena Dachkova, M.S. MA, Economics, UC Davis (1993) Gavilan Mathematics Instructor (1998) Gavilan Math Instructor (007) Gavilan College Math Instructor (001) MS, Applied Math, CSU East Bay (1997) 17 years teaching experience, 15 years teaching math, 1 years as a full time math instructor in California Community Colleges. M.S. Mathematics New York University Courant Institute of Mathematics Hope Jukl has a total of 17 years teaching experience B.A. Mathematics, University of California, Berkeley M.A. Mathematics, University of California, Santa Cruz Jennifer Nari has 4 years teaching experience. M.S. Applied Mathematics Moscow Petroleum University, Russia Elena has 5 years of teaching experience both in community colleges and 4-year institutions. English Expert Panel Mark J. Fuzie, M.A. Jessica Hooper, M.A. Karen Warren, M.A. Jessica Gatewood, M.A. MA English, and Certificate in the Teaching of Composition (post MA) SFSU AB English, SDSU years teaching composition and literature MA, English, SJSU Post-secondary Reading Certificate, SFSU 13 years teaching experience MA, English, SFSU 4 years experience teaching English in community college B.A. English and B.A. History Notre Dame de Namur University M.A English, Notre Dame de Namur University Post-Secondary Reading Certificate, CSU Fullerton 6 years teaching experience in California Community Colleges. Gavilan College Office of Institutional Research 9
Appendix B: Math Placement Flow Chart. Gavilan College Office of Institutional Research 10
Gavilan College Office of Institutional Research 11