Coastline Community College Supporting Documentation for the Initial Approval of a Locally Managed Test MyTest Placement Exam Introduction Coastline Community College is transitioning to a locally developed math placement test. The test uses the MyTest software package to administer and score each placement. The test items were developed by the head of the department and a fellow full-time math faculty member. There are a total 41 questions on the test, including a multiple measures question. The test was designed to progressively get harder as the student moves from one item to the next. The resulting score will place a student into one of eight math levels. For this initial submission, a content validity study was performed, as well as a cut-score and test bias analysis. The college researcher worked closely with the math faculty to determine the appropriateness of this test for placement into all eight levels of the math curriculum. The eight levels of math at Coastline College are: Level 1 C005 Level 2 C008 Level 3 C010 Level 4 C030 Level 5 C100/103/104/106 Level 6 C115/120/150/160 Level 7 C140/170 Level 8 C180 It should be noted that levels 5, 6, and 7 combine a number of math courses. The material taught within each level is the same; they have different course titles to accommodate the type of degree program a student is enrolled in at the college. A placement into any one of these levels will allow a student to enroll into the appropriate course depending on their academic program or degree track (e.g. liberal arts student versus a math or science major). Item # 3 - Content Validity A panel of four math faculty participated in the review of the MyTest. Two types of analysis were completed: 1) a review of the entering level of knowledge and skills needed for success in each math course, and 2) an item by item analysis comparing each test question to the various math levels.
For the first type of analysis, the math faculty was asked to identify prerequisite knowledge and skills for each level of math. A list of the prerequisite skills can be found in Appendix A. In order to determine the comprehensiveness of the measurement, the prerequisite knowledge and skills were matched to the items found in the test. The raw data can be found in Appendix B. This data shows that at least one prerequisite skill for each math level is covered by an item in the placement test. The only exception is the lowest level math course ( C005). There are no prerequisite skills or knowledge needed to enter this remedial course and therefore cannot be matched to any items on the test. As a whole, at least one item if not multiple items correspond to a prerequisite skill or knowledge needed throughout the spectrum of courses offered at Coastline Community College. For the second type of analysis, the math faculty was asked to rate each item according to the following scale: 4 Item is critically important to success in the target course 3 Item is important to the success in the target course 2 Item is slightly important to success in the target course 1 Item is not relevant to success in the target course The following table summarizes the results and the raw data showing the average of all the participating faculties results can be found in Appendix C. C005 The first 8 items were judged important to critically important for C005. Thus, 20% of the items are relevant to success in C005. C008 Progressively, the first 10 items were judged important to critically important for C008. Thus 24% of the items are relevant to success in C008. C010 The first 20 items were judged important to critically important for C010. Thus 49% of the items are relevant to success in C010. C030 The first 34 items were judged important to critically important for C030. Thus 83% C100 /103/104/ /106 C115 /120/150 /160 C140 /170 C180 of the items are relevant to success in C030. The first 30 items as well as item 32 and 33 were judged important to critically important for C100. Thus In addition, items 31 and 34 was considered slightly important for C100. In total, 83% of the items are relevant to success in C100. The first 36 items were judged important to critically important for C115. Thus, 88% of the items are relevant to success in C115. The first 36 items were judged critically important for C140. The remaining five items were judged as slightly important. Thus 88% of the items are relevant to success in C140. All 41 items were judged as critically important for C180. Thus 100% of the items are relevant to success in C180. Content Validity Conclusion Based on the results of the first type of analysis, at least one of the course perquisite skills for each math level are being measured by one or more items in the test. This indicates that the test
comprehensively covers most of the test skills needed at each course level. In addition, the item by item analysis presented in the second set of data indicates that there is sufficient number of items necessary to measure each level of math. Item # 5 Test Bias A panel of 14 people comprised of faculty, staff, and students evaluated the MyTest for anything offensive, confusing, or pose a disadvantage towards any one ethnicity, culture, gender, disability, and language. The panel consisted of individuals representing various ethnicities, age groups, and genders. The following table displays the results of the analysis. Panel Member 1 2 3 4 5 6 7 8 9 10 11 12 13 14 GENDER Male X X X X Female X X X X X X X X X X AGE 16-20 X X X X 21-29 X X X 30-39 X X 40-49 X X X 50 + X X ETHNICITY Asian/Pacific Is. X X X X X African Am. X X Caucasian X X Hispanic X X X X Other X Ethnicity Bias N N N N N N N N N N N N N N Gender Bias N N N N N N N N N N N N N N Age Bias Y N N N N N N N N N N N N N Language Bias N N N Y N N N N Y N N N N N Disability Bias N N N N N N N N N N N N N N Panel Member 1 Age Bias: This person expressed concern that the first question in the test could be discriminating towards someone of an older age. The first question is a multiple measure and awards the student a half point if they took a math course within the past year. Another half point can be earned if the student received an A in their previous math course. The math faculty felt that the student was given two opportunities to get an additional half point and therefore the question is fair. Panel Member 4 Language Bias: For non-native speakers, some of the directions to certain math questions could be confusing. It was suggested to add the word Simplify or Solve the equation to some of the questions directions. The math faculty agreed and ended up adding these simple
statements to the directions of the applicable math questions in order to make them more understandable for non-native speakers. Panel Member 9 Language Bias: The person was concerned about the use of the terms binomial and trinomial in the directions of certain math questions. The math faculty and the panel members felt that these are appropriate math terms that are necessary to understanding the level of math in question. Test Bias Conclusion After a comprehensive review of the MyTest placement exam, the panel agreed that this test does not pose a disadvantage towards any one ethnicity, culture, gender, disability, and language. The comments that where made about some of the questions on the test were addressed by the math faculty and the test is now ready to be offered and an unbiased instrument. Item # 6 - Cut-Score A panel of five math faculty participated in a study that established the Initial Cut-Scores for the MyTest placement exam. The modified Angoff procedure was used as a tool to begin discussion on the appropriate placement levels. After each faculty member filled out their forms, a comprehensive discussion took place that focused on each individual s score, the average faculty derived cut, and the test manuals suggested cut. The following table shows the summary data that the instructors used to settle on a final cut. Faculty Reviewer Cut-Score Conclusion C005 C008 C010 C030 Course C100/103/ 104/106 C115/120/ 150/160 C140/170 FF 5.80 8.00 10.70 16.10 18.50 24.60 27.00 32.80 JK 3.75 6.00 16.25 23.75 30.00 30.50 38.00 39.75 LL 5.80 8.00 11.10 16.10 19.20 26.10 34.70 37.00 RS 6.50 10.50 12.25 15.50 21.00 27.00 29.50 33.50 TC 6.00 12.85 25.35 26.45 28.60 30.40 39.85 41.00 Average facultyderived cut Test manual's suggested cut Cuts selected after reviewing the above data 5.57 9.07 15.13 19.58 23.46 27.72 33.81 36.81 0-4 5-7 8-15 16-23 24-27 28-29 30-33 34-41 0-5 6-8 9-15 16-23 24-27 28-29 30-33 34-41 C180 Most of the variance occurred at the lower level of math ( C005, C008, and C010). The faculty decided to adjust the cut score for these three math levels and settled on the scores as seen in the last row in the above table.
Item # 7 Disproportionate Impact Disproportionate Impact reports will be completed every three years. The first report to be completed and submitted to the State Chancellor s Office will be by April 15, 2015.
Appendix A Prerequisite Skills and Knowledge C005 - This is the lowest level math and therefore doesn t have any prerequisite skills C008 1. Correctly perform the fundamental operations of arithmetic on whole numbers, fractions, and decimals. 2. Correctly solve applications using percentage, ratio, proportion, and measurement. 3. Convert numbers between percentage, decimal, and fraction notation. C010 1. Use the order of operations to add; subtract; multiply; and exponentiate whole numbers, fractions, and decimals. 2. Use rounding techniques to estimate results of operations on whole numbers, fractions, and decimals. 3. Convert rational numbers into decimals, fractions, and percentages. 4. Solve various application problems requiring the use of ratios, proportions, and percentages. C030 1. Solve linear and quadratic equations by factoring, completing the square, and using the quadratic formula. 2. Solve and graph linear inequalities and calculate slopes and intercepts. 3. Solve equations and simplify algebraic expressions involving exponents, polynomial and rational expressions and equations, roots, and radicals. 4. Set up the equation or inequality; then find the solution and explain the reasonableness of the answer a word or application problem. C100/103/104/106 1. Correctly solve linear, quadratic, polynomial, rational, radical, exponential, and logarithmic equations; graph any of these functions with the appropriate domain and range; analyze its behavior and make predictions. 2. Express a real-world problem as an equation or system of equations, estimate the answer, then solve the equation or system to find a solution and judge its reasonableness. 3. Identify and graph conic sections including parabolas, circles, ellipses and hyperbolas; show major and minor axes, intercepts, foci, and asymptotes; given a graph write the equation for the conic section in standard form; solve applications involving conic sections.
Appendix A C115/120/150/160 1. Correctly solve linear, quadratic, polynomial, rational, radical, exponential, and logarithmic equations; graph any of these functions with the appropriate domain and range; analyze its behavior and make predictions. 2. Express a real-world problem as an equation or system of equations, estimate the answer, then solve the equation or system to find a solution and judge its reasonableness. 3. Identify and graph conic sections including parabolas, circles, ellipses and hyperbolas; show major and minor axes, intercepts, foci, and asymptotes; given a graph write the equation for the conic section in standard form; solve applications involving conic sections. C140/170 1. Solve quadratic and ration equations and inequalities. 2. Find the domain, range, inverse, and graph (with the translations) the following: linear, polynomials, rational, exponential, and logarithmic functions. 3. Manipulate polynomials and solve polynomials equations using the Rational Zero Theorem, Synthetic Division, The Remainder Theorem, and Factor Theorem. 4. Solve polynomials equations by factoring and solve radical equations. 5. Simplify expressions involving integers and rational exponents and radicals, and complex fractions. C180 1. Find all real and complex roots or zeros of a polynomial equation. 2. Solve equations involving exponential or logarithmic expressions. 3. Prove or establish trigonometric identities. 4. Decompose a fractional expression via partial fractions.
Appendix B Test Qualification Project - Coastline Community College - Content Review C008 Name of faculty expert who is completing this form:_summary of all faculty data Date: C010 Prerequisite Skills and Knowledge - Item by Item Analysis C030 C100... C115... C140... C180... 1 2 3 1 2 3 4 1 2 3 4 1 2 3 1 2 3 1 2 3 4 5 1 2 3 4 1 2 X 3 X 4 X 5 X X 6 X 7 X X 8 X X 9 X X 10 X 11 X X X 12 X X 13 X 14 X 15 X X X X 16 X X X X 17 X X X 18 X X 19 X X 20 X X X X 21 X 22 X 23 X 24 X 25 X X 26 X X X X X X 27 X X X 28 X X X 29 X X X X X 30 X X X X X X X X 31 X X X 32 X X X X X X X X 33 X X 34 X X X 35 X X 36 X X X 37 X 38 X 39 X 40 X 41 X X X
Appendix C Test Qualification Project - Coastline Community College - Content Review Name of faculty expert who is completing this form: Summary of all faculty data Date: Instructions: Please rate each item's importance for each math course using the scale shown below. How important is the academic knowledge or skill measured by this item for successful acquisition of the skills taught in this course? 4. Critically Important 3. Important 2. Slightly Important 1. Not Relevant Test Item Number C005 C008 C010 C030 C100/103 /104/106 C115/120 /150/160 C140/170 C180 1 4 4 4 4 4 4 4 4 2 4 4 4 4 4 4 4 4 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 3.25 4 4 4 4 4 4 4 6 4 4 4 4 4 4 4 4 7 3.25 4 4 4 4 4 4 4 8 3.25 4 4 4 4 4 4 4 9 1.5 3.75 4 4 4 4 4 4 10 1.25 3.75 4 4 4 4 4 4 11 1.25 2 4 4 4 4 4 4 12 1.25 1.5 4 4 3.5 4 4 4 13 1.25 1.25 4 4 3.5 3.25 4 4 14 1.25 1.25 3.5 4 3.5 3.25 4 4 15 1.25 1.25 3 4 4 3.25 4 4 16 1.25 1.5 3.5 4 4 4 4 4 17 1.25 1.75 3.5 4 3.75 4 4 4 18 1.25 1.75 3.5 4 4 3.5 4 4 19 1.25 1.25 3.5 4 4 4 4 4 20 1.25 1.75 3 4 3.75 3.75 4 4 21 1.25 1.75 2.5 4 4 3.5 4 4 22 1.25 1.5 2.75 4 4 4 4 4 23 1.25 1.5 2.75 4 4 4 4 4 24 1.25 1.5 2.75 4 4 4 4 4 25 1.25 1.5 2.5 4 4 4 4 4 26 1.25 1.25 2.5 4 3.75 4 4 4 27 1.25 1.25 2.5 4 3.75 3.75 4 4 28 1.25 1.25 2.5 4 4 3.75 4 4 29 1.25 1.25 2.25 4 4 4 4 4 30 1.25 1.25 2.25 4 4 4 4 4 31 1.25 1.25 2.25 3.25 2.75 4 4 4 32 1.25 1.25 1.75 3 3.25 3.5 4 4 33 1.25 1.25 2 4 4 4 4 4 34 1.25 1.25 2 3.25 2.75 4 4 4 35 1.25 1.25 1.75 1.75 1.75 3.25 4 4 36 1.25 1.25 1.75 2 2 3.5 4 4 37 1.25 1.25 1.25 1.5 1.5 1.75 2.25 4 38 1.25 1.25 1.25 1.25 1.25 1.5 2.25 4 39 1.25 1.25 1.25 1.25 1.25 1.5 2.25 4 40 1.25 1.25 1.25 1.25 1.25 1.5 2.25 4 41 1.25 1.25 1.25 1.25 1.25 1.5 2.25 4
Revised, January 2004 Chancellor s Office California Community Colleges Request for Approval for the Use or Renewal of a Performance Assessment or Locally Constructed or Managed Test Directions: Provide all requested information. Attach additional pages as needed. Note that this form is to be signed by the identified individuals of the college and submitted with supporting material. When requested, indicate which Standards areas have been investigated or addressed and those areas not as yet addressed. Note: Studies addressing all of the Standards areas need not be completed in order to request approval of an instrument. The minimum requirements are that you provide at least one type of validity evidence and that the test bias standard be addressed. If information exists in a technical report or other sources, summarize the information for this report and draw conclusions from the information on whether you feel a specific standard has been met at a minimal level for your instrument. Submission of extended reports or exhaustive documentation evidence to support your claims is not required or desired for review of this request. 1. Identify the test with its complete title: _MyTest Placement Exam by Pearson 2. For which course(s) is this test used to assist with the placement of students? Please identify: C005, C008, C010, C030, C100/103/104/106, C115/120/150/160, C140/170, and C180 3. Have there been investigations of the validity of the use of scores obtained from this test? (If your response is no to this question, do not submit this request until some validity evidence is available.) X YES, all required studies have been completed. Attach a brief narrative that summarizes the procedures and findings from all such investigations. YES, but not all required studies have been completed. Attach a brief narrative that summarizes the procedures and findings from all such investigations. X Projected completion date for required studies not completed: November 15, 2013 4. Have there been investigations of the reliability of scores obtained from this test? YES. Attach a brief narrative that summarizes the procedures and findings from all such investigations. X NO. Projected completion date: _November 15, 2013_