Assessment Report Mathematics Department. Part I Assessment SUMMARY

Assessment Report Mathematics Department Part I Assessment SUMMARY 2004-05 A. Program/Discipline Mission Statement Mission and Purpose: The mission of the Mathematics Department at Arapahoe Community College is to provide learning-centered mathematics education to students. The department offers courses for both full-time and part-time students supporting both transfer and career opportunities. The department is committed to quality learning-centered mathematics education valuing traditions and incorporating current effective pedagogical trends in the discipline, appropriate technology, and assessment of student learning. B. Intended Outcomes 1) Students will acquire the ability to read, write, listen to, and speak mathematics 2) Students will demonstrate a mastery of competencies identified by the competency-based syllabi for specific courses. 3) Students will use appropriate technology to enhance their mathematical thinking and understanding and to solve mathematical problems and judge the reasonableness of their results. 4) Students will engage in substantial mathematical problem solving. 5) Students will acquire the ability to use multiple approaches-numerical, graphical, symbolic, and verbal-to solve mathematical problems. C. Benchmarks The math department used four comprehensive assessment tools that collectively assessed all five learning outcomes at least twice: the college algebra common final, the college algebra entrance exam, the calculus I common projects, and the survey of algebra student and faculty surveys. From these tools arose the five indicators of Competency,, Score Changes,, and. The five indicators were computed and compared with benchmarks of Strong, Sufficient, and Weak. Tables 1 and 2 below summarize the assessment tools, indicators, benchmarks, and demarcations of indicator values that are associated with the pedagogical strengths of Strong, Sufficient, and Weak. 1

Table 1. Learning Outcome Assessment Tools and Associated Indicators. Assessment Tool Learning Outcome Common Final Entrance Exam / Common Final Common Projects 1 Indicators for Comparison with Benchmarks 2 3 4 5 Competency Competency and Change Scores and Change Scores Student and Faculty Surveys Table 2. Benchmark Values for Indicators. Pedagogical Strength Indicator Strong Sufficient Weak Competency >70% 50% to 70% <50% >70% 50% to 70% <50% Score Changes and Sub- Scores Survey Score Competency: Learning Outcome: Mean Score Change Positive Mean Score Change Negative >70% 50% to 70% <50% >3.5 >4 3 to 3.5 3.5 to 4 <3 <3.5 2

D. Assessment Results 1. Historical Context During AY2001-2002, the math department formulated the mission statement and the five learning outcomes, and assessed learning outcome #2 through the use of the college algebra common final. For AY2002-2003, the department switched to a calculator-based version of the college algebra common final and introduced the college algebra entrance exam to assess learning outcome #2 through the use of two distinct assessment tools. In AY2003-2004, the department implemented calculus one common projects enabling assessment of all five learning outcomes at least one way. Lastly, survey of algebra student and faculty surveys were introduced in AY2004-2005 and in conjunction with the other four assessment tools, the math department now assesses all five learning outcomes at least two ways. Through these four assessment cycles, the math department s assessment efforts have continually evolved and generated new data for analysis and usage in pedagogical improvement. Some key changes in the assessment process and their motivations were Modifying the common final to address extensive use of graphing calculators and to more directly assess students use of technology (learning outcome 3) Modifying the entrance exam to have three identical questions shared between it and the common final, enabling reliable assessment of within-student improvement in skills during the course of an academic semester (learning outcome 2) Modifying the common project rubric to more effectively target all five learning outcomes Pedagogical changes implemented as a direct result of assessment efforts include Modification of college algebra syllabus to include additional time for instruction on exponential functions, resulting in improved student competency Universal implementation of common projects in calculus I, resulting in students skill development across all learning outcomes in a highly applied and interdisciplinary endeavor 2. Current Year Data Results The competencies examined by the common final and contributing to the measurement of learning outcomes #2 and #3 are listed in table 3. Competency correct response rates are graphed in figure 1. 3

Table 3. Colorado Community College System Core Transfer Program Student Learning Outcomes for College Algebra. http://www.cterc.cccoes.edu/cccns/index.html B E F G H I J Perform algebraic manipulations including working with exponents, radicals, polynomial operations, factoring and algebraic fractions. Work with formulas including formula evaluation and solving a formula for any of the variables. Read and analyze problems in the form of word problem applications and obtain solutions using equations. Solve first degree inequalities, higher degree inequalities and inequalities involving absolute value. Recognize and graph linear functions, rational functions, absolute value functions, and graph inequalities in two variables. Work with function notation and demonstrate knowledge of the meaning function. Demonstrate an understanding of function composition, one-to-one functions and inverse functions. K M O Examine, evaluate and graph exponential functions. Work problems and solve equations containing exponential and logarithmic functions. Use at least two of the following techniques to solve linear and non-linear systems of the equations: substitution, addition, Gaussian elimination, Cramer s rule. CC Demonstrate the ability to select and apply contemporary forms of technology to solve problems or compile information. Figure 1. Common Final per Competency, Fall 2004. CC O M K Competency J I H G F E B 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Percentage Answered ly Note that figure 1 depicts a variety of strengths and weaknesses, with several competencies having a correct response rate of 50% or more, indicating pedagogical sufficiency for a majority of the competencies. The entrance exam and common final shared three identical questions that collectively addressed word problem competencies E and F, contributing to assessment of learning 4

outcome #2. For students completing both the entrance exam and common final, word problem score changes were computed and defined as the difference in the number of correctly answered word problems on the final minus the number of correctly answered word problems on the entrance exam. Table 4 presents the score changes. The mean change of 0.79 reflects a statistically significant increase in number of correctly answered word problems on the final versus the entrance exam. Table 4. Distribution of Change Scores Associated with Students Taking both the Entrance Exam and Common Final, Fall 2004. Change Score Frequency -3 0-2 1-1 13 0 35 1 51 2 27 3 5 n 132 mean 0.795454545 sd 1.024424146 se 0.089164677 z 8.921184626 The calculus one common projects were scored using a departmentally designed rubric and rubric categories were linked to all five learning outcomes, as presented in table 5. Figure 2 provides the rubric sub-scores associated with one of the two common projects during Fall 2004. Note that only rubric category 7 appears weak, and then only relative to the other categories as it still achieves Sufficiency according to the benchmark value. Table 5. Linkage Between Common Project Rubric Categories and Learning Outcomes Learning Outcome Rubric Categories 1 2 3 4 5 1) Problem Statement (10 points) X 2) Problem Solution (20 points) X X 3) Mathematical Notation (15 points) X X 4) Graphs (10 points) X X 5) Presentation (15 points) X 6) Creativity (10 points) X 7) Conclusion/Reflection (10 points) X 8) English (10 points) X 5

Figure 2. Common Project Student Performance by Rubric Category, Pendulum Project Fall 2004. 100 Mean Percentage of Possible Rubric Category Points Scored 90 80 70 60 50 40 30 20 10 0 1 2 3 4 5 6 7 8 Rubric Catagory The Survey of Algebra survey s contribution to the measurement of learning outcome #2 is through the student s self-assessment of competency mastery using the following scale: 1=No Knowledge of the Skill, 2=Some Knowledge of the Skill, 3=Proficient in the Skill, and 4=Expert in the skill. The assessed competencies are provided in table 6. Table 6. Student Skill Levels adapted from the list of Colorado Community College System common course competencies for Survey of Algebra. http://cccns.cccs.cccoes.edu/comcrs_display.asp 1 Graphing linear equations and inequalities with two variables. 2 Evaluating functions using "function notation". 3 Solving a system of equations in two or three unknowns. 4 Solving word problems using equations or systems of equations. 5 Solving linear inequalities with one variable and using set and interval notation to write the solution. 6 Graphing linear inequalities with one variable. 7 Solving equations and inequalities involving absolute value. 8 Operations on quadratic equations. 9 Graphing quadratic equations. 10 Operations on polynomials. 11 Graphing polynomials. 12 Simplifying rational expressions and solving rational equations. 13 Applying the laws of exponents. 14 Simplifying radical expressions and solving radical equations. 15 Operations on complex numbers in the form a + bi. 16 Determining composite and inverse functions. 17 Graphing exponential and logarithmic functions. 6

4.00 Figure 3. Students' Self-Perception of Skill Levels, Fall 2004. 3.50 3.00 2.50 Average Score 2.00 1.50 1.00 0.50 0.00 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Skill Number Note that Figure 3 depicts a variety of strengths and weaknesses, with most competencies averaging above a proficient rating. Four competencies fell below the proficient rating, #4, 15, 16 and 17, indicating pedagogical weakness in these competencies. Of these four, the six faculty members surveyed rated competencies 16 and 17 as below proficient. In fact, these were the only two categories on the faculty survey with ratings below the proficient level. Students and faculty were asked to indicate how well the course enabled the students to meet the Mathematics Discipline Intended Learning Outcomes, with the scale 1 = Strongly disagree, 2 = Disagree somewhat, 3 = Neither agree nor disagree, 4 = Agree somewhat, and 5 = Strongly agree. The learning outcomes covered in the student and faculty surveys which contributed to the measurement of learning outcomes #1, 3, 4, and 5 are listed in Table 7, and the average scores for the student survey learning outcomes are graphed in Figure 4. 7

Table 7. Linkage Between Student Survey Learning Outcomes and Mathematics Discipline Intended Learning Outcomes, Fall 2004. Mathematics Discipline Intended Learning Outcome Student Survey Learning Outcome 1 2 3 4 5 1) I acquired or strengthened the ability to read mathematics. X 2) I acquired or strengthened the ability to write mathematics. X 3) I acquired or strengthened the ability to listen to mathematics. X 4) I acquired or strengthened the ability to speak mathematics. X 5) I used technology to enhance my mathematical thinking and X understanding. 6) I used technology to solve mathematical problems and judge X the reasonableness of my results. 7) I engaged in substantial mathematical problem solving. X 8) I acquired or enhanced the ability to use multiple approaches - numerical, graphical, symbolic, and verbal - to solve mathematical problems. X Figure 4. Students' Self-Perception of Achievement of Learning Outcomes, Fall 2004. 4.10 4.00 3.90 3.80 Average Score 3.70 3.60 3.50 3.40 3.30 1 2 3 4 5 6 7 8 Statement Number Notice that all of the 8 statement numbers have scores exceeding 3.5, indicating pedagogical sufficiency across all five learning outcomes. Statement 4 is associated with learning outcome 1, indicating the weakest link in speaking mathematics. 8

3. Learning Outcome Assessment Based Upon Analyses 1) Students will acquire the ability to read, write, listen to, and speak mathematics. The common projects assess learning outcome 1, through the rubric categories of 1 and 8. Figure 2 is representative, though not inclusive, of all common project data collected throughout AY2003-2004. Student common project performance for learning outcome 1 is above the benchmark. The survey assessed learning outcome 1 through statements 1-4 (see Figure 4.) All were either sufficient or strong in pedagogical strength, with the ability to speak mathematics the weakest (though still sufficient.) Collectively, there is strong evidence that students are either sufficient or strong in their achievement of learning outcome 1. 2) Students will demonstrate a mastery of competencies identified by the competency-based syllabi for specific courses. The common final is one of the primary assessment tools for learning outcome 2. Of the 11 competencies addressed by the common final, students are strong/sufficient in the 8 competencies of B, E, G, H, I, J, K, and CC while there is weakness in the competencies of F, M, and O. Despite weakness in competency F, students are improving in their skill level per the entrance exam/common final analyses. The common project applicable rubric categories are 2 and 3, which both exceed 70%, indicating student strength in learning outcome two from this assessment tool. Lastly, in the 13 competencies of 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, and 14 for survey of algebra, the students achieved either strong or sufficient achievement. Competencies 4, 15, 16, and 17 indicated weakness in word problems, operations on complex numbers, composite and inverse functions, and the graphing of exponential and logarithmic functions, respectively. Students overall have at least sufficient achievement of mastery of competencies, with a few exceptions in need of improvement. 3) Students will use appropriate technology to enhance their mathematical thinking and understanding and to solve mathematical problems and judge the reasonableness of their results. The common final, entrance exam, and common projects require the use of technology and address learning outcome 3. The three common final questions requiring the use of a graphing calculator possessed sufficiency (with 1 of the 3 questions achieving over 90% correct response rate), indicating a sufficient student achievement level for learning outcome 3. 9

The common project rubric category 4 was nearly 90%, also indicating departmental strength in learning outcome 3. While students surveyed felt they had attained sufficiency in technology usage, an ancillary study (of too small a sample size) of faculty opinion indicated weakness in this learning outcome. Aside from some faculty concerns over Survey of Algebra, the preponderance of evidence from the assessment tools indicates student achievement to be sufficient in learning outcome 3. 4) Students will engage in substantial mathematical problem solving. Common project rubric categories of 2, 6, and 7 address learning outcome 4, all of which attained sufficiency for student achievement. Category 7 is weakest (in fact of all rubric categories) and indicates room for potential improvement in students ability to draw high-level conclusions and generalize results of mathematical modeling a relevant aspect of learning outcome 4. Survey data indicated that students possessed strength in learning outcome 4, augmenting evidence that students overall programmatically attain sufficient achievement of learning outcome 4. 5) Students will acquire the ability to use multiple approaches-numerical, graphical, symbolic, and verbal-to solve mathematical problems. Common project rubric categories 3, 4, and 5 address learning outcome 5. For all of the common project data, overall student performance exceeded 70% in these categories, indicating student strength for learning outcome 5. Likewise, the highest scoring survey category was associated with learning outcome 5, reinforcing attainment of student strength in learning outcome 5. E. Use of Results Improvement of student learning is the foundation of the assessment process. Rather than being a single event at the end of the process, changes that improve student learning happen continuously throughout an effective assessment cycle. Refinement of the process, increased student awareness of expectations and involvement in assessment activities, constant faculty discussions of ineffective and/or improved pedagogical methods and subsequent decisions to modify pedagogy, and administrative support of program assessment efforts are some components of the assessment cycle that are integral to achieving the goal of improved student learning. Department faculty will study this report (http://www.arapahoe.edu/aboutacc/assessment/mat/04-05.pdf) and discuss student strengths and weaknesses. 10

Student common final correct response ranges indicate departmental strengths and sufficiencies in a majority of the assessed competencies, and specific weaknesses in competencies F, M, and O, which represent word problem solving, usage of exponential and logarithmic functions, and solving non-linear systems of equations. The department will potentially modify the College Algebra syllabus and/or teaching strategies in 2005-2006 based on these results. Common projects, consistent with last year, exhibit high student achievement in all of the five learning outcomes; the only area that potentially warrants improvement is with students drawing high-level conclusions and generalizing mathematical modeling to broader contexts. With the introduction of surveys that assess all five learning outcomes, it is even more evident that students are for the most part attaining at least sufficiency in all of the learning outcomes. However, more data needs to be collected from faculty to determine if students and faculty s assessments of achievement are in conflict over learning outcome 3. 11

Part II Assessment Plan 2005-06 A. Identify Assessment Procedures/Methods Entrance Exam Common Final Common Projects Surveys Mathematical Association of America - Basic Algebra Test Mathematical Association of America - Calculus Readiness Test Modified by the ACC Mathematics Department Interactive and Lively Applications Project (ILAP): Pendulum Activity developed by Bruce MacMillan (University of Colorado at Denver), Sam Welch (University of Colorado at Denver) and Tracy Lawrence (Arapahoe Community College), part of Colorado Institute of Technology (CIT) Foundations of Engineering, Science, and Technology (FEST) Program Somewhere Within the Rainbow by Steven Janke. Colorado College, Colorado Springs, CO acquired from Applications of Calculus, Philip Straffin, editor, MAA Notes #29 Student Survey of Algebra Survey, departmentally designed Faculty Survey of Algebra Survey, departmentally designed Continue AY2004-2005 Assessment Activities in Assessing Learning Outcomes 2 and 3 College Algebra common final o common final administration and data collection o data base data entry o statistical analysis/data presentation o longitudinal studies Continue AY2004-2005 Assessment Activities in Assessing Learning Outcomes 2 and 3 Linked Entrance Exam/Common Final o pretest administration and data collection o data entry o statistical analysis Continue AY2004-2005 Assessment Activities in Assessing Learning Outcomes 1 through 5 Common Projects o project administration o project grading o data entry o statistical analysis Continue AY2004-2005 Assessment Activities in Assessing Learning Outcomes 1 through 5 Student and Faculty Surveys o survey administration o data entry o statistical analysis Write AY2004-2005 Assessment Report 12

B. Benchmarks Tables 8 and 9 define the assessment tool / learning outcome indicators and benchmark values, respectively. Table 8. Learning Outcome Assessment Tools and Associated Indicators. Assessment Tool Learning Outcome Common Final Entrance Exam / Common Final Common Projects 1 Indicators for Comparison with Benchmarks 2 3 4 5 Competency Competency and Change Scores and Change Scores Student and Faculty Surveys Table 9. Benchmark Values for Indicators. Pedagogical Strength Indicator Strong Sufficient Weak Competency >70% 50% to 70% <50% >70% 50% to 70% <50% Score Changes and Sub-Scores Mean Score Change Positive Mean Score Change Negative >70% 50% to 70% <50% >70% 50% to 70% <50% 13