Instructor Contact Information Anne Seraphine, PhD seraphine@coe.ufl.edu 352-284-0831 (texts are welcome) Office: 1207 Norman Hall Office Hours: By appointment David Miller, PhD dmiller@coe.ufl.edu 352-273-4306 Office: 1215D Office Hours: By appointment EDF 7491 Evaluation of Educational Products and Systems Course Credit Hours: 3 Fall 2016 Course Description Provides broad coverage of model and methods for formative and summative evaluation of educational products and programs. The need for effective program evaluations is on the rise, particularly in the face of demands for greater accountability. As a result, policy makers, funding agencies, and other stakeholders are under increased pressure to show how programs work and what can be done to improve them. Figuring out how programs work is the primary job of the program evaluator. The work of the evaluator draws on a wide range of disciplines, including evaluation theory, quantitative and qualitative research methodology, project management; multicultural communication and group facilitation; evaluation and research ethics; and organizational theory. The purpose of this course is to provide a broad, introductory coverage of these and other basic topics relevant for the program evaluator. Prerequisite NONE Course and Student Learning Objectives By the end of the course students will: Identify a breach of ethics when given various evaluation examples and describe the ethical violations and what can be done to prevent such a violation. Describe the five attributes of the Standards of Evaluation and apply them to a evaluation situation in your discipline. Compare and contrast the Standards of Evaluation and the Guiding Principles of Evaluation. Make a distinction between research misconduct and unethical research conduct when discussing responsible conduct of research. Describe and critique in terms of strengths and limitation various approaches to evaluation, including accreditation, expert versus consumer, the CIPP model, Patton s Utilization-Focused Evaluation, Logic Models, and Naturalistic Evaluations. Be able to describe each approach in terms of its epistemological assumptions and paradigm. page 1
When given open-ended exam questions, describe, analyze, and evaluate various evaluation approaches; evaluation theory and concepts; the four validities internal, construct, statistical conclusion, and external, as they apply to evaluation; preexperimental, quasi-experimental, and randomized experimental designs; the politics of evaluation; and the collection of evaluation data. When given written descriptions of evaluations, critique in writing the evaluations in terms of sampling, measurement, and research design. Provide recommendations how to improve the evaluation to enhance its internal, statistical conclusion, external, and construct validity. Apply to evaluation approaches to an evaluation plan for program that is somehow related to the students chosen field of study. Course Texts and Readings Required Texts: Fitzpatrick, J. L., Sanders, J., & Worthen, B.R. (2011). Program evaluation: Alternative approaches and practical guidelines (4 th ed.). Upper Saddle River, NJ: Pearson Education, Inc. (FSW) Recommended Texts: Joint Committee on Standards for Educational Evaluation. (2011). The Program evaluation standards (3 rd ed). Thousand Oaks, CA: Sage Publications. (Standards) (JCSEE) Cook and Campbell (1979). Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin Company. (CC) Additional Readings: Chelimsky, E. (2008). A clash of cultures: Improving the fit between evaluative independence and the political requirements of a democratic society. American Journal of Evaluation, 29(4), 400-415. doi: 10.1177/1098214008324465. Christie, C.A. & Alkin, M.C. (2013). An evaluation theory tree. In M.C. Alkin (Ed.) Evaluation roots: A wider perspective of theorist s views and influences (pp. 11-57). Thousand Oaks: Sage Publications. House, E. R. (1983). Assumptions underlying evaluation models. In In G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam (eds.) Evaluation Models: Viewpoints on educational and human services evaluation (pp.45-64). Boston: Kluwer Academic Publishers. Maduas, G. F. & Kellaghan, T. (2000). Models, metaphors, and definitions in education. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (eds.) Evaluation Models: Viewpoints on educational and human services evaluation (2 nd ed.) (pp. 19-32). Boston: Kluwer Academic Publishers. Madaus, G.F., & Stufflebeam, D. L. (2000). Program evaluation: A historical overview. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (eds.) Evaluation Models: Viewpoints on educational and human services evaluation (2 nd ed.) (pp. 3-18). Boston: Kluwer Academic Publishers. Scriven (2000). Evaluation Ideologies. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (eds.) Evaluation Models: Viewpoints on educational and human services evaluation (2 nd ed.) (pp. 33-84). Boston: Kluwer Academic Publishers. W.K. Kellogg Foundation (2004). Logic Model Development Guide. (pp. 1-25). page 2
Weekly Schedule of Topics, Readings, and Assignment Week Topics Readings Week 1 Evaluation Introduction (A & B Intro) Syllabus, Example, Evaluation Definitions & Research vs. Evaluation FSW 1 M-1 Week 1 Evaluation History, Ethics with RCR/IRB FSW 2, 3 M-2 Week 2 Guiding Principles for Evaluators, Standards for Evaluation, Five attributes utility, feasibility, propriety, accuracy, accountability parts 1-4 FSW 3; JCSEE M-3 Week 3 Week 4 Week 4 No Class--Holiday Overview of Models (Goal-based, Judgement-based, Decision- Based, Naturalistic, Participant-Based, Program-Based) and Overview of Paradigms Expert & Consumer-Oriented (Accreditation, Scriven guidelines) and Program-Based (In-depth Logic Models) Week 5 Decision-Based Model(Stufflebeam s CIPP Model and Context, Input, Process, and Product Evaluations) Week 6 Context Evaluation Technique (Delphi) and Decision-Based Model (Patton s Utilization Focused-Evaluation) Week 7 Politics and Evaluation Utilization: Practicalities of Power, Politics, and Ethics Week 7 EXAM 1 FSW 4, 10 M-4 FSW 5, 6, 12 Kellogg M-5 FSW 7 M-6 FSW 7 M-7 FSW 11 M-8 Week 8 Naturalistic Evaluation (naturalistic approach, constructivist paradigm and assumptions, the role of the evaluator) FSW 8 M-9 Week 9 Culturally Responsive Evaluations FSW 9 M-17 Week 10 No Class--American Evaluation Conference Week 11 Beyond the Model Practicalities and Considerations (Key Considerations in Planning, Chelimsky Five Summary Points, FSW 11,12,13 M-10 Week 12 Construct Validity Scales, psychometric characteristics-reliability, and validity FSW 16; CC 2 M-11 Week 13 Sources and Collection of Evaluative Data Interviews, Surveys and Focus groups FSW 14, 16 M-12 Week 14 Design Conditions of causality, threats to internal validity, quasi- and randomized-experimental designs. FSW 15; CC 1, 2, 3 M-13, M-14 Week 15 Statistical Conclusion and External Validity FSW 15, 16; CC 2 M-15 Week 16 Reports and Communication Meta-evaluations. FSW 17 M-16 Week 16 EXAM 2 page 3
Methods by Which Students Will Be Evaluated and Grades Determined Exams 2 (20% each, 40% total): Each exam will focus on the knowledge and application of content presented in readings and lectures. Item format will be a combination of short open ended answers. Logic Model (15%): Describe an evaluation briefly and develop a logic model for the evaluation. Include a brief written description of the logic model. The paper should be about 2 pages of writing and a logic model, 12-pt font, 1.5 spaced. CIPP Model (15%). Apply the CIPP evaluation approach to an evaluation design of interest. Identify and briefly describe a program for evaluation (1-2 paragraphs). Describe a CIPP evaluation for the program (Maximum 2 pages for each of the four types of evaluation). Patton Model (15%). Apply Patton s evaluation approach to an evaluation design of interest. Identify and briefly describe a program for evaluation (1-2 paragraphs). Describe Patton s evaluation for the program (Maximum 4 pages). Culturally Responsive (15%). Apply a culturally responsive evaluation approach to an evaluation design of interest. Identify and briefly describe a program for evaluation (1-2 paragraphs). Describe a culturally responsive evaluation for the program (Maximum 4 pages). For each of the 4 projects, it is expected that you would use the same program to be evaluated with an emphasis on the different approaches. Consequently, the same program description should be used for each assignment. Final Grade =.20(Exam 1) +.20(Exam 2) +.15(Logic Model) +.15(CIPP Model) +.15(Patton Model) +.15(Culturally Responsive) Course Grading Scale Score Grade 93.4-100 A 90.0-93.3 A- 86.7-89.9 B+ 83.4-86.6 B 80.0-83.3 B- 76.7-76.9 C+ 73.4-76.6 C 70.0-73.3 C- 66.7-69.9 D+ 63.4.66.6 D 60.0-63.3 D- 0-59.9 E For more information about university policies, see: http://www.isis.ufl.edu/minusgrades.html page 4
Course and University Policies Attendance, Make-Up Exams, and Assignments Requirements for class attendance and make-up exams, assignments, and other work in this course is consistent with university policies that can be found at: http://gradcatalog.ufl.edu/content.php?catoid=4&navoid=907#attendance Accommodations for Students with Disabilities Students requesting accommodation for disabilities must first register with the Disability Resource Center - www.dso.ufl.edu/drc/. The DRC will provide documentation to the student who must then provide this documentation to the instructor when requesting accommodations. Students should contact the DRC and complete this process as early as possible in the term for which they are seeking accommodations. UF Student Honor Code UF students are bound by the Honor Pledge which states, We, the members of the UF community, pledge to hold ourselves and our peers to the highest standards of honor and integrity by abiding by the Honor Code. On all work submitted for credit by students at UF, the following pledge is either required or implied, On my honor, I have neither given nor received unauthorized aid in doing this assignment. The Honor Code (http://www.dso.ufl.edu/sccr/process/student-conduct-honor-code/) specifies a number of behaviors that are in violation of this code and the possible sanctions. Furthermore, you are obliged to report any condition that facilitates academic misconduct to appropriate personnel. If you have any questions or concerns, please consult with the instructor in this class. Online Course Evaluation Process Students are expected to provide feedback on the quality of instruction in this course based on 10 criteria. These evaluations are conducted online at https://evaluations.ufl.edu. Evaluations are typically open during the last two or three weeks of the semester, but students will be given specific times when they are open. Summary results of these assessments are available to students at https://evaluations.ufl.edu. Student Assistance and Emergencies University support services are available to students who are experiencing significant distress and/or personal emergencies. As appropriate please contact: UF Counseling & Wellness Center: www.counseling.ufl.edu or 352-392-1575 University Police Department: 352-392-1111 or 9-1-1 for emergencies Software Use All faculty, staff, and students of the University are required and expected to obey the laws and legal agreements governing software use. Failure to do so can lead to monetary damages and/or criminal penalties for the individual violator. Because such violations are also against University policies and rules, disciplinary action will be taken as appropriate. Information Specific to Online Course Delivery Course Structure This course is taught online and requires active participation. Students are expected to be present and contributing to the online discussion each week. If unable to participate while away, students should contact the instructor in advance to avoid being considered absent. Throughout the course, the instructor will: Read and respond to all posts and emails within a 48-hour time frame; and, page 5
Post online office hours twice a week for direct contact via email or phone--tuesdays and Thursdays at 5-8 PM. Course Website and Communication The course website will run via Canvas through the UF e-learning website; go to http://elearning.ufl.edu and click on the Canvas Login button. The course site will be used to post relevant announcements, reading, lecture materials, links, assignments and quizzes, etc. You are responsible for checking this site for announcements and to verify that your grades are recorded correctly. Technical Assistance If you require any technical assistance, please click the Help button on the upper right side of Canvas screen to submit a request for help. You can also call the UF Help Desk at 352-392-4357. page 6