EPE-EDP-ANT/SOC 620/622 Tuesday - Thursday 12:30-1:45 131 Dickey Hall Texts Cook, T.D. and Reichardt C. (eds) Qualitative and Quantitative Methods in Evaluation Research. Beverly Hills: Sage, 1979(CR) This is out of print. Articles from the book and others are available in a packet at IKON. Madaus, G., M. Scriven, and D. Stufflebeam. Evaluation Models. Boston: Kluwer-Nijhoff Publishing, 1983 (MSS) Packet of Readings at Johnny Print Tentative Schedule August 26 Getting our act together August 31 - September 3 History, Models & Tyler Readings: Tyler in packet, Chaps I,1,2,3 (MSS) September 7 - September 16 Extending the core Readings: Chaps 4,6,7 (MSS) Scriven paper in packet September 21 - September 28 Additional Views Readings Chaps 9,14,16,17 (MSS) September 30 - October 8 Qualitative Alternatives Ethnography: John?? Readings to be assigned. Who favors it Readings: Chaps 18,19,20 (MSS) Does it work Readings Chaps in C&R October 12 - November 9 Guest Speakers Persons and topics to be announced. November 11- December 16 Student Reports Depending on the size of the class, each student will have about 1/3 to 1/2 of a class period to present verbally his or her project. Expectations 1. Read the assigned material and come to class prepared to discuss it. 2. Present verbally to class and in written form one of the following: (a) a report of an evaluation you are working on; (b) a report of an evaluation you think should be done, including what should be done, how it should be done, how data should be collected/analyzed, and how results should be reported (c) a report of an evaluation that has been done with your extended critique of how well it was done.
General Description This course is structured to examine a subset of evaluation methods, topics and problems. It is an introductory course with minimal emphasis on quantitative methods. The content of the course is considered appropriate for students in the social sciences and education generally. The course is designed to provide: a perspective from which evaluation studies may be viewed; and, experiences for those who will learn from or conduct evaluation. Introduction Evaluation can be defined as disciplined inquiry to determine the worth of things, where things may include programs, products, procedures or objects. Typically, one attempts to distinguish activities construed as evaluation from those usually associated with research. Although there are no simple ways to differentiate between the two, there are three dimensions on which they may differ. First, evaluation need not have as its objective the generation of knowledge. Evaluation is applied; research is basic. Second, evaluation, presumably, produces information that is used to make decisions or form the basis of policy. Evaluation yields information that has immediate use; research need not. Third, evaluation is, finally, a judgment of worth. Evaluation result in value judgments; research need not and some would say should not. There are a plethora of evaluation models, problems and examples. They come from most of the traditional disciplines of the social sciences and, of course, the field of education. There is no way that a course could cover all of these. Rather, this course will attempt to look at a subset of them. The choice of the subset, however, is not random. An attempt is made to choose those models, problems and examples which may generalize across disciplines. The intent is to engage students in questions of broad methodological significance with studies from different areas of the social sciences and education, so that knowledge and skills can be transferred into their particular discipline or area of interest and form a basis for understanding evaluation studies that have been conducted in a field with which they are most familiar. Figure 1 suggests a way to look at evaluation studies. An evaluator chooses one or more of the existing evaluation models, applies his or her skills to a problem and conducts a study within a set of constraints. The three dimensions are general. The particular labels represent the subset of skills and problems that will be addressed in the course. The content used as a vehicle to discuss these skills and problems will be reports of evaluation studies conducted by persons from various disciplines. the intent is to pursue evaluation skills through content that is familiar to students in various social science disciplines. Figure 1. An heuristic for viewing evaluations. Further Specification
Models 1. Traditional - Tyler Rationale. An emphasis on consistency between goals, experiences and outcomes. 2. Decision-oriented - CIPP. An emphasis on collecting information from a variety of sources to provide a basis for making decisions. 3. Case study - Ethnographic. An emphasis on understanding activities and values, being responsive to diverse audiences. 4. Integrative - Scriven. An emphasis on crucial issues and value judgments. Skills 1. Designing - A emphasis on asking appropriate questions and gathering evidence that responds to those questions. 2. Implementing - An emphasis on procedures and instrumentation that are congruent with the proposed evaluation design. 3. Assessing - An emphasis on data collection and data analysis; a search for consistency between this and (1) and (2). 4. Reporting - An emphasis on writing for an appropriate audience; reporting complex results in straightforward, understandable ways. Constraints 1. Values - An emphasis on what it means to evaluate (make judgments of worth). A look at value orientation and potential conflicts. 2. Contexts - An emphasis on how a setting of an evaluation impinges on what can be done effectively. 3. Design - An emphasis on how a design of an evaluation can affect the type and strength of conclusions that might be reached. Some Sources Abt, Clark C. (ed.) The evaluation of social programs. Beverly Hills: Sage Publications, 1976 Anderson R. B., Education as experimentation: a planned variation model. Cambridge, Mass Abt Associates 1977 Bloom, B. et. al. Handbook of formative and summative evaluation of student learning. New York: McGraw-Hill, 1971 Britan, Gerald, Experimental and conceptual models of program evaluation. Evaluation and Program Planning I, 1978 Campbell, D. T. Reforms as experiments. American Psychologist, 1969, 24, 409-429 Campbell, D. T. and Stanley J. Experimental and quasi-experimental design for research. Chicago: Rand-McNally, 1966 Caro, Francis G. (ed) Readings in Evaluation Research, N.Y.: Russell Sage Foundation, 1977 Carroll, John Bissell The teaching of French as a foreign language in eight countries. New York: Wiley, 1975 Cook, T. (ed) Evaluation studies annual review. Beverly Hills: Sage Publications
Cooley, William W., Evaluation research in education. New York: Halsted Press, 1976. Cronback, Lee J. et. al. Toward reform of program evaluation. San Francisco: Jossey Bass, 1980 Dressel, Paul Leroy, Handbook of academic evaluation (1 st ed) San Francisco: Jossey-Bass, 1976. Eisner, Elliot W. The art of educational evaluation: a personal view. Philadelphia: Falmer Press, 1985. Gottman, John Mordechai, Evaluation in education; a practitioner's guide Itasca, Ill: F. E. Peacock Publishers 1972 Gronlund, Norman Edward (ed), Readings in measurement and evaluation. New York: Macmillan 1968 Green, Joan L. Curriculum evaluation: theory and practice, with a case study from nursing education. New York: Springer Pub. Co., 1977. Guba, Egon G., Toward a methodology of naturalistic inquiry in educational evaluation. Los Angeles: Center for the Study of Evaluation, UCLA Graduate School of Education, University of California, 1978. House, Ernest R. The logic of evaluative argument. Los Angeles: Center for the Study of Evaluation, UCLA Graduate School of Education, University of California, University of California, Los Angeles. Center for the Study of Evaluation, 1977 Isaac, Stephen, Handbook in research and evaluation; a collection of principles, methods, and strategies useful in the planning, design, and evaluation of studies in education and the behavioral sciences. San Diego: Calif., R. R. Knapp, 1971 Joint Committee on Standards for Educational Evaluation, The personnel evaluation standards: how to assess systems for evaluating educators. Newbury Park, Calif. Sage Publications, 1988. King, L.M., and Horkay, N., (eds),the NAEP guide: a description of the content and methods of the 1997 and 1998 assessments. National Center for Education Statistics,Washington, DC: Office of Educational Research and Improvement, U.S. Dept. of Education, 1997 Koretz, Daniel, et. al..perceived effects of the Kentucky Instructional Results Information System (KIRIS) Santa Monica, CA: Rand, 1996. Levin, Henry M. Cost-effectiveness : a primer. Beverly Hills: Sage Publications, 1983. Levin, Henry M. Gene V. Glass, Gail R. Meister, Cost-effectiveness of four educational interventions Stanford, Calif: Institute for Research on Educational Finance and Governance, School of Education, Stanford University, 1984 Mark, R. and Lance Shotland, (eds),.multiple methods in program evaluation. San Francisco: Jossey-Bass, 1987. Millman, Jason, A primer on decision analysis. Portland OR: Research on Evaluation Program, Northwest Regional Educational Laboratory, 1983 Mislevy, Robert J. What can we learn from international assessments. Princeton, N.J.: Educational Testing Service; Washington, DC U.S. Dept. of Education, Office of Educational Research and Improvement, Educational Resources Information Center, 1995 Madaus, G. F. & Stufflebeam, D.L. Educational evaluation: classic works of Ralph W. Tyler. Boston: Kluwer Academic, 1989 Morris, Lynn Lyons. How to measure program implementation - Beverly Hills: Sage Publications, 1978 Patton, Michael Quinn, Qualitative evaluation methods. Beverly Hills: Sage Publications, 1980
Popham, J. Evaluation in education: current applications. Berkeley, Calif: McCutchan Pub. Corp, 1974 Popham, J. Educational evaluation Englewood Cliffs, N.J: Prentice-Hall, 1975 Riecken and Boruch (eds) Social experimentation: A method for planning and evaluating social intervention. Rossi, Peter H., Freeman, H.E. & Wright, S.R. Evaluation: A systematic approach (2 nd ed) Beverly Hills: Sage Publications, 1979 Scriven, M., Evaluation thesaurus. (4 th ed) Beverly Hills: Sage Publications, 1991 Sechrest, L. Evaluation studies annual review. Beverly Hills: Sage Publications, 1979 Smith, J. K. (editor) Proceedings of the seminar "Cost analysis in educational evaluation: where to from here. Portland, OR: Research on Evaluation Program, Northwest Regional Educational Laboratory, 1983 Smith, Mary Lee, The politics of assessment: a view from the political culture of Arizona. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Graduate School of Education & Information Studies, University of California, Los Angeles 1996 Stonehill, Robert M., An evaluation of ESEA Title I: program operations and educational effects. Washington, D.C: U.S. Dept. of Education, Office of Planning, Budget, and Evaluation, 1982 Subcommittee on Education, Arts, and Humanities. Evaluation of Title I. United States Senate, One Hundred Fourth Congress, second session, February 20, 1996. Washington: U.S. G.P.O., 1996 Tyler, R. Basic principles of curriculum and instruction. Chicago: University of Chicago Press, 1949 Tyler, Ralph Winfred, Robert M. Gagne & Michael Scriven. Perspectives of curriculum evaluation. Chicago: Rand McNally 1967 United States Office of Education. Office of Evaluation and Dissemination, Annual evaluation report on programs administered by the U.S. Office of Education. Washington DC: Dept. of Health, Education and Welfare, Weiss, C. (ed) Evaluating social action programs. Boston: Allyn & Bacon, 1972 Worthen, Blaine R. Evaluating educational and social programs: guidelines for proposal review, onsite evaluation, evaluation contracts, and technical expertise. Boston: Kluwer-Nijhoff, 1987 Worthen, Blaine R and Sanders J. R. Educational evaluation: theory and practice. Worthington, Ohio: C. A. Jones Pub. Co., 1973