University of Massachusetts Lowell Graduate School of Education Program Evaluation 07.642 Spring 2014 - Online Instructor: Ellen J. OʼBrien, Ed.D. Phone: 413.441.2455 (cell), 978.934.1943 (office) Email: Please use the email through Blackboard Learn Office Hour: Mondays 7:00 8:00 PM, beginning January 27, 2014 COURSE DESCRIPTION Evaluation tasks will be identified and the policy issues attendant to evaluation will be examined. Students will identify and discuss several models of program evaluation, understand what needs to be considered and addressed in needs assessment, learn to identify an appropriate design for a new evaluation. Students will be expected to conduct program evaluation, present their ideas and illustrate how evaluation results can be useful for program decision making. CONCEPTUAL FRAMEWORK The mission of the University of Massachusetts Lowell is to enhance the intellectual, personal and cultural development of its students through excellent, affordable educational programs. The University seeks to meet the needs of the Commonwealth today and into the future and supports the development of sustainable technologies and communities through its teaching, research, scholarship and engagement. The Graduate School of Education (GSE) contributes to this mission by developing professionals who help transform the region and beyond through leadership roles in education. Through this course, the GSEʼs commitment to Education for Transformation produces graduates who: Demonstrate excellent knowledge, judgment and skills in their professional fields Promote equity of educational opportunity for all learners Collaborate with other educators, parents, and community representatives to support educational excellence Use inquiry and research to address educational challenges Possess the collaborative capability to transform relationships among people in schools, mobilizing them to accomplish purposes they value The central tenets of our conceptual framework are Excellence, Equity, Inquiry and Collaboration. In this class we will strive to create an online environment in which those values are essential. Our collaborative learning environment will center around the inquiry and equity of the students in the class, as well as respect for equity and the use of inquiry as we work with programs to help create evaluation plans outside of class.
Your work both in the classroom and with program directors and/or stakeholders for the programs you will be working with is expected to reflect your desire for educational excellence. STUDENTS OUTCOMES This course is designed to enable students to: Identify and discuss several models of program evaluation Understand what needs to be considered and addressed in a needs assessment Be able to identify an appropriate design for an evaluation Complete tasks important in creating an evaluation plan Become comfortable presenting ideas and plans to stakeholders Understand how evaluation results can be useful for program decision making. REQUIRED COURSE TEXTS Posovac, E. J. (2011). Program evaluation: Methods and case studies. 8 th Edition. Upper Saddle River, New Jersey: Prentice Hall. ISBN-10: 0205804977 ISBN-13: 978-205804979. The National Science Foundation. (2010). The 2010 User Friendly Handbook for Project Evaluation. This book may be obtained through: http://www.westat.com/pdf/projects/2010ufhb.pdf. This should be done immediately. RECOMMENDED COURSE TEXT American Psychological Association. (2009). Publication manual of the American Psychological Association (6 th Edition). Washington, DC: American Psychological Association. STUDENTS WITH SPECIAL NEEDS Students with documented special needs are responsible for making their needs known to the instructor by the end of the first week of class, or as documented. Students are responsible for seeking and arranging available assistance from the Office of Student Disability Services, 240 OʼLeary, 978-934-4574, or by contacting http://www.uml.edu/student-services/disability/services.html. This course adheres to the University's policy to provide reasonable accommodations for persons with disabilities and for religious reasons. http://www.uml.edu/equal/accommodations/accomodations.html.
COURSE POLICIES What I Expect of You Full participation in each class module On-time completion of all assignments Sufficient outside time spent on the coursework to meet the course objectives Demonstrated effort toward excellence in class work Collaboration with classmates as appropriate Respect for and consideration of classmates Enthusiasm toward professional growth, exploration, and change What You Can Expect of Me Planned and organized class sessions Demonstration of good instructional practices Fair evaluation procedures (including constructive feedback) Creation of an environment in which ideas may be openly shared and discussed Availability for questions and support A sense of humor and understanding ACADEMIC INTEGRITY Students are expected to adhere to the Universityʼs policies for Academic Integrity. These can be accessed at http://www.uml.edu/catalog/undergraduate/policies/academic_dishonesty.htm. Cheating and plagiarism are not acceptable and will not be condoned. Cheating and/or plagiarism will result in a failing grade for the assignment, and could result in a failing grade for the class. CLASS FORMAT The class, like good program evaluation, is dependent upon collaboration among the stakeholders (you), the program director (me), and the evaluators (us). You will be asked to lead online discussions, contribute to discussions, and create and post materials for presentation. Although I hope to teach you many things about evaluation, you are ultimately responsible for what you learn in the class. So come online ready to
contribute and learn! I will be available to chat with you during my weekly chat hour. If you cannot make the chat time, but would like to chat with me either online or on the phone, please email me and let me know! The new week begins on Mondays at 12:01 am. You will have until the following Sunday at midnight to post responses to discussion questions. This course is being offered in a 10-week format. There is a great deal to accomplish in this timeframe. Please plan accordingly. MAJOR COURSE ASSESSMENTS Assessment of Evaluation Needs 20% Discussion Leading 10% Discussion Participation 15% Evaluation Plan 35% Final Examination 20% Participation Participation in class will be assessed weekly, based on you assignment postings, timeliness, and any other elements of participation expected for the week. The assessment will be holistic, spanning all of your contributions during the week. Unlike major projects, there will not be regular feedback on your participation assessment. Unlike other assessed work in this course, participation may NOT be revised once a learning module is completed. If you do not participate in a particular week, you do not have the option to make up the work. Posts need to be substantive, and on topic. General/non-descript statements are not considered substantive ( I agree with you, I never thought of that, I feel the same way. ). Substantive posts are directly related to the reading material, and classmatesʼ posts. Therefore, citations from reading are expected. Citations from the required textbooks need only the authorʼs name (Posavac, or NSF), and page number. However, all other citations should be in APA format. SUBMISSION OF WORK Work is due on the day and time indicated for each assignment, as indicated below, and found in the Week at a Glance resource provided each week. Formal assignments must adhere to APA style. Weekly postings may be more informal, however, appropriate citations are always required. If you anticipate having trouble meeting a due date, please contact me in advance so that we may negotiate an appropriate alternative arrangement. All deadlines are in Eastern Standard Time; so for those of you in different zones, please take this into account.
ALL written work must be submitted in Word format. It is your responsibility to convert to Word prior to submission. ELECTRONIC SUBMISSION GUIDELINES The required assignments for this class will be submitted within the appropriate Learning Module, within the provided DropBox. Assignments must be formatted as a Word document, or as a Power Point document. The course calendar showing each learning module of the course, together with the reading for that learning module and assignments, are in this syllabus. A learning module is the course content for one week. Much more detail about topics, readings, and assignments than what appears below will be provided within each learning module. Each learning module begins on Monday. Discussion threads occur throughout the week. Classes run from Monday through Sunday. Based on the above, a typical week will proceed as follows: Monday: New learning module released Monday-Sunday: Review of information; reading; collaborative dialogue; response to prompts COMMUNICATION AND PARTICIPATION Email: Blackboard works with its own internal mail system, which means all messages are sent, stored, and read within the course, not through your own external mail. To access mail, you must be logged in to this class. It is recommended that you check your email at least once a day, so that you donʼt miss important course information. I will respond to your email within 48 hours of your sending it. Please send all email to me via the course email. If I do not respond to you, please assume that I did not receive the email, and send it again. Discussion Boards: Each Learning Module (week) has its own discussion boards. Each discussion board consists of weekly questions. Make the most of this information sharing resource. The discussion boards serve as an extension of the face-to-face classroom in a traditional class. It is an important part of the online learning experience and should be maximized by all students. It is expected that you will contribute not only with your own thoughts/reflections, but also by responding to othersʼ posts. You will do a great deal of this work within your work groups. Netiquette: Netiquette stands for Network Etiquette. It refers to proper behavior while interacting
online. The golden rule of netiquette is essentially to treat your classmates and colleagues as you would want to be treated. Please be polite and considerate. Think about whether your comment could cause hurt feelings. Be careful about how your words can come across because misunderstandings can be common online. Calling Me: Please feel free to call me! There are some things you may want to discuss or question, that you may not feel comfortable with in email. I am more than fine with this! If I donʼt answer my phone, please leave a message for me (and remind me if you are in a different time zone), and let me know when it would be convenient for me to call you back. My cell phone is 413.441.2455. My office phone is 978.934.1943. Chat: I will be available to chat with you during my weekly chat hour. If you cannot make the chat time, but would like to chat with me either online or by telephone, please email me and let me know. Weekly chats are Mondays, from 7:00 8:00 PM.
COURSE SCHEDULE Learning Topic Assessed Work due Module Week 1 Introduction to course, Overview ~ Participation Week 2 Planning an evaluation ~ Participation Week 3 Developing and Using Program Theory, Measures of Implementation ~ Participation ~ Program Approval and Outcomes Week 4 Ethics; Needs Assessments ~ Participation ~ Letter of Agreement Week 5 Program Monitoring ~ Participation ~ Summary of Program/Eval. needs Week 6 Single Group Designs; Quasi- Experimental Approaches ~ Participation ~ Logic Model Week 7 Conducting Research while ~ Participation Evaluating: Qualitative and Quantitative Methods Week 8 Cost Analyses & Evaluation Reports ~ Participation ~ Evaluation Plan Week 9 Encouraging Utilization ~ Participation ~ Presentation - Evaluation Plans Week 10 Pulling It All Together/Summary ~ Participation ~ Letter from program director ~ Final exam
Reading Assignments by Chapter Titles Week Dates Program Evaluation Chapters User Friendly Handbook Pages 1 Jan. 21-26 1-2 Jan. 27 - Feb. 2 2 1-14 3 Feb. 3-9 3-4 15 37 4 Feb. 10-16 5-6 75 93 5 Feb. 17-23 7-6 Feb. 24 - Mar. 2 9-10 - 7 Mar. 3-9 8 & 11 52-74 8 Mar. 10-16 12-13 39-51 9 Mar. 24-30 14-10 Mar. 31 - Apr. 5 - - NOTE: Other readings may be assigned as the semester progresses.
EVALUATION OF LEARNING ASSIGNMENT DUE DATE POINT VALUE 1. Assessment of Evaluation 20% of Final Grade Needs a. Approval of Program Feb. 9, 5:00 pm 5 points b. Letter of Agreement from Feb. 14, end of day 10 points Program Manager/Director c. Summary of program & their Feb. 23, end of day 50 points evaluation needs d. Logic Model for program Mar. 2, end of day 35 points 2. Discussion Leading 10 % of Final Grade a. Approval of questions Thursday before 15 points getting them to me on time and editing as needed. leading week b. Timely responses to posts Daily during leading 20 points week. c. Content of responses and 30 points posts d. Content of summary 20 points e. Timely posting of summary By following Wednesday, 5:00 PM 15 points 3. Discussion Participation By Saturday 5 PM for initial responses, by Sunday end of day for responses to classmates each week (you get to drop one week; your discussion leading week does not count in this grade either). 4. Evaluation Plan (see Mar. 16, end of day 15% of Final Grade (The grades for 8 weeks will be totaled to comprise this). 35 % of Final Grade guidelines) a. Summary Paper 30 points b. Budget 10 points c. Timeline 10 points d. PowerPoint presentation 30 points e. Logic Model 10 points f. Letter from program director indicating receipt of paper &/or presentation Apr. 4, end of day 10 points 5. Final Exam Apr. 5, 5:00 pm EDT 20% of Final Grade
Course Requirements: 1) Online discussion leader. You will create discussion questions one week. You will each be expected to lead the discussion for a question that you decide to post. This means that you must carefully read all responses to your question and respond appropriately to the comments and questions your classmates post. You will be required to post your discussion questions by 8:00 am on Monday of the week you are assigned. You should plan to get my approval for your discussion ideas by the Thursday before your assigned discussion leading week (unless you choose week 2!). You will be asked to complete an additional reading on your discussion topic, so that you can contribute additional knowledge to the discussion. You must also post a summary of the discussion by the following Wednesday at 5:00 p.m. 2) Class participation in Discussion Sessions. You will be assigned to a discussion group for the duration of the course. Each week you will discuss the readings online with that group. In addition to coming to each discussion prepared to discuss the readings, you will be expected to contribute at least two ideas or comments on othersʼ ideas each week. Your comments/ideas must be relevant to the topic of the week and indicate that you did complete the readings and/or assignments for the week. See the document entitled Class Discussions in the Course Resources folder for more detailed information about the grading scheme for discussion participation. 3) Final exam. The exam will be comprehensive, and it will include essay and a few short answer questions. It will be posted by 5:00 p.m. EDT March 31st and must be completed by April 5th 5:00 p.m. EDT 4) Program Summary Paper & Evaluation Plan. You will be required to find a program to work with this semester in order to create an evaluation plan. This may be a program that you are currently working with, or it may be a new program to you. You need to get approval from me by Feb. 9th and you must meet with the program director prior to Feb. 14th. You will probably need to meet with the program director a few times during the semester. You will ultimately give them a copy of your evaluation plan and/or present to them your findings. The papers that you turn in must be in APA format. If you do not know APA format, please take the time to learn it. The program assessment summary paper (due Feb. 23rd) must be no more than 5 double spaced pages, including any figures and graphs. The point of this paper is to allow you to assess the programʼs need for an evaluation. It will also indicate that you are thoroughly familiar with the program. Your paper must include a program description that states the primary purpose of the program, the
mission, a description of the stakeholders and their roles in the program, and any other any relevant program issues. You will also need to include a description of any evaluation activities that have already been done or are currently being done for the program. You must clearly explain why an evaluation plan is needed for the program. You must have a title page and references that are formatted according to APA style. The title page, reference and any appendices are not included in the page count. The evaluation plan (due Mar. 16th) must include evaluation questions and hypotheses, the design and criteria, the analysis plan (including a schedule), a budget and plan for dissemination of results. Your plan must be based on a literature review and material covered in class. You will be required to post a Power Point presentation of the plan with the stakeholders as the presumed audience. You will also turn a brief written description of the plan which may be no more than 7 double spaced pages, plus a budget, timeline and a list of references. See additional posted materials for details on the requirements for this assignment. RUBRIC Graduate School of Education Overall Grade Rubric Grade GPA Point structure Comments A+ 4.0 99-100% Work of the highest professional standard demonstrating independence and exemplary performance. A 4.0 96-98% Excellent work demonstrating independence and high quality performance. A- 3.7 91-95% Very good work, indicating consistent and careful thought and attention to the task, but requiring some areas of improvement. B+ 3.3 86-90% Good work, carefully executed for the most part, yet requiring several areas of improvement. B 3.0 80-85% Work of graduate standard, but omissions exist or careful analysis is not evident. Below Graduate Standard B- 2.7 76-79% Effort is evident, but work indicates lack of understanding of the demands of the task. C+ 2.3 70-75% Poor quality work with little attention to detail and the demands of the task. C 2.0 65-69% Work of very poor quality, indicating no understanding of the depth of analysis required. F 0.0 Below 65% Serious neglect or evidence of academic dishonesty.
Weekly Learning Modules/Calendar Spring Week # Dates 1 Jan. 21 26 2 Jan. 27 Feb. 2 3 Feb. 3 9 4 Feb. 10-16 5 Feb. 17 23 6 Feb. 24 Mar. 2 7 Mar. 3-9 8 Mar. 10-16 9 Mar. 24-30 10 Mar. 31 Apr. 5 Classes run from Monday through Sunday each week. End of day is using Eastern Time/Eastern Daylight Savings Time There is no class the week of Spring Break March 17-23 Week 10 is a shorter week, ending on Saturday Chat is Monday nights from 7:00 8:00 PM