Speech Assessment Report (2006 07) Part 1 Assessment Summary for 2005 06 A. Program/Discipline Mission Statement The ACC Speech discipline facilitates student learning through well defined theoretical and performance based instructional activities in a learning centered environment. Through continued assessment, we prepare students for academic transfer and careers in communication. B. Intended Learning Outcomes Students will be able to: 1. Phrase clear purpose statements for a well chosen topic that is appropriate for the audience and occasion. 2. Develop specific, well focused thesis statements for appropriately chosen topics. 3. Use an organizational plan that is appropriate for the topic, purpose, audience and occasion. 4. Develop appropriate supporting material for the audience and occasion. 5. Establish credibility by demonstrating knowledge of the topic and incorporating material from appropriate sources, using proper verbal citations. 6. Use transitions effectively to promote a smooth flow of information. 7. Prepare and use presentation aids that promote clarity and interest. 8. Use language that is appropriate for the audience, purpose and occasion. 9. Use pronunciation, grammar and articulation appropriate to the audience. 10. Use vocal variety to engage the audience and sustain interest. 11. Use nonverbal behaviors to support the verbal message. 12. Demonstrate the ability to cite external sources using MLA format. C. Benchmark Our benchmark is that 70% of students will achieve a proficiency of 4 or higher on the assessment criteria for outcomes #1, #2, and #12. 2005 2006 Speech: Revised Assessment (3.29.07) Page 1
D. Assessment Results 1. Historical Context Based upon the assessment results of the 2004 05 academic year, the Speech faculty proposed: 1) Increasing the focus of the program in several areas: Spend more class time linking the use of citations to credibility and plagiarism. Revise the library assignment to require students to write both oral and written citations. Devote more class time to the importance of creating well developed speech conclusions. 2) Revising the assessment instrument and rubric to clarify language for the students and give greater emphasis and weight in grading on the use of oral citations in student speeches. 3) Gathering demographic information from students to get additional data for qualitative analysis of student performance, specifically to determine if performance is affected by gender, age or ethnicity. 2005 2006 Speech: Revised Assessment (3.29.07) Page 2
2. Current Year Data Results and Analysis The following chart shows the average scores for the learning outcomes identified in Section B. Comparing this year s results with the assessment results from the 2004 2005 academic year is difficult because criteria totals were based on a 5 point scale and the 05 06 criteria were based on a 4 point scale. Nevertheless, the chart below shows that Conclusion and Outline are still somewhat of a weak point (comparatively). Speech 2005-06 Assessment: Avg Criteria Totals 3.50 2.50 1.50 0.50 0.00 2.91 Audience/ Topic Introduction 2.67 2.63 2.71 2.83 2.55 2.57 2.45 Body Conclusion Organization Language Delivery Visual Aids 2.39 Outline 1. The following charts were produced by the teacher evaluation of the students. The first chart reveals that lower values were ability of the student to capture the audience s attention and their preview of the main ideas. Topic Choice and Introduction: Average Score Comparisons 2.92 2.90 2.85 2.62 2.54 Topic is appropriate for speech assignment Appropriate, culturally sensitive Captures audience's attention Clearly identifies topic/thesis Previews main ideas [24] 2005 2006 Speech: Revised Assessment (3.29.07) Page 3
3. It seems that students could use some work on continuity of presentation as shown here in the category of smooth flow of information. Body: Average Score Comparisons 2.68 2.57 Well developed main ideas Smooth flow of information 4. 36 students scored a 0 or a 1 in memorable concluding device. Students relatively low ability to use a memorable concluding device combined with their low score in the first (Topic Choice) of previewing main ideas indicates that they need to encapsulate their speech in a way that follows the old adage: Tell them what you are going to tell them, tell them, and tell them what you told them. Conclusion: Average Score Comparisons 2.64 2.42 2.28 Restates topic/thesis Reviews main ideas [26] Uses memorable concluding device [36] 2005 2006 Speech: Revised Assessment (3.29.07) Page 4
One flaw in the teacher assessment instrument was including clear transitions and smooth flow of information under Organization. The instrument is being revised to put smooth flow of information under Delivery. Organization: Average Score Comparisons 2.69 2.74 Clear organizational pattern Clear transitions 6. Language scores were relatively high. Language: Average Score Comparisons 2.81 2.86 2.83 Clear and concise Proper grammar Vivid language use 2005 2006 Speech: Revised Assessment (3.29.07) Page 5
7. Delivery was about average compared to the other categories. 30 students scored a 0 or 1 in Appropriate Movement and Gestures, however. Delivery: Average Score Comparisons 2.51 2.57 2.57 2.38 Vocally expressive Conversational Adequate eye Appropriate style [20] contact movement and gestures [30] 8. Visual aids were about average. Visual Aids: Average Score Comparisons 2.51 2.56 2.52 2.67 Professional quality Simple & Visuals meet Displayed at [21] uncomplicated [20] guidelines for appropriate time proper use [21] 2005 2006 Speech: Revised Assessment (3.29.07) Page 6
9. This graph shows a relative lack of expertise in creating outlines when it comes to labeling and expressing purpose. 8.00 7.00 6.00 5.00 Outline: Average Score Comparisons 7.65 2.56 2.31 Typed & in correct Provides clear purpose Transitions included format & thesis statements [24] and labeled [33] 10. Citations were the lowest scoring area. There were many students who scored less than 2 in the various categories. Citations 2.09 1.82 1.83 1.78 Required written Credible citations External sources Citations orally citations in MLA orally cited at orally cited [62] cited correctly [68] format [49] appropriate times [67] 2005 2006 Speech: Revised Assessment (3.29.07) Page 7
Student Self Evaluation 11. The biggest need in this area is that the student self evaluation have the same scale and the same rubric as the teacher evaluation. In that way, we can compare more easily the difference between teacher and student evaluations. Self Evaluation: Audience, Intro, Body, and Conclusion 5 4 4.35 4.16 4.44 3.73 3 Audience Analysis/Topic Choice Introduction Body Conclusion Self Evaluation: Organization 5.00 4.17 3.63 Organized main ideas using a plan that was appropriate for the speech purpose Used transitions effectively to maintain the flow of information 2005 2006 Speech: Revised Assessment (3.29.07) Page 8
Self Evaluation: Language 5.00 4.19 3.99 Used language that was appropriate for the audience and occasion Used accurate pronunciation, grammar and articulation Self Evaluation: Delivery 5.00 3.85 3.87 3.77 3.48 Was vocally Used body Had good eye Used an expressive language to contact extemporaneous support the vocal delivery message 2005 2006 Speech: Revised Assessment (3.29.07) Page 9
Self Evaluation: Visual Aids 5.00 Used presentation aids that were clear and interesting There is a wide discrepancy between the teacher evaluations for citations and the student evaluation of their performance on citations. So students not only are doing poorly in the citations group as we saw earlier, but they were reportedly of the opinion that they had done fairly well. Students rated themselves between 4 and 5, yet the teacher average was 3. Self Evaluation: Citations 5.00 4.27 4.12 3.63 Followed ethical guidelines in research and presentation Used proper verbal citations Used appropriate and credible sources Although Following ethical guidelines was a self evaluation category, there was no such category within the Citation group that was scored by the instructors. This discrepancy will be corrected with the revised rubric. 2005 2006 Speech: Revised Assessment (3.29.07) Page 10
F. Use of Results Faculty members in the speech program met to discuss the data and analysis and agreed to implement the following revisions in the curriculum: 1) We will revise the teacher and student rubrics to make them more comparable for comparison and analysis of student learning. 2) We have decided to gather demographic data on student age only because past analyses have shown little difference in student learning based on gender and ethnicity. We will put the ages into categories, such as 18 to 25, 26 to 39, and 40 & above in order to make the data easier to work with and to allow people some confidentiality in this category. 3) We will increase class time spent on developing a specific purpose and thesis for speeches (Learning Outcomes #1 and #2) since these elements drive every other step in the speech process. It is our hope that improvement in these areas will lead to improvement in organization, outlining, and the development of the introduction, body and conclusion of the speech. 4) We will use more interactive activities to give students additional hands on practice with Learning Outcomes #1, #2 and #12 which we have identified as our focus for the next academic year. This assessment summary will be e mailed to all speech instructors and the chair of English, Speech and Developmental Studies. The assessment results will be reflected in the syllabi and handouts that will be distributed to future SPE 115 students. We will continue to discuss the progress of our new initiatives. 2005 2006 Speech: Revised Assessment (3.29.07) Page 11
Part II Assessment Plan for 2006 07 A. Outcome #12 has been changed from Demonstrate acceptable ethical standards in research and presentation of materials to Demonstrate the ability to cite external sources using MLA format. Outcomes will be assessed in two ways during the academic year. B. Assessment Methods Method 1 All speech instructors teaching SPE 115 during the spring semester will participate in an assessment activity designed to assess students abilities to develop and deliver effective persuasive speeches. (We have decided to assess the persuasive speeches instead of the informative speeches that we assessed in the past. This is the most challenging speaking that the students do, and it is one of the last speeches they give. We believe that this will more accurately assess student learning in the course.) They will use a quantitative rubric that will determine the level of skill in speech development and delivery. The criteria that will be used are derived from a list of competencies and assessment criteria identified in College Sophomore Speaking and Listening Competencies, a document published by the National Communication Association (formerly the Speech Communication Association) in 1990. This document makes national recommendations on communication competence for college students, and includes the twelve (12) learning outcomes that are identified in Part IB. Method 2 The speech program will assess student learning in a second way. All speech instructors will videotape student persuasive speeches. Students will be given a quantitative evaluation instrument similar to the one used by their instructors for the informative speech. Each student will view the videotape of his/her persuasive speech and evaluate his/her ability to achieve the intended outcomes. Teachers will collect the student evaluations before giving the students written feedback so that the student assessments are not influenced by instructor comments on the persuasive speech. The student instrument will include two additional sections: 1) A list of questions that will provide additional data for qualitative analysis. 2) A request for demographic information that will allow for additional analysis of student performance to determine if performance is affected by age. 2005 2006 Speech: Revised Assessment (3.29.07) Page 12
The speech program coordinator will collect blind copies of teacher evaluations for the persuasive speech and student evaluations of videotapes of their persuasive speeches. Blind copies will be collected for each section of SPE 115 that is taught during the spring semester. The coordinator will randomly select 100 student and 100 faculty evaluations and send them to the Institutional Research department for quantitative analysis of the intended outcomes. The coordinator will ask IR to seek correlations between the two sets of evaluations. The coordinator will analyze qualitative information that is provided on both the teacher and student evaluations. These analyses should provide us with valuable information that will identify: 1) Faculty assessment of student proficiency in the identified outcomes, 2) Student perceptions of success in achieving proficiency in the identified outcomes, 3) Similarities in student and teacher perceptions of proficiency in the identified outcomes, and 4) Demographic factors that may be influencing student success. The program coordinator will write the assessment summary based on the results and send them to the assessment committee. Members of the speech faculty will meet to review the data and determine how the results will impact the speech curriculum. C. Benchmarks Our benchmark is that 70% of students will achieve a proficiency of 4 or higher on a 5.0 scale on the assessment criteria for the intended outcomes on the teacher evaluations. D. Budget Requirements None 2005 2006 Speech: Revised Assessment (3.29.07) Page 13