PRESENTER EFFECTS 1 Presenter Effectiveness in Project PAVE Jake Kahn Kelly Lavin Deanna Metropoulos University of Denver Research Methods PSYC 3050-1
PRESENTER EFFECTS 2 Abstract This study examined presenter effectiveness as it relates to TDV knowledge, attitudes, and behaviors. Two thousand thirty seven middle and high school students completed surveys assessing knowledge, behavior, and attitude changes before and after participating in a TDV prevention course by Project PAVE. The hypothesis was that different degrees of TDV knowledge, attitude, and behavior change would occur based on which presenter taught a given class. Furthermore, presenter ratings were predicted to be correlated with gains in TDV knowledge, attitudes, and behaviors. Facilitator 1 demonstrated the largest increase in TDV knowledge, attitude, and behavior changes. These results supported our hypothesis. We also hypothesized there would be a positive correlation between facilitator likeability ratings and how much the students learned. This hypothesis was not supported. The implications suggest that Project PAVE should spend more time supporting and preparing their facilitators for teaching.
PRESENTER EFFECTS 3 Presenter Effectiveness in Project PAVE There are over sixty million adolescents between the ages of thirteen and nineteen in the United States (U.S. Census Bureau, 2013). More than half of them will experience violence including, but not limited to, teen dating violence (TDV) (Finkelhor, Ormrod, Turner, & Hamby, 2005). TDV can be defined as a pattern of behavior to gain power and control over another person using various tactics. Victims of adolescent violence have been shown to suffer from poor academic performance, personality distortions, and significant declines in mental health (Finkelhor & Hasima, 2001). These adolescents also have an increased risk for delinquent behavior (Finkelhor & Hasima, 2001). In an effort to prevent TDV and its harmful consequences, Project Promoting Alternatives to Violence through Education (PAVE) was established in 1986 (Project PAVE Statistics, 2013). Project PAVE offers TDV prevention curriculum in over thirty schools in the Denver area. The six-day program aims to educate students on healthy relationships, help students to identify warning signs of TDV, and break the gender binary opposition (Project PAVE; Dating Violence Prevention, 2013). The focus of the current study was to evaluate the effectiveness of Project PAVE s facilitators, specifically in the TDV prevention school curriculum. Past research on TDV prevention programs have shown them to be effective at decreasing the acceptance of TDV. For instance, youth that participated in Ending Violence (a TDV prevention program), showed more likelihood to seek help against relationship violence than students who did not participate (Jaycox et al., 2006). The ninth year students in ten Los Angeles schools who participated in this prevention program were less accepting of violence aimed towards men, knew more about laws
PRESENTER EFFECTS 4 relating to TDV, and were more likely to seek help for dating violence than the students who never were in a TDV prevention program (Jaycox et al., 2006). Although past research on the effectiveness of TDV prevention programs has been encouraging, we do not know much about the impact TDV prevention program presenters have on the efficacy of the programs. Both a presenters personality and personal history related to the topic in discussion have been shown to affect their evaluations and teaching quality. First, regarding personality, research has shown that students in various courses in a university setting gave better course evaluations to their teachers who displayed four of the Big Five personality traits: extraversion, openness, agreeableness, and conscientiousness (Patrick, 2011). The perceived amount of information learned by the students had no significant effect on the teacher ratings (Patrick, 2011). Furthermore, the relationship between the subject matter and the presenters personal ties to the topic have also been found to affect the way the program s message was delivered and perceived. Facilitators who had a personal tie to the topic they presented on were more likely to be viewed by the participants positively (i.e., receive better ratings) than those who had no personal ties (Schwartz, Thomas, Bohan, & Vartanian, 2007). In sum, teachers were rated higher when they had certain personality traits and when they had a personal affiliation with the topic on which they were presenting. Presenters peer reviewer s evaluations and on-site training affects their teaching abilities and overall effectiveness as well. Dahlgren et al. (2006) demonstrated that presenters who had a peer reviewer, or a critical friend, were provided with positive and valuable feedback about their teaching skills that ultimately lead them to develop more
PRESENTER EFFECTS 5 effective presentation skills. They also showed that acting as a critical friend taught presenters about different methods of teaching that later were implemented into their own teaching styles (Dahlgren et al., 2006). Researchers have also examined whether variables such as onsite training impact presenter effectiveness. For example, Ahsam, Shepherd, and Warren (2006) examined whether on-site teacher training, which included communication workshops and background information in the teacher s subject matter, affected teaching ability. Teachers who received on-site training improved their communication skills with the children and led to higher rates of teacher confidence (Ahsam, Shepherd, & Warren, 2006). TDV prevention groups can utilize similar on-site training in order to improve their presenters communication skills with the adolescents in their groups. Project PAVE aims to diminish TDV; therefore, the effectiveness of their presenters is of high importance. Past research has shown that improving the quality of teaching skills through on-site training, peer reviewers evaluations, and students personal evaluations can increase the facilitator s confidence and/or likeability. The current study focused more basically on presenter effectiveness and perceived learning within Project PAVE s TDV classroom intervention. We hypothesized that there would be different degrees of TDV knowledge, attitude, and behavior obtained based on which presenter taught a given class. Furthermore, we hypothesized that there would be positive correlations between facilitator likeability ratings and how much the students learned. Method Participants
PRESENTER EFFECTS 6 Three thousand two hundred thirty students in the Denver metro area participated in the TDV Prevention program offered by Project Pave. A total of 32 middle- and high schools participated. Of the participants involved, 22.6% were in the 12th grade (n= 731), 19.3% in the 11th grade (n= 625), 13.5% in the 10th grade (n= 436), 26.9% in the 9th grade (n= 870), 7.2% in the 8th grade (n= 233), 1.1% in the 7th grade (n=34), 0.7% in the 6th grade (n=22), and 273 students left the field for grade unanswered. The Hispanic/Latino/Mexican participants comprised 37.5% of the sample (n= 1,211), the Native American/American Indian participants comprised 1.5% (n= 48), Asian or Pacific Islander participants comprised 3.2% (n= 103), White/Caucasian participants comprised 22.4% (n= 723), Black/African-American comprised 13.6% of the sample (n= 440), and participants classified as more than one ethnicity, or other were comprised to 10.8% (n= 348) and 1.2% (n= 38) of the sample. Of the total participants, 316 did not provide ethnicity or racial information. The male participants comprised 32.5% of the sample (n= 1,049), the female participants comprised 54.3% (n= 1,753), the participants that reported their gender as other comprised as.1% (n= 4), and the participants that did not respond comprised of 13% (n= 421). One thousand four hundred ninety eight participants reported as heterosexual (46.4%), 16 participants reported as homosexual (.5%), and 116 participants reported as bisexual (3.6%). Thirty participants reported their sexual orientation as not sure (.9%), and eight participants reported other (.2%), whereas 3 participants did not provide a feasible answer. In the income category, 2.5% (n= 81) of the participants reported a family income of $0.00 to $10,000, 2.1% (n= 69) reported a family income of $10,000 to $25,000, 2.2% (n= 70) reported a family income of $25,000 to $35,000, 2.9% (n= 95) reported a family income of $35,000 to $50,000, 4.2% (n= 135)
PRESENTER EFFECTS 7 reported a family income of $50,000 or more, 35.9% (n= 1160) reported that they did not know their family s annual income, and 50.1% (n=1,1617) of participants did not provide an answer for annual family income. The schools in the study were considered to be in underserved areas within various Denver communities. Presenters were rated by participants at post-test. In some classes there were two presenters teaching at the same time. The mean score of the primary facilitator was 4.57 (SD=.75), and the mean score of the secondary facilitator was 4.53 (SD=.77). TDV knowledge gained, attitudes changed, and behaviors changed were computed by subtracting the pre-test scores of knowledge, attitude, and behavior calculated before from the post-test score for each domain. Measures The differences in responses on two surveys, one given before and one given after the TDV class series were used to measure the overall effectiveness of the program. Project PAVE created their own survey. For the pre-test, knowledge, attitudes, and behavior towards TDV as well as demographic questions were measured on a 5-point Likert scale, from strongly disagree (1) to strongly agree (5). Twenty Likert scale questions were asked, including: If someone I know wants to stay in an abusive relationship there is not much I can do to help. At the end of the pre-test, the questions were scored and summed up to come up with a score for TDV knowledge, attitudes, and behaviors. For the post-test, participants were asked the same questions as the pre-test plus seven additional questions evaluate the presenters effectiveness. One of the questions that they asked was, What did you like about the presentation? The questions answered
PRESENTER EFFECTS 8 on the post-test were then scored and summed up to come up with a score for TDV knowledge, attitudes, and behaviors. The participants answer changes between the pre- and post-test were used as a measure to assess the effectiveness of Project PAVE s facilitators, knowledge, attitude, and behavior changes. The questions in the pre-test were first scored individually and summed together to for all of the following sums: knowledge, attitude, and behavior. The post-test was score in a similar manner for each of the three domains. The changes for knowledge, attitude, and behavior is then resulted by taking the sums of each domain from the pre-test and subtracting them from the sums of each of the domains from the post-test. The presenter effectiveness was a combination of all 3 of the domains. This was calculated after the pre-test, post-test, and the difference of both scores. Procedure Middle- and high schools in the Denver area were sought out to become involved with Project PAVE. When a school agreed to participate, specific classrooms chose to participate in the program. The specific classrooms then coordinated with Project PAVE to establish when to carry out the TDV prevention curriculum. The program itself is a 6- day, 5 session interactive curriculum aimed at providing students with information about healthy relationships in addition to TDV. Topics covered in the program included helping the participants understand and identify warning signs, gender roles, appropriate social skills, and how healthy relationships and bystander intervention could help their peer in the short- and long-term. Before a specific classroom began the TDV curriculum, students completed a pre-test assessing demographic variables in addition to knowledge, attitudes, and
PRESENTER EFFECTS 9 behaviors related to TDV. Students were informed that the surveys were anonymous and that they did not have to answer any question they were uncomfortable with. After the completion of the TDV curriculum, students were then asked to complete and almost identical post-test survey, which assessed not only the same demographic variables and TDV-related information, but also assessed the facilitator and presentation qualities. There was no compensation for either completing the course or completing the surveys. Results Descriptive Statistics For facilitator 1, knowledge mean change was 4.66 (SD=3.98), attitude mean change was 3.28 (SD=5.20), and the behavior mean change was 0.83 (SD=2.30). For facilitator 2, knowledge mean change was 4.17 (SD=4.07), the attitude mean change was 1.17 (SD=4.95), and the behavior mean change was 0.71 (SD=2.01). For facilitator 4, knowledge mean change was 4.23 (SD=3.65), the attitude mean change was 2.33 (SD=4.43), and the behavior mean change was -0.06 (SD=2.30). For facilitator 5, knowledge mean change was 3.19 (SD=3.87), the attitude mean change was 0.69 (SD=4.88), and the behavior mean change was 0.48 (SD=2.02). For facilitator 6, knowledge mean change was 3.59 (SD=4.00), the attitude mean change was 1.47 (SD=4.80), and the behavior mean change was.48 (SD=2.33). Table included over. Inferential Statistics For all statistical tests, the alpha level was set at 0.05. First, we used the Pearson correlation coefficient to examine the relationships between attitude, behavior, and knowledge changes. The knowledge change had a correlation coefficient r=.41, p<.05 in relation to attitude change, and r(0.11) in relation to behavior change. Knowledge change
PRESENTER EFFECTS 10 and attitude change were positively correlated, r=.41, p<.05. All three correlations showed positive relationships, therefore, for example, as knowledge scores increased, so did behavior and attitude scores. When two presenters simultaneously taught one class, they both received relatively the same ratings. In a paired sample t test to assess differences in dual presenters, we failed any differences in their ratings, t(271)=.95, p= 0.34. These results showed that there was no significant difference between likeability in dual presenters; therefore, the results contradicted our hypothesis. We first hypothesized that there would be positive correlations between facilitator ratings and how much the students changed within knowledge, attitudes, and behaviors. We found a correlation between presenter effectiveness and the knowledge difference of r= 0.17. We then found presenter effectiveness had a positive correlation with attitude differences (r= 0.12). There was also a positive correlation between presenter effectiveness and behavior differences (r= 0.15). We then hypothesized that there would be different degrees of TDV knowledge obtained based on which presenter taught a given class. An omnibus one-way ANOVA test was performed to test whether different presenters had differential effects on knowledge gained by the participants. The one-way ANOVA was conducted because there was one IV with five levels representing each of the presenters, and one continuous DV. Results of the omnibus indicated that there were differences between presenters with respect to the amount of knowledge gains made by students, F (4, 120) = 2.45, p< 0.05; therefore, we rejected the null hypothesis. In a post hoc Tukey HSD test, the results showed a significant difference in TDV knowledge gained by students taught by
PRESENTER EFFECTS 11 presenters 1 and 5 (MD= 1.48), 1 and 6 (MD= 1.07), and 2 and 5 (MD= 0.98). Presenter 1 was statistically significantly better in helping their students gain knowledge than presenters 5 and 6. Presenter 2 was statistically significantly better than presenter 5. There were no significant differences between any of the other presenters. A one-way ANOVA testing whether there were significantly different changes in TDV attitudes for the five presenters was significant, F(4, 120) = 2.45, p < 0.05. A Tukey HSD test for attitude differences among presenters indicated a significant difference for presenter 1 with presenters 2, 5, and 6. Presenter 1 s students had statistically significantly greater changes in TDV attitudes than did the students of presenter 2 (MD= 2.11), 5 (MD= 2.58), and 6 (MD= 1.81). In a one-way ANOVA test assessing whether presenters were differentially associated with levels of TDV behavior change, we found significant results F= 0.01; therefore, we rejected the null hypothesis. In a post hoc Tukey HSD test for behavior differences, we found no significant differences between any pairs of presenters. Discussion The current study explored the relationship between changes in TDV violence knowledge, attitudes, and behaviors as a result of a TDV prevention program and facilitator effectiveness. There were two hypotheses in the study: 1) there would be different degrees of TDV knowledge, behaviors, and attitudes obtained based on which presenter taught a given class, and 2) there would be positive correlations between facilitator likeability ratings and how much the students learned. We also explored whether co-presenters of a specific class would receive different ratings from students.
PRESENTER EFFECTS 12 The results supported the first hypothesis such that different presenters students demonstrated different degrees of TDV knowledge, behavior and attitudes. Specifically, we found when Facilitator 1 taught a TDV class, students gained the most in TDV knowledge, attitudes, and behaviors compared to the students in many of the other facilitators classes. Our results showed that Facilitator 1 reported the largest increase in knowledge differences followed by Facilitators 4, 2, 6, and 5 respectively. For attitude differences, Facilitator 1 reported the largest increase in the amount the students gained followed respectively by Facilitator 4, 6, 2, and 5. Lastly, for the behavior differences, Facilitator 1 had the largest amount gained, closely followed by Facilitators 2, 5, 6, with Facilitator 4 having a small loss for his/her behavior differences. These results are similar to those found by past researchers who have reported that certain facilitators were more effective in teaching students than others. Facilitators teaching ratings and styles could vary due to numerous variables including years of experience and peer review assistance. In the future, Project PAVE could include peer reviews, given that past research (e.g., Dahlgren et al., 2006) has shown that method to be effective in improving teaching outcomes. Project PAVE can use the data collected to infer which facilitators are doing the best jobs in their teaching. In order to improve everyone s overall scores, on-site training sessions could be used to have a positive impact on the presenter effectiveness (Ahsam, Shepherd, & Warren, 2006) and peer reviewers evaluations could be used to provide the facilitators with positive and valuable feedback about their teaching skills, possibly leading them to develop more effective presentation skills and increase their presenter effectiveness (Dahlgren et al., 2006). The possible benefit of identifying Facilitator 1 as
PRESENTER EFFECTS 13 the most effective presenter, in knowledge, attitude, and behavior change, is that the other presenters may learn from him/her to improve their personal teaching skills. Our results also indicated that how much students learned in each domain was correlated with presenter ratings, which is consistent with the findings of past studies. However, studies such as Patrick (2011) found that the amount of information learned by the students was not related to teacher ratings. This implies that whether the facilitators are likable, students will still be able to obtain knowledge from their presentations. Past research has separately evaluated TDV educational and prevention programs (Jaycox et al., 2006; Veldwijk, Hoving, Van Gelder, & Feenstra, 2012) and facilitator effectiveness (Praxmarer, 2011; Gilbert, Lee, & Chiddix, 1981; Schwartz, Thomas, Bohan, & Vartanian, 2007), but we were unable to find research on correlations or influences of both facilitator effectiveness and TDV prevention. The importance of the current study is that it combines the two once distinct areas of literature in order to bring forth new knowledge to this specific topic. This study was presented with few, but significant limitations. Project PAVE s staff compiled the surveys used in the current study. They included items of interest to the agency; however, the survey did not have documented validity or reliability. Future researchers can avoid this problem by using well-validated measures when examining TDV prevention programs. Another limitation resided with the sample used. According to Project PAVE, the programs were offered to underserved Denver communities. Therefore, the current results are not generalizable to all public and private schools in the United States. An additional limitation was that some facilitators had more classroom exposure than others. Specifically, Facilitator 4 only taught 30 students, Facilitators 1 and
PRESENTER EFFECTS 14 6 taught over 200 students each, and Facilitators 2 and 5 each taught over 700 students. Because each presenter taught a different amount of students, their level of experience within Project PAVE may be related to their overall effectiveness (not analyzed in this study). Future researchers could improve on the current study in several ways. First, a TDV prevention program survey with documented validity and reliability could be used in place of the survey Project PAVE currently uses. Second, the sample could be expanded to include not only the schools considered to be in underserved within various Denver communities. Finally, in a follow-up study, facilitators could be assigned to teach the same amount of students, thereby potentially increasing the internal validity of the analyses.
PRESENTER EFFECTS 15 References Ahsam, S., Shepherd, J., & Warren-Adamson, C. (2006). Working with pre-school practitioners to improve interactions. Child Language Teaching and Therapy, 22(2), 197-217. doi: 10.1191/0265659006ct303oa. Dahlgren, L. O., Eriksson, B. E., Gyllenhammar, H., Korkeila, M., Sääf-Rothoff, A., Wernerson, A., & Seeberger, A. (2006), To be and to have a critical friend in medical teaching. Medical Education, 40, 72 78. doi: 10.1111/j.1365-2929.2005.02349. Finkelhor, D., & Hashima, P. Y., (2001). The victimization of children and youth: A comprehensive overview. Handbook of Youth and Justice. doi: 10.1007/978-1- 4615-1289-9_4. Finkelhor, D., Ormrod, R., Turner, H., & Hamby, S. L., (2005). The victimization of children and youth: A comprehensive, national survey. Child Maltreatment. doi: 10.1177/1077559504271287.
PRESENTER EFFECTS 16 Gilbert, L. A., Lee, R. N., & Chiddix, S. (1981). Influence of presenter's gender on students' evaluations of presenters discussing sex fairness in counseling: An analogue study. Journal of Counseling Psychology, 28(3), 258-264. doi: 10.1037/0022-0167.28.3.258. Jaycox, L. H., McCaffrey, D., Eiseman, B., Aronoff, J., Shelley, G. A., Collins, R. L., & Marshall, G. N. (2006). Impact of a school-based dating violence prevention program among Latino teens: Randomized controlled effectiveness trial. Journal of Adolescent Health. doi: 10.1016/j.jadohealth.2006.05.002. Patrick, C. (2011). Student evaluations of teaching: Effects of the big five personality traits, grades, and the validity hypothesis. Assessment & Evaluation in Higher Education, 36, 239-249. doi: 10.1080/02602930903308358. Praxmarer, S. (2011). How a presenter's perceived attractiveness affects persuasion for attractiveness-unrelated products. International Journal of Advertising: The Quarterly Review of Marketing. Communications, 30(5), 839-865. Project PAVE. (n.d.). Project PAVE Statistics. In Project PAVE. Retrieved April 12, 2013, from http://www.projectpave.org. Schwartz, M. B., Thomas, J. J., Bohan, K. M., & Vartanian, L. R. (2007). Intended and unintended effects of an eating disorder educational program: Impact of presenter identity. International Journal of Eating Disorders, 40(2), 187-192. doi: 10.1002/eat.20301. U.S. Census Bureau. (2013). U.S. and World Population Clock. Retrieved from http://www.census.gov/popclock/
PRESENTER EFFECTS 17 Veldwijk, J., Hoving C., Van Gelder, B.M., & Feenstra, T.L. (2012). Potential reach of effective smoking prevention programmes in vocational schools: Determinants of school directors intention to adopt these programmes.public Health, 126, 338 342. doi: 10.1016/j.puhe.2012.01.002. Tables Appendix