Effect of teaching context and tutor workshop on tutorial skills

Medical Teacher, Vol. 28, No. 4, 2006, e112 e119 WEB PAPER Effect of teaching context and tutor workshop on tutorial skills ANNE BAROFFIO, MATHIEU R. NENDAZ, ARNAUD PERRIER, CARINE LAYAT, BERNARD VERMEULEN & NU V. VU University of Geneva, Switzerland ABSTRACT Effective faculty development workshops are essential to develop and sustain the quality of faculty s teaching. In an integrated problem-based curriculum, tutors expressed the needs to further develop their skills in facilitating students content learning and small-group functioning. Based on the authors prior observations that tutors performance depends on their teaching context, a workshop was designed not only tailored to the tutors needs but also organized within their respective teaching unit. The purposes of this study are (1) to evaluate whether this workshop is effective and improves tutors teaching skills, and (2) to assess whether workshop effectiveness depends on tutors performance before the workshop and on their teaching unit environment. Workshop effectiveness was assessed using (a) tutors perception of workshop usefulness and of their improvement in tutorial skills, and (b) students ratings of tutor performance before and after the workshop. In addition, an analysis of variance model was designed to analyse how tutors performance before the workshop and their teaching unit influence workshop effectiveness. Tutors judged the workshop as helpful in providing them with new teaching strategies and reported having improved their tutorial skills. Workshop attendance enhanced students ratings of tutors knowledge of problem content and ability to guide their learning. This improvement was also long-lasting. The workshop effect on tutor performance was relative: it varied across teaching units and was higher for tutors with low scores before the workshop. A workshop tailored to tutors needs and adapted to their teaching unit improves their tutorial skills. Its effectiveness is, however, influenced by tutors level of performance before the workshop and by the environment of their teaching unit. Thus, to be efficient, the design of a workshop should consider not only individual tutors needs, but also the background of their teaching units, with special attention to their internal organization and tutor group functioning. Introduction To be effective, a problem-based-learning (PBL) tutor should know well both the content of the problems and how to facilitate the small-group learning process (Dolmans et al., 2002). This has implications for recruiting tutors, in particular those who are familiar with the problem content (Wilkerson, 1994). It also highlights the need for faculty development programmes. Most programmes emphasize a general understanding of PBL and its values, exercising the Practice points. Effective faculty development workshops are essential to develop and sustain the quality of teaching.. A faculty development workshop tailored to tutors needs within their teaching unit improves their problem-content knowledge and ability to guide students learning, especially for tutors with low students ratings.. To be efficient, the design of workshops should consider specificities of teaching contexts since they affect the effectiveness of workshops. tutorial process, and developing general and content-specific tutor knowledge and skills (Holmes & Kaufman, 1994; Irby, 1996). The few studies that have documented the effectiveness of these training programmes show that they help develop new tutorial skills (Wilkerson & Irby, 1998; Skeff et al., 1998; Skeff et al., 1999; Griffith, 2000; Hewson et al., 2001). They specifically indicate that workshops, followed by tutorial practice and/or feedback from students or peers, are effective strategies in improving teachers knowledge, attitude and performance ratings. With the introduction of a new integrated six-year problem-based curriculum in 1995 (Vu et al., 1997), the University of Geneva Faculty of Medicine (UGFM) implemented a multilevel tutor-training programme. All tutors in the medical sciences years were required to attend an introductory workshop, which consisted of (1) an introduction to the curriculum, the rationale for PBL, the tutorial process and the tutor role; (2) an observation and/or participation in a tutorial session in their assigned teaching unit (to put the tutor in the teaching context); and (3) a discussion with other tutors to share observations and strategies in tutoring. Students ratings indicated that the workshop has adequately prepared tutors (Baroffio et al., 1999). However, although tutors improved their overall performance during their first three years of practice (Baroffio et al., 1999), we observed that students ratings of Correspondence: Anne Baroffio, Unit of Development and Research in Medical Education, University of Geneva, School of Medicine, 1 rue Michel Servet, CH-1211 Genève 4, Switzerland. Tel: þ41 22 379 59 39; fax: þ41 22 379 51 21; email: anne.baroffiobarbier@medecine.unige.ch e112 ISSN 0142 159X print/issn 1466 187X online/06/040112 8 ß 2006 Taylor & Francis DOI: 10.1080/01421590600726961

Fostering tutorial skills tutors abilities to give feedback and to help students synthesize the problem remained low. To assist tutors in maintaining and improving their teaching skills, we offered to all experienced tutors an advanced optional workshop, emphasizing and working on tutorial situations that they commonly identified as difficult. This second workshop improved significantly the ratings of tutors with low performance ratings, specifically their overall performance and ability to guide students process of problem synthesis (Baroffio et al., 1999). In addition, we showed that this workshop improved the tutors ratings above and beyond the self-improvement demonstrated by most tutors with increased experience in tutoring. However, over the years we have noticed that tutor ratings tended to vary and differ according to the teaching unit they taught in. Based on our observations and on the findings that faculty development programmes are most effective when they target the learners needs (Davis et al., 1995; Skeff et al., 1997) and address these needs within their teaching context (Skeff et al., 1997), we designed a third advanced workshop, organized by teaching unit and focusing on the tutors and the unit s needs. It was introduced in 1999 2000 for all practising tutors. As the tutors major needs were to become better prepared in the content in order to guide students learning and to improve their skills in facilitating small-group functioning, the workshops were organized into two parts. The first was devoted particularly to providing the tutors with both pedagogical content knowledge and an organizational approach to their functioning within the unit by stimulating them to have regular meetings to preview cases, to negotiate key concepts to be covered in the tutorials, and to construct a common understanding of problem-content and the learning process (Holmes & Kaufman, 1994; Irby, 1996). The second part addressed small-group facilitation skills, specifically how to handle students or small groups dysfunctioning (Kaufman & Holmes, 1996), by analysing and working through various difficult tutorial situations. The content of each workshop was specifically adapted to the questions and needs of the tutors within the teaching unit, thus not only addressing their individual needs but also considering the functioning of each tutor within his teaching context (Skeff et al., 1997). The purposes of this study are to evaluate whether this workshop is effective and further improves tutors teaching skills, and to assess whether workshop effectiveness may depend on the performance of tutors before the workshop and on their teaching context. Methods Context and subjects This study was conducted over three academic years (1998 99; 1999 2000; and 2000 01). The second and third years of our pre-clerkship curriculum are composed of 13 teaching units (e.g. circulation, nutrition and digestion, infectious disease, etc.). Each unit is taught once a year, lasts four weeks, and consists of 16 two-hour tutorial sessions. Each tutor teaches one unit to a group of students (mean: seven students, range: four to 15). Students rate their tutor at the end of each teaching unit. Each class counts about 150 students. During the 1999 2000 academic year, 203 tutors were invited to attend an advanced workshop and were sent one month beforehand a pre-workshop questionnaire. Of these 203 tutors, 149 (73%) completed the pre-workshop questionnaire and 145 (71%) attended the workshop. Of the 145 workshop participants, 112 (75%) completed the workshop evaluation questionnaire and 78 (54%) returned the self-assessment questionnaire, which was administered one year after the workshop. Of the 203 tutors, 126 were completely evaluated by students, namely before, one month after and one year after the workshop. Among them, 96 attended the workshop and 30 could not attend for different reasons, thus constituting a convenience control group. Needs assessment: pre-workshop questionnaire The pre-workshop questionnaire consisted of 38 true/false items probing different activities that the tutors carried out before (15 items), during (13 items) and after (10 items) their teaching in the unit. These items mainly referred to activities defined and referred to in the literature as useful in developing and guiding students acquisition of problemcontent knowledge and in facilitating the PBL tutorial process and small-group functioning (Holmes & Kaufman, 1994; Tipping et al., 1995; Kaufman & Holmes, 1996). In addition, tutors were asked with open-ended questions to identify aspects for which they needed specific training and teaching situations they found difficult to manage. Intervention workshop The design of our advanced PBL workshop for practising tutors is based on evidence-based workshop practices (Wilkerson & Irby, 1998; Griffith, 2000). Using a tailored approach it integrates in its programme learner-centred, interactive and reflective teaching strategies (Hewson, 2000). Overall, we designed each workshop to address specifically the needs of the tutors within each of the 13 teaching units and as identified in the pre-workshop survey. The needs commonly identified by tutors consisted of how better to guide students in learning the unit problem content and facilitate small-group functioning. Towards these purposes, we planned each workshop with the teaching unit director(s). It usually took place during the month preceding the teaching period, in a session where tutors met to finalize their teaching plans. It consisted of two parts and lasted three hours. Specifically, the first part of the workshop session was set up (1) to facilitate discussions among unit directors and tutors on their perceived difficulties and (2) to help tutors reflect and discuss strategies on how better to prepare for the problem content of the teaching unit, guide students learning and define the depth to which students should address the learning objectives of each problem. In a second part, a situation-based approach was used to assist tutors in developing advanced strategies to facilitate small-group functioning. For this purpose, we presented pre-selected scenarios representing common difficult tutorial situations (i.e. a disruptive or silent student, a non-responsive group,...), and asked tutors to analyse, discuss and share strategies that they would use to deal with these situations. When needed, we proposed additional alternative solutions. e113

A. Baroffio et al. Instruments and outcomes measures (a) Tutors evaluation of the workshop. At the end of the workshop we asked the participants to rate on a fivepoint Likert scale questionnaire whether the workshop (a) was useful for their role as a tutor, (b) furthered their understanding of the tutorial group functioning and of the importance of small-group facilitation in PBL, and (c) provided new strategies in preparing the problem content of the teaching unit and facilitating small-group functioning. (b) Tutors self-assessment of changes in teaching strategies. One year after the workshop, we sent a questionnaire to all participants. They indicated on nine yes/no items changes they had adopted as a tutor and changes that occurred to the tutors functioning in their teaching unit. They also commented freely on the new teaching strategies they had adopted. (c) Students ratings of tutor performance. Each group of students rated their tutor at the end of the teaching unit. The 16-item evaluation questionnaire has been adapted from the one developed and validated at the Maastricht medical school (Dolmans et al., 1994) and has been described in an earlier paper (Baroffio et al., 1999). For this study, we selected the four items of the questionnaire that were directly related to the content of the workshop. Students were asked to rate on a fivepoint Likert scale (from 5 ¼ completely agree to 1 ¼ completely disagree) the following three items: (i) My tutor has good knowledge of the problem content of the teaching unit; (ii) My tutor guides me through the PBL process; and (iii) Students participate actively in the small group. The fourth item asked the students to rate on a three-point scale (5 ¼ outstanding, 3 ¼ good or 1 ¼ to be improved) the tutor s overall performance. Students ratings were obtained before the workshop (tutor baseline score), then one month and one year after the workshop. Statistical analyses A quasi-experimental design was used to compare the 96 tutors who attended the workshop with the 30 who could not attend. The data consisted of a total of 7938 ratings performed by 450 students from the classes of 1998 to 2000. We compared students baseline ratings of the 126 tutors with the ratings they received one month and one year after the workshop. We conducted multivariate analyses of variance (ANOVAs) with the score differences after the workshop (one month and one year respectively) as dependent variables; workshop attendance and teaching unit as between-subject factors; baseline score and teaching experience (number of teaching years at the onset of the study) as covariates (SPSS version 11.0, Chicago). We considered an alpha value under 0.05 to be significant. Results Tutors baseline performance Students ratings of tutor performance obtained before the workshop revealed marked differences not only between specific tutorial skills but also across teaching units. Students rated 12% of the 126 tutors (all teaching units combined) as insufficient in their overall performance, 13% in their knowledge of the problem content of the teaching unit, 10% in their guiding of students through the PBL process, and 38% in their ability to facilitate students participation to their tutorial group. Furthermore, these insufficient ratings varied across teaching units for tutors overall performance (0% to 36%), knowledge of the problem content (0% to 43%), PBL guide (0% to 50%), and facilitation of students participation (0% to 86%). Tutors needs A total of 149 tutors completed the pre-workshop questionnaire and 116 listed their specific needs. Overall, 12% of the tutors did not feel well prepared in guiding students learning of the unit problem content and 23% in facilitating small-group functioning. However, when each individual teaching unit was taken into consideration, a different picture emerged from the tutors answers and comments. There were considerable differences in tutors perception of their level of preparedness across the teaching units. While in some teaching units none of the tutors reported any difficulties, in some others up to 27% of the tutors felt insufficiently prepared in guiding students learning of the problem content, and up to 42% had difficulties in handling student and group functioning. Most importantly, in some teaching units these difficulties were raised by more than 90% of the tutors in their written comments. Many tutors also indicated a need to further define, with the tutors in their teaching unit, the level of depth to which the problem-learning objectives should be achieved. Tutors evaluation of the workshop In all, 112 workshop participants completed the workshop evaluation questionnaire (Table 1). Overall, the tutors rated the workshop as very useful for their role as a tutor. The workshop was judged helpful in furthering tutors understanding of the tutorial group learning process and the importance of small-group facilitation skills. It also Table 1. Tutors evaluation of the workshop usefulness (n ¼ 112 tutors) on a five-point Likert scale (5 ¼ I totally agree to 1 ¼ I totally disagree). Overall, the workshop: Mean SD was very useful for my role as a tutor 4.36 0.77 furthered my understanding of the 4.19 0.82 tutorial group functioning process furthered my understanding of the 4.13 0.82 importance of small-group facilitation in PBL provided useful strategies to improve 4.28 0.95 content knowledge with the group of tutors in my teaching unit provided useful strategies to improve the functioning of small groups 4.27 0.89 e114

Fostering tutorial skills provided them with useful strategies in improving their skills to facilitate small-group functioning and in preparing the unit problem content by working with members of the unit teaching group. Tutors self-assessment of changes in their tutoring strategies The self-assessment questionnaire that was administered one year after the workshop was returned by 145 workshop participants. Regarding their own performance, 58% of the tutors self-reported as more active guides of their students learning and 53% as better small-group facilitators, in particular by stimulating students to discuss their group functioning. Regarding their teaching unit s functioning, 43% of the tutors judged that the workshop helped modify their preparation on the unit problem content. Written comments revealed more frequent tutor meetings within the unit, active attempts at improving the quality of the problems, and in seeking consensus among tutors on the depth to which the learning objectives should be addressed and studied. However, these percentages varied across teaching units, suggesting that not all tutors benefited to the same extent from the strategies proposed during the workshop. Change in tutor performance after the workshop Table 2 describes students mean ratings of tutors overall performance, problem-content knowledge, skill in guiding students through the PBL process, and ability to facilitate student participation before the workshop, one month after and one year after the workshop. Before the workshop, although tutors who attended the workshop had a slightly lower baseline score for these skills, both groups (with and without workshop attendance) did not differ significantly. The differences in scores over time between baseline and one-month and one-year scores respectively were analysed by multivariate ANOVAs. The single and interactive effects of workshop attendance, tutors baseline score, teaching unit and teaching experience on the score difference one year after the workshop are summarized in Table 3. This revealed that (a) the score difference in tutors overall performance depended on the baseline score and on the teaching unit (61% of variance explained); (b) the score difference in tutors problem-content knowledge depended on the baseline score, teaching unit, and on an interaction effect between workshop attendance by teaching unit (74% of variance explained); (c) the score difference in the tutors skill in guiding students through the PBL process depended on workshop attendance, baseline score and teaching unit (55% of variance explained); the score difference on tutors ability to facilitate student participation depended on baseline score, teaching unit and teaching experience (74% of variance explained). Compared with the results obtained one year after the workshop, we found no effect of the workshop or of the teaching experience on the change in tutors ratings one month after the workshop (results not shown). The teaching unit only affected the change in student participation. The baseline score, however, had a significant effect on all four variables. In order to show separately the different effects of workshop, teaching unit and baseline score, Figure 1 displays the effects of workshop attendance and teaching unit (1a), and of workshop attendance and baseline score (1b) on the score differences after the workshop for problem-content knowledge. It illustrates first (1a) that score differences were larger for workshop participants (0.18 0.65 vs 0.10 0.57). Furthermore, for some teaching units, score differences after the workshop were small and did not differ according to workshop attendance (e.g. units 1, 7, 13), whereas for others they were larger for tutors who attended the workshop (e.g. units 5, 9). In addition, for the tutors who attended the workshop, the score differences after the workshop were negatively correlated to their baseline scores (1b). In other words, the lower the baseline score of a tutor, the higher was his improvement in content knowledge after the workshop. This was also true for their overall performance and PBL guide (Table 4). Conversely, for the tutors who did not attend the workshop, the score differences were either weakly (for overall performance and PBL guide ) or not (for content knowledge ) correlated to their baseline scores (Table 4). In other words, tutors with a low baseline Table 2. Scores before workshop (baseline), one month and one year after the workshop of students mean (SD) ratings of 126 tutors on several tutorial skills (overall performance, problem-content knowledge, skill in guiding students through the PBL process, and ability to facilitate student participation). Overall performance a Content knowledge b Guide in PBL b Student participation b Workshop not attended (n ¼ 96) Baseline 4.3 0.7 4.7 0.4 4.6 0.3 4.1 0.6 One month 4.1 0.8 4.6 0.4 4.4 0.4 4.1 0.7 One year 4.2 0.9 4.6 0.6 4.3 0.5 4.3 0.4 Workshop attended (n ¼ 30) Baseline 4.1 0.9 4.5 0.7 4.5 0.4 4.1 0.7 One month 4.1 0.9 4.5 0.7 4.5 0.6 4.2 0.5 One year 4.3 0.7 4.6 0.4 4.6 0.4 4.4 0.4 Notes: a The overall performance is rated on a three-point scale (5 ¼ outstanding, 3 ¼ good, 1 ¼ to be improved). b The problem-content knowledge, skill in guiding students through the PBL process, and ability to facilitate student participation are rated on a five-point Likert scale (from 5 ¼ completely agree to 1 ¼ completely disagree). e115

A. Baroffio et al. Table 3. Effect of workshop attendance, tutors baseline score, teaching unit and teaching experience, and their interactions on the score differences in students ratings of several tutorial skills one year after the workshop (overall performance, problem-content knowledge, skill in guiding students through the PBL process, and ability to facilitate student participation). Mean squares a F (df) a p a Overall performance: b workshop 0.143 0.314 (1;100) 0.576 baseline score 28.065 61.977 (1;100) 0.000* teaching unit 1.375 3.036 (12;100) 0.001* workshop teaching unit 0.351 0.774 (10;100) 0.653 teaching experience 0.208 0.460 (1;100) 0.499 score before the workshop improved more after attending the workshop compared with those who did not. On the other hand, the correlations on student participation were similar for both groups of tutors, showing no specific effect of the workshop. In summary, the workshop improved tutors knowledge of the problem content of the teaching unit and their ability to guide the learning of their students. This effect worked on a long-term basis and was more effective for low-scoring tutors but it seemed to vary across teaching units. On the other hand, our workshop did not seem to improve tutors ability to facilitate students participation, a skill that appeared, however, to improve with teaching experience. Discussion Content knowledge: c workshop 0.106 0.797 (1;100) 0.374 baseline score 18.126 135.663 (1;100) 0.000* teaching unit 0.698 5.227 (12;100) 0.001* workshop teaching unit 0.357 2.672 (10;100) 0.006* teaching experience 0.011 0.083 (1;100) 0.774 PBL guide: d workshop 1.071 7.776 (1;100) 0.006* baseline score 7.619 55.308 (1;100) 0.000* teaching unit 0.307 2.231 (12;100) 0.015* workshop teaching unit 0.143 1.040 (10;100) 0.416 teaching experience 0.003 0.024 (1;100) 0.878 Student participation: e workshop 0.085 0.610 (1;100) 0.436 baseline score 29.044 208.259 (1;100) 0.000* teaching unit 0.299 2.141 (12;100) 0.020* workshop teaching unit 0.102 0.732 (1;100) 0.692 teaching experience 0.594 4.263 (1;100) 0.042* Notes: a Multivariate analyses of variance with workshop attendance and teaching unit as betweensubject factors; score before workshop and teaching experience as covariates. All factors and interactions were tested. Interactions are designated by. b R 2 ¼ 0.612; c R 2 ¼ 0.743; d R 2 ¼ 0.546; e R 2 ¼ 0.738. Our study demonstrates that our workshop, designed to address tutors specific needs and organized within their respective teaching unit, does help them develop new teaching strategies and improves their teaching skills. This confirms earlier findings on the effectiveness of specifically designed advanced training workshops (Holmes & Kaufman, 1994; Irby, 1996; Wilkerson & Irby, 1998; Skeff et al., 1998; Skeff et al., 1999; Baroffio et al., 1999; Hewson, 2000; Griffith, 2000; Hewson et al., 2001; Pandachuk et al., 2004). In addition, our study reveals that the effectiveness of our workshop is relative and depends on tutors performance before the workshop, and on the teaching unit they are teaching in. The design of our advanced PBL workshop for practising tutors is based on evidence-based workshop practices (Wilkerson & Irby, 1998). Our strategy was to tailor the workshop to the needs of the tutors in the context of their teaching unit (Davis et al., 1995; Skeff et al., 1997). This was justified by our previous observation that tutors performance varies across teaching units and by this study, showing that tutors needs are different for each teaching unit. Tutors reported that the workshop, as designed, provided them with a better understanding of the tutorial group functioning and the small-group facilitation processes. Moreover, they stated that they had acquired new strategies on how to develop advanced problem-content knowledge and small-group facilitation skills. They confirmed these perceptions one year after the workshop, by reporting being e116

Fostering tutorial skills (a) Score difference after workshop (b) Score difference after workshop 3 2 1 0 1 2 3 0 3 2 1 0 1 2 3 1.5 1 2 2.0 3 2.5 4 5 6 7 8 9 Teaching unit Workshop No workshop 3.0 Baseline score Figure 1. Effect of workshop attendance, teaching unit and baseline score on the score differences in content knowledge of 126 tutors one year after the workshop. 1a: score differences for the 30 tutors who did not attend the workshop () and for the 96 tutors who attended the workshop () for each of the 13 teaching units studied. 1b: correlation between the score difference 1 year after workshop and the baseline score obtained before the workshop for individual tutors. No workshop (, dashed line): Pearson s R ¼ 0.147: p ¼ 0.438; n ¼ 30. Workshop (, full line): Pearson s R ¼ 0.802: p ¼ 0.000; n ¼ 96. better guides of their students learning and more efficient small-group facilitators. Moreover, many tutors spontaneously recounted in free comments that the workshop modified the functioning of their teaching unit, in particular through more frequent tutor meetings. Thus, the workshop also seemed to improve the organizational vitality of the teaching units. Planning the workshop with the teaching unit director(s) and integrating it as a part of the unit regular preparation meeting for tutors presented several advantages. 3.5 4.0 10 4.5 11 12 5.0 13 14 5.5 First, as the workshop was not perceived as an additional activity to be attended, most tutors participated, thus preventing the usual self-selection of the best and most motivated tutors who usually attend such faculty development activities (Baroffio et al., 1999; Griffith, 2000). Second, it helped create or re-create the working dynamics among the unit tutors, thus favouring the necessary discussions within their own teaching unit. Finally, it facilitated the renewing of tutorial skills immediately before tutors started teaching. Analyses of the change in students ratings of tutor performance after the workshop revealed that the workshop specifically improved tutors problem-content knowledge and ability to guide the learning of their students. This effect was not detectable when tutors taught right after the workshop but one year later. One possible explanation for these findings could be the following: after the workshop, the group of tutors from the teaching unit meets more frequently, as reported by the tutors; this leads to an amelioration of the quality of the problems, a common understanding among tutors regarding problem content, and better concordance on the depth to which the learning objectives should be covered; as a result, the tutors get then better knowledge of the content of the problems and become better guides of the learning process. However, we found that, depending on the teaching unit they were teaching in, tutors problem knowledge progressed diversely after the workshop. This means that, even tailored to tutors needs within their teaching unit, our workshop did not seem to work for all of them. These differences among teaching units could be related to the quality of the problems, the structure of the unit, students prior knowledge (Schmidt, 1994), or tutorial groups productivity (Dolmans et al., 1999), which have been shown to influence tutor performance. More work is clearly needed to understand why and how the workshop did not elicit all the needed changes in the functioning of all teaching units. On the other hand, we have found that the workshop predominantly improves the tutorial skills of tutors with low baseline scores, thus reinforcing our previous results (Baroffio et al., 1999). This finding might be due to the sharing of strategies between expert and/or skilled tutors with inexperienced and/or untrained tutors during the workshop and the more frequent tutor meetings of the teaching unit. This implies that even though the low-baseline tutors do benefit more from the workshops, training should not be offered to these tutors only, since the above results suggest that they learn from better tutors. Finally, students ratings revealed that tutors improved their ability in facilitating student participation, and tutors assessed themselves as being better facilitators of small-group functioning after the workshop. This effect, which is also stronger for low-baseline tutors irrespective of workshop attendance, appears not to be due to the workshop but simply to teaching experience. This study has several limitations. First, we defined our control group as those tutors who could not attend the workshop. Whereas we have no indication of the reasons why these tutors could not attend, we think that they did not selfselect ( I do not need to go ), since their scores before the workshop were similar to those of the tutors who attended the workshop. They might nevertheless be influenced by secondary effects of the workshop, such as more frequent e117

A. Baroffio et al. Table 4. Correlations between the score difference after workshop and the baseline score obtained before the workshop for individual tutors. Workshop not attended Workshop attended Pearson s R p Pearson s R p Overall performance 0.411 0.024* 0.707 0.000* PBL guide 0.448 0.013* 0.663 0.000* Content knowledge 0.147 0.438 0.802 0.000* Student participation 0.715 0.000* 0.821 0.000* teaching unit meetings, which may limit the specific appraisal of workshop effectiveness. Second, the questionnaires that we used to assess tutors needs, to evaluate the workshop and to appraise tutors changes in teaching after the workshop have been elaborated for the purpose of this study but have not been validated at other institutions. This might represent a shortcoming of our study. In conclusion, based on prior observations that tutors performance greatly depends on their teaching context, we designed a workshop to address tutors needs within their teaching unit. We found that this workshop improved tutors understanding of the PBL principles, their repertoire of teaching strategies, their knowledge of problem content and their ability to guide students learning. This workshop was found to be more effective for tutors who had low scores before the workshop. Furthermore, its long-term effect on tutor performance may presumably result from modifications of teaching units functioning. Nevertheless, despite the fact that each workshop was tailored to each teaching unit, it was not equally effective for all teaching units. We do not know the reason for this finding, but hypothesize that for many different reasons modifications in functioning may not occur in all teaching units. Our results have implications for the design of tutor workshops, suggesting that such interventions should be tailored not only to individual tutors needs but also to the needs of specific teaching units, with special attention to their internal organization. However, more research is needed to further explore the type of internal organization that improves the functioning of teaching units and hence the performance of their tutors, and to understand how a workshop can elicit a change in this functioning. Notes on contributors ANNE BAROFFIO, PhD, is assistant professor at the Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Switzerland. MATHIEU N NENDAZ, MD MHPE, is Médecin Adjoint of the Division of General Internal Medicine, University Hospitals, Geneva, and a faculty member of the Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Switzerland. ARNAUD PERRIER, MD, is Professor of Internal Medicine of the Division of General Internal Medicine, University Hospitals, Geneva, and Faculty of Medicine, University of Geneva, Switzerland. CARINE LAYAT, MS, is research assistant at the Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Switzerland. BERNARD VERMEULEN, MD, is Médecin Adjoint and the head of the Emergency Service, University Hospitals, Geneva, Switzerland. NU VIET VU, PhD, is full professor and Director of the Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Switzerland. References BAROFFIO, A., KAYSER, B., VERMEULEN, B., JACQUET, J.& VU, N.V. (1999) Improvement of tutorial skills: an effect of workshops or experience? Academic Medicine, 74, pp. S75 S77. DAVIS, D.A., THOMSON, M.A., OXMAN, A.D. & HAYNES, R.B. (1995) Changing physician performance: a systematic review of the effect of continuing medical education strategies, Journal of the American Medical Association, 274, pp. 700 705. DOLMANS, D.H., GIJSELAERS, W.H., MOUST, J.H., DE GRAVE, W.S., WOLFHAGEN, I.H. & VAN DER VLEUTEN, C.P. (2002) Trends in research on the tutor in problem-based learning: conclusions and implications for educational practice and research, Medical Teacher, 24, pp. 173 180. DOLMANS, D.H., WOLFHAGEN, H.A.P., HOOGENBOOM, R.J.I. & VAN DES VLEUTEN, C.P.M. (1999) Is tutor performance dependent on the tutorial group productivity? Toward further resolving of inconsistencies in tutor performance, Teaching and Learning in Medicine, 11, pp. 186 191. DOLMANS, D.H., WOLFHAGEN, I.H., SCHMIDT, H.G.& VAN DER VLEUTEN, C.P. (1994) A rating scale for tutor evaluation in a problembased curriculum: validity and reliability, Medical Education, 28, pp. 550 558. GRIFFITH, C.H. (2000) Evidence-based educational practice: the case for faculty development in teaching, American Journal of Medicine, 109, pp. 749 752. HEWSON, M.G. (2000) A theory-based faculty development program for clinician-educators, Academic Medicine, 75, pp. 498 501. HEWSON, M.G., COPELAND, H.L. & FISHLEDER, A.J. (2001) What s the use of faculty development? Program evaluation using retrospective self-assessments and independent performance ratings, Teaching and Learning in Medicine, 13, pp. 153 160. HOLMES, D.B. & KAUFMAN, D.M. (1994) Tutoring in problem-based learning: a teacher development process, Medical Education, 28, pp. 275 283. IRBY, D. (1996) Models of faculty development for problem-based learning, Advances in Health Sciences Education, 1, pp. 69 81. KAUFMAN, D.M. & HOLMES, D.B. (1996) Tutoring in problem-based learning: perceptions of teachers and students, Medical Education, 30, pp. 371 377. PANDACHUCK, K., HARLEY, D. & COOK, D. (2004) Effectiveness of a brief workshop designed to improve teaching performance at the University of Alberta, Academic Medicine, 79, pp. 798 804. e118

Fostering tutorial skills SCHMIDT, H.G. (1994) Resolving inconsistencies in tutor expertise research: does lack of structure cause students to seek tutor guidance, Academic Medicine, 69, pp. 656 662. SKEFF, K.M., STRATOS, G.A., BERGEN, M.R. & REGULA, D.P. (1998) A pilot study of faculty development for basic science teachers, Academic Medicine, 73, pp. 701 704. SKEFF, K.M., STRATOS, G.A., BERGEN, M.R., SAMPSON, K.& DEUTSCH, S.L. (1999) Regional teaching improvement programs for community-based teachers, American Journal of Medicine, 106, pp. 77 80. SKEFF, K.M., STRATOS, G.A., MYGDAL, W., DEWITT, T.A., MANFRED, L., QUIRK, M., ROBERTS, K., GREENBERG, L.& BLAND, C.J. (1997) Faculty development: a resource for clinical teachers, Journal of General Internal Medicine, 12, pp. S56 S63. TIPPING, J., FREEMAN, R.F. & RACHLIS, A.R. (1995) Using faculty and student perceptions of group dynamics to develop recommendations for PBL training, Academic Medicine, 70, pp. 1050 1052. VU, N.V., BADER, C.R. & VASSALLI, J.D. (1997) The redesigned undergraduate medical curriculum at the University of Geneva, in: A.J.J.A. Scherpbier, C.P.M. van der Vleuten, J.J. Rethans & A.F.W. van der Steg (Eds) Advances in Medical Education, pp. 532 535 (Kluwer, Dordrecht). WILKERSON, L. (1994) The next best thing to an answer about tutors content expertise in PBL, Academic Medicine, 69, pp. 646 648. WILKERSON, L.A. & IRBY, D.M. (1998) Strategies for improving teaching practices: a comprehensive approach to faculty development, Academic Medicine, 73, pp. 387 396. e119