This article was downloaded by: [Consorci de Biblioteques Universitaries de Catalunya] On: 11 December 2008 Access details: Access Details: [subscription number 789296667] Publisher Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Higher Education in Europe Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t713423578 Preliminary Evaluation of the Implementation of a Mentorship Plan in the Faculty of Education at the University of Lleida (UdL), Spain Gemma Filella; Isabel Lara; Anna Soldevila; Jesús Nadal; Ramona Ribes; Maria Jesús Agulló; Fernando Carrillo Online Publication Date: 01 December 2008 To cite this Article Filella, Gemma, Lara, Isabel, Soldevila, Anna, Nadal, Jesús, Ribes, Ramona, Agulló, Maria Jesús and Carrillo, Fernando(2008)'Preliminary Evaluation of the Implementation of a Mentorship Plan in the Faculty of Education at the University of Lleida (UdL), Spain',Higher Education in Europe,33:4,447 456 To link to this Article: DOI: 10.1080/03797720802522718 URL: http://dx.doi.org/10.1080/03797720802522718 PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.
Higher Education in Europe, Vol. 33, No. 4, December 2008 TRIBUNE Preliminary Evaluation of the Implementation of a Mentorship Plan in the Faculty of Education at the University of Lleida (UdL), Spain GEMMA FILELLA, ISABEL LARA, ANNA SOLDEVILA, JESU S NADAL, RAMONA RIBES, MARIA JESU S AGULLO and FERNANDO CARRILLO The evaluation of a preliminary study on the possible implementation of a Mentorship Plan in the Faculty of Teacher Education at the University of Lleida (UdL) is described. A total of sixty-seven first-year Special- and Nursery School-education teacher students participated in this survey. Results were evaluated in both qualitative and quantitative terms. Participating students declared themselves satisfied with the contents of the Plan. However, further analysis of results indicates the convenience of adapting the Plan to the particular needs of each university degree, and of improving some organisational and methodological aspects observed during this preliminary survey. Introduction Tutorship during higher education is a possible way of addressing some current issues arising from increasing heterogeneity of university students (foreign origin, special educational needs, part-time working situations, etc.) as well as, in the case of European universities, from the adaptation of university curricula to the new European Higher Education Space (Rodrı guez Espinar, 2004). In order to deal with these needs, university instructors are expected not only to excel in research and to have theoretical knowledge, practical competence and commitment, but also to be able to guide the learning process in a personalised fashion. Tutorship is understood as a set of activities, strategies and methodologies leading to easier and more efficient training, in addition to better social and working integration of students. This cognitive and behavioural intervention comprises three aspects of student education, namely academic, personal and professional (Arbizu, 1994; Campoy and Pantoja, 1999; Pantoja et al., 2003; Rodrı guez Espinar, 2004; Rodrı guez Moreno, 2002; Sa nchez Garcı a, 2001), with the general purpose of optimising decision-making during university education and thereafter. Although such a mentorship plan has been implemented in several Spanish universities during the last few years, references on the evaluation of results in terms of efficiency and student satisfaction are, to our knowledge, still scarce. This paper therefore describes the implementation and evaluation of a mentorship plan in the Faculty of Education at the UdL. Results are examined and discussed in relation to the initial objectives. ISSN 0379-7724 print/issn 1469-8358 online/08/040447-10 # 2008 UNESCO DOI: 10.1080/03797720802522718
448 G. FILELLA ET AL. Objectives and Contents of the Mentorship Plan The general objectives of the Plan were: (a) to help first-year student integration in university life; (b) to optimise the learning process of the students; and (c) to help student development in personal and professional terms. In order to address these objectives, the contents of the Plan were split into academic, personal and professional guidance: i. Academic guidance included general information about the institution and possibilities of participation in university life, as well as on methods for optimising the learning process. This aspect was considered necessary in order to help adaptation to the new circumstances, frequently involving important lifestyle changes including residence, friends and substantial modifications in the learning environment. Thus academic guidance should provide the student with tools for: - Knowing the institution in terms of services, organisational chart, curricula offered, etc. - Becoming familiar with their own learning styles, skills and limitations. - Improving time organisation by means of adopting proper study techniques and habits. ii. Personal guidance was meant to address human and social aspects by providing strategies aimed at adaptation to new situations through development of two basic abilities: - Stress-management strategies. - Communicative skills. iii. Professional guidance was aimed at providing information of interest for first-year students regarding: - Curricula structure in the chosen university degree. - Public and private job opportunities. Methodology Participants The project was developed in the Faculty of Education at UdL during the 2004 2005 academic year, on a group of sixty-seven first-year teacher students being trained in the specialties of Special Education (twenty-seven) and Nursery School Education (forty). Age of participants ranged between 18 20 years old, 94 per cent being female. Four volunteer instructors interested in the mentorship system took part in the project after specific training. Procedures The project was developed in several consecutive steps as follows: (a) Analysis of needs: the students were asked to suggest contents of interest to be included in the mentorship plan;
IMPLEMENTATION OF A MENTORSHIP PLAN 449 (b) (c) (d) (e) Definition of the Plan contents according to suggestions made by both the participant students and the trained instructors; Objective pre-test: a qualitative test designed according to the objectives of the Plan, aiming at ascertaining previous student acquaintance with its contents; Implementation of the Mentorship Plan; Objective post-test: the purpose was to determine the degree of familiarity with the contents of the Plan acquired by the students after its implementation. The contents of the Plan were developed throughout the academic year in a total of six tutorship sessions (two hours per session). The first two sessions took place in early- October and mid-november, respectively, and were devoted to academic guidance. Contents related to personal guidance were considered during the third and forth sessions (late-december and late-january, in that order), while the last two (mid- March and late-april) dealt with professional aspects. Teaching methodology included traditional magisterial lessons, group dynamic techniques including face-to-face tutorial meetings and dramatisations, as well as individual computer-assisted learning activities (<www.ice.udl.es/uou>). Evaluation of the Plan The evaluation design focused on three different moments along the development of the Plan; namely before (step c), during (step d) and after (step e) of the implementation process. Evaluation techniques were both quantitative and qualitative. Process Evaluation At the end of the third tutorship session, both participant students and instructors were asked to complete a questionnaire meant to assess their opinion regarding the suitability of methodology and resources (technical and other) used, as well as on the appropriateness of the time schedule and the activities developed. The questionnaire also included organisational and general coordination aspects, in addition to estimations (addressed to the instructors) on the degree of participation and apparent interest by the students. The student questionnaire was anonymous, and consisted of seven questions aimed at evaluating the tutor s attitude and connection with the students both individually and as a group, contents, methodology and resources. To each question the individual could reply as very satisfactory, satisfactory, unsatisfactory or very unsatisfactory. The instructor questionnaire also consisted of seven questions intended to estimate students attitudes towards the tutorship sessions, attainment of initial objectives, resources and methodology available, and working environment. To each question the individual could likewise reply as very satisfactory, satisfactory, unsatisfactory or very unsatisfactory. Final Evaluation Three different instruments were used for the final evaluation of the Plan: - Objective pre- and post-tests, aiming at assessing previous and subsequent student acquaintance with the contents of the Plan. These tests consisted of twenty questions
450 G. FILELLA ET AL. regarding the three aspects (academic, personal and professional) considered within five, seven and eight questions, respectively. Each question included a choice of five possible answers, to which the student could reply as true, near, false or unknown. True meant the right answer, while near denoted not false and close to true, false stood for the wrong option, and unknown indicated the student did not know the answer. - An anonymous questionnaire fulfilled by the participant students on the degree of satisfaction with the contents and the implementation process of the Plan. This questionnaire consisted of five questions. The first one asked about the most interesting contents in each of the three aspects considered in the Plan, the other four being freeanswer questions in which the students were welcome to suggest modifications in the methodology, organisation and/or contents of the Plan. Data obtained were analysed separately for each of both specialties considered in the survey. Answers to the first question were considered objective (interesting/uninteresting contents), while the rest of data were considered subjective (suggestions and proposals). - A questionnaire completed by the participant instructors in order to evaluate the tutorship sessions, and consisting of eight free-answer questions on students degree of participation in the sessions, overall attainment of initial objectives, and appropriateness of contents and of resources available. Results Process Evaluation: Students The anonymous questionnaire at the end of the third tutorship session provided some relevant information, 89 per cent of students rating as very satisfactory the tutors attitudes to the group during the meetings. However, only 53 per cent of participants declared themselves entirely satisfied with the contents developed during the sessions. 90 per cent were pleased with materials and resources used, but only 49 per cent were satisfied with the methodology utilised. Overall evaluation of the tutorial meetings ranged from satisfactory (11 per cent) to very satisfactory (89 per cent). Process Evaluation: Tutors Students interests and attitudes were positively evaluated by all four participant instructors, in spite of complaints about students passivity and reluctance to individual interviews with the tutor, as only two out of sixty-seven participant students had hitherto used that possibility. Observations on students passivity led to the introduction of group discussions and other active teaching methodologies in order to encourage participation during the second half of the implementation period. Working environment was considered as satisfactory, while contents related to learning techniques were considered unsuitable. Final Evaluation: Students Objective Pre- and Post-tests Because data obtained represent frequencies for each of four possible answers (true, near, false or unknown) to every statement in the test, they are given as absolute, nonparametric scores, requiring a log-linear frequency analysis (Zar, 1984). Analyses were
IMPLEMENTATION OF A MENTORSHIP PLAN 451 structured from the general model (Special Education and Nursery School Education students together) to the specific features of each specialty in search of significant changes, if any, in test results before and after implementation of the mentorship plan. Taken as a whole, the tests failed to detect significant changes in students acquaintance with the contents of the Plan after its implementation. Answers were not chosen randomly, as they were not distributed equally among all possible options. However, when analysing each item separately, three different trends were detected in test results, namely (a) significant differences in favour of the true answer; (b) small, non significant changes favouring the near option and leading to a decrease in the frequency of the unknown option; and (c) no differences in answers to the test prior to and after implementation of the Plan. Some examples are given below. As regards academic guidance, there was an increase in the number of students choosing correctly active techniques of study after the tutorship sessions (df53, x 2 58.88, P(0.03). In contrast, no significant differences were found in relation to the way of taking class notes, to study efficiently, or advantages of programming a learning time schedule, probably because students already had correct habits concerning these aspects prior to participation in the Plan. Contrarily, a large increase in correct answers to the question UdL grants and scholarships was observed after implementation of the Plan (df53, x 2 553.15, P(0.00). Concerning personal guidance, the concept of assertiveness shifted from totally unknown to correctly identified after the sessions (Table 1). Students in the specialty of Nursery School Education changed significantly their answers to the item stress can be defined as after the tutorial sessions (df53, x 2 58.78, P(0.03). Nonetheless, this change favoured the true answer in only 15 per cent of cases, while another 15 per cent shifted to the false option (Table 2). In a third example, the answers to the items strategies to decrease anxiety before exams or methods to relax before an exam were not modified, possibly owing to previous knowledge of these aspects. As to professional guidance, significant differences were observed for library services (df53, x 2 510.59, P(0.00), required subjects (df53, x 2 550.44, P(0.00), previous requirements for the Practicum (df53, x 2 536.45, P(0.00) and certificate in Catalan proficiency (df53, x 2 526.14, P(0.00), modifications leading to increased choice of the true and the near options, and to strongly decreased choice of the unknown answer (Table 3). Other items similarly showing significantly higher choice of the correct answer after implementation of the mentorship plan were required subjects (df53, x 2 553.02, P(0.00) and temporary academic jobs (df53, x 2 539.39, P(0.00). TABLE 1. ANSWER FREQUENCIES TO THE CONCEPT OF ASSERTIVITY BEFORE AND AFTER IMPLEMENTATION OF THE TUTORSHIP PLAN Specialty Nursery School Special Total Answer Pre-test Post-test Pre-test Post-test Pre-test Post-test True 8 26 4 9 12 35 False 0 0 1 0 1 0 Near 9 14 2 12 11 26 Unknown 23 0 20 0 43 0 Source: Objective pre and post-test related before and after tutorial sessions according with their training speciality; Zar, 1984.
452 G. FILELLA ET AL. TABLE 2. ANSWER FREQUENCIES TO THE QUESTION STRESS CAN BE DEFINED AS IN THE PRE- SCHOOL EDUCATION GROUP OF STUDENTS BEFORE AND AFTER IMPLEMENTATION OF THE TUTORSHIP PLAN Answer Pre-test Post-test True 2 8 False 4 11 Near 33 21 Unknown 1 0 Source: Objective pre- and post-test related before and after tutorial sessions; Zar, 1984. TABLE 3. ANSWER FREQUENCIES TO THE ITEMS REQUIRED SUBJECTS, PREVIOUS REQUIREMENTS FOR THE PRACTICUM AND CERTIFICATE IN CATALAN PROFICIENCY BEFORE AND AFTER IMPLEMENTATION OF THE TUTORSHIP PLAN Required subjects Practicum Catalan proficiency Answer Pre-test Post-test Pre-test Post-test Pre-test Post-test True 5 30 0 0 8 9 False 5 6 5 0 5 8 Near 12 17 37 61 16 35 Unknown 45 7 25 0 38 9 Source: Objective pre and post-test related before and after tutorial sessions; Zar, 1984. FIGURE 1. EVALUATION OF THE CONTENTS DEVELOPED DURING THE TUTORIAL SESSIONS BY THE STUDENTS PARTICIPATING IN THE SURVEY. PERCENTAGES REPRESENT THE FREQUENCY OF INDIVIDUALS FINDING INTERESTING A GIVEN ISSUE Source: Student questionnaires in the final process evaluation of tutorial plan; Zar, 1984.
IMPLEMENTATION OF A MENTORSHIP PLAN 453 Anonymous Questionnaire In relation to contents considered interesting (Question 1), differences were found between both specialties participating in the survey (Figure 1) as summarised below. As to academic guidance, noticeable differences were observed between both specialties participating in the Plan. All three aspects considered within (UdL services, learning styles and learning techniques) were found interesting by more than 50 per cent of Nursery School Education students. However, only those contents related to general services offered by UdL generated significant interest among Special Education students (65 per cent), information on learning styles and techniques having low (35 per cent) or very low (20 per cent) acceptance, respectively. At any rate, these results show clearly that first-year students frequently lack information on general university services and structure, which leads to great eagerness for any initiative favouring familiarity with these aspects. Lower interest for contents relating to learning styles and techniques might have arisen from previous learning habits, perceived as adequate for their needs by the participant students, which is in accordance with results of the objective post-test which failed to find significant differences in acquaintance with these aspects after implementation of the Plan. Remarkable differences were also found in the appreciation of those sessions dedicated to personal guidance. Fifty-five per cent of Special Education students appreciated positively contents dealing with communicative skills and stress control during examination periods. These results differed considerably from those obtained among Nursery School Education students, whose appreciation of sessions dealing with assertiveness was low (33 per cent), maybe due to the perception that they will not need such skills during their interactions with little children. In contrast, the session devoted to exam-induced stress was widely approved (75 per cent) among these same students. Sessions devoted to professional guidance were, as expected, positively evaluated by the participant students regardless of specialty, with similar results for them both, although scores for the contents related to curricula offered were somewhat higher among Nursery School Education students. The rest of the questions in the questionnaire invited students to suggest additional contents to be included, as well as modifications to be taken in mind in order to help optimise efficiency of the Plan in the future. Thus, Question 2 asked about suitable modifications in the informative contents of the sessions. Blank answers have been taken to mean that no changes were considered necessary. Hence, 40 per cent (Nursery School Education) and 35 per cent (Special Education) of students participating in the survey were totally satisfied with the information obtained during the tutorial sessions, and considered it unnecessary to make any change in the contents of the Plan. Another fraction of the participating students were satisfied with the contents overall, but suggested a deeper approach to some particular issues. If these students are added to those completely satisfied, it can be concluded that 47.5 per cent (Nursery School Education) and 55 per cent (Special Education) of students found the contents of the Plan entirely adequate. The rest of the participant students considered that some interesting or important contents had not been sufficiently addressed, or addressed at all, during the tutorial sessions, and consequently made suggestions for future improvement of the Plan (data not shown). Suggestions ranged from deeper treatment of contents related to academic guidance (35 per cent and 40 per cent for Nursery School- and Special Education
454 G. FILELLA ET AL. students, respectively), to strengthening treatment of professional issues (22 per cent and 20 per cent, in that same order). Curiously, only Nursery School Education students demanded more information about personal issues, 15 per cent of them requesting the possibility of discussing personal problems during the sessions, possibly in agreement with higher appreciation scores for the session related to stress control strategies (Figure 1). Question 3 dealt with methodology changes that, in the opinion of the participant students, could improve the mentorship plan in the future. As in the previous case, blank answers were deemed to mean that no modification was considered necessary. 45 per cent (Nursery School Education) and 40 per cent (Special Education) of participant students stated their complete approval of the methodology and resources used during the development of the project. The most conspicuous of recorded suggestions were related to the time schedule of tutorial sessions, with which 25 per cent (Nursery School Education) and 35 per cent (Special Education) of individuals participating in the survey disagreed. However, most students admitted explicitly the intrinsic difficulties of this organisational aspect. In spite of disagreement with the time schedule of the sessions, average attendance at the sessions was very high (95 per cent). It was also suggested (27.5 per cent and 15 per cent for Nursery School and Special Education students, respectively) that participative rather than theoretical sessions should be encouraged in the future. In accordance, it was also suggested it might be better to work with smaller groups or even individually. Suggestions on individual mentorship sessions are in clear contradiction with the reluctance for individual interviews with the tutor, repeatedly pointed out by the participating instructors. Nevertheless, differences were again found between both specialties considered, as the suitability of working in little groups was pointed out by 40 per cent of Nursery Schoolbut only by 20 per cent of Special Education students, illustrating again different needs and expectations for students of different specialties and showing the convenience of adapting future mentorship plans to the specific requirements of each particular university degree. In spite of suggestions on modifications and proposals for improvement of the Plan, students taking part in the survey indicated that they were satisfied with this preliminary experience, as 100 per cent (Nursery School Education) and 95 per cent (Special Education) of individuals answered affirmatively to Question 4, asking whether they would recommend the Plan to other students. Among various remarks and comments collected (Question 5), students participating in the Plan emphasised that it would be particularly useful for first-year students. Contrarily, no clear opinion was gathered as whether such a mentorship plan should be implemented as one of the services offered by UdL or, alternatively, acknowledged academically as a free-choice subject. Final Evaluation: Tutors Participation in the tutorial sessions was considered satisfactory, although instructors complained that students did not make use of the possibility of individual interviews with the tutor (only eight out of sixty-seven participants). Concerning the contents, those related to professional and personal guidance were perceived as most helpful. Overall, tutors considered the general objectives of the Plan had been achieved. Methodology and resources available were also regarded as suitable.
IMPLEMENTATION OF A MENTORSHIP PLAN 455 Conclusions Taken as a whole, results of the evaluation process indicate students were widely satisfied with this preliminary experience. Nevertheless, the need to improve some organisational and methodological aspects of the Plan was also suggested. Process evaluation by the participant instructors plainly showed that during the first half of the experimental period, participation of students in the tutorial sessions was not entirely satisfactory. When accordingly more active teaching methodologies (dramatisations, group discussions) were introduced, student participation was clearly higher. It is considered, however, that such methodologies were not suitable for learning of theoretical concepts, as measured by the objective pre- and post-tests. Contents dealing with learning techniques and habits could be suppressed, as students participating in the Plan already had their own, and perceived them as satisfactory. Contrarily, information on learning styles was found more interesting, although not to the same extent by students of both specialties participating in the Plan. From an organisational point of view, an exclusive and specific time schedule should be arranged for the tutorial sessions in the future. General lack of variation in choice of true answers in the objective test after implementation of the Plan suggests at least two non-excluding hypotheses: (a) teaching methodology was not suitable for learning of theoretical contents of the Plan; and/or (b) there was no correspondence between contents developed by the instructors and knowledge evaluated in the objective post-test. In the first case, a feasible approach to this problem might be working with smaller groups, consisting of 10 12 students, in which more active participation would be achievable, as often requested by the students themselves in the anonymous questionnaire. In the second instance, better coordination between developed contents and items in the test would be advisable. Results of the evaluation process illustrate that the appropriateness of choosing more active teaching methodologies is dependent on the contents to be taught. Data also suggest the suitability of adapting the Plan and its contents to the particular needs of each university degree, which ideally implies the carrying out of a preliminary project as the one described herein prior to the ultimate implementation of a mentorship plan. References ARBIZU, F. La labor orientadora del profesor universitario desde la perspectiva del alumnado y del profesorado [The Guiding Task of University Teachers from a Student and a Lecturer Points of View], Revista de Investigacio n Educativa [Journal of Educative Research] 23 2 (1994): 614 622. CAMPOY, T. and PANTOJA, A. Necesidades de orientacio n para la mejora de la calidad de ensen anza. Un estudio descriptivo en la Universidad de Jae n [Training Needs for the Improvement of Education Quality. A Descriptive Study at the University of Jae n], in, Asociacio n Interuniversitaria de Investigacio n Pedago gica (AIDIPE), ed. Nuevas realidades educativas, nuevas necesidades metodolo gicas [New Educational Realities, New Methodological Needs]. Ma laga: CEDMA, 1999, pp. 301 317. PANTOJA, A., CAMPOY, T. and CAN AS, A. Un estudio multidimensional sobre la orientacio n y la accio n tutorial en las diferentes etapas del sistema educativo [A Multidimensional Study
456 G. FILELLA ET AL. on Guiding and Mentorship at Different Stages of the Education System], Revista de Investigacio n Educativa [Journal of Educative Research] 21 1 (2003): 67 91. RODRI GUEZ ESPINAR, S., ed. Manual de tutorıá universitaria [Handbook of University Mentorship]. Barcelona: Octaedro/ICE, 2004. RODRI GUEZ MORENO, M.L. Hacia una nueva orientacio n universitaria modelos integrados de accio n tutorial, orientacio n curricular y construccio n del proyecto profesional [Towards A New System for University Mentorship: Integrated Models, Curricular Guide and Construction of the Professional Project]. Barcelona: Edicions de la Universitat de Barcelona, 2002. SA NCHEZ GARCI A, M.F. La orientacio n universitaria y las circunstancias de eleccio n de los estudios [University Mentorship and Circumstances Underlying the Choice of Higher Studies], Revista de Investigacio n Educativa [Journal of Educative Research] 19 1 (2001): 39 61. ZAR, J.H. Biostatistical Analysis. New Jersey: Prentice-Hall, 1984.