UWIC Teaching Fellows Call 9 2005-2006 Dr. Louise Fielding Cardiff School of Health Sciences The role of postgraduate research students in undergraduate dissertation management Summary The aim of this teaching Fellowship project was to enhance the process of research informing the curriculum, through improved dissertation choice and outcomes for undergraduate students. Harmonisation of practices throughout the School would be the ultimate goal, through a pilot scheme introduced in the undergraduate programmes in the food science area. The research methods module at level 2 of the programme was modified to allow a more overt progression to the dissertation module at level 3. This was achieved through introduction of a literature review assignment and modification of the statistics element of the programme to provide examples of current research, providing a more solid context for the students. The dissertation was modified in terms of the final submission only current practices regarding topic selection and assigning supervisors remained unchanged so that the effects of changing the assessment profile could be analysed. The traditional thesis was replaced with four components of assessment a literature review, an academic paper, an academic poster and a supervisor s mark. The implementation of these changes did not lead to any significant changes in the marks obtained for either module compared with the marks obtained in the previous academic year. The poster component of the dissertation scored significantly better than any of the other components of the dissertation, although the overall marks were not significantly different. Feedback from students, through module evaluation forms, and staff, through informal feedback, were positive for both modules,
but particularly so for the modified dissertation submission. Further dissemination of this project is underway. Introduction Developing the link between teaching and research is crucial to maintaining effective teaching strategies and to the further development of research at UWIC. The Cardiff School of Health Sciences has a high research profile within UWIC, with a strong history of research informing the curriculum. The School s teaching and learning strategies are also well developed, with many staff being members of the Higher Education Academy. While all undergraduates undertaking honours degrees in the School are required to undertake a dissertation, some areas of the School lacked a clearly defined system for development of dissertation topics or dissemination of dissertation project results. There was an ad hoc approach to the involvement of research students and assistants with dissertations in the previous School of Applied Sciences. Collaboration has been successful, with the results of one undergraduate dissertation being disseminated at an international conference. In 2005-6, there were 3 dissertations in the area of food science and technology that came about by the involvement of KTP associates in the dissertation process and this collaboration brought all three of UWIC s missions together teaching, research and enterprise in the local community. It is recognised that the consistency of assessment of a module that has a significant impact on the final degree classification is important (Pathirage et al. 2004). If a large number of students are being assessed by large numbers of lecturers, this can lead to inconsistency (Todd et al. 2004). While the groups undertaking dissertations in this project were not large, the whole teaching team are responsible for marking the components, potentially leading to inconsistency. Double marking is, therefore, employed in all cases (with the exceptions of the poster where all academic staff marks are collated and the supervisor s mark, which is individual). If we add the stress that naturally accompanies the dissertation process (Webster et al. 2000), it is vital that the process of dissertation management and, of particular importance from the students point of view, how we assess their performance, must be transparent and open to academic debate. The original project aim was to develop a programme to allow postgraduate research students to engage with dissertation management and dissemination of results and enhance research informing the curriculum, through improved dissertation choice and outcomes for undergraduate students. A meeting with interested stakeholders from the School, however, exposed some concerns among the academic staff and technicians in particular about the role of research assistants in the dissertation module and also highlighted some more pressing issues with the dissertation. It was decided, therefore, to alter the focus of this project to address some of these issues. A key emerging issue was the relevance of the traditional dissertation report for graduates entering vocational careers as opposed to pursuing an academic career. The final project aim
was to develop the dissertation module to reflect the demands on graduates in their relevant careers. The modified programme would still allow postgraduate research students to engage with dissertation management where relevant. This was to be achieved by: 1. Development of the research methods module at level 2 to more fully support the dissertation module at level 3. 2. Piloting of a modified dissertation submission format to enhance student s engagement with and understanding of the research process. Methods It was decided that the project would concentrate on the programmes (BSc and HND where relevant) in Food Science & Technology, Food & Consumer Science and Nutrition & Food Science. These programmes have a common programme director who is not research active but values its importance in ensuring the currency of teaching. Research Methods Module This module runs as a 20 credit module for the students on the BSc and HND programmes identified above. The teaching team is closely involved in both research and teaching activities. The aims and learning outcomes of this module were examined and it was felt that the main aims and learning outcomes remained valid. The indicative content was also, broadly still valid although it was felt that the emphasis on certain aspects had been overstated the past and that more emphasis needed to be placed on laboratory based, quantitative techniques. It was also apparent from module evaluation that the statistics content of the module required review. It was felt by the students that this aspect came too late in the module timetable so they were unprepared for the coursework based upon statistics and, consequently, the planning of the statistics required for their dissertation in year three. The issue of the dissertation was also addressed. Following the meeting interested stakeholders from the School of Health Sciences, it was decided that the concept and requirements for the dissertation module should be introduced at the very start of the second year. The importance of the module in the students programme of study was highlighted and the requirements in terms of commitment and output fully explained. The issue of ethics was also addressed in term 2 of this module in the form of a lecture. As students are now expected to consider their chosen area of research in term 1, ethics guidance is also given in term 1. The students were introduced to the School and University ethics procedures and were expected to complete the relevant ethical approval forms before the end of term 3 in their second year. It was also decided that the assessment for the research methods module would be linked with the planning for the dissertation module. This enabled the students to focus more clearly on the direction they wished to take for the dissertation and allowed them to begin the research process
as soon as they returned in the year of their dissertation. The coursework would comprise two separate reports. The first was a concise literature review on the broad subject area of choice for a dissertation. If the student already had a clear idea at this point, their literature review was much more focused than those of their peers and this advantaged them when writing the literature review for their dissertation. The second item of coursework was based around statistical analysis. The students were provided with a real data set and they were given the task of deciding (with a clearly stated rationale) exactly which statistical tests to perform, to analyse the data and to comment on the results in terms of statistical significance. They were also expected to comment upon the strengths and weaknesses of the data as well as the statistical tests they used. The planned outcome of these modifications to the level 2 Research Methods module was that the students would enter level 3 and their dissertation module running. There should be very little time delay between the student returning for their final year (or year in which they are taking this module) and commencement of the data collection phase of the research. This entailed having an approved dissertation topic and ethical approval letter. Dissertation Module This module is worth 40 credit points (one third of the final year marks) so has a significant bearing on the degree classification and, as a result, students tend to become stressed during this module. There were some issues with the way the dissertation module had been run for students of the programmes involved in this project, particularly the way in which the traditional assessment. From the stakeholder meeting, as well as personal involvement in a number of validation events, I gained an insight into other strategies for the dissertation. Discussions were held with all members of the course team, the School s Directors of Learning and Teaching and Research. The Director of the Food Industry Centre was also consulted as he has considerable experience in the requirements of industry (which is where the majority of our graduates find employment). After some discussion the following changes were decided upon for the dissertation module. 1. The aims, learning outcomes and indicative content remain valid and were not, therefore, changed. The ethos of the dissertation was not in question, merely the way in which the research process is managed and reported by the student. 2. It is the responsibility of the supervisor to ensure that the student completes all relevant ethical approval documentation and risk assessments. 3. The outcome of the dissertation will be a focused literature review. The review will be 3000 words and will include references that conform strictly to the Harvard system. This review will form 15% of the final mark.
4. The student will identify an academic journal into which their chosen area of work could be published, which must be agreed with the supervisor. The student will write up their research project according to the Instructions for Authors for their chosen journal. The resulting paper will be submitted, along with the relevant instructions for authors. This will comprise 50% of the total mark. 5. The student will design a poster presentation of their work. Posters have been identified as a valuable means of disseminating research results into the wider scientific community and benefit both students (in how to present their work in a meaningful and interactive way) and staff (assessment of different skills and the opportunity to directly question the student regarding understanding of their project) (Kuri et al. 2007). The poster will be presented in an open session, with visitors from industry and the relevant professional bodies. The student will be present by their poster and will answer any questions posed by the visitors and examiners. This will comprise 15% of the final mark. 6. The supervisors will also be required to submit an assessment of each of their students in terms of the way in which that student performed throughout the whole process. This will comprise 20% of the final mark. The assessment method used for marking is broadly analytical rather than holistic, although the supervisor s mark does allow a certain degree of flexibility in assessment (Pathirage et al. 2004). The success of the new format of the research methods module was assessed by the normal end of module evaluation form as well as by formative assessment by discussions with the students. The evaluation of the dissertation module is much more complex and must be designed in such a way that the results for successive academic years can be directly compared to assess the impact of the research methods module modifications on the outcome of the dissertation module. An evaluation tool was constructed to allow accurate recording of the students experience in the dissertation module. Results and Discussion The restructured research methods module was delivered to level 2 students in the 2006-7 academic year. Table 1 below shows the mean marks obtained, with the standard deviation and range for the whole module and for the literature review and statistics coursework elements. Data were analysed using a one way ANOVA. It can be seen that the mean mark for the literature review (2006-7) was lower than that for the previous assessment (2005-6) (ns, P 0.05) but with a greater standard deviation. The difference between the statistics coursework for each group and the overall marks at the end of the module were not significantly different. It can be concluded, therefore, that the two sets of students did not perform significantly differently to each other (ns, P 0.05).
COMPONENT METRIC YEAR 2005-6 2006-7 Literature review Mean 63.4 Range 30-80 SD 13.13 Epidemiology Mean 65.5 Range 52-76 SD 7.14 Statistics Mean 61.2 56.1 Range 40-72 5-87 SD 9.11 23.23 Overall Mean 63.6 60.5 Range 53-73 38-83 SD 6.57 14.82 Table 1. The marks obtained in the Research Methods module at Level 2 before and after the changes were implemented One cohort of students has completed module feedback since the changes were implemented in the content and assessment of this module. The literature review was seen as a positive strategy as the students had a degree of control over the choice of topic and could easily see the relevance of undertaking such a review. For some of students, this review formed the basis of their final year dissertation and these students had an advantage over those who chose to change their topic between the second and third year of the programme. This is discussed in more detail below. While changes were made with regard to the delivery of the statistics component of the module, this was the area that received the least positive feedback and accounted for the majority of failures in the module (see table 1). There were 10 dissertations undertaken in the academic year 2006-7 where the students followed the modified submission protocol. These have been compared with the 19 students undertaking the traditional dissertation submission the year before (2005-6). Table 2 below shows the mean marks obtained, with the standard deviation and range. Comparing the overall marks obtained, the one way ANOVA shows that there was no significant difference between the two groups of students. It is apparent, however, that the marks obtained for the poster component of the assessment were significantly higher than all other components (P 0.05, one-way ANOVA). This means that the poster marking was too generous or that the students found this component of the dissertation particularly rewarding (there is some evidence for the latter explanation from the module feedback forms).
COMPONENT METRIC YEAR 2005-6 2006-7 Literature review Mean 65.91 Range 58-78 SD 7.20 Academic Paper Mean 62.27 Range 50-70 SD 6.28 Poster Mean 78.36 Range 62-87 SD 7.09 Supervisor mark Mean 63.09 Range 50-75 SD 7.65 Overall Mean 58.47 65.45 Range 20-73 54-73 SD 12.18 5.97 Table 2. The marks obtained in the Dissertation module at Level 3 before and after the changes were implemented As with research methods, one year of module feedback has been completed since the modifications were implemented. Generally, the students preferred the new format although many found writing the academic paper challenging as they had to follow very detailed instructions, which is a departure from the majority of their other coursework assessments. One student commented that approaching the dissertation in three distinct phases has enhanced their understanding of the research. The changes implemented in the research methods module and the dissertation module led to no significant difference in the marks obtained between 2005-6 and 2006-7 for either module. This can be seen as an indication that the new model for either assessment did not significantly advantage or disadvantage the students in terms of overall marks or final degree classification in the first year of operation. The marks for 2007-8 will, when available, be similarly compared. Feedback was generally positive and the value of the changes, particularly with regard to the dissertation assessment, has been recognised in the form of module evaluation. The final evaluation of the impact of modifying the research methods module on subsequent dissertation performance and module evaluation will not be known until the end of the 2007-8 academic year.
Conclusion & Recommendations Further amendments to the process are planned. To refocus the project on its original aim of enhancing the dissertation process by using research assistants and students as supervisors, the dissertation topics for the academic year 2008-9 will be targeted at current research and will be proposed by different staff to widen participation from all the staff in the food science area As stated above the delivery of the statistics element of the research methods module is still in need of some attention. This is in place for the current academic year. While emphasis has been placed on questionnaire generated data in the past (analysed using SPSS), this year we will also deliver more laboratory generated data that will be analysed in Minitab. It is hoped that this will engage all students and lead to a reduced failure rate for this piece of coursework. Although the initial aim and objectives changed early on in the process, the initial plan of work will be implemented in the academic year 2008-9 when the current system is embedded (there will have been two cohorts of students going through each of the revised modules). References Kuri V., Clowes G., Folland E. and Brennan C. (2007) The use of poster presentations as a teaching tool providing opportunities for student centred learning and developing science communication skills. http://www.bioscience.heacademy.ac.uk/ftp/projects/ltr/postervk.doc Pathirage C., Haigh R., Amaratunga D., Baldry D. and Green C. (2004). Improving Dissertation Assessment. Education in a Changing Environment, 13-14 September 2004, Salford University Todd M., Bannister P. and Clegg S. (2004) Independent inquiry and the undergraduate dissertation: perceptions and experiences of final-year social science students. Assessment and Evaluation in Higher Education 29 (3) 335-355 Webster F., Pepper D. and Jenkins A. (2000) Assessing the undergraduate dissertation. Assessment and Evaluation in Higher Education 25 (1) 71-80