Title:Survey of e-learning implementation and faculty support strategies in a cluster of mid-european medical schools

Transcription

1 Author's response to reviews Title:Survey of e-learning implementation and faculty support strategies in a cluster of mid-european medical schools Authors: David A Back (david.back@charite.de) Tina Harms (tina.harms@charite.de) Joachim Plener (joachim.plener@charite.de) Kai Sostmann (kai.sostmann@charite.de) Harm Peters (harm.peters@charite.de) Version:2Date:14 June 2015 Author's response to reviews: see over

2 Charité Campus Mitte Berlin BMC Medical Education BioMed Central 236 Gray's Inn Road London WC1X 8HB United Kingdom Deans Office for Students Affairs Dieter Scheffner Centre for Medical Education and Educational Research Phone Fax Director: Professor Harm Peters, MD June 2015 Dear Professor Sigrid Harendza, Dear Mr Aldrin Ulep, We would like to thank you for the opportunity to submit a revised version of the manuscript entitled Survey of e-learning implementation and faculty support strategies in a cluster of mid-european medical schools (No ). We consider the comments of the reviewers Saran Shantikumar and Marc Triola to be very useful, as they indeed facilitated a marked improvement of the manuscript. Additionally, we corrected the citation style in the Reference section as advised. It is our impression that all issues raised by the reviewers have now been addressed appropriately. Enclosed please find the revised version of our manuscript for a new consideration to be published in the BMC Medical Education. Yours Sincerely, David Back CHARITÉ - UNIVERSITÄTSMEDIZIN BERLIN Gliedkörperschaft der Freien Universität Berlin und der Humboldt-Universität zu Berlin Charitéplatz Berlin Telefon

3 - 2 - General revision information: Mr. Florian Behringer contributed markedly to the revision of this manuscript. He has now been added to the author list (please see title page). Subsequently, the authors contributions were changed into: DAB and KS designed the study. DB, TH, JP and KS collected the data. DAB, FB, TH, KS and HP analyzed and interpreted the results. DAB drafted the article with substantial support by KS and HP. FB, TH, JP, HP and KS revised the paper critically. All authors approved the submitted version to be published. (Authors contributions, page 16, lines 4-7) First reviewer: Saran Shantikumar Major revisions medical schools were chosen. How were they chosen? (State inclusion/exclusion criteria, if applicable.) Right, this aspect needed clarification in the manuscript. The 48 medical schools (MS) addressed were the sum of all current MS in the area covered by the Society for Medical Education (there are 36 MS in Germany, 4 in Austria and 8 in Switzerland). None of them was excluded. We changed the manuscript text accordingly into: The target group of the survey consisted of all existing 48 medical schools in Austria (n = 4), Germany (n = 36) and Switzerland (n = 8). (Methods, page 6, line 2) 2. The questionnaire link was ed to the medical schools. We know who ended up responding to the questionnaire, but who was actually ed initially, and how were these people chosen? We agree. This relevant information was missing indeed. To clarify this, we added the following sentence to the text of the manuscript: Contact persons of the survey were members of the working group New Media of the Society for Medical Education and faculty members at the deaneries of student affairs responsible for e- learning. (Methods, page 6, lines 3-5) 3. How many times were reminders sent for non-respondents? Did you try sending the link to different people to try and get a response? We added the number of reminders and alternative attempts we chose in order to get responses from the various medical schools in cases of non-responses. All persons addressed were confirmed to have received the questionnaire. We changed the sentence and completed the context as follows: The survey started in February 2013 and after 4 reminders, data collection ended in September Reminders consisted of reminder s to the correspondent involved. In cases where an answer was still missing, we tried to contact assistants of the deaneries of student affairs by . In a last attempt, the deanery was called by phone. (Methods, page 7, lines 3-8) 4. Most schools included were in Germany. Out of interest, was there any difference between the practices inside and outside of Germany? This is indeed an interesting question, which cannot be addressed sufficiently in this paper due to the complexity and large differences of not only internationally, but also intra-nationally in curricular design and teaching methods in practice. Regarding the curricular teaching practice, it has to be stated that there are large differences - 3 -

4 - 3 - in the curricular design and teaching methods between each of the three countries addressed here. The structures of the individual undergraduate medical curricula range from a rather "traditional" approach with separation of basic and clinical sciences up to integrated, outcomebased programs, where clinical aspects are integrated from the beginning onwards. Regarding the e-learning practice, we unfortunately cannot draw any credible conclusions from the data gained here as most participating medical schools made use of the anonymity that was offered. In general, e-learning practice in different learning environments is markedly influenced by the local educational systems. 5. An r statistic from the Spearman s rank is given. Please provide a p-value for this analysis, and report the interpretation after that (e.g. weak positive correlation that is not statistically significant). We used this helpful remark to re-calculate the given data and changed the particular paragraph into the following: There was a weak negative correlation trend that was not statistically significant for the size of the medical school and their financial means (r(10) = , p = 0.20). A weak positive correlation trend which was not statistically significant could be observed for the size of the medical school and the number of employees within an e-learning department (r(14) = 0.17, p = 0.53). (Results, page 9, lines 7-11) Minor 1. Results, p9. Instead of using, say, (26x) after the percentages, use conventional notation, i.e. (n=26) As recommended, we changed the particular enumerations into the conventional notation form in the Results section. 2. In figure 2, there are some disciplines in the other disciplines section which are clinical, e.g. emergency medicine, infectious diseases. Also there is an artefact present ( Diagrammbereich - near the top). We apologize for the artefact, and this has been removed. The misallocated clinical disciplines are now in the right section of Figure 2. The Figure itself was changed according to the remarks of reviewers 2 (please see below). 3. Manuscript needs language correction throughout. We apologize for this need. A native English speaker has corrected the manuscript before resubmission

5 - 4 - Second reviewer: Marc Triola Major Compulsory Revisions More detail is needed about how the survey was administered and its contents. Including the actual survey instrument as part of the manuscript would greatly help the interpretation of the data. We agree. The first part about the way the survey was administered is in line with the comments of the first reviewer and is addressed in the section above. Regarding the second mentioned aspect about the content of the survey, we added the translated survey as Supplement 1 to the manuscript. We added to the manuscript: The final questionnaire, which for this study contained a total of 49 items, with 27 closed questions and 22 questions for free text answers (Supplement 1). (Methods, page 6, line 23) Also, we added on the last page in the new section Supplements : Supplement 1: The questionnaire served as survey instrument (the text was translated from German into English). It was made available to the addressed medical schools in Austria, Germany and Switzerland with the online program SurveyMonkey (SurveyMonkey, Oregon, USA). The survey consisted of 49 items, with 27 closed questions and 22 questions for free text answers. (Supplements, page 25, lines 3-8) Seven free text questions were not included in the manuscript as the received responses yielded a low response rate (n 5) or ambiguous answers, i.e. statements that were not accurate or precise enough to allow a clear interpretation by the authors. We added to the manuscript: A total of 34 out of the 48 online questionnaires was returned (71%). After analysis of all data, seven questions for free text answers were excluded due to a low return rate (n 5) or due to ambiguous responses (Supplement 1). (Results, page 8, lines 2-4) The questions that gathered lists of LMSs and authoring tools would likely be very helpful to readers looking to implement similar solutions. Also the list of challenges resonated with what many schools are facing. We agree and included the specific questions directly into the subheadings: Table 3: Software programs used for the creation of e-learning contents recording of lectures. The questions were: What is/are the name/s of the LMS you are using at your medical school?, Which programs do you offer your teachers for the development and creation of e-learning contents?, Which programs are used alongside those you offer by your teachers on their own initiative (as far as you know of?), If you record lectures which tools do you use? (Table 3, page 23, lines 3-7) We also followed the advice and added the original (translated) answers of the challenges which many medical schools are facing as a further supplement (Supplement 2) to this re-submission. We added to the manuscript: With regards to a question about challenges for the future in the field of e-learning, the 21 responses (Supplement 2) could be crystallized into the following themes: (Results, page 10, line 13) We also added on the last page in the new section Supplements : Supplement 2: Original answers (n = 21) of the participating medical schools to the question Which challenges do you see in the field of e-learning for your medical school in the next years? (This was translated from German into English.) Each number represents commentaries of one medical school (anonymous). (Supplements, page 25, lines 10-14) - 5 -

6 - 5 - I did not fully understand Figure 2. Were these the result of yes/no questions asking if any e-learning of any type were used in that topic area? Including the specific questions that these data were abstract from would help their interpretation. We agree with the reviewer that some more clarification would be helpful for Figure 2 and have now included the specific questions in the subheadings. Figure 2: Relative distribution of e-learning offerings in various disciplines at medical schools. The number of the answering medical schools per items is shown in brackets. The main question asked was: Which of the following disciplines at your medical school offer e-learning? (individual disciplines and their grouping into preclinical/clinical/ interdisciplinary clinical disciplines were given). (Figures, page 24, lines 12-17) Though the great majority of respondents chose to remain anonymous, one addition that could greatly strengthen this paper is some sort of table or figure that allowed us to see a synthetic view of the data according to each school. Are each of the 34 schools along a similar spectrum? Do some do just a few pre-clinical activities and no clinical? vice versa? Do schools that have mandatory use also have faculty rewards? Does the presence of e- learning staff (50%) predict the use of any modalities? Does the budget for implementation correlate with the level of available tools and support? We consider these remarks to be very helpful indeed. We actually tried (very hard) to create a synthetic Table or Figure, but no sense-making or readable synthetic view was able to emerge. This was due to the many different perspectives potentially analyzable (presence of e-learning staff, budget, relation pre-clinical versus clinical activities, mandatory use, etc.). There were also large numbers of different items included and the amounts of data actually available resulted in statistically nonsignificant factors (often due to underpowered data numbers). We instead followed the specific questions raised by the review itself and performed the following analysis. 1) Do some do just a few pre-clinical activities and no clinical? vice versa? In the first step, we formed groups of low, average and high implementation for both the pre-clinical disciplines and the clinical disciplines: legend low average high low average 1 to 2 disciplines 3 to 5 disciplines 6 to 8 disciplines 1 to 7 disciplines 8 to 14 disciplines pre-clinical curriculum 8 items clinical curriculum 21 items high 15 to 21 disciplines - 6 -

7 - 6 - Level of implementation of e-learning formats 70% 60% 50% 40% 30% 20% 10% 0% pre-clinical curriculum none low high clinical curriculum We compared the level of implementation of e-learning for both pre-clinical and clinical curricula. A chi-square test of independence indicated that the level of implementation of e-learning was not associated with the section of curriculum, χ 2 = , p = Due to the lack of statistically relevant findings, we did not include this aspect into the manuscript 2) Do schools that have mandatory use also have faculty rewards? Use of e-learning was only mandatory in only 3 medical schools, thus leaving this underpowered for statistical analysis. When we combined mandatory use of e-learning and blended learning the number of medical schools increased to 15. We compared the mandatory use of e-learning with the presence of a faculty reward. A chi-square test of independence indicated that the presence of a faculty reward was not associated with the mandatory use of e-learning, χ 2 = 2.848, p = 0.092, Cramer s V = Here also, we relinquished to include this aspect into the manuscript due to the absence of any statistically relevant findings. 3) Does the presence of e-learning staff (50%) predict the use of any modalities? We analyzed this aspect for the results shown in Table 1 and 2 as well as in Figure 1 and 2. Some significant relations could be revealed which are explained below. Table 1 (version just for the cover letter): General information about e-learning provision at medical schools (n=34) (results are expressed as percentage per total answers per item) Question General information about e-learning at the medical schools Do e-learning tools for the education of students exist at your medical school? predominantly mandatory predominantly optional Is there a specific set of recommendations for the application of e-learning by your medical school from the academic board or deanery? Yes (%) 100 % 9.4 % 90.6 % e-learning staff present yes no 100 % 3,1% 46,9% 100 % 6.3 % 43,8% 43.8 % 34.4 %* 9.4 %* - 7 -

8 - 7 - Do they stipulate the use of a quality assurance code for your e-learning activities? - If so, are these quality criteria designed to comply with the standards stipulated by the German Medical Association or similar (please see the PDF file attached for details) Is there a faculty-wide strategy for increasing the scope and quality of e-learning tools over the coming years in your medical school? Do you take gender criteria into account when developing e-learning activities? Do you wish more support from relevant discipline societies (GMA/GMDS etc.) and academic institutions or public agencies (DFG/BMBF/State Ministries) for the development of e-learning activities? * p < 0.01 (chi-square test of independence) 48.4 % 20.0 % 29.0 % 15.0 % 19.4 % 5.0 % 58.1 % 35.5 % 22.6 % 39.3 % 17.9 % 21.4 % 75.9 % 41.4 % 34.5 % The item Is there a specific set of recommendations for the application of e-learning by your medical school from the academic board or deanery? was significantly different between medical schools with and without e-learning staff. For all other items, this was not the case. For the manuscript we decided to mark the significant difference in Table 1 and to indicate the result in the manuscript text. We chose this option instead of including the two extra rows, each with large numbers of values with no significant difference. We felt this would distract the potential reader from the main result (percentage of yes-answers) and would provide the presence or non-presence of e-learning staff a far greater visual weight as the analyses did in fact reveal. Table 2 (version just for the cover letter): Infrastructural information about e-learning provisions at medical schools (n= 34) (results are expressed as percentage per total answers per item) Questions Infrastructural information about the provision of e-learning Does your medical school offer a) performance-orientated financial rewards (LOM)? b) specific LOM for teaching? c) Is e-learning associated with the award of LOM? Yes (%) 75.9 % 66.7 % 16.7 % e-learning staff present yes no 41.4 % 33.3 % 6.7 % 34.5 % 33.3 % 10.0 % Do you have permanent staff in your medical school who are employed to deal with e-learning (an e-learning team/department)? Do you offer training or qualification programs for teachers......that deal directly with the authoring systems of programs in use at your medical school?...on the topic of e-learning (general information)? Would you make use of training programs that have been developed at other universities for the training of your teachers? Does your medical school use electronic means to carry out summative (mandatory) exams? Does your medical school also offer e-learning formative exams to students? Do you reward your teachers in some form for... a) the development of e-learning tools/courses? b) the implementation of e-learning tools/courses? 50.0 % % 70.0% 84.6 % 45.2 % 43.3 % 42.3 % 35.5 % 26.7 % 42.3 % 53.3 % 26.7 % 26.7 % 58.6 % 34.5 % 24.1 % 32.1 % 33.3 % 17.9 % 18.5 % 14.3 % 14.8 % - 8 -

9 - 8 - Do the development and implementation of e-learning tools/courses count towards teaching activities or load? If not, is this planned at your medical school for the future? Would you consider offering e-learning opportunities developed by other medical schools for inclusion in teaching by your medical school, if they fitted into the curriculum? Are teachers at your medical school encouraged (through instructions, study regulations, etc.) to prepare e-learning tools? Do you regularly evaluate the opinions, attitudes and experiences of your teachers on the subject of e- learning in your medical school? Do you regularly evaluate the opinions, attitudes and experiences of your students on the subject of e-learning in your medical school? Does your medical school or do your students recognise outstanding e-learning opportunities with awards? * p < 0.01 (chi-square test of independence) 24.1 % 39.1 % 17.2 % 21.7 % 6.9 % 17.4 % 96.6 % 51.7 % 44.8 % 56.7 % 43.3 % 13.3 % 31.0 % 20.7 % 10.3 % 62.1 % 44.8 %* 17.2 %* 13.3 % 10.0 % 3.3 % The item Do you regularly evaluate the opinions, attitudes and experiences of your students on the subject of e-learning in your medical school? was significantly different between medical schools with and without e-learning staff. For all other items, this was not the case. In a manner similar to that used for Table 1, we decided to mark the significant difference in Table 2 and to indicate the result in the manuscript text. Also here, we chose this option instead of including the two extra rows, each with large number of values with no significant difference. Figure 1 (changed also for the revised submission) In the context of Figure 1, a statistically significant higher implementation level was found over all items when e-learning staff was present in the medical school (Mann-Whitney U p < 0.05, / % versus /-13.3 %). The former Figure 1has now been replaced in the manuscript. The new graph now shows the relative implementation level of each item in relation to the fact whether or not e-learning staff was present

10 - 9 - Figure 2 (changed also for the revised submission) In the context of Figure 2, a highly statistically significant higher implementation level was shown over all individual disciplines when e-learning staff was present in the medical school (Mann- Whitney U p < 0.001, /-8.5 % versus /-6.5. %). Like Figure 1, the former Figure 2 was replaced in the manuscript. The new graph now shows the relative implementation level of all disciplines in relation to the fact whether or not e-learning staff was present. Following the above-mentioned analyses, we changed the following parts within the manuscript: Results: Online learning platforms were used by 97% of the schools. Full-time e-learning staff was employed by 50%, and these had a positive and significant effect on the presence of e- learning in the corresponding medical schools. (Abstract, page 2, lines 14-16) Conclusions: While all participating medical schools used e-learning concepts, this survey revealed also a reasonable support by institutional infrastructure and the importance of staff for the implementation level of e-learning offerings. (Abstract, page 2, lines 22-23) Additionally, the chi-square test was used to analyze if the presence of e-learning staff predicted the use of any modalities, indicated by the medical schools in the closed questions. Alternatively, the Mann-Whitney U test was performed to analyze the implementation level of e-learning in the existing disciplines at the medical schools, and that of e-learning tools both in general and in dependence of the presence of e-learning staff. (Methods, page 7, lines 16-21) Concerning the impact of the presence of e-learning staff, a significant correlation could be shown for the existence of recommendations for the application of e-learning by a medical school (Table 1). Also, the evaluation of students on the subject of e-learning was performed significantly more often when e-learning staff was present (Table 2). There were also significantly higher implementation levels of the provided total amount of e-learning items (Figure 1), as well as of the general existence of e-learning offers in the disciplines of medical schools (Figure 2). (Results, page 9, lines 12-18)

11 However, it could also be shown that the existence of e-learning staff is positively correlate with the presence of e-learning offers at a medical school. This stresses the use of employing personnel merely dedicated to managing a medical schools e-learning portfolio. (Discussion, page 13, lines 1-4). Table 1: General information about e-learning provision at medical schools (n=34; results are expressed as percentage of total answers per item). The symbol * indicates a significant difference between medical schools with (34.4%) and without e-learning staff (9.4%) with a p < 0.01 (chisquare test of independence). (Tables, page 21, lines 3-7) Table 2: Infrastructural information about e-learning provisions at medical schools (n= 34; results are expressed as percentage of total answers per item). The symbol * indicates a significant difference between medical schools with (44.8%) and without e-learning staff (17.2%) with a p < 0.01 (chi-square test of independence). (Tables, page 22, lines 1-5) Figure 1: Relative use of various e-learning formats at medical schools. The numbers of medical schools answering these items are shown in brackets. The graph shows the relative implementation level of each item in relation to where e-learning staff was present or not. There was a statistically significant higher implementation level of the provided total amount of e-learning items when e-learning staff was present (35.9 +/-13.1 %) versus not present (26.1 +/-13.3 %) in the medical school (p < 0.05, Mann-Whitney U test). (Figures, page 24, lines 3-10) Figure 2: Relative distribution of e-learning offerings in various disciplines at medical schools. The numbers of the medical schools answering these items are shown in brackets. The main question asked was: Which of the following disciplines at your medical school offer e-learning? (Individual disciplines and their grouping into preclinical/clinical/ other disciplines were given). It was possible to add as answers individual disciplines for any medical school. The graph shows also the relative implementation level of all disciplines in relation to where e-learning staff was present or not. There was a statistically significant higher implementation level of the general existence of e-learning offers in the individual disciplines when e-learning staff was present (40.0 +/-8.5 %) or not (26.2 +/-6.6 %) in the medical schools (p < 0.001, Mann-Whitney U test). (Figures, page 24, changes in this context in lines 17-22) 4) Does the budget for implementation correlate with the level of available tools and support? For these analyses, 13 medical schools provided information about their budget. With these numbers and the budget categories of the survey we explored several groupings but that did not yield any significant relations. This was mainly due to the underpowered number of data that could be analyzed. Thus, we did not include this aspect into the manuscript 16/34 responses came from e-learning staff. Would they be able to accurately report the faculty perspective? This question is interesting. But as the addressed e-learning staff was part of the faculty of education in all medical schools, it can be assumed that they were able to report accurately about the faculty perspective. Additionally, the learning management platforms of the schools are usually also run by the e-learning staff, giving these employees a perfect overview over the individual e- learning activities not only of the single disciplines but also from the faculty side

12 Minor Essential Revisions The following line could use greater clarification: "E-learning was used neither in 100% nor 0% of the evaluated subjects at all medical schools, what has already been reported for single subjects [26]." We admit that this sentence was not easy to understand. The part has been re-phrased as follows: There was an inhomogeneous representation of e-learning offers for the single medical disciplines in this survey. However, the individual number of medical schools which provided offers was also inhomogeneous for every particular discipline, an observation which has already been reported e.g. for radiology [26]. The presented results suggest that there is a lower level of implementation in smaller disciplines, such as otorhinolaryngology or human genetics. (Discussion, page 12, lines 6-11) One of the survey questions was "Do you take gender criteria into account when developing e-learning activities? This is not referred to in the text. What was the intent of the question and how do you think the responders interpreted it? We kindly thank the reviewer for having pointed out this not major, but important aspect. We think that this question covers gender issues in general. This may cover use of gender-neutral language, avoiding of gender stereotypes (doctor = man, nurse = female) or contents of gender medicine as a discipline itself. The revised document now contains the following sentence: In this study, gender issues were taken into account for developing e-learning activities in general by less than half of the participating medical schools. The survey did not go into any details, for instance into the use of gender-neutral language, avoiding gender stereotypes or gender medicine as a topic itself. (Discussion, page 11, lines 17-21)