Online Lessons ICT. GCP Training. Six months after training. Pre-training. Immediately after training

Transcription

1 ICT GCP Training Lessons Sabine Veigel and Andreas Grund at GCP-Service International provide a comparison of training methods for site staff, including an analysis of face-to-face versus online training methodology Conducting a clinical study is only justified when it has been appropriately planned, implemented, evaluated and reported from ethical and scientific standpoints. In the field of clinical research, national laws, international guidelines and their implementation have been subject to constant development over the past few years. Germany has particularly strict requirements in place in terms of investigator training courses (1). German ethics committees require that all investigators attend a one or two day training, which covers national regulations, as well as ICH GCP (2). These intensive training activity demands are criticised quite frequently by investigators and sponsors, although this requirement has not been invented by German ethics committees themselves, but is required according to ICH GCP (ICH GCP 4.3.1) (3). It is expected that investigators as well as other site staff, sponsors and CRO personnel involved in clinical studies have the expertise to ensure the correct conduct of clinical studies. For this reason, both the pharmaceutical and medical device industries spend a lot of time, money and resources for training personnel, investigational sites and CRO staff. Whether these investments are an effective way of building up required competencies is rarely questioned. Often, a lot of time and money is spent on long seminars and face-to-face training. However, from educational science we know that these training methods are only successful for a limited number of people. When participants are asked about the content of the training course they attended a few weeks later, the results are often disappointing (see Figure 1). Most course participants need learning methods with a more systematic approach and regular revision as well as testing of the newly acquired knowledge. This enables participants to retain the learnt material over a longer period of time and successfully apply it in practice. Figure 1: Normal learning curve before, immediately after training and six months after training later Pre-training 98 Immediately after training 75 Six months after training ICT l November 211

2 METHODOLOGY Figure 2: Number of attendees in both training courses between 29 and 211 Recently, a prospective survey was designed in order to compare a lecture-based GCP course with an e-learning GCP course of identical content. The participants were investigators who had completed face-to-face training on GCP topics and a participant group, who had completed an online GCP course with identical contents. Both eight-hour training courses have been accredited by the Medical Association of Bremen (Germany) and include the following topics: Introduction: development of drugs and medical devices Basics of biometrics: different study designs and sample size calculation The declaration of Helsinki and the GCP guidelines GCP definitions/important GCP documents Investigator responsibilities part one Investigator responsibilities part two/sponsor responsibilities/audits Inspections and the reporting of safety-related events Current legislation and regulation on clinical trials in Europe In order to receive a training certificate from a lecture-based course, the Medical Associations require that the success of the training is measured. In this case study, the test contained 15 questions about the training content, which the attendees needed to complete immediately after or during the training. The 15 questions cover all eight topics of the training. Investigators have been trained face-to-face in the eight training modules between 29 and 211. In 29 a training course was developed alongside an online equivalent. The attendees of the online course received an individual username and password which must be kept confidential. Each of the online training modules consisted of an animated video presentation that took approximately 45 minutes. Afterwards a multiple choice test needed to be passed for each module. Each test consisted of 15 questions that were generated randomly from a pool of 4 questions, and 8 per cent of the questions needed to be answered correctly to pass the test. If a trainee failed the test, three test repeats were possible. With each attempt, new questions were generated. The attendees had access for two months, in which all eight tests needed to be passed to receive the training certificate automatically. Technical as well as GCP-related In the face-to-face training an average of 16 investigators per month could be trained between 29 and 211, whereas 27 investigators were trained in the online training (p<.5) Number of participants January 29 March 29 May 29 July 29 September 29 November 29 January 21 March 21 questions could be asked by , and the trainees received an answer within 48 hours. The statistics of the online training platform and the results from the face-to-face test could be used to compare shortterm test results in both groups to evaluate the advantages and disadvantages of each training intervention. The t-test was used as a point of comparison if differences were significant. The number of attendees varied between 17 and 128 for the online training and 58 to 578 in the face-toface training, because not all attendees gave feedback. To compare the groups the percentages per group were used. Furthermore, investigators were given the opportunity to complete evaluation questionnaires consisting of quantitative components and open-ended qualitative responses to assess different parameters of the training courses in respect of acceptance and satisfaction of the attendees with the respective course. The following parameters were evaluated: degree of satisfaction; usability; intellectual approach; failure rate; duration; practicability and direct/indirect costs. By calculating the potential travel expenses based on the places of residences of the online training participants, the courses could also be compared in respect of their costs. RESULTS May 21 July 21 September 21 November 21 January 21 R 2 =.9663 R 2 =.9498 The first result from the survey was that the increasing requirements for training in GCP and the national requirements correlate with the number of training attendees. Figure 2 shows the development of training attendees on the two different courses from January 29 until August 211. In both courses the total number of March 211 May 211 July l ICT 17

3 Figure 3: Overall satisfaction of trainees in both courses Overall evaluation of training in 211 training training Very good Good Satisfactory Bad Very bad Figure 4: Satisfaction of trainees in terms of course severity Evaluation of course content regarding lectures participants participants Too difficult Just right Too easy Figure 5: Satisfaction of trainees in terms of course practicality Evaluation of course content regarding examples of use attendees increased significantly, with online attendees growing the most. In terms of the satisfaction of the attendees, the results demonstrated overall satisfaction with both training methods. The usability of the course contents in professional practice was regarded as high or very high by 68 per cent of the investigators. Eighty-six per cent of online training participants and 94 per cent of face-to-face training participants who completed the evaluation form rated the GCP training as good or very good, which results in an overall grade of 1.8 for the online and 1.5 for the lecture-based course (see Figure 3). On a scale from too difficult to too easy, the course contents regarding the lectures were evaluated as just right by 71 per cent of the online course participants and 91 per cent of the lecture-based course (see Figure 4). The practical examples used were found as right by even 1 per cent of investigators trained online and by 91 per cent trained face-to-face (see Figure 5). The practicality of both courses has been judged very similar. Both groups of attendees appreciated the practical examples used in the courses to explain certain rules and requirements. As results from the open questions answered by investigators trained online show, the flexibility of the online training programme was very much appreciated. Expectations of the online training participants could be met particularly in terms of the quality of content (grade of 1.7), professional benefit (grade of 1.8) and examples of use (grade of 1.9). However, online participants reported that there was too much content for the allotted timeframe of 45 minutes per module. participants participants Too difficult Just right Too easy In order to determine the total period of time needed by the participants to complete the online training, the dates of the test visits of the investigators in the statistical database of the online platform ILIAS were analysed. By means of the time difference between test visit one (first visit) and test visit eight (last visit) an average of days ( to 39 days) could be calculated. This analysis does not acquire the net working hours, but the absolute timeframe in which the training was performed. Three participants completed all eight modules of the online training within one day, another three even within hours ( days), whereby the shortest processing time lasted nine hours, 28 minutes. The average duration of each online training test covering 15 questions 18 ICT l

4 Figure 6: Mean direct and indirect costs per trainee for both courses 7, Comparison of training costs 6, 5, 4, Direct costs ( ) 3, Indirect costs ( ) 2, Total costs ( ) 1, was 14 minutes and varied between (±8) minutes. All investigators trained online or face-to-face passed the required tests. On average, 87 per cent of the tests questions were answered correctly in both groups. In addition to the time the costs were compared, the direct as well as the indirect costs (see Figure 6). The price for a face-to-face training session is 4, which reflects the mean price for one day face-to-face GCP trainings in Germany. The price for an online GCP training is 35, which makes a difference of 12.5 per cent (p<.5). The difference by taking into account the indirect costs was even higher. There are no indirect travelling costs to online training, whereas Figure 7: Development of certified online training (1998 until 29) All figures: GCP CME physician participants Year the estimated travel costs for the travel to the training are (p<.1). If attendees needed to stay overnight the costs increased, but these additional costs have not been considered. The overall cost difference between both methods is 313. Furthermore, the loss of pay for selfemployed investigators is significantly higher for the face-toface training, which took place during the week compared to the online training, which was usually done in the evening or during weekend. Considering that 58 per cent of the investigators had little or no experience with new online media, the majority responded that they felt comfortable in using this course format and would prefer the online training over classroom training. According to the participants, the main strengths of online training include the free timing, no travelling effort, lower costs, convenient learning conditions in personal environments, the possibility of repeating individual chapters and selfdetermination of the amount of subject matter or speed of learning. Potential disadvantages mentioned include the lack of communication with experts and with any group of other participants, a need for free time, technical prerequisites and self-motivation/ discipline needing to be raised. 29 DISCUSSION The ever increasing scientific complexity of clinical studies has enlarged the number of study patients as well as the number of study sites during the last couple of years. In parallel, the regulatory complexity has changed the responsibility of sponsors, CROs, ethics committees and competent authorities. A few years ago research personnel needed to have simple ICH GCP certificates in place, which was sufficient to comply with the expectations of the inspectors and ethics committees. The method of gaining the certificate was usually not scrutinized; 11 ICT l

5 short GCP courses were conducted during investigator meetings. However, with the implementation of the EU Directive 21/2/EC, and thereby ICH GCP, into the national legislations of the EU member states, the requirements became stricter (4). Nowadays, clinical research personnel do not need to prove training attendance; instead the focus is on competence in research-related topics. This is reflected by the fact that most of the inspection and audit findings reported by the authorities and sponsors on a regular basis refer to deviations against ICH GCP, as well as national regulations. Hence, training in ICH GCP alone cannot be considered sufficient to ensure data integrity, or the rights and welfare of study subjects. The consequence of this is that the training efforts spent for clinical studies exploded. The common way of providing training to research staff cannot be considered adequate as it is not possible to train such a high number of new staff in a systematic and cost-effective way. Furthermore, the fluctuations in site staff is quite high in clinical research, which increases even more the efforts, costs and energy that sponsors need to conduct training. The evaluation of the two different training methods showed that both methods have advantages and disadvantages. Both training methods were much appreciated by the trainees, who liked the content of the training modules. The satisfaction grade of face-to-face training is not significantly higher than online training. The conditions for obtaining a training certificate for face-to-face training are easier to meet as only a limited number of test questions are required, which can even be answered during the training. For online training, a much higher number of randomised test questions needs to be answered correctly, which also extends throughout the duration of the training until a certificate is obtained. Nevertheless, the total time from registration for the online training until obtaining the training certificate is days on average, which is extremely fast, and usually lasts much longer for face-to-face training, as the trainee is dependent on the date and location of the training course. Cost reduction is another main advantage of online training. According to the results of this evaluation, online training is about 3 less expensive per trainee compared to face-toface training. For midsize studies, on which 1 investigators (principal investigators and sub-investigators) are involved, the cost difference would be 3,. Hence, online training is an economic way of providing knowledge, saving time as well as direct and indirect costs through reduced travel expenses. This advantage has generally been recognised in medicine, which caused an increase of certified online training in medical topics of 41.9 per cent between 1998 and 29 as shown in Figure 7 (5). The new technologies now available make use of well-developed GCP online training courses covering scientific backgrounds, as well as national regulations, resulting in a logical consequence for the clinical research business. CONCLUSION Taken together, this observation study has shown that both methods seem to have the same learning effect in the short term. These results have already been shown in other studies; from an economic point of view, online training has significant advantages as the costs per investigator are significantly reduced (6). The continuously and steeply increasing number of online attendees also indicates that a web-based training form is very much welcome by the trainees and the advantages are well recognised. Whether the long-term learning effect of both training methods is comparable needs to be evaluated in future. However, it can be assumed that the learning effect of an online training, which takes place over several days when combined with tests, is at least not less beneficial for the memory of investigators compared to a one day crash course, as it is known from different educational areas as well (for example, learning of languages). References 1. German Medical Association, Announcement of the German Medical Association, Recommendations by Ethics Committees in clinical trials for evaluating the qualifi cation of investigators and the suitability of trial sites according to German Drug Law, Dtsch Arztebl 17(1-2): A48-A51, Bergtholdt B and Gaertner U, Investigator and trial site qualifi cation the requirements are becoming more severe, DGPharMedNews 12(3): pp , ICH Topic E6(R1), Guideline for Good Clinical Practice 4. Directive 21/2/EC of the European Parliament and of the Council of 4 April Mack J, Pharma Support for Accredited CME Continues to Decline, Pharma Marketing News, Cook DA, Levinson AJ and Garside S et al, Internet-based learning in the health professions: a meta-analysis, JAMA 3: pp1,181-96, 28 About the authors Sabine Veigel (MSc) works in the clinical research area since 21, when she carried out her Master s thesis on the topic Scientifi c evaluation of a GCP online training at GCP-Service International as part of her studies in Nutritional Science at the University of Giessen. Since May 211 her main responsibility within GCP-Service International is working as Clinical Research Associate and Training Manager. sveigel@gcp-service.com Andreas Grund (MSc, PhD) has worked since 1997 in the clinical research sector. He worked in clinical monitoring, quality assurance, project management and training for the University Kiel, and global CROs. In 24 the CRO GCP-Service International was founded by Andreas in Bremen (Germany). Currently his main responsibility within the company is consulting and training of pharmaceutical and medical device companies in regulatory requirements, national laws and international guidelines. agrund@gcp-service.com 112 ICT l