L-UNIVERSITÀ TA MALTA Msida Malta SKOLA MEDIKA Sptar Mater Dei Prof. Charles Savona-Ventura MD, DScMed, FRCOG, AccrCOG, MRCPI Head Department of Obstetrics & Gynaecology UNIVERSITY OF MALTA Msida Malta MEDICAL SCHOOL Mater Dei Hospital 16 th June 2011 FINAL EXAMINATION OBG4000 AUDIT June 2011 SESSION WRITTEN COMPONENT & LOGBOOK ASSESSMENT The overall Final Assessment in Obstetrics & Gynaecology [Study unit OBG4000] for the June Session 2011 is composed of two major sections a written component and a clinical assessment. Each section carries an equal proportion of the final mark, i.e. 50% each section, though a pass in the clinical section of the examination is essential for an overall pass. The grading criteria defined by the University of Malta are detailed in the table below: Descriptor Mark Range Grade Work of exceptional quality. 95%- 100% A+ Work of excellent quality. 80%- 94% A Work of very good quality. 75%- 79% B+ Work of good quality. 70%- 74% B Work of average quality. 65%- 69% C+ Work of fair quality. 55%- 64% C Work of rather low quality. 50%- 54% D+ Marginal Pass. 45%- 49% D Fail <45% F The overall student grade distribution for the whole examination and for the two major sections are shown in the Figure below. The written section grade distribution showed a clear Gaussian distribution; however the clinical assessment grade distribution shows a definite skewed distribution towards the higher grades. This overall grade resulted in a slightly skewed distribution towards the higher grades with an average mark of 70.6 + 7.6 [equivalent to Grade B reflecting work of good quality ]. A total of nine students scored a 1
Grade A mark thus passing the examination with distinction. There were no students who failed their examination [Grade F] or obtained a grade reflecting work of exceptional quality [Grade A+]. 60.0 F D C B A 50.0 40.0 30.0 20.0 10.0 0.0 TOTAL GRADE TOTAL WRITTEN TOTAL CLINICAL Figure 1: Overall and sectional grade distribution The inter-relationship between the written and clinical sections of the assessment shown in the Figure below suggests that the students generally obtained higher grades in the clinical section as compared to the written assessment. This is quite reassuring since it strongly suggests that the increased emphases on clinical and small group teaching this year has reaped its benefits. 100 Clinical percentage mark 80 60 40 20 0 0 20 40 60 80 100 Written percentage mark Figure 2: Inter-relationship between Written & Clinical Assessments 2
The class grades distribution for the previous years is shown in the following figure. 80 F D C B A 70 60 percent class 50 40 30 20 10 0 2003-04 2004-05 2005-06 2006-07 2007-08 2008-09 2009-10 2010-11 Figure 3: Class grade distribution 2003-2011 3
Written section This year the written examination structure introduced last year was retained whereby the paper was composed of one three-hour paper that incorporates both obstetrics and gynaecology. This overall format for the paper included: A: Multiple Choice Question Section [20x5 questions negative marking]; B: Objective Structured Clinical Examination [4 questions]; C: Extended Matching Questions [4 questions]; and D: Short Answer Questions [6 questions]. Paper Question Type % global written marks distribution & maximum mark possible Mean + s.d. grade Mean % grade OBG4000 A: MCQs 40% 100 63.5 + 12.0 63.5% C B: OSCEs 40% 100 68.6 + 9.3 68.6% C+ C: EMQs 8% 20 12.7 + 2.8 63.5% C D: SAQs 12% 30 17.3 + 4.5 57.8% C OVERALL 100% 250 64.8 + 9.1 64.8% C/C+ Mark distribution by each section and question type in written section The questions were targeted to be clinical and core subject oriented to better serve the objectives set by the Course Study Unit defined as The Study Unit augments the clinical knowledge and skills learned in previous years and prepares the student to have a comprehensive outlook to primary and preventive care for women s health care needs. Each section of the written open-ended examination was corrected by two examiners. This actually meant that each student s written assessment was collectively seen by fivesix examiners. Each question was designed with a model answer, so that inter-examiner variability was minimal even when the questions were open-ended. The overall results of the written part of the assessment show a Gaussian distribution [Figure 1] with a slight skew to the higher grades. There were two students who obtained a grade less than 45% [one 36% and one 44%]. There were two student who obtained a Grade A result [none with a Grade A+]. The average grade obtained by the class was 64.8 + 9.1% equivalent to a Grade C. The overall distribution suggests a statistically fair distribution of the assessment tool with the students performing slightly above average in the subject. This is not surprising since the written assessment had set out to 4
concentrate in assessing mainly core knowledge essential material and not material which could be considered of a specialist level. The written paper was composed of four sections using different assessment techniques carrying a variable proportion distribution of marks [see Table above]. The mean + s.d. score obtained by the class in the Multiple Choice Questions [MCQs] section was 63.5 + 12.0 [maximum possible mark = 100] equivalent to Grade C. This carried negative marking for wrong answers. On average, the unknown unanswered mean was 20.6 + 9.7, whereas the wrong answer mean score was 8.0 + 4.2. This gave a mean positive response of 71.4 + 10.2. A similar mean percentage score [63.5% or Grade C] was obtained in the Extended Matching [EMQs] section score = 12.7 + 2.8 [maximum possible mark = 20]. It should be noted that MCQs, and EMQs are closed-ended questions where the answer is either right or wrong and hence carry a very closed system of marking allowing no room for inter-examiner variability. OSCEs and SAQs are open-ended questions where, in spite of designing model answers, inter-examiner variability can play a role. The mean + s.d. score obtained by the class in the Short Answer Questions [SAQs] section was 17.3 + 4.5 [maximum possible mark = 30] equivalent to 57.8% or Grade C. The mean + s.d. score obtained by the class in the Objective Structured Clinical Examination [OSCEs] section was 68.6 + 9.3 [maximum possible mark = 100] equivalent to Grade C+. The inter-examiner variability was sometimes quite marked [see Table below]. Examiners OSCE Max = 100 SAQs Max = 30 1 vs 2 63.7 vs 67.6 13.3 vs 16.7 3 vs 4 65.2 vs 84.4-5 vs 6 65.4 vs 68.0 18.1 vs 17.8 7 vs 8 69.8 vs 64.3 18.5 vs 17.5 3 vs 9-18.2 vs 18.6 Inter-Examiner variation 5
60.0 F D C B A 50.0 40.0 30.0 20.0 10.0 0.0 TOTAL EMQs OSCEs MCQs SAQs Figure 3: Grade distribution by different question type A definite improvement in overall assessment can be noted when comparing the results obtained in the fourth year of studies [mean + s.d. = 59.7 + 9.5] with those obtained in the written section of the fifth year assessment [64.8 + 9.1] [Figure 2], 100 80 Vth year 60 40 20 0 0 20 40 60 80 100 IVth year Figure 4: Grade comparison by Years IV and V results in written section 6
Clinical section The clinical assessment structure this year will be drastically changed to introduce the concept of stations to allow for the new examination management that will be necessary with the increasing number of students projected in the coming years. The clinical assessment part of the examination further incorporates the assessment mark obtained for the Clinical Placement Logbook prepared throughout the student attachments in the IV th and V th years of studies. This was felt an essential inclusion, since it encourages the students to really involve themselves and participate in the clinical training programme. The overall distribution of marks in the clinical section of the examination is as follows: Assessment % marks distribution Mean + s.d. actual mark obtained OBG4000 Station 1 20 15.0 + 2.8 Clinical History & Presentation OBG4000 Station 2 20 14.9 + 2.8 Obstetric examination OBG4000 Station 3 20 15.1 + 2.9 Gynaecological examination OBG4000 Station 4 10 7.7 + 1.8 Clinical acumen 1 OBG4000 Station 5 10 8.0 + 1.8 Clinical acumen 1 OBG4000 Station 6 10 7.7 + 1.8 Clinical acumen 1 OBG4000 Logbook mark obtained 10 8.3 + 0.7 % mark distribution by each section in the Clinical assessment The clinical examination structure was newly designed to be composed of six stations, all with specific assessment objectives: A. Stations 1 & 2 [lasting 30 minutes, with 10 minutes for the student to take a patient s history on his own] assessed the practical clinical skills of the student in relation to the obstetric patient data collection, assimilation, presentation, and finally examination of the pregnant abdomen. Assessment was defined according to the following criteria: 7
outright fail borderline pass good pass excellent History # a. Introductory sentence sets the scene # b. Logical & organised progression of presentation 1 2 3 4 5 1 2 3 4 5 # c. Focuses on relevant features 1 2 3 4 5 # d. Fluent progress & keeps to time 1 2 3 4 5 Obstetric Examination # a. Interaction skills with the patient with appropriate explanation of process outright fail borderline pass good pass excellent 1 2 3 4 5 # b. Appropriate technique 3 6 9 12 15 B. Station 3 [lasting 10 minutes] assessed the student s competence relating to vaginal examination using the simulation models. Assessment was defined according to the following criteria: Gynaecological Examination outright fail borderline pass good pass excellent # a. Appropriate technique - abdominal 2 4 6 8 10 # b. Appropriate technique - pelvic 2 4 6 8 10 C. Stations 4-6 [lasting 10 minutes each] assessed the student s ability to manage the obstetric and gynaecological patients in various clinical situations. A number of specific clinical situations with appropriate answers have been prepared to guide the examiners. Assessment was defined according to the following criteria: Clinical acumen # a. knowledge level outright fail borderline pass good pass excellent 1 2 3 4 5 8
The objectives set by the clinical examination structure were to assess: 1. the practical clinical skills through Stations 1-3, these carrying a total of 60% of the overall mark mean overall mark obtained in this section was 45.0 + 6.3 [equivalent to 75% grade C+]; 2. the clinical management skills through Section 4-6, these carrying a total of 30% of the overall mark mean overall mark was 23.4 + 3.7 [equivalent to 77.8% grade B+]; The grade distribution of these sections as they compare to the total clinical examination is shown in the Figure below. The clinical assessment grade distribution shows a definite skewed distribution towards the higher grades with an average mark of 76.6 + 8.7%. This probably reflects the markedly increased emphasis placed on the clinical placement and the significant increase in small group teaching during the last year of the course that has been introduced this year. This policy has certainly paid dividends. Small group academic teaching in the final academic years of the course in the department now accounts for a total of 6 hours of case-based discussion and six hours of tutorials/simulation workshops during the first semester, followed by 28 further hours of clinical-based tutorials throughout the second semester. 70.0 F D C B A 60.0 50.0 40.0 30.0 20.0 10.0 0.0 TOTAL CLINICAL SKILLS ACUMEN Figure 6: Grade distribution by different sector type of clinical assessment 9
The logbook assessment carried 10% of the total mark in this section or an overall 5% of the total overall grade. It was graded on the criteria listed in the Table below. Reproductive health history taking [assessment based on case presentations] Obstetric Abdominal examination [assessment based on clinic session attendances] Pelvic examination including cervical smear taking [assessment based on observation] Familiarity with normal and abnormal deliveries [assessment based on signed cases registered in logbook] Capacity to discuss clinical options [e.g. differential diagnosis, management options] in a case scenario setting [assessment based on case presentations] Overall Attendance record during Clinical attachment [assessment based on registered signed attendances] each marked as 1. outright fail; 2. borderline; 3. pass; 4. good pass; 5. excellent The logbook prepared during the students placement in the department was generally taken very seriously and was very well prepared. This is reflected by the rather high mean + s.d. score of 8.3 + 0.7 achieved by the class with a range of 5.4 to 9.0[maximum score = 10]. There was a very good correlation between the logbook marks obtained in the 4 th and 5 th year of studies [% mark - mean + s.d.: 81.5 + 8.3 vs. 84.3 + 7.7] [see Figure below]. One student improved significantly his percentage score from 44% to 92%; while another reduced the score from 84% to 52%. 100 90 80 Vth year % mark 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 IVth year % m ark Figure 5: Percentage grade distribution of Logbook assessment in IV th & V th years 10
Conclusions The results obtained by the students this academic year clearly support the policy taken by the Department to concentrate significantly on small-group teaching and the strong encouragement given towards students attending regularly their clinical attachments. The introduction of simulation workshop sessions this outgoing academic has also apparently increased the practical clinical skills required in the speciality. Further investment needs to be made to upgrade the Obstetrics & Gynaecology simulation facilities with the purchase of support parts for the current models and the purchase of new models designed for obstetrics examination/demonstrations. Acknowledgements Acknowledgement is definitely due to the rest of the academic team who have contributed significantly to the undergraduate teaching programme of the department. Their contribution far outweighs the remuneration they actually receive. Further acknowledgement is due to the non-academic members of the clinical staff who in spite of receiving no remuneration still contribute to the teaching programme. Their association with the university should be truly acknowledged and made a formal one. 11