University of Alabama at Birmingham Detailed Assessment Report 2013-14 Medicine, MD As of: 8/26/2014 05:46 AM EST (Includes those Action Plans with Budget Amounts marked One-Time, Recurring, No Request.) Mission / Purpose Contribution to School/Unit Mission: The MD program's mission is entirely consistent with that of the School of Medicine. The ultimate outcome of the MD program is production of undifferentiated physicians who are prepared to enter graduate (residency) training in any medical discipline. Contribution to Program or Department Mission: Individual departments contribute to the MD program, and each department has its role in the MD program as part of its departmental mission. Elements of departmental mission that are not directly related to the MD program, such as research and service (and clinical care for clinical departments), provide an enhanced framework for their contribution to the education of MD's. Student Learning Outcomes/Objectives, with Any Associations and Related Measures, Targets, Findings, and Action Plans SLO 1: Acquisition of Requisite Basic Science Knowledge All School of Medicine students will acquire and demonstrate by the end of their second year a comprehensive understanding of the basic science information that is essential for success in the clinical education program of the third and fourth years. Related Measures M 1: Performance on USMLE Step 1 for the first time The USMLE Step 1 is a nation-wide knowledge exam developed and administered by the National Board of Medical Examiners (NBME) to cover all of the pre-clinical (basic science) disciplines that are requisite to success in the clinical education program. Students can retake the exam and are required to pass the exam in order to proceed in the program.. Source of Evidence: Standardized test of subject matter knowledge Target: The 2012 goal is to meet or exceed the national pass rate for students taking Step 1 for the first time, per the School of Medicine Scorecard. Finding (2012-13) - Target: Met The national pass rate for 2012 was 95.3%, and our pass rate was 95.2%. Finding (2011-12) - Target: Not Met The national pass rate for 2011 was 93.7%, and our pass rate was 93.4%. Finding (2010-11) - Target: Partially Met For students taking Step 1 for the first time in 2010, our pass rate was 90.75%, which is below the goal of 96.0%, but within rounding error of the national pass rate (91.34%). These students were the second class in our new integrated curriculum, and Step 1 scores typically drop after a curriculum change. Our goal of 96.0% was set before plans for the new curriculum were in place and is not realistic. https://app.weaveonline.com/reports/dar.aspx 1/12
Related Action Plans (by Established cycle, then alpha): Identify Step 1 content gaps and unwanted redundancies in preclerkship curriculum For the new integrated curriculum, beginning in 2007-08, preclerkship module learning activities and exam questions are being related to the Step 1 Content Outline, to identify content gaps and unwanted redundancies. Established in Cycle: 2007-08 Measure: Performance on USMLE Step 1 for the first time Outcome/Objective: Acquisition of Requisite Basic Science Knowledge Implementation Description: Module directors will use Step 1 content outline to classify exam items and module activities to identify gaps and unwanted redundancies in the curriculum. Responsible Person/Group: Pre-clerkship module directors; Office of Undergraduate Medical Education staff Add more NBME-type items to preclerkship module exams Preclerkship module exams contain too many items that do not meet standards described by the NBME for Step 1 items. All exam items will be reviewed and module directors will be assisted in rewriting items. Measure: Performance on USMLE Step 1 for the first time Outcome/Objective: Acquisition of Requisite Basic Science Knowledge Implementation Description: Staff in the Office of Undergraduate Medical Education will review exam items and assist module directors in re-writing items as needed. Responsible Person/Group: Preclerkship module directors and faculty; Office of Undergraduate Medical Education staff. Continue to monitor Step 1 content and NBME-type exam items in preclerkship modules For the new integrated curriculum, beginning in 2007-08, preclerkship module learning activities and exam questions are being related to the Step 1 Content Outline, to identify content gaps and unwanted redundancies. Preclerkship module exams contain too many items that do not meet standards described by the NBME for Step 1 items. All exam items will be reviewed and module directors will be assisted in re-writing items. Established in Cycle: 2010-11 https://app.weaveonline.com/reports/dar.aspx 2/12
Measure: Performance on USMLE Step 1 for the first time Outcome/Objective: Acquisition of Requisite Basic Science Knowledge Implementation Description: Module directors will use Step 1 content outline to classify exam items and module activities to identify gaps and unwanted redundancies in the curriculum. Staff in the Office of Undergraduate Medical Education will review exam items and assist module directors in re-writing items as needed. Responsible Person/Group: Preclerkship module directors and faculty; Office of Undergraduate Medical Education staff. Additional Resources: None SLO 2: Acquisition of Requisite Clinical Knowledge&Skills Before graduation, all School of Medicine students will acquire and demonstrate a comprehensive understanding of clinical problems, their signs and symptoms, and management issues as is appropriate for the MD at the beginning of residency training. Related Measures M 2: Performance on USMLE Step 2 for the first time The United States Medical Licensure Exam (USMLE) Step 2 is a nation-wide exam developed and administered by the National Board of Medical Examiners (NBME) to assess student clinical knowledge. Students can retake the exam and are required to pass the exam in order to graduate. Source of Evidence: Standardized test of subject matter knowledge Target: The 2012 goal is to meet or exceed the national pass rate for students taking Step 2 for the first time, per the School of Medicine Scorecard. Finding (2012-13) - Target: Not Met The national pass rate for 2012 was 97.2%, and our pass rate was 98.0%. Finding (2011-12) - Target: Met The national pass rate for 2011 was 97.1%, and our pass rate was 97.0%. Finding (2010-11) - Target: Met For students taking Step 2 for the first time in 2009-10, our pass rate was 98.25%, which exceeds the goal of 96.0%. Finding (2009-10) - Target: Met In 2009, our Step 2 pass rate was 99.38%, exceeding the stated goal of 95.5%. Related Action Plans (by Established cycle, then alpha): Identify Step 2 content gaps and unwanted redundancies in clerkship curriculum Clerkship directors will classify curriculm activities based on Step 2 content areas to identify gaps and unwanted redundancies. Implementation Status: Planned https://app.weaveonline.com/reports/dar.aspx 3/12
Measure: Performance on USMLE Step 2 for the first time Outcome/Objective: Acquisition of Requisite Clinical Knowledge&Skills Implementation Description: The classification of activities based on the Step 2 content outline will be part of the new clerkship review protocol being planned by the Clinical Medical Education Committee. Responsible Person/Group: Clerkship directors; Medical Education Committee; Office of Undergraduate Medical Education staff SLO 3: Inculcation of Patient-Centered Attitudes Prior to graduation, all School of Medicine students will acquire and demonstrate knowledge and skills of interacting with patients professionally, ethically, and compassionately. Related Measures M 3: Rating of student interactions with patients In the required Junior clerkships, a 9-point rating scale is used to evaluate students' interactions with patients. The Patient Interaction item is part of an evaluation instrument that also includes items for History & Physical, Case Presentation, Data Analysis and Factual Knowledge, all using the same 9-point rating scale. The 9 points are divided into the three categories of Below Expectations (1-3), Meets Expectations (4-6) and Exceeds Expectations (7-9), and descriptors are provided for each category. Students are rated by faculty and residents and the number of ratings per student varies by clerkship and campus. If a student receives a rating less than 4, the clerkship director is notified and meets with the student. Source of Evidence: Field work, internship, or teaching evaluation Target: No student who receives a rating of less than 4 (i.e., Below Expectations) will receive another rating of less than 4 in a subsequent clerkship. Finding (2012-13) - Target: Met In 2012-13, two students received a rating less than 4, and none of these received a rating less than 4 in a subsequent clerkship. Overall, 6082 ratings were recorded across eight clerkships, and only two of these ratings were less than 4. Finding (2011-12) - Target: Met In 2011-12, four students received a rating less than 4, and none of these received a rating less than 4 in a subsequent clerkship. Overall, 6355 ratings were recorded across eight clerkships, and only four of these ratings were less than 4. Finding (2010-11) - Target: Met In 2010-11, three students received a rating less than 4, and none of these received a rating less than 4 in a subsequent clerkship. Overall, 6463 ratings were recorded across eight clerkships, and only five of these ratings were less than 4. Finding (2009-10) - Target: Met In 2009-10, seven students received a rating less than 4, and none of these received a rating less than 4 in a subsequent clerkship. Overall, 6003 ratings were recorded across eight clerkships, and only nine https://app.weaveonline.com/reports/dar.aspx 4/12
of these ratngs were less than 4. Finding (2008-09) - Target: Met In 2008-09, three students received a rating less than 4, and one of these received a rating less than 4 in a subsequent clerkship. However, this student received a consensus rating of 7 from the clerkship director, so the second 4 was considered an outlier. Overall, 6770 ratings were recorded across eight clerkships, and only four of these ratngs were less than 4. Finding (2007-08) - Target: Met In 2007-08, six students received a rating less than 4, and none of these received a rating less than 4 in a subsequesnt clerkship. Overall, 6236 ratings were recorded across the seven required clerkships, and only seven of these ratngs were less than 4. Note that one of the six students had two ratings of less than 4, but these were in the same clerkship. Related Action Plans (by Established cycle, then alpha): Field-test NBME instrument for assessing professionalism in clinical settings A new instrument for assessing professionalism is being developed by the NBME in conjunction with five medical schools (including UAB). UAB is field-testing the instrument with students, faculty, residents and staff in the BIrmingham Medicine Clerkship. Implementation Status: Finished Measure: Rating of student interactions with patients Outcome/Objective: Inculcation of Patient-Centered Attitudes Implementation Description: The instrument is delivered via E*Value, and participation is voluntarry. IRB approval was obtained for the study. Responsible Person/Group: Medical Stduent Services staff; Office of Undergraduate Medical Education staff Other Outcomes/Objectives, with Any Associations and Related Measures, Targets, Findings, and Action Plans O/O 4: Assurance of Quality Instruction As an accredited medical school, the School of Medicine requires that faculty be appropriately trained for the area in which they teach and that their teaching performance be monitored and assessed routinely. Related Measures M 4: Instructional evaluation questionnaires Instructional performance by lecturers and small group preceptors in preclerkship modules and by clinical educators (faculty and residents) in Junior clerkships is evaluated by students using on-line evaluation questionnaires. The questionnaire for lecturers in pre-clerkship modules contains six items rated on a 5-point scale where 1=Unsatisfactory, 2=Occasionally Fails, 3=Meets https://app.weaveonline.com/reports/dar.aspx 5/12
Expectations, 4=Exceeds Expectations, and 5=Far Exceeds Expectations; four experimental items in interactive instruction rated Level 0 to Level 4 with specific descriptors for the Levels on each item; and an Overall Teaching Effectiveness item rated 9-point scale where 1-3=Needs Improvement, 4-6=Satisfactory and 7-9=Excellent. The questionnaire for small group preceptors in pre-clerkship modules contains 10 items, each rated on a 9-point scale where 1-3=Needs Improvement, 4-6=Satisfactory and 7-9=Excellent. The questionnaire for clinical educators in Junior clerkships contains 18 items, each rated on a 9-point scale where 1-3=Needs Improvement, 4-6=Satisfactory and 7-9=Excellent. The questionnaires for small group preceptors and clinical educators also contain an item for Overall Teaching Effectiveness on the 9-point scale. Source of Evidence: Client satisfaction survey (student, faculty) Target: Ninety percent of lecturers in each preclerkship module, 90% of small group preceptors in each preclerkship module and 90% of clinical educators in each clerkship will receive a mean of at least 5.0 on the Overall Teaching Effectiveness item. Finding (2012-13) - Target: Met For all pre-clerkship modules combined, 95.5% of lecturers received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all modules exceeded 90%, ranging from 91.3% to 100%. For the preclerkship modules combined, 96.3% of small group preceptors received a mean Overall Teaching Effectiveness rating of 5.0 or above, and 9 of 10 modules exceeded 90%, ranging from 69.2% to 100%. For all clerkships combined, 99.5% of clinical educators received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all clerkships exceeded 90%, ranging from 97.6% to 100%. Finding (2011-12) - Target: Met For all pre-clerkship modules combined, 97.1% of lecturers received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all modules exceeded 90%, ranging from 93.0% to 100%. For the preclerkship modules combined, 96.8% of small group preceptors received a mean Overall Teaching Effectiveness rating of 5.0 or above, and 11 of 12 modules exceeded 90%, ranging from 70.0% to 100%. For all clerkships combined, 99.6% of clinical educators received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all clerkships exceeded 90%, ranging from 99.0% to 100%. Finding (2010-11) - Target: Met For all pre-clerkship modules combined, 97.8% of lecturers received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all modules exceeded 90%, ranging from 93.3% to 100%. For the preclerkship modules combined, 95.3% of small group preceptors received a mean Overall Teaching Effectiveness rating of 5.0 or above, and seven of 10 modules exceeded 90%, ranging from 81.8% to 100%. For all clerkships combined, 99.7% of clinical educators received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all clerkships exceeded 90%, ranging from 97.9% to 100%. Finding (2009-10) - Target: Met For all pre-clerkship modules combined, 97.7% of lecturers received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all modules exceeded 90%, ranging from92.5% to 100%. For the preclerkship modules combined, 99.0% of small group preceptors received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all modules exceeded 90%, ranging from 94.7% to 100%. For all clerkships https://app.weaveonline.com/reports/dar.aspx 6/12
combined, 99.8% of clinical educators received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all clerkships exceeded 90%, ranging from 99.2% to 100%. Finding (2008-09) - Target: Met For all pre-clerkship modules combined, 93.9% of lecturers received a mean Overall rating of 3.0 or above. For the 12 modules, 10 exceeded 90%, and the two that didn't had 88.0% and 89.7% with a mean of 3.0 or greater. For the pre-clerkship modules combined, 97.5% of small group preceptors received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all modules but one (89.5%) exceeded 90%, ranging from 93.8% to 100%. For all clerkships combined, 99.5% of clinical educators received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all clerkships exceeded 90%, ranging from 98.1% to 100%. Finding (2007-08) - Target: Partially Met For all pre-clerkship modules combined, 89.8% of lecturers received a mean Overall rating of 3.0 or above. For the eight modules requiring lecturer evaluations, five exceeded 90%, and the other three ranged from 79.2% to 86.5%. For all pre-clerkship modules combined, 98.2% of small group preceptors received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all modules exceeded 90%, ranging from 94.4% to 100%. For all clerkships combined, 99.3% of clinical educators received a mean Overall Teaching Effectiveness rating of 5.0 or above, and all clerkships exceeded 90%, ranging from 95.2% to 100%. Related Action Plans (by Established cycle, then alpha): Add new items to lecturer evaluation form to identify candidates for faculty development A new evaluation form based on a rubric for teaching in the new curriculum will be used. Faculty development will be provided for lecturers not meeting expectations. Established in Cycle: 2007-08 Measure: Instructional evaluation questionnaires Outcome/Objective: Assurance of Quality Instruction Implementation Description: Four additional items will be added to the lecturer evaluation form. Module directors will review evaluation results and refer selected lecturers for faculty devlopment. Responsible Person/Group: Office of Medical Education staff O/O 5: Preparation for Entrance into Residency Program The School of Medicine will graduate students who are prepared to enter residency programs in a wide range of specialties and at a wide range of residency training institutions. Related Measures M 5: Residency Match results The National Residency Matching Program (NRMP) is a nationwide program that conducts the "Match" between medical school graduates and residency https://app.weaveonline.com/reports/dar.aspx 7/12
programs. Students who do not match can contact residency programs with unfilled positions through a post-match process called the Supplemental Offer and Acceptance Program (SOAP). The NRMP provides the School with Match results for our students and the School assists students who did not match with finding a position. Source of Evidence: Job placement data, esp. for career/tech areas Target: All of our graduates will be placed in accredited residency programs and the percentage of our graduates matching through the NRMP will meet or exceed the national percentage. Finding (2012-13) - Target: Met For 2012-13, all of our graduates were placed in an accredited residency program and 96.9% of our graduates matched through the NRMP, exceeding the national percentage of 93.7%. Finding (2011-12) - Target: Partially Met For 2011-12, all but one of our graduates were placed in an accredited residency program and 95.3% of our graduates matched through the NRMP, exceeding the national percentage of 95.1%. Finding (2010-11) - Target: Met For 2010-11, all of our graduates were placed in an accredited residency program and 94.7% of our graduates matched through the NRMP, exceeding the national percentage of 94.1%. Finding (2009-10) - Target: Met For 2009-10, all of our graduates were placed in an accredited residency program and 98.7% of our graduates matched through the NRMP, exceeding the national percentage of 93.3%. Finding (2008-09) - Target: Partially Met For 2008-09, all of our graduates were placed in an accredited residency program and 92.3% of our graduates matched through the NRMP, which was slightly below the national percentage of 93.1%. Finding (2007-08) - Target: Met For 2007-08, all of our graduates were placed in an accredited residency program and 95.3% of our graduates matched through the NRMP, exceeding the national percentage of 94.2%. Related Action Plans (by Established cycle, then alpha): Survey students to determine how to improve Match advising Survey students to determine how to improve Match advising Implementation Status: Finished Measure: Residency Match results Outcome/Objective: Preparation for Entrance into Residency Program Responsible Person/Group: Medical Student Services staff; Office of Undergarduate Medical Education staff O/O 6: Assurance of Student Success To the extent possible, students' academic success will be assured as a function of https://app.weaveonline.com/reports/dar.aspx 8/12
the School of Medicine's comprehensive educational support program. Related Measures M 6: Participation in academic skills program There is a formal academic skills program that is designed to be individual, confidential, and non-judgmental. Students are referred by course directors or clinical faculty or are self-referred. While some students start their Freshman year by participating in this academic skills program, other students who may or may not have participated at the outset do in fact participate as their needs (performance in courses) dictate throughout the academic year. Source of Evidence: Activity volume Target: All students who are performing in a sub-standard fashion in one or more pre-clerkship modules will be referred to an academic advisor for assistance. Finding (2012-13) - Target: Met In 2012-13, all students performing in a sub-standard fashion were referred. Approximately 99% of eligible students came for one or more academic counseling sessions, and those students were monitored at least through the next two exams. Finding (2011-12) - Target: Met In 2011-12, 100% of eligible students came for one or more academic counseling sessions, and those students were monitored at least through the next two exams. Finding (2010-11) - Target: Met In 2010-11, 100% of eligible students came for one or more academic counseling sessions, and those students were monitored at least through the next two exams. Finding (2009-10) - Target: Met In 2009-10, 100% of eligible students came for one or more academic counseling sessions. In addition, those students were monitored at least through the next two exams. Finding (2008-09) - Target: Met In 2008-09, 100% of eligible students came for one or more academic counseling sessions. In addition, those students were monitored at least through the next two exams. Finding (2007-08) - Target: Met In 2007-08, 100% of eligible students came for one or more academic counseling sessions. In addition, those students were monitored at least through the next two exams. Related Action Plans (by Established cycle, then alpha): Use Learning Style Inventory to enhance academic advising Use Learning Style Inventory to enhance academic advising Measure: Participation in academic skills program Outcome/Objective: Assurance of Student Success Implementation Description: Research is underway to investigate the relationship between scales of the Kolb Learning https://app.weaveonline.com/reports/dar.aspx 9/12
Style Inventory and medical school performance. These results will be used to enhance academic advising. Responsible Person/Group: Medical Student Services staff; Office of Undergraduate Medical Education staff Details of Action Plans for This Cycle (by Established cycle, then alpha) Add new items to lecturer evaluation form to identify candidates for faculty development A new evaluation form based on a rubric for teaching in the new curriculum will be used. Faculty development will be provided for lecturers not meeting expectations. Established in Cycle: 2007-08 Measure: Instructional evaluation questionnaires Outcome/Objective: Assurance of Quality Instruction Implementation Description: Four additional items will be added to the lecturer evaluation form. Module directors will review evaluation results and refer selected lecturers for faculty devlopment. Responsible Person/Group: Office of Medical Education staff Identify Step 1 content gaps and unwanted redundancies in preclerkship curriculum For the new integrated curriculum, beginning in 2007-08, pre-clerkship module learning activities and exam questions are being related to the Step 1 Content Outline, to identify content gaps and unwanted redundancies. Established in Cycle: 2007-08 Measure: Performance on USMLE Step 1 for the first time Outcome/Objective: Acquisition of Requisite Basic Science Knowledge Implementation Description: Module directors will use Step 1 content outline to classify exam items and module activities to identify gaps and unwanted redundancies in the curriculum. Responsible Person/Group: Pre-clerkship module directors; Office of Undergraduate Medical Education staff Add more NBME-type items to preclerkship module exams Preclerkship module exams contain too many items that do not meet standards described by the NBME for Step 1 items. All exam items will be reviewed and module directors will be assisted in re-writing items. https://app.weaveonline.com/reports/dar.aspx 10/12
Measure: Performance on USMLE Step 1 for the first time Outcome/Objective: Acquisition of Requisite Basic Science Knowledge Implementation Description: Staff in the Office of Undergraduate Medical Education will review exam items and assist module directors in re-writing items as needed. Responsible Person/Group: Preclerkship module directors and faculty; Office of Undergraduate Medical Education staff. Field-test NBME instrument for assessing professionalism in clinical settings A new instrument for assessing professionalism is being developed by the NBME in conjunction with five medical schools (including UAB). UAB is field-testing the instrument with students, faculty, residents and staff in the BIrmingham Medicine Clerkship. Implementation Status: Finished Measure: Rating of student interactions with patients Outcome/Objective: Inculcation of Patient-Centered Attitudes Implementation Description: The instrument is delivered via E*Value, and participation is voluntarry. IRB approval was obtained for the study. Responsible Person/Group: Medical Stduent Services staff; Office of Undergraduate Medical Education staff Identify Step 2 content gaps and unwanted redundancies in clerkship curriculum Clerkship directors will classify curriculm activities based on Step 2 content areas to identify gaps and unwanted redundancies. Implementation Status: Planned Measure: Performance on USMLE Step 2 for the first time Outcome/Objective: Acquisition of Requisite Clinical Knowledge&Skills Implementation Description: The classification of activities based on the Step 2 content outline will be part of the new clerkship review protocol being planned by the Clinical Medical Education Committee. Responsible Person/Group: Clerkship directors; Medical Education Committee; Office of Undergraduate Medical Education staff Survey students to determine how to improve Match advising Survey students to determine how to improve Match advising https://app.weaveonline.com/reports/dar.aspx 11/12
Implementation Status: Finished Measure: Residency Match results Outcome/Objective: Preparation for Entrance into Residency Program Responsible Person/Group: Medical Student Services staff; Office of Undergarduate Medical Education staff Use Learning Style Inventory to enhance academic advising Use Learning Style Inventory to enhance academic advising Measure: Participation in academic skills program Outcome/Objective: Assurance of Student Success Implementation Description: Research is underway to investigate the relationship between scales of the Kolb Learning Style Inventory and medical school performance. These results will be used to enhance academic advising. Responsible Person/Group: Medical Student Services staff; Office of Undergraduate Medical Education staff Continue to monitor Step 1 content and NBME-type exam items in preclerkship modules For the new integrated curriculum, beginning in 2007-08, pre-clerkship module learning activities and exam questions are being related to the Step 1 Content Outline, to identify content gaps and unwanted redundancies. Preclerkship module exams contain too many items that do not meet standards described by the NBME for Step 1 items. All exam items will be reviewed and module directors will be assisted in re-writing items. Established in Cycle: 2010-11 Measure: Performance on USMLE Step 1 for the first time Outcome/Objective: Acquisition of Requisite Basic Science Knowledge Implementation Description: Module directors will use Step 1 content outline to classify exam items and module activities to identify gaps and unwanted redundancies in the curriculum. Staff in the Office of Undergraduate Medical Education will review exam items and assist module directors in re-writing items as needed. Responsible Person/Group: Preclerkship module directors and faculty; Office of Undergraduate Medical Education staff. Additional Resources: None https://app.weaveonline.com/reports/dar.aspx 12/12