Annual Program Evaluation: Data-Driven Evaluation Practices

Transcription

1 Heidi Kromrei, PhD AVP Academic Affairs, ADIO Detroit Medical Center Presented a t the SEMCME New Program Director Workshop Annual Program Evaluation: Data-Driven Evaluation Practices

2 Session Goals Recall the ACGME Requirements for Annual Program Evaluation (APE) and Program Evaluation Committee (PEC) Understand the role of the APE in the ACGME Next Accreditation System Identify Strategies to Develop a Successful Annual Program Evaluation Protocol

3 Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture

4 APE: Continuous Evaluation and Improvement Source ACGME

5 ACGME Self-Study Model APE is important!!

6 ACGME Self-Study Model ACGME RRCs Evaluate Programs Annually!!

7 ACGME Self-Study Model

8 ACGME Residency Review Committee Annual Review Data RRC will evaluate this data every year. Will you?

9 Annual Program Evaluation ACGME Requirements for APE and PEC

10 ACGME PEC Membership Requirements: Program Director must appoint PEC At least 2 faculty members At least 1 resident Must have a written description of responsibilities

11 The PEC Responsibilities The Program must monitor & track: 1. Resident Performance 2. Faculty Development 3. Graduate Performance 4. Program Quality The Program must ensure that. Residents and Faculty have the opportunity to evaluation the program confidentially and in writing at least annually The Program must use the results of residents and faculty members assessments along with other program evaluation results to improve the program

12 Activity: Define Your Current APE Practices Who? What? When? How?

13 Annual Program Evaluation Design ACGME Mandates Performance Areas: 1. Resident Performance 2. Graduate Performance 3. Faculty Development 4. Program Quality Questions to be answered: 1. What are our strengths, weaknesses, opportunities and threats in these 4 areas? 2. What is working? 3. What is not working? 4. Are we meeting our AIMS? 5. What should we do to improve?

14 The APE Meeting APE Meeting Planning

15 APE Agenda 1. Attendees, Meeting Date & Time 2. Previous Year s Action Plan Progress Report, Review of Program Aims 3. SWOT Analyses of Resident Performance, Graduate Performance, Faculty Development & Program Quality 4. Development of Action Plan for Program Improvement for upcoming academic year

16 DMC PEC Responsibilities Template

17 Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture

18 How will you measure success? What data will you use?

19 ACGME AIMS Example

20 Data Collection & Analyses How is data used in the APE?

21 Data-Driven Evaluation Why is it important? How is it done? How does it relate to APE?

22

23 Resident Performance Data Sources 1. End of rotation evaluations 2. In-Service and other exams 3. Simulation Results 4. Milestone Review 5. Case Logs/Clinical Volume Summary Report 6. Scholarly Activity

24 Graduate Performance Data Sources 1. Board Certification Exam Report (5-year Summary) 2. Graduate Satisfaction Survey 3. Graduate Employment Report 4. Graduate Employer Survey

25 Faculty Development Data Sources 1. Faculty Scholarly Activity Report 2. Faculty Clinical, Educational, Evaluation Development Activities 3. Timeliness of Evaluation Completion Report

26 Program Quality Data Sources 1. Accreditation Status and Citations 2. Block Rotation Schedule 3. Survey Reports (ACGME Resident/Faculty; Program Surveys) 4. Duty Hour Audit Reports 5. Program Curriculum/Milestone Review

27 APE Data Planning Performance Domain Resident Performance Graduate Performance Data Source/Docum ents In-Service Exam Report Board Pass Rate Report Data Analysis Plan Assigned To Other Data Needed Trend Analysis APD Rotation Quiz Trend Analysis PD Alumni Survey Faculty Development Faculty CVs Descriptive (ed/eval training) Program Coordinator Scholarly Activity (pub, presentations, etc.) Program Quality ACGME RRC Letter Descriptive (status, citations, AFI, trends) Faculty Program Specific Annual Resident/Facul ty Surveys

28 SWOT Analyses and APE Reporting Conducting SWOT Analyses and the APE Report

29 Conduct SWOT Analyses Strengths of the Program What should definitely be continued? Weaknesses of the Program What aspects of the program should be improved? Prioritized by relevance to program aims, importance to stakeholders, compliance with ACGME requirements Opportunities in the Program Environment What are opportunities that favor the program? New areas for the curriculum? Changes to enhance program performance? Threats in the Program Environment What are the threats to the program? Factors that threaten program success and sustainability Local or national

30 Document Findings: The APE Report

31 Document Findings: The APE Report SWOT Analyses Results: 1. Resident Performance 2. Graduate Performance 3. Faculty Development 4. Program Quality Weaknesses/Threats: 1. Action Plan for Program Improvement

32 Document Findings: The APE Action Plan for Program Improvement

33 Communicate APE Findings 1. Prepare Final APE Report 2. Submit for Faculty Approval 3. Share with Residents 4. Submit to GMEC 5. Track and provide quarterly progress report to all stakeholders

34 Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture

35 ACGME Annual Update Report Major Changes Section: 1. Opportunity to acknowledge deficits to ACGME before a citation issued (i.e., case log report, ACGME surveys, Board Pass Rate, etc.) 1. Describe APE action plan items that address deficits 1. Describe improved performance 1. Let the RRC know you are working on improving areas of noncompliance

36 Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture

37 Annual Program Evaluation and Self-Study Process Provide a brief description of the process for the Annual Program Evaluation, including action plan tracking, and the self-study process for this program. Question 6: Annual Program Evaluation Process Describe the Annual Program Evaluation. How is information from the Annual Program Evaluation aggregated? How are action plans tracked? What follow-up occurs? (Maximum 250 words) ACGME Self-Study Summary

38 APE Action Plan and the Self-Study

39 Questions?

40 ACGME website: References ACGME Common Program Requirements: pdf ACGME Self-Study Page: Study-and-Site-Visit