Heidi Kromrei, PhD AVP Academic Affairs, ADIO Detroit Medical Center Presented a t the SEMCME New Program Director Workshop Annual Program Evaluation: Data-Driven Evaluation Practices
Session Goals Recall the ACGME Requirements for Annual Program Evaluation (APE) and Program Evaluation Committee (PEC) Understand the role of the APE in the ACGME Next Accreditation System Identify Strategies to Develop a Successful Annual Program Evaluation Protocol
Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture
APE: Continuous Evaluation and Improvement Source ACGME Webinar @: http://acgme.org/acgmeweb/portals/0/pfassets/nov4nasimpphaseii.pdf
ACGME Self-Study Model APE is important!!
ACGME Self-Study Model ACGME RRCs Evaluate Programs Annually!!
ACGME Self-Study Model
ACGME Residency Review Committee Annual Review Data RRC will evaluate this data every year. Will you?
Annual Program Evaluation ACGME Requirements for APE and PEC
ACGME PEC Membership Requirements: Program Director must appoint PEC At least 2 faculty members At least 1 resident Must have a written description of responsibilities
The PEC Responsibilities The Program must monitor & track: 1. Resident Performance 2. Faculty Development 3. Graduate Performance 4. Program Quality The Program must ensure that. Residents and Faculty have the opportunity to evaluation the program confidentially and in writing at least annually The Program must use the results of residents and faculty members assessments along with other program evaluation results to improve the program
Activity: Define Your Current APE Practices Who? What? When? How?
Annual Program Evaluation Design ACGME Mandates Performance Areas: 1. Resident Performance 2. Graduate Performance 3. Faculty Development 4. Program Quality Questions to be answered: 1. What are our strengths, weaknesses, opportunities and threats in these 4 areas? 2. What is working? 3. What is not working? 4. Are we meeting our AIMS? 5. What should we do to improve?
The APE Meeting APE Meeting Planning
APE Agenda 1. Attendees, Meeting Date & Time 2. Previous Year s Action Plan Progress Report, Review of Program Aims 3. SWOT Analyses of Resident Performance, Graduate Performance, Faculty Development & Program Quality 4. Development of Action Plan for Program Improvement for upcoming academic year
DMC PEC Responsibilities Template
Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture
How will you measure success? What data will you use?
ACGME AIMS Example
Data Collection & Analyses How is data used in the APE?
Data-Driven Evaluation Why is it important? How is it done? How does it relate to APE?
Resident Performance Data Sources 1. End of rotation evaluations 2. In-Service and other exams 3. Simulation Results 4. Milestone Review 5. Case Logs/Clinical Volume Summary Report 6. Scholarly Activity
Graduate Performance Data Sources 1. Board Certification Exam Report (5-year Summary) 2. Graduate Satisfaction Survey 3. Graduate Employment Report 4. Graduate Employer Survey
Faculty Development Data Sources 1. Faculty Scholarly Activity Report 2. Faculty Clinical, Educational, Evaluation Development Activities 3. Timeliness of Evaluation Completion Report
Program Quality Data Sources 1. Accreditation Status and Citations 2. Block Rotation Schedule 3. Survey Reports (ACGME Resident/Faculty; Program Surveys) 4. Duty Hour Audit Reports 5. Program Curriculum/Milestone Review
APE Data Planning Performance Domain Resident Performance Graduate Performance Data Source/Docum ents In-Service Exam Report Board Pass Rate Report Data Analysis Plan Assigned To Other Data Needed Trend Analysis APD Rotation Quiz Trend Analysis PD Alumni Survey Faculty Development Faculty CVs Descriptive (ed/eval training) Program Coordinator Scholarly Activity (pub, presentations, etc.) Program Quality ACGME RRC Letter Descriptive (status, citations, AFI, trends) Faculty Program Specific Annual Resident/Facul ty Surveys
SWOT Analyses and APE Reporting Conducting SWOT Analyses and the APE Report
Conduct SWOT Analyses Strengths of the Program What should definitely be continued? Weaknesses of the Program What aspects of the program should be improved? Prioritized by relevance to program aims, importance to stakeholders, compliance with ACGME requirements Opportunities in the Program Environment What are opportunities that favor the program? New areas for the curriculum? Changes to enhance program performance? Threats in the Program Environment What are the threats to the program? Factors that threaten program success and sustainability Local or national
Document Findings: The APE Report
Document Findings: The APE Report SWOT Analyses Results: 1. Resident Performance 2. Graduate Performance 3. Faculty Development 4. Program Quality Weaknesses/Threats: 1. Action Plan for Program Improvement
Document Findings: The APE Action Plan for Program Improvement
Communicate APE Findings 1. Prepare Final APE Report 2. Submit for Faculty Approval 3. Share with Residents 4. Submit to GMEC 5. Track and provide quarterly progress report to all stakeholders
Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture
ACGME Annual Update Report Major Changes Section: 1. Opportunity to acknowledge deficits to ACGME before a citation issued (i.e., case log report, ACGME surveys, Board Pass Rate, etc.) 1. Describe APE action plan items that address deficits 1. Describe improved performance 1. Let the RRC know you are working on improving areas of noncompliance
Program Aims Annual Program Evaluation ACGME Annual Update ACGME Self-Study The BIG Picture
Annual Program Evaluation and Self-Study Process Provide a brief description of the process for the Annual Program Evaluation, including action plan tracking, and the self-study process for this program. Question 6: Annual Program Evaluation Process Describe the Annual Program Evaluation. How is information from the Annual Program Evaluation aggregated? How are action plans tracked? What follow-up occurs? (Maximum 250 words) ACGME Self-Study Summary
APE Action Plan and the Self-Study
Questions? hkromrei@dmc.org
ACGME website: http://www.acgme.org References ACGME Common Program Requirements: http://www.acgme.org/portals/0/pfassets/programrequirements/cprs_07012016. pdf ACGME Self-Study Page: http://www.acgme.org/what-we-do/accreditation/self- Study-and-Site-Visit