ACGME Data Systems Update How will the systems change in NAS?

ACGME Data Systems Update 2011-2012 How will the systems change in NAS? Rebecca Miller, MS SVP, Applications Development and Data Analysis

Session Overview Briefly Review NAS & New Policies Discuss the ACGME Accreditation Data Systems - Major Features and Changes Specifically Discuss: General Annual ADS Reporting new UI Faculty Survey Resident Survey Milestones Resident Competency Evaluation Resident Case Log System (RCLS)

Transitioning to the Next Accreditation System - Goals Begin to the realize of the promise of Outcomes Free good programs to innovate Assist poor programs to improve Reduce the burden of accreditation Provide accountability for outcomes to the Public

The Next Accreditation System Focused on continuous accreditation and improvement Increased emphasis on annually submitted data Enhanced oversight to ensure high quality education and a safe and effective learning environment RRCs assess program performance each year High performing programs free to innovate Programs with identified problems require in-depth look RRC decisions more timely and tailored

Accredited Residency Programs

Levels of Accreditation Initial Accreditation New Programs Accreditation with Warning Accredited Programs on Probation or Warning Probationary Accreditation Maintenance of Accreditation Accredited Programs without Major Concerns Continued Accreditation STANDARDS Structure Resources Core Process Detailed Process Outcomes Structure Resources Core Process Detailed Process Outcomes Structure Resources Core Process Detailed Process Outcomes Structure Resources Core Process Detailed Process Outcomes Withhold Accreditation Withdrawal of Accreditation

The Building Blocks of The Next Accreditation System Program Self Study Visits 10 years Institutional and Program Self Study Site Visits as Needed Continuous RRC Oversight and Accreditation Core Program Oversight of Subs Sponsor Oversight of All CLER Visits every 18-24 months

Next Accreditation System Rollout The Next Accreditation System begins 7/2013 for 7 specialties with subs Pediatrics Internal Medicine Diagnostic Radiology Emergency Medicine Orthopedic Surgery Neurological Surgery Urology The remaining specialties/rrcs begin using the Next Accreditation System 7/2014 ALL programs will adhere to the new policies effective 7/2013

Continuous Data Collection (Focus on Existing) Annual ADS Update Resident and Faculty Information Major Changes Citation Response Program Characteristics Structure and Resources Scholarly Activity data driven - New Block Diagram - New Board Pass Rate Data (external) Resident Clinical Experience Resident Survey Faculty Survey - New Semi-Annual Resident Evaluation Milestone Reporting - New Reporting Only )

Data Reporting Timeframe ADS Annual Update July- Sept Milestone Reporting May Resident Case Log Reporting July-June Faculty Survey Jan-May Milestone Reporting Dec Resident Survey Jan- May

Data Review Timeframe Data collected throughout the year Data analytically compiled late Summer Annual review of information by RRC teams Fall Pre-meeting actions taken if needed Accreditation Decisions made at Winter/Spring RRC Meetings

Pre-meeting Actions Focused Site Visit Full Site Visit Request Clarification or Progress Report Send Material out to RRC for Review Highlighting the Problem(s) for Peer Decision No Pre-meeting action Continued Accreditation )

Accreditation Decisions - After Initial Accreditation Continued Accreditation Continued Accreditation with warning Probationary Accreditation (limit 2 years) Requires Site Visit Withdrawal of Accreditation Requires Site Visit Recommend Administrative Withdrawal Voluntary Withdrawal of Accreditation OR Further Postpone Accreditation Decision )

Areas of Concern Not Citations Annual Data submission may have raised an issue that would be noted in the Annual Notification Letter Does not have to be linked to specific Program Requirement Concerns do not require response unless specifically requested Slate clean based on each annual submissions Not the same as level as Citations )

Citations Can be issued without a site visit Must be linked to standards Will be reviewed annually when program is on probation or accreditation with warning reconfirmed vs. resolved Must be reviewed as part of any site visit Issued prior to July 1, 2013 (phase 1), removed if program is issued continued accreditation for 2 years with no concerns Newly issued citations will need to be reviewed working on details )

Multi-disciplinary Subs New Policy Multiple Residency Review Committee Model PR may and may not be the same Each RRC reviews the subspecialties sponsored by it specialty Co-sponsored Multiple RRC One set of PR Each RRC reviews the subspecialties sponsored by it specialty RRCs interested in joining notifies ACGME Board Note: New applications may require review of subspecialty expertise )

No Independent Subs New Policy Effective July 1, 2013 all new fellowships must be accredited as a dependent sub Effective July 1, 2015 all existing independent subs must: Operate under the oversight of an accredited sponsoring institution Associate with a core residency program under the oversight of an accredited sponsoring institution )

Getting Back to the Data Systems )

Data Systems Integration- Background Case Log Data Collection Accreditation Activity System Survey Data ADS Reporting System External Data: Board Pass Rate )

Data Systems Revised and Integrated Created an enhanced human-computer interface More intuitive Better flow Reducing need for support Adheres to web standards A common record for each resident linked to all previous ACGME training Programs have ability to verify prior training for newly-entered residents

System Access Requirements ACGME Data Systems can be accessed anywhere with an internet connection via a computer using Microsoft Windows or Apple Mac operating systems ACGME supports IE 8 and 9, Mozilla Firefox, Google Chrome, and Apple Safari. ACGME no longer supports IE 7!! Add www.acgme.org as acceptable site

Easy Access Login Login links for Accreditation Data System (ADS), ACGME Resident Survey, and Resident Case Log System located on ACGME homepage ADS and RCLS for programs accessible through single common login screen; sharing the same resident, faculty, and rotation/institution information )

Secured Login Access

Overview Page

What are ADS Required Tasks and How Have they changed? Initial Application Completion Annual ADS Update Changes (minor and significant) Faculty Survey Administration Resident Survey Administration Resident Competency Evaluation Resident Case Log System

Program Initial Accreditation Application Process Electronic Process July 1, 2013 Initiated by DIO Program Director sent User Name and Password for further completion Two sections: Common and Specialty Specific Requires DIO sign-off Locked after submission Must have a site visit within 2 years of initial accreditation effective date Within 2 years of Withheld, may reapply only to same RC

What are ADS Required Tasks? Initial Application Completion Annual ADS Update Changes (minor and significant) Faculty Survey Administration Resident Survey Administration Resident Competency Evaluation Resident Case Log System

Annual Data Collection Changes Transition to more Quantitative Measures No PIFs/ PIFmanship Reduce Describe the Reduce Does the program comply.. No Faculty CVs except for new applications Quantify scholarly activity Measure significant changes Begin to assess residents attainment of milestones Begin Resident Registration

DIOs Must Update sponsor institution data annually Add/delete participating sites available for programs Monitor/manage program annual reporting Electronically approve significant changes PD Resident complement changes Voluntary withdrawals New applications

Programs Must Provide annual data Submit changes Update and generate site visit materials as requested Monitor resident and faculty survey participation Report resident milestone data (phase 1) Oversee resident participation in Case Logs Requires PD annual sign-off coming soon

Program Annual Update - Overview

Program Annual Reporting Update Program Data Basic program information PD/Coordinator information Major changes Respond to Citations Participating sites DH and Patient Safety data General Competency Assessment Methods Block Diagram Typical Rotation Schedule

Program Annual Reporting Update Resident Data Add new residents- verify prior training Confirm all active and graduating residents unconfirmed each year Enter scholarly activity (2013) Update Faculty Data Add / remove faculty with credentials Enter scholarly activity (2013)

Adding a New Resident A common record for each resident linked to all previous ACGME training Programs have ability to verify prior training for newly-entered residents Must have accurate email address New residents (RCLS) will be emailed ACGME user ID and password

Confirm Existing Residents

Resident Status Newly Added Residents Active Full Time Active Part Time (counted as 0.5) Started Program Off Cycle (automatically chosen depending on start dates) Completed Training Completed All Accredited Training (for this specialty) and prepared for independent practice

Resident Status continued Inactive Residents In Program but Doing Research/Other Training (intends to resume accredited training in this program) Not in Program Yet and/or Doing Preliminary Year Elsewhere Leave of Absence Left Program Completed all training but NOT PREPARED for independent practice Withdrew from Program Transferred to Another Program (prior to completing required training) Dismissed Deceased

Changing Historical Data Accuracy is Important Critical for Surveys, Case Logs, Scholarly Activity and Milestone Reporting Changes can occur only during current academic year Final at the end of the academic year

Active Residents Have access to Case Log System Participate in Resident Survey Annually Require the input of Scholarly Activity Require 2 summative evaluations per year Need confirmation every year

Resident Scholarly Activity Based on previous academic year activity An entry for each resident/fellow who was last year Active full time or part time In program doing research/other training Completed all training Excludes preliminary trainees and active Year 1 residents (no previous record) IM and PEDs exceptions

Resident Scholarly Activity Publications recognized by the National Library of Medicine - Pub Med IDs up to 3 Number of abstracts, posters, and presentations given at international, national, or regional meetings Number of chapters or textbooks published Participated in funded or non-funded basic science or clinical outcome research project Lecture or teaching presentation (such as grand rounds or case presentations) of at least 30 minutes duration at the sponsoring institution or program

Active Faculty Must have sufficient number of faculty (including core) with documented qualifications Ratio of teaching faculty to residents has a significant influence on the performance of a program Types of Faculty Non-Physician Faculty Attending Physicians case logs Faculty all Core Faculty

Active Faculty There must be a sufficient number of (physician and nonphysician) faculty with documented qualifications to instruct and supervise all residents for the program. A portion of the faculty must be core physician faculty. All physicians who devote at least 15 hours per week to resident and administration are designated as core faculty: All core physician faculty should: teach and advise residents evaluate the competency domains work closely with and support the program director assist in developing and implementing evaluation systems PD is not considered core

Active Physician Faculty Credentials Degree Primary and Secondary Specialty Certification Medical School Primary Institution location Date of first appointment Time teaching in specialty Hours spent devoted to residency program Clinical supervision of residents Administration of the program Research/scholarly activity with residents Didactics/teaching with residents

Faculty Scholarly Activity Based on previous academic year activity An entry for each core faculty active in the program Replaces core faculty CVs (after initial accreditation achieved)

Faculty Scholarly Activity Publications recognized by the National Library of Medicine - Pub Med IDs up to 4 Number of abstracts, posters, and presentations given at international, national, or regional meetings Number of presentation given (grand rounds, invited professorships), materials developed or other work presented in non-peer review publications include peer reviewed by not recognized by NLM Number of chapters or textbooks published

Faculty Scholarly Activity Continued Number of grants for which faculty member had leadership role (PI, Co-PI, or site director) Had an active leadership role (such as serving on committees or governing boards) in national medical organizations or served as reviewer or editorial board member for a peer-reviewed journal (Y/N) Held teaching responsibility for seminar, conference series, or course coordination (such as arrangement of materials, assessment of participants performance) for any didactic training within the sponsoring institution or program. Not a single presentation or lecture.

Make Changes in ADS Immediately All data should be current (resident, faculty, and program level) Major changes require sign-off and approval (DIO & RRC Approved resident complement PD initiates New Program Director DIO initiates Participating site affiliations DIO initiates Request voluntary withdrawal - PD initiates Major structural changes Citation responses annual responses stored RRCs review changes

Faculty Survey- Background The faculty survey seeks input regarding the overall educational environment and compliance with the standards Implemented for Core faculty & PD in accredited specialty programs phase 1 in 2013 Implemented for all faculty in subspecialties - with rollout exception Email addresses source for contact accuracy important

Faculty Survey Content Questions focusing on residents and overall program similar to Resident Survey Faculty Supervision / teaching Educational content Resources Patient Safety Teamwork AND Program overall assessment question

Faculty Survey - Administration Administered annually Jan- May (5 weeks) Managed at the program level monitor respondents Core faculty assigned username (program ID) and password (last name first initial) All data are maintained anonymously and confidentially Aggregate reports available if 3 respondents and 60% response rate Areas of deficiency should be noted and addressed

Resident Survey - Background Monitors resident clinical education and provides early warning of potential non- compliance Assesses the resident perspective Aligned with Faculty Survey Anonymous and confidential Includes some specialty specific questions (n=12)

Resident Survey- Additional Questions Emergency Medicine Pathology Family Medicine Internal Medicine Ophthalmology Radiation Oncology Transitional Year Preventive Medicine Allergy and Immunology Internal Medicine/Pediatrics Anesthesiology Pediatrics

Resident Survey Content Not a single, unique survey Using a bank of questions that differ depending on responses and level of training Focus on general content areas Duty hours Resources Faculty supervision/teaching Evaluation Educational content Patient safety Teamwork AND Program overall assessment question

Resident Survey Level of Training The first question asks for level of training: Beginning, Intermediate, and Advanced. Available responses based on specialty requirements.

Resident Survey- Administration Administered annually January-May (5 weeks) Includes all residents in ALL accredited programs Includes residents in combined specialty tracks 2013 Requires 70% compliance Managed by your Program Requires accurate resident information in ADS

Resident Survey- Administration Program provides residents login information and instructions- NOT ACGME Login ID = program ID Password = DOB plus last 2 digits of last name (mmddyyyyln) Monitor resident completion rate Monitor list of non-responders Unable to update data after the 5 week period ends Reminders sent to program

Resident Survey- Results After completion (70%) PDs and DIOs see aggregate resident reports - programs with 4 or more residents Reports for the smaller programs combine data across 4 years. Requires: At least 3 out of 4 years of data At least 4 respondents during 4 year period 70% compliance each year RRCs follow-up with programs that appear to be in substantial non-compliance

Resident Competency Evaluation Summative Evaluation for all Residents in the 6 competencies Dec 1 and May 15 Phase 1 Core specialties in 2013-2014 Clinical Competency Committee should have reached consensus on residents reported level of experience Summarized data will be part of the information considered during the accreditation process

Resident Competency Evaluation Reporting Form Each specialty has unique Milestones Moves from Numbers to Narratives Numerical systems produce range restriction Narratives are easily discerned by faculty and have shown to produce data without range restriction

Professionalism Resident seeks out opportunities to demonstrate compassion and empathy in the care of all patients; and demonstrates respect and is sensitive to the needs and concerns of all patients, family members, and members of the health care team. Resident demonstrates compassion and empathy in care of some patients, but lacks the skills to apply them in more complex clinical situations or settings. Occasionally requires guidance in how to show respect for patients, family members, or other members of the health care team.

Resident Competency Evaluation Data and Feedback Summarized data will be part of the information considered during the accreditation process - looking for patterns Reports will be provided to programs displaying overall performance by cohort Narrative reports will be provided to programs for formal feedback

Resident Case Log System Web-based application developed in 2000 for residents to track clinical experiences Still in Data Migration Phase 2 Procedures and cases grouped into categories created by Review Committees Review Committees establish key indicators and minimum expectations - critical in NAS Review Committees assess program performance and assess residents ability to meet the minimums for each key indicator

Case Log Statistics 15,000,000 10,000,000 5,000,000 0 Number of procedures logged per year

Specialties Using Resident Case Logs Required Optional Surgery Vascular Surgery Orthopaedic Surgery Plastic Surgery Urology Ophthalmology Radiation Oncology Allergy and Immunology Dermatology During 2011-2012 Obstetrics and Gynecology Neurological Surgery Anesthesiology Otolaryngology Diagnostic Radiology Colon and Rectal Surgery Pathology PM&R Medical Genetics Nuclear Medicine (7/1) Thoracic Surgery Med/Peds Pediatrics Emergency Medicine Neurology 13.3 million procedures were entered 2,873 programs used the Resident Case Log System 44,361 residents accessed and logged entries

Subspecialties Using Resident Case Logs Required Pediatric Anesthesiology Procedural Dermatology Ophthalmic Plastic and Reconstructive Surgery Orthopaedic Surgery Subs Neurotology Pediatric Otolaryngology Craniofacial Surgery Plastic Hand Surgery Surgery (Peds, Critical Care, Vascular) Pediatric Urology Cardiovascular Disease Neurodevelopmental Disabilities Child Neurology Blood Banking/Transfusion Chemical Pathology Cytopathology Forensic Pathology Hematology Medical Microbiology Neuropathology 77.4% of all specialties have logs representing 56% of specialty programs Optional Pediatric Pathology Pediatric Hematology/Oncology

Case Logs Mapping Structure

Specialties without Case Logs Need to measure Residents Clinical experience

Accessing Case Logs Programs have logins to use Case Log System (same logins used to access ADS) Residents automatically assigned unique logins and passwords as they are added to the program rosters in ADS and notified via email. No user set up or maintenance required by program less burden

Case Log Data Flow Residents enter cases Assess set minimums Programs monitor progress by utilizing reports National Data Reports Created in ADS Residents graduate Programs verify completing graduate data accuracy & electronically submit to ACGME

Access Agreement

Case Log Development Mobile Website: acgme.org/mobilercl

Case log Development - continued

Frequently Asked 1. How often do the residents need to log their data? 2. Do residents have the ability to enter data from the past? 3. Can the Program Director Log cases or at least have access to the logging mechanism? 4. What happens if a resident transfers to a different program?

Resident Level Data Residents should know the data are used for accreditation purposes with other benefits: Secure record of cases with export feature Data is widely used for privileges and various posttraining positions Depending on specialty data may be used for Boards (data access agreement must be submitted electronically by resident)

Data Download

ACGME Data Systems Assistance WebADS@acgme.org OpLog@acgme.org Ressurvey@acgme.org FacSurvey@acgme.org

Technical Support Walk-in Technical Support for ACGME Data Systems 9:30 5:00 pm on Friday 8:30 5:00 pm on Saturday

Questions?