Improving the Evaluation of Interprofessional Education: Moving beyond Attitudinal Measures

Improving the Evaluation of Interprofessional Education: Moving beyond Attitudinal Measures VIRGINIA COMMONWEALTH UNIVERSITY UNIVERSITY OF CALIFORNIA SAN FRANCISCO OREGON HEALTH & SCIENCE UNIVERSITY

Panel Agenda Virginia Commonwealth University: Use of multisource data to evaluate classroom-based IPE for early learners University of California, San Francisco: Multisource feedback in a longitudinal clinical skills course where medical students are embedded in an interprofessional clinical team Multisource feedback for students and residents who are trained in a patient-centered medical home Oregon Health & Science University: Multisource data for evaluating a large, national study focused on implementing IPE in primary care physician residency continuity clinics Use of Q methodology to evaluate rural IPE

Use of Multisource Data to Evaluate Classroom-based IPE for Early Learners Kelly Lockeman, PhD Virginia Commonwealth University

Background Two large-scale required IPE courses Fall: Foundations of Interprofessional Practice Spring: Interprofessional Quality Improvement and Patient Safety Classroom-based 1 credit each, pass/fail grading ~500 students each semester 88 interprofessional teams 12-16 faculty facilitators

Assessment Challenges Many Students Few Faculty Limited Time Assessing individual students is resource intensive.

Our Approach To Assessment Use student learning objectives as a guide. Measure outcomes of individual students and teams. Build measures into course activities. Use validated measures when feasible. Assess validity for course-specific measures. Utilize measures that are easily scored. Provide formative feedback for students. Iterate!

Typical Assessment Measures Learner Team Knowledge Quizzes Work products Skills/Performance Attitudes Self-assessment Peer assessment Self-report using validated tools (e.g., SPICE-R2) Work products (Aggregate) Self-report using validated tools (e.g., TDM) Narrative reflections

Sample Findings: Individual Learners KNOWLEDGE: Average quiz score 85% (passing = 70%) SKILLS/PERFORMANCE: Peer assessment: Scores depend on rating method Rubric with rating criteria scores highly skewed Budget-based approach normal distribution that clearly identifies low and high performers Self-assessment: Correlates w/ budget-based peer assessment (r =.61, p <.001) ATTITUDES: Significant increase on SPICE-R2 (pre/post)

Sample Findings: Teams KNOWLEDGE: Depiction of collaborative care in team work products suggests poor understanding of IPC SKILLS/PERFORMANCE: High scores suggest they can use teamwork to complete tasks effectively ATTITUDES: Students acknowledge the need for and value IPE. Comments suggest that interprofessional socialization as a primary outcome.

Considerations Correlation between individual measures is low. Knowledge Skills/Performance Attitudes Students (and faculty) don't know they are lousy assessors. Peer assessment requires practice. Interrater reliability between faculty is challenging. Qualitative analysis of narrative reflections is resource intensive.

Conclusions Data from multiple sources useful information for program evaluation, but Keep seeking evidence for validity Review data annually and revise measures as needed Draw from the experience of other institutions Focus on faculty development

Thank you! For more information Visit our website: http://ipe.vcu.edu/ Follow us: Twitter: @VCUCIPE Facebook: https://www.facebook.com/vcuipe Contact me directly: Kelly Lockeman, PhD Director of Evaluation and Assessment VCU Center for Interprofessional Education and Collaborative Care Kelly.Lockeman@vcuhealth.org

An Interprofessional Multi- Source Feedback Tool for Early Learners: Lessons Learned from a Pilot Study Josette Rivera, MD Associate Professor of Medicine Department of Medicine University of California, San Francisco

The UCSF Bridges Curriculum To prepare the 21st century physician to work collaboratively in promoting health and reducing suffering while continually improving our health care system Guiding Principles Integration into Interprofessional Collaborative Care Immersion in Authentic Workplace Learning Engagement with Health Care Delivery Systems Campus-wide 5 quarter IPE curriculum, SOM additional IPE coursework 6/19/2015

The Clinical Microsystem Clerkship (CMC) Who Medical students (150): First 18 months Faculty coaches (25): each assigned 6 students. Teach direct patient care skills, shepherd students through their microsystems projects Staff: members of the healthcare team and other staff in the microsystem What Students embedded in a clinical microsystem one day/week throughout SF Quality improvement, direct patient care skills, interprofessional collaboration Systems-based project in the microsystem

an interprofessional collaboration (IPC) assessment tool for CMC and beyond Opportunity: assess relevant student behaviors in clinical settings Goal: formative assessment at multiple time points, emphasize importance of IPC to students and coaches Considerations: # items, narrative vs. rating scales, #/timing of assessments, # raters, training of raters Tensions: raters in diverse hectic clinical environments, infeasible to train, avoid rating scales causing reluctance to judge Gap: brief, narrative, formative assessment, not setting or specialty specific

The Multi-source Feedback Tool Tool Consider both observations & direct interactions with the student Identify their profession Questions: What does X do well to collaborate with other health professionals? What could X improve to collaborate more effectively with other health professionals? Did you work with this student: On their systems improvement project? To provide patient care? Both? Please estimate the number of days you interacted with this student Process Students identify and request feedback from one non-md in their microsystem Request feedback in fall and spring; not required to have same raters Coaches review feedback with individual students, develop individualized plan/goals

Compliance Fall 2016: 152 students 111 completed (73%) 41 missing Spring 2017: 152 students 106 completed (70%) 46 missing 9/152 (6%) never received feedback -> written reflection on why Projects only involved physicians Setting chaotic with rotating team members, non-overlapping schedules, only brief interactions

Fall 2017 Results 98/111 (88%) raters had 1-8 days of interaction with student Majority positive; only 18 students received constructive feedback desire to understand the job descriptions/responsibilities of each health care professional and is open to seeing how every member of the health care team can contribute never hesitates to reach out to different health care providers for their perspectives or recommendations Good at sharing his ideas and asking for feedback explains his thought processes yet accepts constructive criticism and uses it to improve

Spring 2017 Results 86/106 (81%) 1-8 days of interaction with student; 11/106 (10%) >13 days Many more students (35) received constructive feedback become more vocal and express your ideas and thoughts improve upon finding your own unique professional voice and speaking up more in larger interdisciplinary meetings Don t be afraid to make a mistake. You don't need to be perfect and you don't need to know everything. Keep asking questions and continue to put yourself out there.

Lessons Learned Feasible: optimistic can increase compliance due to lessons learned Useful: thoughtful & fairly specific feedback without rater training Messaging as important as the tool itself Emails to students, coaches, and staff explaining purpose of tool Emphasize formative for student development First iteration in fall too early: insufficient contact with others This year->earlier messaging to new coaches, push timeline back

Implications for Program Evaluation Although an explicit course expectation, some students not interacting with other professionals in their microsystem Qualitative study with student interviews Outside of projects and assignments, IP interactions did not happen spontaneously or organically Students reluctant to disturb busy staff Coach discussions reveal buy-in and guidance needed Interview microsystem stakeholders to determine what/how students contributed to the microsystem

Next Steps Include more structured activities early on to facilitate IP interactions (more team member interviews/shadowing opportunities) Faculty development for current and future coaches Tips for coaches by coaches (ex. select projects that facilitate IP interactions) Workshops on IPE and the hidden curriculum Setting expectations: assist students with compliance, review feedback Expand use of multi-source feedback tool to select clerkships in 3 rd /4 th years Demonstrate trajectory Consistent assessment, continued emphasis on importance of IPC in all 4 years

Questions? Josette.Rivera@ucsf.edu

Multi-level Evaluation of IPE in the Workplace Individuals, Teams, Systems Bridget O Brien, PhD Center for Faculty Educators & Dept of Medicine, University of California, San Francisco Center of Excellence in Primary Care Education, San Francisco VAMC

2 San Francisco VA EdPACT Program The Workplace Primary Care clinics for Veterans IP Team-based model of care 3 clinic sites ~18,000 patients The Educational Program Integrated with the clinics Learners part of clinic teams Curriculum emphasizes communication, relationships, & systems MSA NP Student RN Social Work PACT Patient TEAM LVN Pharm Mental Health IM Residen t

IPE: An Individual and Collective Enterprise EdPACT evaluation must address goals and objectives of Each professional program: individual competence in communication, interpersonal, and teamwork skills VA primary care clinic: collective competence in team-based care

Individual Performance in a Team Evaluation Question Does EdPACT help learners achieve competence in team-based care? Communication, Collaboration, Empowering team members, Situational Awareness Multi-source Feedback Formative assessment to individual learners Aggregate data for program evaluation Instrument Modified version of American Board of Internal Medicine multi-source feedback tool (TEAM) 1 Covers domains such as giving & receiving feedback; respect; empowering others Process All team members rate each learner on the team Learners self-assess using same survey Learners receive a report of their results and review with a faculty member Chesluk BJ et al, 2012 Health Affairs; Chesluk BJ et al, 2015 J Cont Ed Health Prof 2016 Education Showcase

Individual Performance in a Team: Example 2016 Education Showcase

Program Evaluation: Individual Performance in a Team Does EdPACT help learners achieve competence in team-based care? In 2017, 36 learners rated per year by 63 team members (~80% response rate) Learners rate own performance highest in empowering others and relational domains (respect, acknowledging others contributions), lowest in seeking and giving feedback Team members generally rate learners performance higher than learners rate themselves; similar patterns of highest and lowest performance seeking and giving feedback lowest Implications Broadly focus on feedback Customized learning goals for each learner Communication with team Self-awareness 31

Interprofessional Teamwork Evaluation Question To what extent are teams achieving cohesiveness, effective communication, role clarity, clear goals? Team Development Formative feedback for teams to review and use for improvement Program level feedback to address culture, coaching, etc Instrument Team Development Measure 1 Process Administered twice per year Teams review reports and have opportunities to discuss 1 Stock et al, 2013; Survey available at: http://www.peacehealth.org/about-peacehealth/medical-professionals/eugenespringfield-cottage-grove/team-measure/pages/measure 2016 Education Showcase

2016 Education Showcase Interprofessional Teamwork: Example

2016 Education Showcase Program Evaluation: Interprofessional Teamwork To what extent are teams achieving cohesiveness, effective communication, role clarity, clear goals? 57 teams over the past 6 years, 8-10 per year 194 out of 247 (79%) individual responses; Response rate within team: 50% to 100% TDM scores range from 55 to 89 Generally increase from fall to spring Spring scores were lowest in 2012 (64.9), peaked in 2013 (70.8), gradually fell 2014-17 (67.8) Scores consistently highest in communication, then cohesiveness, role clarity, goal clarity Huddle coaches & trainees rated goal clarity lower, on average, than other team members Teams with higher TDM scores tend to have more dispersed distribution of responses Implications Consider workplace factors staff turnover, shortages Additional team building activities

A Systems-Based Assessment Framework 1 for IPE? Level 3: System readiness for future change -What changes can we anticipate and do our teams and metrics align with these changes? Level 2: Program performance -Is the program & clinic meeting current expectations? Level 1: Individual curriculum components & stakeholders -Are learners & teams meeting expectations? Bowe C, Armstrong E. Assessment for systems learning. Acad Med. 2017; 92: 585-592

3 2016 Education Showcase 10/1 Acknowledgements Support for the SFVA Center of Excellence for Primary Care Education is provided by the Veterans Affairs Office of Academic Affiliations Many thanks to the team members who complete the survey each year and to our data manager, Gillian Earnest, who collects and compiles these data THANK YOU!

Improving the Evaluation of Interprofessional Education: Moving Beyond Attitudinal Measures Patricia (Patty) Carney, PhD Professor of Family Medicine Oregon Health & Science University

PACER (Professionals Accelerating Clinical & Educational Redesign) 3 Year Quasi-Experimental Study Designed to Improve IPE and Co-Learning Among the # Primary Care Disciplines Family Medicine General Internal Medicine General Pediatrics Funded by the Foundations of the ABFM, ABIM, ABP, Josiah Macy Jr. Foundation, ACGME

PACER MAP of Sites 27 continuity clinics at 9 institutions across the U.S.

Learners included in PACER - FM, IM, Peds Residents & Faculty - PA Students - NP Students - MA Students - Behavioral Health Students - Pharmacy Students

Evaluation Design Mixed Methods Approach (Qualitative/Quantitative) - Design is Pre-Post Test Repeated Measures (Quasi-Experimental) No Recreational Data Collection!!!

Core Data Collection Instruments Quantitative PACER Training Post Program Survey Attendance logs PCMH Monitor Continuity Clinic Survey Educational Program Survey Faculty Skills Self-Assessment Survey Qualitative Key Informant interviews (Telephone) Focus Groups (at site visits) Direct Observations (collected via Field notes) at site visits and all joint activities

Instrument Domains # of Variables Post Program Survey Session quality Overall usefulness Intention to make changes at institution Patient-Centered Medical Home Monitor - Validated - 7 Domains Leadership and engagement QI Team Process Data capacity Patient self-management support Team-based care Cost containment and care management Access and continuity Continuity Clinic Survey Practice characteristics Patient population Educational Program Survey Learning community activities Coaching Activities/ assessment Practice re-design efforts Training re-design efforts Sustainability activities Sustainability and dissemination Faculty Skills Self-Assessment Survey Interprofessional care and education Leadership for change Patient-centered care Stewardship of resources Competency assessment skills Individual characteristics 43 39 15 40 25

Longitudinal Coaching Key component of this project Coaches will attend the collaborative site visits Coaching calls planned quarterly Collecting data as part of these calls and during the collaborative site visits

Collaborative Site Visits Provide a Check in opportunity Designed as a Learning Community NOT an Audit Formal Progress Report produced using all contacts and used for evaluation

First paper fully drafted: The Educational Value of Interprofessional Learners Applying their Developing Team-based Care Skills in Primary Care Ambulatory Training Settings Benefits of Co-learning: Development of Personal Relationships Improved Education, Improved Patient Care Improved Job Satisfaction Enablers of Perceived Benefits: Clinic Culture With open Leadership Empoweres TBC Systemic Factors Clinic layout that fostered TBC (e.g., places to huddle and co-work) Barriers of Realizing Perceived Benefits: Clinic Culture With Top Down Leadership Stymies TBC Systemic Factors Clinic layout that inhibited TBC (e.g., no co-learning space)

We are having more fun than we ve ever had in our lives

Use of Q methodology to evaluate rural IPE Curt Stilp, EdD, PA-C Director, Oregon Area Health Education Center (AHEC) Assistant Professor, Physician Assistant Program Oregon Health & Science University Portland, Oregon

Rural IPE Study Purpose Examine how rural IPE influences student perspectives Search for shared perspectives Rural life Rural team-based care What factors form the perspective

Q Method Research Question What factors do students participating in a rural IPE experience consider most important and least important in making a decision to practice teambased care in a rural setting?

Q Methodology Stephenson, 1953; Brown, 1993 Concourse (Rural, Team-based care, Rural IPE) Literature Previous student s rural IPE journals Q set Set of statements representing thoughts, ideas, opinions on the topic

Q Set Statement Category Source 35 total statements 17 - student journals 18 - literature 4 5 from each category Time and sustained presence in a community helped build trust and familiarity. Working together in the clinic serves as great peer support that is needed. Rural communities have limited funds which restrict what care can be provided. IPE leads to a greater understanding of my own role on the health care team. Social Henry & Hooker, 2007 Team Student reflection journal Community Student reflection journal Team Ponzer et al., 2004 The availability of outdoor activities attracts me to the rural setting. Personal Student reflection journal The most effective rural IPE allows for engagement in the community. Education Deutchman et al., 2012

Q Sort Post-experience sorting of statements using FlashQ software Sorting grid Most agree Most disagree Subjective ordering of statements in relation to each other and the experience Reveals perspectives

Q Sort

Data Collection Rural IPE students Two rural locations Four health care professions MD, PA, Dental, Pharmacy 15 from each location 45 participants

Q Sort Demographics Late twenties (mean 28.5) Majority PA students (24) Near even split female and male (22 and 23) Majority married/partnered without children (24) Majority not from a rural background (27) Majority were in Coos Bay (28)

Q sort Interpretation Factor Analysis Four-step process 1. Correlation 2. Rotation 3. Factor scores 4. Factor array Representative Q sort of participants who sorted similarly shared perspective

Q sort Interpretation Factor Analysis Three Perspectives 1. The Team-Oriented Rural Optimist + Team, age, profession, length of time Living together, resources, project, community connection 2. The Independent Rural Impartial + Isolation from family/friends, natural resources, length of time Project, community engagement, team-based approach 3. The Team-Willing Rural Skeptic + First-hand experience, no children, location, length of time Recreation, profession, clinical environment, community

Study Conclusions General Rural IPE is useful for understanding rural life and making decisions about where to live and work after graduation Rural IPE needs to have a sustained presence Connection between community involvement and decision to return Specific Rural IPE motivates students to return to practice team-based care Feeling of isolation/remoteness deter a student from returning Rural heritage leads to increased likelihood a student will return

Limitations Only 4 health care professions No correlation between qualitative and quantitative No educational sequence data Bias in Q statement development

Thank you!

Questions & Discussion Alan Dow, MD, MSHA Assistant Vice President of Health Sciences for Interprofessional Education & Collaborative Care Virginia Commonwealth University