HOW to develop an evaluation tool for IPE evaluation the NUS experience Ms Ang Chai Yun Rachel Department of Pharmacy RA of IPE Study, NUS 1
Quality of program evaluation can only be as good as the tools it uses Stone (2006) 2
The background Uni-profession education before 2010 Since 2010, IPE implemented in NUS for undergraduates in 5 Academic Units (Dentistry, Medicine, Nursing, Pharmacy, & Social Work) Evaluation of effectiveness of IPE warranted 3
Aim: Evaluate the effectiveness of the IPE program on NUS Dental, Medical, Nursing, Pharmacy & Social Work undergraduates in achieving 5 interprofessional competencies The Evaluation Plan Hypothesis: IPE will equip students with interprofessional competencies by improving their reaction, attitude, knowledge & skills towards collaborative practice Design: Longitudinal study adopting a pre-post test design 4
NUS IPE Core Competencies Teamwork Ethical Practice Roles & Responsibility IPE Learning & Reflection Communication Patient, Family & Community Focus 5
Surveying the landscape of IPE evaluation tools Buffet of existing tools (e.g. RIPLS, IPEC, ISVS, CPAT) Most common tool is survey; each assesses different areas or subscales (CIHC, 2012) Distribution of quantitative tools across outcome levels from CIHC review 1. Attitudes 64 tools 2. Knowledge, skills & abilities 20 tools 3. Behaviour 34 tools 4. Organizational level 6 tools 5. Patient satisfaction 8 tools 6. Provider satisfaction 14 tools 6
Limitation of existing tools No universal operational definition for each competency Measure 1-2 competencies Developed without psychometric properties Adapted from existing tools without validation Applicable to a specific setting i.e. practitioners or students Developed exclusively for post-intervention Used by their author(s) only once Profession specific 7
The Tool Development Plan Aim To develop a psychometrically robust questionnaire that can comprehensively assess the effectiveness of IPE on NUS health professional undergraduates on 5 selected IPE competencies Framework Modified Kirkpatrick s model of Learning Outcomes 1. Reaction 2a. Modification of attitudes/perceptions/beliefs 2b. Acquisition of knowledge/skills 3. Behavioral change 4a. Change in organizational practice 4b. Benefits to patients/clients 8
Evaluation Needs Format 1 2 Preliminary decisions in tool development 3 Method of Item Generation 9
Phase 1: Literature Review Constructs/themes/topics/issues in interprofessional education & collaborative practice Conceptual & Operational definition of the competency RESOURCES CIHC tool inventory 10
Phase NUS Presentation 2 : Title Identify 2001 constructs for each competency HONESTY TRUST Ethical Practice RESPECT ACCEPTANCE 11
GOAL Coverage of Construct(s) & Level(s) of learning outcome Phase 3 : Generate initial list of items Self generated Existing tools Choice of tool What does it measure Setting/population/ sample size Psychometric properties 12
Phase 4: Refinement of items DOUBLE BARRELED AMBIGUOUS TIME SPECIFIC DOUBLE NEGATIVE LEADING QUESTION 13
Ensuring face validity Omission Actual example Revision I more highly value open and honest communication with team members I value open and honest communication among IPE team members Retention 14
Phase 5: Choice of response option Dichotomous Rating No. of response options 2 >2 Likert Scale Labeling How many points E.g. E.g. 15
Phase 6: Format Length Instruction 16
CAPTURING 17
The Developed Tool Demographics 7 questions Scale 76 items on a 6 point Likert Scale (Strongly Agree=5; Agree=4; Somewhat agree, somewhat disagree=3; Disagree=2; Strongly Disagree=1; I don t know=scoring not entered). Reverse coding for 6 negative items Items measure reactions, attitude, knowledge and skills in 5 selected IPE competency domains: Roles & Responsibility = 18 items Communication = 18 items Teamwork = 17 items Patient, Family & Community Focus = 6 items Ethical Practice = 17 items 18
Sample items from scale Roles & Responsibility My professional expertise complements the roles of other members working on the same team with me. Communication I have the necessary skills to negotiate issues within the IPE team. Teamwork When in an IPE team, I should actively contribute to the team. Patient, Family & Community Focus IPE will help me gain knowledge of community resources. Ethical Practice IPE will help me to conduct myself with honesty in the delivery of team-based care. 19
Tool Validation Feedback from students on tool Finalized Tool 20
21
22
Bibliography CIHC (2012). An inventory of quantitative tools measuring Interprofessional Education and Collaborative Practice Outcomes. A report by the Canadian Interprofessional Health Collaborative. Retrieved from http://ipe.utoronto.ca/sites/default/files/cihc_tools_report_aug 26%202012%20%281%29.pdf Stone, N. (2006). Evaluating interprofessional education: The tautological need for interdisciplinary approaches. Journal of Interprofessional Care, 20(3), 260 275. 23