HOW to develop an evaluation tool for IPE evaluation the NUS experience

Transcription

1 HOW to develop an evaluation tool for IPE evaluation the NUS experience Ms Ang Chai Yun Rachel Department of Pharmacy RA of IPE Study, NUS 1

2 Quality of program evaluation can only be as good as the tools it uses Stone (2006) 2

3 The background Uni-profession education before 2010 Since 2010, IPE implemented in NUS for undergraduates in 5 Academic Units (Dentistry, Medicine, Nursing, Pharmacy, & Social Work) Evaluation of effectiveness of IPE warranted 3

4 Aim: Evaluate the effectiveness of the IPE program on NUS Dental, Medical, Nursing, Pharmacy & Social Work undergraduates in achieving 5 interprofessional competencies The Evaluation Plan Hypothesis: IPE will equip students with interprofessional competencies by improving their reaction, attitude, knowledge & skills towards collaborative practice Design: Longitudinal study adopting a pre-post test design 4

5 NUS IPE Core Competencies Teamwork Ethical Practice Roles & Responsibility IPE Learning & Reflection Communication Patient, Family & Community Focus 5

6 Surveying the landscape of IPE evaluation tools Buffet of existing tools (e.g. RIPLS, IPEC, ISVS, CPAT) Most common tool is survey; each assesses different areas or subscales (CIHC, 2012) Distribution of quantitative tools across outcome levels from CIHC review 1. Attitudes 64 tools 2. Knowledge, skills & abilities 20 tools 3. Behaviour 34 tools 4. Organizational level 6 tools 5. Patient satisfaction 8 tools 6. Provider satisfaction 14 tools 6

7 Limitation of existing tools No universal operational definition for each competency Measure 1-2 competencies Developed without psychometric properties Adapted from existing tools without validation Applicable to a specific setting i.e. practitioners or students Developed exclusively for post-intervention Used by their author(s) only once Profession specific 7

8 The Tool Development Plan Aim To develop a psychometrically robust questionnaire that can comprehensively assess the effectiveness of IPE on NUS health professional undergraduates on 5 selected IPE competencies Framework Modified Kirkpatrick s model of Learning Outcomes 1. Reaction 2a. Modification of attitudes/perceptions/beliefs 2b. Acquisition of knowledge/skills 3. Behavioral change 4a. Change in organizational practice 4b. Benefits to patients/clients 8

9 Evaluation Needs Format 1 2 Preliminary decisions in tool development 3 Method of Item Generation 9

10 Phase 1: Literature Review Constructs/themes/topics/issues in interprofessional education & collaborative practice Conceptual & Operational definition of the competency RESOURCES CIHC tool inventory 10

11 Phase NUS Presentation 2 : Title Identify 2001 constructs for each competency HONESTY TRUST Ethical Practice RESPECT ACCEPTANCE 11

12 GOAL Coverage of Construct(s) & Level(s) of learning outcome Phase 3 : Generate initial list of items Self generated Existing tools Choice of tool What does it measure Setting/population/ sample size Psychometric properties 12

13 Phase 4: Refinement of items DOUBLE BARRELED AMBIGUOUS TIME SPECIFIC DOUBLE NEGATIVE LEADING QUESTION 13

14 Ensuring face validity Omission Actual example Revision I more highly value open and honest communication with team members I value open and honest communication among IPE team members Retention 14

15 Phase 5: Choice of response option Dichotomous Rating No. of response options 2 >2 Likert Scale Labeling How many points E.g. E.g. 15

16 Phase 6: Format Length Instruction 16

17 CAPTURING 17

18 The Developed Tool Demographics 7 questions Scale 76 items on a 6 point Likert Scale (Strongly Agree=5; Agree=4; Somewhat agree, somewhat disagree=3; Disagree=2; Strongly Disagree=1; I don t know=scoring not entered). Reverse coding for 6 negative items Items measure reactions, attitude, knowledge and skills in 5 selected IPE competency domains: Roles & Responsibility = 18 items Communication = 18 items Teamwork = 17 items Patient, Family & Community Focus = 6 items Ethical Practice = 17 items 18

19 Sample items from scale Roles & Responsibility My professional expertise complements the roles of other members working on the same team with me. Communication I have the necessary skills to negotiate issues within the IPE team. Teamwork When in an IPE team, I should actively contribute to the team. Patient, Family & Community Focus IPE will help me gain knowledge of community resources. Ethical Practice IPE will help me to conduct myself with honesty in the delivery of team-based care. 19

20 Tool Validation Feedback from students on tool Finalized Tool 20

21 21

22 22

23 Bibliography CIHC (2012). An inventory of quantitative tools measuring Interprofessional Education and Collaborative Practice Outcomes. A report by the Canadian Interprofessional Health Collaborative. Retrieved from 26%202012%20%281%29.pdf Stone, N. (2006). Evaluating interprofessional education: The tautological need for interdisciplinary approaches. Journal of Interprofessional Care, 20(3),