INSTRUMENT VALIDATION AND EVALUATION OF PROBLEM-BASED LEARNING TUTORIAL PERFORMANCE OF UNDERGRADUATE NURSING STUDENTS

Transcription

1 INSTRUMENT VALIDATION AND EVALUATION OF PROBLEM-BASED LEARNING TUTORIAL PERFORMANCE OF UNDERGRADUATE NURSING STUDENTS Melanie Lack A dissertation submitted to the Faculty of Health Sciences, University of the Witwatersrand, Johannesburg in fulfilment of the requirements for the degree of Master of Science in Nursing Johannesburg, 2009

2 DECLARATION I, Melanie Lack declare that this dissertation is my own unaided work. It is being submitted for the degree of Master of Science in Nursing in the University of the Witwatersrand, Johannesburg. It has not been submitted before for any degree or examination at this or any other University. Melanie Lack day of 2009 ii

3 PUBLICATIONS 1. USING SUBJECTIVE JUDGEMENT TO DETERMINE THE VALIDITY OF A TUTORIAL PERFORMANCE EVALUATION INSTRUMENT Health SA Gesondheid Volume 14 Number 1 (March 2009 edition). 2. WEIGHTING OF ITEMS IN A TUTORIAL PERFORMANCE EVALUATION INSTRUMENT: STATISTICAL ANALYSIS AND RESULTS. Health SA Gesondheid Volume 14 Number 1 (March 2009 edition). iii

4 ABSTRACT PURPOSE: The purpose of this two-phased study was to determine the performance of undergraduate nursing students in problem-based learning (PBL) tutorials using a validated evaluation instrument. RELEVANCE: to determine the effectiveness of the PBL learning approach relative to the South African student. Phase 1 led to the validation of an instrument and Phase 2 evaluated the performance of nursing students in PBL tutorials using the validated instrument. PARTICIPANTS: Phase 1 participants included academic experts (n=8) selected by means of purposive, maximum variation sampling. Phase 2 participants included the total population of undergraduate nursing students (n=53) and facilitators (n=6). METHODS: A quantitative research approach was used to inform the overarching design that was descriptive and comparative in Phase 2 of the study. Phase 1 employed statistical techniques for instrument validation and refinement. Phase 1 data were collected in three round of a Delphi survey. After completion of the first two rounds a rating instrument with a 4-point (0-3) rating scale was developed referred to as the Tutorial Performance Rating Instrument. During the third and final round of the Delphi survey weighting of each main-item, sub-items and the rating scale took place, incorporating the Subjective Judgement Model using pair-wise comparisons on linear visual analogue scales. ANALYSIS: Relative weights were determined and following statistical analysis ratio scales were developed creating a unique weight to each item and the rating scale. This weight was represented in a percentage allowing each main-item construct and each sub-item to be placed in a hierarchy from highest to lowest percentage. Calculation on a student assessment would become time consuming and subject to error if done manually. A computer-based program referred to as the Tutorial Performance Evaluator was developed to carry out all the calculations of the percentages allocated to the items and rating scale. A replica of the Tutorial Performance Rating Instrument was built into the programme. In Phase 2 of the study a self-assessment and facilitator-assessment on each student s tutorial performance first-to fourth-year was carried out using the Tutorial Performance Rating Instrument. Following a one on one meeting between the student and the facilitator consensus agreement was reached on an acceptable rating against each item on the evaluation instrument. The latter was entered into the computer and a percentage for each main-item construct and a total percentage was calculated efficiently and accurately in 20 seconds. RESULTS: The results showed that first-year students struggled in all aspects of the PBL tutorial. Furthermore they did not possess the skills required for self-assessment. A small group of the second-year students struggled in the tutorials but were better able to carry out a self-assessment. The third-year students showed a slight drop in score when compared with the second-year students and this could be attributed to the new subjects introduced in the third-year of study. There was a substantial improvement in the results achieved by the fourth-year students in all the aspects of the PBL tutorial and showed a good correlation in carrying out a self-assessment when compared with the facilitator assessment. iv

5 RECOMMENDATIONS: This was a cross-sectional study and a longitudinal study should be carried out in future research to assess the individual s progress from first-to fourth-year in the PBL approach to learning. Greater academic support should be given to first-year students or alternatively the at risk students should be given a foundation course to assist them with communication skills and learning skills. v

6 ACKNOWLEDGEMENTS I wish to acknowledge the contribution of the following people without whose help this study could not have materialised: Professor Judith Bruce for her guidance and supervision from conception to completion of this study. The nursing students who participated in this study. The facilitators from the Department of Nursing Education University of the Witwatersrand. The Experts from the Problem-Based Learning expert group. The Department of Nursing Education at the University of the Witwatersrand, for allowing me to use the original Tutorial Performance Evaluation Instrument that became the starting point of this study. Mr. P.Ferreira for his support in helping me develop the computer programme. Professor Pieter Becker for his support in statistical analysis and interpretation in the early part of the study and Mr Musenge, the Bio-statistician at the Epidemiology Data Centre, Faculty of Health Sciences, for his support in statistical analysis and interpretation in the latter part of the study. The Faculty Research committee for their grant of R Mrs L Voicht - Professional Independent Editor vi

7 TABLE OF CONTENTS Page DECLARATION PUBLICATIONS ARISING FROM THIS STUDY ABSTRACT ACKNOWLEDGEMENTS TABLE OF CONTENTS LIST OF FIGURES LIST OF TABLES LIST OF ABBREVIATIONS LIST OF ANNEXURES ii iii iv vi vii xiii xv xviii xix CHAPTER ONE: ORIENTATION TO THE STUDY 1.1 INTRODUCTION BACKGROUND TO THE STUDY Problem-Based Approach to Teaching and Learning and Constructivism Requirements for South African Undergraduate Nursing Students PROBLEM STATEMENT RATIONALE FOR THE STUDY PURPOSE OF THE STUDY RESEARCH QUESTIONS RESEARCH OBJECTIVES Phase-One Objectives Phase-Two Objectives DEFINITION OF TERMS 14 vii

8 1.9 OVERVIEW OF METHODOLOGY SIGNIFICANCE OF THE STUDY ETHICAL CONSIDERATIONS CONCLUSION OUTLINE OF THE STUDY 18 CHAPTER TWO: LITERATURE REVIEW 2.1 INTRODUCTION PROBLEM-BASED LEARNING Historical Perspective PBL Process Facilitator Group Process Evaluation CHALLENGES OF PBL UNDERGRADUATE NURSING STUDENT CONCLUSION 42 CHAPTER THREE: RESEARCH METHODOLOGY PHASE INTRODUCTION RESEARCH DESIGN RESEARCH METHODS Population and Sample The Sampling Method Sample Eligibility Criteria Data Collection Techniques 48 viii

9 The Delphi Technique The Subjective Judgement Model Paired Comparison Analysis Visual Analogue Scales Data Collection Procedure Round One Survey Round Two Survey Pilot Study Round Three Survey VALIDITY AND RELIABILITY CONCLUSION 73 CHAPTER FOUR: DATA ANALYSIS AND RESULTS: PHASE INTRODUCTION APPROACH TO DATA ANALYSIS Determining Content Validity Determining Construct Validity THE STUDY RESULTS Content Validity Construct Validity and weighting of items DEVELOPMENT OF COMPUTER-BASED PROGRAMME CONCLUSION 105 CHAPTER FIVE: RESEARCH METHODOLOGY PHASE INTRODUCTION RESEARCH DESIGN 106 ix

10 5.3 RESEARCH METODS Population and Sample The Research Setting Approach to Data Collection Data Collection Procedure APPROACH TO DATA ANALYSIS VALIDITY AND RELIABILITY Validity Reliability CONCLUSION 117 CHAPTER SIX: DATA ANALYSIS AND RESULTS: PHASE INTRODUCTION DATA ANALYSIS THE STUDY RESULTS Inter-Class Comparison of Student Tutorial Performance Scores Problem Solving Skills Contributions Communication Critical Thinking Skills Learning Skills Personal Growth Leadership Skills Reliability between Facilitator and Student CONCLUSION 174 x

11 CHAPTER SEVEN: DISCUSSION, MAIN FINDINGS, RECOMMENDATIONS, LIMITATIONS AND CONCLUSIONS 7.1 INTRODUCTION SUMMARY OF THE STUDY DISCUSSION OF RESULTS: PHASE Personal Growth Creative Thinking Leadership Learning Skills Contributions Problem-Solving Skills Communications Critical Thinking Skills Weighting of Items Tutorial Performance Evaluator (TPE) DISCUSSION OF RESULTS: PHASE First-Year Students Second-Year Students Third-Year Students Fourth-Year Students Comparison between Facilitators and Students Scores MAIN FINDINGS STUDY LIMITATIONS RECOMMENDATIONS Nursing Education Research 208 xi

12 7.8 CONCLUSION 208 REFERENCE LIST 210 xii

13 LIST OF FIGURES Figure Page 1.1 The Problem-Based Learning Characteristics Delphi Technique Sequence Pilot Study Mean Scores: First-Year Students Pilot Study Mean Scores: Second-Year Students Pilot Study Mean Scores: Third-Year Students Pilot Study Mean Scores: Fourth-Year Students Example of a Visual Analogue Scale Example of Weighting on the Rating Scale Visual Analogue Scale Descriptors Main-items Visual Analogue Scale Descriptors sub-items Distribution of Students First-year to Fourth-year Gender Distribution of Students Total Population Student Mean Age First-Year to Fourth-Year Data-Sets in Relation to Data Analysis First-year Students Tutorial Performance Scores Second-year Students Tutorial Performance Scores Third-year Students Tutorial Performance Scores Fourth-year Students Tutorial Performance Scores Mean Tutorial Performance Scores First-year to Fourth-year Mean Score First-year to Fourth-year Problem Solving Mean Score First-year to Fourth-year Contributions Mean Score First-year to Fourth-year Communications Mean Score First-year to Fourth-year Critical Thinking 145 xiii

14 6.10 Mean Score First-year to Fourth-year Learning Skills Mean Score First-year to Fourth-year Personal Growth Mean Score First-year to Fourth-year Leadership Correlation of Mean Scores between Facilitator and Student 173 xiv

15 LIST OF TABLES Table Page 1.1 Overview of Research Methodology Percentage of L2 Learners in the First-Year Intake of Nursing Students S.A.N.C. Gender Demographics of registered Nursing Students Demographic Data on the Expert Group Nominal Scale Lynn (1986) Expert Ratings in Round One Delphi Expert Ratings in Round Two Delphi Proportion of Experts whose Endorsement is Required to Establish Content Validity Weighting Scores on 4- point Likert Scale Measurement in mm. of Main-items on the VAS Measurement in mm. of sub-items on the VAS Relative weights of Main-items and sub-items Hierarchy of items Weights (%) on the Original Likert Scale Weights (%) on the Ordinal Scale Percentages allocated to Main-items Weighting of Main-items and sub-items and relative weights Completed Tutorial Performance Evaluator Total Number of Data Sets in each Year of Study Summary of Scores between First-year to Fourth-year xv

16 Students in Problem-Solving Skills Analysis of Variance: Problem-Solving Skills Comparison of Problem-Solving Skills by year of study Percentage of Students obtaining a 0-3 Rating in Problem-Solving Skills Summary of Scores between First-year to Fourth-year Students in Contributions Analysis of Variance: Contributions Comparison of Student Contributions by Year of Study Percentage of Students obtaining a 0-3 Rating in Contributions Summary of Scores between First-year to Fourth-year Students in Communication Skills Analysis of Variance: Communication Skills Comparison of Communication Skills by Year of Study Percentage of Students obtaining a 0-3 Rating in Communication Skills Summary of Scores between First-year to Fourth-year Students in Critical Thinking Skills Analysis of Variance: Critical Thinking Skills Comparison of Critical Thinking Skills by Year of Study Percentage of Students obtaining a 0-3 Rating in Critical Thinking Skills Summary of Scores between First-year to Fourth-year Students in Learning Skills Analysis of Variance: Learning Skills Comparison of Learning Skills by Year of Study Percentage of Students obtaining a 0-3 Rating in Learning Skills Summary of Scores between First-year to Fourth-year Students in Personal Growth Analysis of Variance: Personal Growth 153 xvi

17 6.23 Comparison of Score by year in Personal Growth Percentage of Students obtaining a 0-3 Rating in Personal Growth Summary of Scores between First-year to Fourth-year Students in Leadership Skills Analysis of Variance: Leadership Skills Comparison of Leadership Skills by Year of Study Percentage of Students obtaining a 0-3 Rating in Leadership Skills Results of First-year Facilitator Assessment Results of First-year Students Self-Assessment Results of Second-year Facilitator Assessment Results of Second-year Students Self-Assessment Results of Third-year Facilitator Assessment Results of Third-year Students Self-Assessment Results of Fourth-year Facilitator Assessment Results of Fourth-year Students Self-Assessment Comparison of Facilitator and Student Mean Scores First-Year to Fourth-Year 174 xvii

18 ABBREVIATIONS BA (Cur): Bachelor Curationis B.Ed (Hons): Bachelor of Education (Honours) B.Nurs: BSc (Hons): D (Cur): L2 Learner: MA (Cur): M.Ed: MSc (Nurs): MSc (OT): PhD: PBL: SANC: TPEI: TPRI: TPE: VAS WC: WI: WS: Bachelor of Nursing Bachelor of Science (Honours) Doctor Curationis English second language Master Curationis Master Of Education Master of Science (Nursing) Master of Science (Occupational Therapy) Doctor of Philosophy Problem-Based Learning South African Nursing Council Tutorial Performance Evaluation Instrument Tutorial Performance Rating Instrument Tutorial Performance Evaluator Visual Analogue Scale Weight of Construct Weight of Item Weight of Ordinal Scale xviii

19 LIST OF ANNEXURES Page ANNEXURE A Original Tutorial Performance Evaluation Instrument 222 ANNEXURE B Postgraduate Committee Approval Letter 223 ANNEXURE C Protocol Approval Letter 224 ANNEXURE D Clearance Certificate and Protocol Number 225 ANNEXURE E Information Letter to Expert Group 226 ANNEXURE F Student Letter and Consent Form 227 ANNEXURE G Round-One Questionnaire 229 ANNEXURE H Round-Two Questionnaire 232 ANNEXURE I Tutorial Performance Rating Instrument 235 ANNEXURE J Descriptors of the Rating Scale 236 ANNEXURE K Round-Three Instructions and Visual Analogue Scales 237 ANNEXURE L Weights of Main-items and sub-items 248 ANNEXURE M Computer-based Tutorial Performance Evaluator 249 ANNEXURE N Phase 2 Statistical Data 250 xix