Tun your everyday simulation activity into research Chaoyan Dong, PhD, Sengkang Health, SingHealth Md Khairulamin Sungkai, UBD Pre-conference workshop presented at the inaugual conference Pan Asia Simulation Society in Healthcare (PASSH)
This talk is based on a white paper drafted by IMSH Research Committee, which is currently under review for publication.
State of simulation research Minimize threats to validity Types of simulation research Identify proper study design Formulate research questions
Roadmap for Simulation Research 1. Understand the state of simulation research 2. Determine types of simulation research 3. Formulate a research question 4. Identify proper study design 5. Minimize threats to internal validity
1. Understand the state of simulation research 1) What is already known, and what constitutes the current frontier of knowledge? 2) Is it necessary to study this? 3) What is the contribution your study makes to the field?
Example - Scenario A school purchased a SimMan3G. Emergency Dept started using SimMan for the airway management course. After a few months, the course director said, It s been a few months using SimMan 3G. Why don t we do a study to show that SimMan 3G helps students to learn airway management?
Databases for Literature Search PubMed MEDLINE (OVID) CINAHL Web of Science
Healthcare Simulation Journals Simulation in Healthcare Clinical Simulation in Nursing BMJ Simulation & Technology Enhanced Learning Academic Medicine Medical Education Teaching and Learning in Medicine
Group Activity 1 Define a topic & conduct literature search [10 min] Present your findings [5 min/group] Discuss your challenges [10 min]
Roadmap for Simulation Research 1. Understand the state of simulation research 2. Determine types of simulation research 3. Formulate a research question 4. Identify proper study design 5. Minimize threats to internal validity
2. Determine type of simulation research 1. Evaluate the effects of simulation a. Does it or does it not improve learning outcomes? b. Should simulation be implemented as a preferred educational approach in a specific arena? c. How do we improve the delivery of simulation-based programs?
2. Use simulation as an investigation tool to evaluate another process, such as patient care a. Using a standardized simulated environment to study the quality of care being offered in various stages of clinician fatigue b. Using a simulated cardiac arrest patient to investigate how stress affects patient care [example]
3. Collaborative simulation research a. Simulation research with multi-centre trials
Group Discussion 2 Work in pair Share a simulation research idea Decide the type of simulation research [10 min] Present your findings [5 min/group] Discuss your challenges [5 min]
Roadmap for Simulation Research 1. Understand the state of simulation research 2. Determine types of simulation research 3.Formulate a research question 4. Identify proper study design 5. Minimize threats to internal validity
3. Formulate a research question Where do the questions come from? 1. Your own clinical practice and teaching activities 2. Inspiration from others at professional meetings 3. Gaps in literature as assessed during a rigorous review
To formulate a research question Literature review must be done to: 1. Refine the potential research question 2. Identify theoretical frameworks 3. Finalize the research questions
Theoretical Framework 1. Provide guidance to ask the right questions in the appropriate way 2. Function as mental hooks on which the research project can be hung to assure that they are on the right track
Structure of Research Questions F: feasibility of the question I : interest level to the researcher N: novel nature with regard to the established literature E: ethical status R: relevance to current and future practice
P.I.C.O. P: Participants I : Intervention (independent variable) C: Comparison O: Outcome (dependent variable)
Group Activity 3 Based on previous group activities, construct the research questions for your project [10 minutes] Present & receive feedback [5 minutes] Share challenges in constructing research questions [5 minutes]
Break
Roadmap for Simulation Research 1. Understand the state of simulation research 2. Determine types of simulation research 3. Formulate a research question 4.Identify proper study design 5. Minimize threats to internal validity
4. Identify appropriate study design Qualitative Research Design Quantitative Research Design
Quantitative Research Design 1. Includes two variables 2. Explains relationships between the variables
Quantitative Research Design 1. Observational designs: descriptive cohort studies, casecontrol studies, cross-sectional studies 2. Quasi-experimental designs: pre- & post-tests 3. True experimental designs: randomized controlled trials, cross-over trials
Observational Designs Example
Quasi-experimental Design Pre- & post-design 1. No randomization 2. Contain specific prospective interventions 3. Confounding variables can be controlled by inclusion & exclusion criteria and carefully performed statistical analysis
Example
True Experimental Designs 1. Randomized controlled trial 2. Cross-over trial
Steps for RCT design 1. Select participants 2. Measure baseline variables 3. Randomize 4. Blind the intervention 5. Collect data 6. Measure outcome 7. Analyze data
Group Activity 4 Decide whether you will use the quantitative method Develop details for the study design [10 minutes] Present & receive feedback [5 minutes/group] Challenges in developing the quantitative study design [5 minutes]
Qualitative Research Design 1. Goal: to generate an appropriately testable question 2. Format: a. focus group discussion, b. Interview 3. Qualitative data analysis
Qualitative Research OBSERVATION PATTERNS HYPOTHESIS IMSH 2015
When to Use Qualitative Research? Variables cannot be quantified best understood in their natural settings studied over real time Studying roles, processes & groups Overall objective is understanding IMSH2015 Simulation Celebration January 2015
Units of Analysis Individuals Certain experiences In particular settings Identities, e.g., specific group Groups Demographic groups Intervention groups Types of people, e.g., medical students, residents One setting vs. another Organizations
Qualitative & Quantitative Methods Criteria Qualitative Quantitative Purpose To understand & interpret social interactions Data Words, images, or objects Numbers Form of Data Data Analysis Results Open-ended responses, interviews, participant observations, field notes, & reflections Identify patterns & themes Particular or specialized findings To test hypotheses, look at cause & effect, and make predictions Measurements using structured & validated datacollection instruments Identify statistical relationships Generalizable findings January 2015
Sample Size Saturation: no new evidence emerging from the data In grounded theory the notion of saturation means that categories are fully accounted for, the variability between them is explained and the relationships between them are tested and validated and thus a theory can emergy.
Two dimensions Breadth vs. Depth A study that has repeat engagements with participants would require less participants than a study with single data collection point per participant Pragmatic Is the study funded? Is it single or multiple-site? Are there multiple data source and methods of data collection?
Data Collection Methods 1.Interview 2.Focus group 3.Participant observation 4.Document analysis 5.Field notes
Interviews Collect in-depth info to gain insight into the issues Unstructured: no preconceived ideas about content or with no organization, encourage participants to tell their story, begin with broad question, subsequent questions are more focused and are guided by initial responses Semi-structured: particular interests, written interview guide to ensure all questions covered, most widely used, freely talk about on the guide
Audio/Video tape for accuracy
Develop Interview Questions Review existing similar studies Review relevant theories Refine key issues to address
Tips Use open-ended questions What? Where? Who? When? How? Why? Avoid leading questions Do you think vomiting during diarrhea is serious? How do you think vomiting during diarrhea? Let the interviewee lead Probe issues in depth Repeat the last thing an interviewee said and ask to continue Remain quiet & wait for interviewee to continue
Do not being interview right away. Greeting first. Listen & express interest in what you hear Use interviewee s language to ask new questions Learn how to re-phrase questions Avoid using why questions
Step 1. Transcription A.What is transcribed: written texts, audio/visual data B.Brings the researcher closer to the data C.Builds theoretical sensitivity (Strauss & Corbin, 1990)
Step 2. Data Analysis A.Produces summaries, abstracts, coding, and memos B.Summarizes results/data (metrics, frequency counts) C.Draws conclusion
Group Activity 5 Decide whether you will use the quantitative method Develop details for the study design [10 minutes] Present & receive feedback [5 minutes/group] Challenges in developing the quantitative study design [5 minutes]
Roadmap for Simulation Research 1. Understand the state of simulation research 2. Determine types of simulation research 3. Formulate a research question 4. Identify proper study design 5.Minimize threats to internal validity
Ethical Review Protocol 1. Research question with or without study hypothesis 2. Definitions of independent and dependent variables 3. Potential confounding variables 4. Sampling techniques and plan 5. Randomization technique if applicable
Ethical Review Protocol 6. Sample size calculations if applicable 7. Inclusion and exclusion criteria 8. Study measurements (measurement types, instruments, timing, validity, and reliability) 9. Data storage/confidentiality
Step 5: Minimize threats to validity Internal validity whether an experimental treatment makes a difference or not Whether there is sufficient evidence to support the claim External validity The generalizability of the treatment
Validity Criteria Diagnostic Tests Did clinicians face diagnostic uncertainty? Was there a blind comparison with an independent gold standard applied similarly to the treatment group and control group? Did the result of the test being evaluated influence the decision to perform the reference standard? Users' Guide to the Medical Literature - AMA Press 2002
Validity Criteria Derivation of Clinical Prediction Rules Were all important predictors included in the derivation process? Were all important predictors present in a significant proportion of the population? Were the outcome event and predictors clearly defined? Were those assessing the outcome blinded to presence of the predictors and were those assessing the presence of predictors blinded to the outcome event? Was the sample size adequate ( including a significant number of outcome events )? Does the rule make clinical sense? Users' Guide to the Medical Literature - AMA Press 2002
Bias Selection bias Study groups differ in confounding variables Strict criteria inclusion, definitions Matched control groups Measurement bias Inconsistent measurement Ex. Recall, reporting, investigator bias Confounding bias Two factors are associated, confusing effect Matching, stratification analysis
Error Random Error Error due to chance non-directional Statistically derived P value, confidence intervals Correction Adequate sample size ( power ) Systematic error ( AKA Bias, validity ) Related to study design How well does the study design answer the question? Directional bias in one direction
Qualitative Research Trustworthiness: Rigor and quality in qualitative paradigm Triangulation: to check and establish validity in studies by analyzing a research question from multiple perspectives
Triangulation (Denzin, 1978; Patton, 1999) Methods: consistency of findings generated by different data collection methods Sources: consistency of different data sources from within the same method At different points in time In public vs. private settings People with different viewpoints Analysts: multiple analysts to review findings Theories: multiple theoretical perspectives to interpret data
Group Activity 6 For your quantitative design, what could cause the validity threats? [5 minutes] For your qualitative design, what could be done for triangulation? [5 minutes] Present your group findings [5 minutes] Share your challenges [5 minutes]
Take-Away Points
Contact Us Dong Chaoyan, PhD Sengkang Health, SingHealth Dong.chaoyan@skh.com.sg Md Khairulamin A.S PAPRSB, Institute of Health Sciences Universiti Brunei Darussalam khairulamin.abdullah@ubd.edu.bn