Using Script Concordance for Clinical Reasoning Assessment Rebecca Maldonado, MS, PA-C Associate Professor, University of Colorado Physician Assistant Program Kristine Himmerick, MS, PA-C Assistant Professor, Northern Arizona University Physician Assistant Program
Session Objectives 1. Describe the theory of script concordance testing (SCT) as a measure of clinical reasoning. 2. Understand the reliability of script concordance testing. 3. Recognize the parameters for constructing an expert clinician panel. 4. Comprehend the development techniques for a script concordance test item. 5. Understand the use of the SCT calculator for performing item analyses for both the expert panel responses and learner responses to SCT items.
What is Clinical Reasoning? the thinking and decisionmaking processes associated with professional practice o-higgs
Hierarchy of Clinical Reasoning Knowledge o o Didactic knowledge Clinical skills Cognition o Application of knowledge to practice Metacognition o Thinking about the decision making process o Thinking about thinking (Flavell, J. H. 1979)
Metacognition Awareness of the learning process Recognition of the limitations of memory Appreciation of perspective Capacity for self-critique Ability to select appropriate strategies for different situations (Croskerry, 2003)
Foundational knowledge and experience are required to progress o Novice to Expert Continuum Novice - Advanced beginner Competence Proficiency Expert Beginners are taught rules with little context. As the learner gains experience in applying the knowledge to real world situations, he/she advances toward expertise. (Benner, 1982)
Novice to Expert Thinking More Novice Monitoring, Reflection Cost, Time, Effort Developing Expert Less Experienced Low High Reliability, Accuracy (Graber 2009)
How can medical educators evaluate clinical reasoning? Can the novice to expert continuum be objectively measured?
Scripts are Medical knowledge structures which link relevant clinical and pathophysiologic information Scripts frame an individual s expectations (hypotheses) in clinical scenarios Initiated during early medical training AND developed through clinical exposure and practice.
Scripts Concordance Testing is a written assessment of clinical reasoning under conditions of uncertainty Specific facet of clinical reasoning competence: clinical data interpretation Hinges on: quality of acquired scripts A proxy indicator of: EXPERTISE Gold Standard : Acknowledged experts in field
Script Concordance Test (SCT) Multiple Choice Questionnaire (MCQ) Ill-defined problem with context of uncertainty Well-defined problem with no uncertainty More than one acceptable answer Only one correct answer Assess Reasoning Assess Knowledge
Clinical Reasoning: Key Steps Patient history and nonverbal communication Hypothesis Generation Script Activation SCT Question Format Case Vignette "If you were thinking " Data Collection Hypothesis Evaluation Data Interpretation "And then you find " "Your hypothesis becomes more or less likely?" (-2, -1, 0, +1, +2)
SCT Principles 1. Examinees are faced with challenging, realistic clinical situations and must choose between several appropriate response options 2. Item format reflects the way information is processed in clinical problem-solving situations 3. Scoring takes into account the variability of responses of experts to clinical situations
SCT Reliability Alpha-coefficients of 0.70-.090 across multiple disciplines (Lubarsky, 2011)
Creating SCT Items Create a small case vignette Must contain ambiguity or be missing information Appropriate for examinee s training Example: Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.
Creating SCT Items Select hypotheses for testing Must be plausible Represent common or life-threatening possibilities Pneumonia Pulmonary embolism Acute coronary syndrome Examples: Spontaneous pneumothorax Rib fracture/chest trauma Anxiety/panic attack
Creating SCT Items Write the SCT item Test the link of new information into the context of the clinical vignette Introduce key features which may have a broad range of possible responses among clinicians Write items which assess reasoning within the case vignette across the spectrum of the response scale If you were thinking: And then you find: Example: This diagnosis becomes: Pneumonia The patient is afebrile -2-1 0 1 2 Spontaneous Pneumothorax The onset of chest pain/sob was while running -2-1 0 1 2 Acute coronary syndrome The patient uses cocaine -2-1 0 1 2-2: Very unlikely, -1: Somewhat unlikely, 0: Neither more or less likely, 1: Somewhat likely, 2: Very likely
Creating Investigative/Therapeutic SCT Items Write the SCT item Test the link of new information into the context of the clinical vignette Introduce key features which may have a broad range of possible responses among clinicians Write items which assess reasoning within the case vignette across the spectrum of the response scale If you were thinking of: And then you Example: find: This treatment becomes: Ordering a chest x-ray The patient is not tachypneic -2-1 0 1 2 Ordering troponin levels The patient s EKG is normal -2-1 0 1 2 Ordering a D -dimer The patient s Well s criteria score was <2-2 -1 0 1 2-2: Contraindicated, -1: Less indicated 0: Neither more or less indicated, 1: Somewhat indicated, 2: Very indicated
Creating SCT Items: More Cases or More Questions? Gagnon et al., 2008
Vetting the SCT items: 1. Construct an Expert Panel A. Utilize faculty or community providers with experience in item content areas(petrucci, 2013) B. Expert panel size 10-15 members (Gagnon, 2005) 0,80 0,75 0,70 0,65 0,60 0,55 0,50 0,45 0,40 n=5 n=10 n=15 n=20 n=25 n=30 n=38 mean (sd) min value max value
Vetting the SCT items: 2. Scoring Expert Panel Responses A. Discard or re-write items identified by panel as confusing, erroneous, etc. B. Use aggregate responses, don t look for consensus (Norman,1985) C. Score the responses using modal distribution (Charlin, 2010) D. In panels <15, remove outlier responses (Gagnon, 2011) Scoring Key for 1 SCT item (panel size= 10) Answers -2-1 0 1 2 (1) (2) (3) (4) (5) Number of panel members Who chose this answer 0 0 2 5 3 Number of panel members Who chose this answer divided By the modal answer 0/5 0/5 2/5 5/5 3/5 Score for each response On this item 0 0.4 1.6
Vetting the SCT items: Choosing Moderate Variability Items Use SCT calculator to determine variability of panel responses for each item. Staff-students 1,5 1 0,5 0 Low variability Moderate variability High variability Charlin, 2006
SCT Calculator: Expert Panel Response Entry 1. Must be entered using 5-point Likert numbering 2. Once expert panel responses are entered, click to the 3 rd worksheet tab to see item statistics
SCT Calculator: Expert Panel Item Stats http://www.cpass.umontreal.ca/sct.html Rebecca.maldonado@ucdenver.edu for Excel file
SCT Item #1 - Audience Response: Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath. 20% 20% 20% 20% 20% A. B. C. D. E. If you were thinking: And then you find: This diagnosis becomes: A) B) C) D) E) Pleuritis Chest pain began with Exercise Very likely Somewhat more Neither more or Somewhat unlikely Very unlikely likely less likely
SCT Item #2 - Audience Response: Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath. 20% 20% 20% 20% 20% A. B. C. D. E. If you were thinking: And then you find: This diagnosis becomes: Chest trauma / pain / strain Chest pain was improved by lying down A) Very likely B) Somewhat more likely C) Neither more or less likely D) Somewhat unlikely E) Very unlikely
SCT Item #3 - Audience Response: Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath. 20% 20% 20% 20% 20% A. B. C. D. E. If you were thinking: And then you find: This diagnosis becomes: Pneumothorax Patient exam revealed a mid-line trachea A) Very likely B) Somewhat more likely C) Neither more or less likely D) Somewhat unlikely E) Very unlikely
Expert Panel Responses: SCT Items 1-3
Administering the SCT Items 1. Options for student exams 2. SCT calculator for student scoring & item analysis
SCT Calculator- Scores &Item Analysis of Learner Responses
Avoiding Pitfalls: Use this Do list 1. Write short clinical vignettes 2. Write questions for which the answer is not clearly defined in books or clinical guidelines. 3. Keep the number of questions per case to < 5. 4. Solicit more expert panel members than you need by at least 30%.
Questions Many thanks to Kevin Bogart, MS, PA-C for his contributions to this presentation.
References Benner, P. (1982). From novice to expert. American Journal of Nursing, 82(3), 402-407. Charlin, B. (2010). Assessment in the context of uncertainty using the Script Concordance test: more meaning for scores. Teaching and Learning in Medicine, 22(30), 180-186. Charlin B, Boshuizen H, Custer E, Feltovich P. (2007 ) Scripts and clinical reasoning. Medical Education. 41:1178-84. Charlin B, Gagnon R, Pelletier J, et al. (2006 ) Assessment in the context of uncertainty: the effect of variability within the panel of reference. Medical Education.18: 22-7. Croskerry, P. (2003). "Cognitive forcing strategies in clinical decisionmaking." Ann Emerg Med 41(1): 110-120. Flavell, J.H. (1979). Metacognition and cognitive monitoring: A new area of cognitive developmental inquiry. American Psychologist, 34(10), 906-911. Fournier J, Demeester A, Charlin B. (2008 ) Script Concordance Tests: Guidelines for construction. BioMedCentral, Medical Informatics and Decision Making. 8: 18. Gagnon, R, et al. (2008) Script Concordance testing: more cases or more questions. Adv. Health Sci Educ. 14: 367-375. Gagnon R, Charlin B, Coletti M, Sauve E, van der Vleuten C. (2005) Assessment in the context of uncertainty: How many members are needed on the panel of reference of a script concordance test. Medical Education. 39: 284-91. Graber, M. L. (2009). "Educational strategies to reduce diagnostic error: can you teach this stuff?" Adv Health Sci Educ Theory Pract 14 Suppl 1: 63-69. Higgs, J. and M. A. Jones (2000). Clinical reasoning in the health professions. Oxford ; Boston, Butterworth-Heinemann. Lubarsky S, Charlin B, Cook D, Chalk C, van der Vleuten C. (2011 ) Script concordance testing: a review of published validity evidence. Medical Education. 45:329-338. Norman, G. (2005). Research in clinical reasoning: past history and current trends. Med Educ, 39(4), 418-427. Petrucci, AM, et al. (2013) Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key. Am J Surg. 205(2): 137-140.