Using Script Concordance for Clinical Reasoning Assessment: A Workshop for PA Educators Rebecca Maldonado, MSHPE, PA-C Associate Professor, University of Colorado Physician Assistant Program Kevin Bogart, MSPA, PA-C Instructor, University of Colorado Physician Assistant Program
Session Objectives 1. Describe the theory of script concordance testing (SCT) as a measure of clinical reasoning. 2. Understand the reliability of script concordance testing. 3. Recognize the parameters for constructing an expert clinician panel. 4. Comprehend the development techniques for a script concordance test item. 5. Understand the use of the SCT calculator for performing item analyses for both the expert panel responses and learner responses to SCT items.
Development of Clinical Reasoning Knowledge Didactic knowledge Clinical skills Cognition Application of knowledge to practice Metacognition Thinking about the decision making process Thinking about thinking (Flavell, J. H. 1979)
Novice to Expert Thinking More Novice Monitoring, Reflection Cost, Time, Effort Developing Expert Less Experienced Low High Reliability, Accuracy (Graber 2009)
Scripts are Medical knowledge structures which link relevant clinical and pathophysiologic information (key features of diseases) Based on hypothetical-deductive model of thinking Scripts frame an individual s expectations (hypotheses) in clinical scenarios Initiated during early medical training AND developed through clinical exposure and practice
Scripts Concordance Testing is a written assessment of clinical reasoning under conditions of uncertainty Specific facet of clinical reasoning competence: clinical data interpretation Hinges on: quality of acquired scripts A proxy indicator of: EXPERTISE Gold Standard : Acknowledged experts in field
Script Concordance Test (SCT) Multiple Choice Questionnaire (MCQ) Ill-defined problem with context of uncertainty Well-defined problem with no uncertainty More than one acceptable answer Only one correct answer Assess Reasoning Assess Knowledge
Clinical Reasoning: Key Steps Pa$ent history and nonverbal communica$on Hypothesis Genera$on Script Ac$va$on SCT Ques6on Format Case Vigne;e "If you were thinking " Data Collec$on Hypothesis Evalua$on Data Interpreta$on "And then you find " "Your hypothesis becomes more or less likely?" (- 2, - 1, 0, +1, +2)
SCT Reliability Alpha-coefficients of 0.70-.090 across multiple disciplines (Lubarsky, 2011)
Creating SCT Items Create a small case vignette Must contain ambiguity or be missing information Appropriate for examinee s training Example: Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.
Creating SCT Items Select hypotheses for testing Must be plausible Represent common or life-threatening possibilities Examples: Pneumonia Pulmonary embolism Acute coronary syndrome Spontaneous pneumothorax Rib fracture/chest trauma Anxiety/panic attack
Creating SCT Items Write the SCT item Test the link of new information into the context of the clinical vignette Introduce key features which may have a broad range of possible responses among clinicians Write items which assess reasoning within the case vignette across the spectrum of the response scale Example: If you were thinking: And then you find: This diagnosis becomes: Pneumonia The patient is afebrile -2-1 0 1 2 Spontaneous Pneumothorax The onset of chest pain/sob was while running -2-1 0 1 2 Acute coronary syndrome The patient uses cocaine -2-1 0 1 2-2: Very unlikely, -1: Somewhat unlikely, 0: Neither more or less likely, 1: Somewhat likely, 2: Very likely
Creating Investigative/Therapeutic SCT Items Write the SCT item Test the link of new information into the context of the clinical vignette Introduce key features which may have a broad range of possible responses among clinicians Write items which assess reasoning within the case vignette across the spectrum of the response scale Example: If you were thinking of: And then you find: This treatment becomes: Ordering a chest x-ray The patient is not tachypneic -2-1 0 1 2 Ordering troponin levels The patient s EKG is normal -2-1 0 1 2 Ordering a D -dimer The patient s Well s criteria score was <2-2 -1 0 1 2-2: Contraindicated, -1: Less indicated 0: Neither more or less indicated, 1: Somewhat indicated, 2: Very indicated
Creating SCT Items: More Cases or More Questions? Gagnon et al., 2008
Vetting the SCT items: 1. Construct an Expert Panel A. Utilize faculty or community providers with experience in item content areas(petrucci, 2013) B. Expert panel size 10-15 members (Gagnon, 2005) 0,80 0,75 0,70 0,65 0,60 0,55 0,50 0,45 0,40 n=5 n=10 n=15 n=20 n=25 n=30 n=38 mean (sd) min value max value
Vetting the SCT items: 2. Scoring Expert Panel Responses A. Discard or re-write items identified by panel as confusing, erroneous, etc. B. Use aggregate responses, don t look for consensus (Norman,1985) C. Score the responses using modal distribution (Charlin, 2010) D. In panels <15, remove outlier responses (Gagnon, 2011) Scoring Key for 1 SCT item (panel size= 10) Answers -2-1 0 1 2 (1) (2) (3) (4) (5) Number of panel members Who chose this answer 0 0 2 5 3 Number of panel members Who chose this answer divided By the modal answer 0/5 0/5 2/5 5/5 3/5 Score for each response On this item 0 0.4 1.6
Vetting the SCT items: Choosing Moderate Variability Items Use SCT calculator to determine variability of panel responses for each item. Staff-students 1,5 1 0,5 0 Low variability Moderate variability High variability Charlin, 2006
SCT Calculator: Expert Panel Response Entry 1. Must be entered using 5-point Likert numbering 2. Once expert panel responses are entered, click to the 3 rd worksheet tab to see item statistics
SCT Calculator: Expert Panel Item Stats http://www.cpass.umontreal.ca/sct.html Rebecca.maldonado@ucdenver.edu for Excel file
Expert Panel Responses:
Administering the SCT Items 1. Options for student exams 2. SCT calculator for student scoring & item analysis
SCT Calculator- Scores &Item Analysis of Learner Responses
SCT Calculator- Scores &Item Analysis of Learner Responses
Avoiding Pitfalls: Use this Do list 1. Write short clinical vignettes 2. Write questions for which the answer is not clearly defined in books or clinical guidelines. 3. Keep the number of questions per case to < 5. 4. Solicit more expert panel members than you need by at least 30%.
Questions
References Benner, P. (1982). From novice to expert. American Journal of Nursing, 82(3), 402-407. Charlin, B. (2010). Assessment in the context of uncertainty using the Script Concordance test: more meaning for scores. Teaching and Learning in Medicine, 22(30), 180-186. Charlin B, Boshuizen H, Custer E, Feltovich P. (2007 ) Scripts and clinical reasoning. Medical Education. 41:1178-84. Charlin B, Gagnon R, Pelletier J, et al. (2006 ) Assessment in the context of uncertainty: the effect of variability within the panel of reference. Medical Education.18: 22-7. Croskerry, P. (2003). "Cognitive forcing strategies in clinical decisionmaking." Ann Emerg Med 41(1): 110-120. Flavell, J.H. (1979). Metacognition and cognitive monitoring: A new area of cognitive developmental inquiry. American Psychologist, 34(10), 906-911. Fournier J, Demeester A, Charlin B. (2008 ) Script Concordance Tests: Guidelines for construction. BioMedCentral, Medical Informatics and Decision Making. 8: 18. Gagnon, R, et al. (2008) Script Concordance testing: more cases or more questions. Adv. Health Sci Educ. 14: 367-375. Gagnon R, Charlin B, Coletti M, Sauve E, van der Vleuten C. (2005) Assessment in the context of uncertainty: How many members are needed on the panel of reference of a script concordance test. Medical Education. 39: 284-91. Graber, M. L. (2009). "Educational strategies to reduce diagnostic error: can you teach this stuff?" Adv Health Sci Educ Theory Pract 14 Suppl 1: 63-69. Higgs, J. and M. A. Jones (2000). Clinical reasoning in the health professions. Oxford ; Boston, Butterworth-Heinemann. Lubarsky S, Charlin B, Cook D, Chalk C, van der Vleuten C. (2011 ) Script concordance testing: a review of published validity evidence. Medical Education. 45:329-338. Norman, G. (2005). Research in clinical reasoning: past history and current trends. Med Educ, 39(4), 418-427. Petrucci, AM, et al. (2013) Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key. Am J Surg. 205(2): 137-140.