The Assessment of Metacognitive Skillfulness: On-line vs. Off-line Measures. Marcel V. J. Veenman Developmental and Educational Psychology, Leiden University & Graduate School of Teaching and Learning, University of Amsterdam.
Assessing Metacognition: Prospective assessment (prior to task) Questionnaires. Concurrent assessment (during the task) Protocol analysis, Systematic observ, Logfile registration, Eye movements. Retrospective assessment Questionnaires, (afterwards) Interviews, Stimulated recall. Learning performance: Posttest, GPA, Credits, etc.
Prospective: Questionnaires MSLQ (Pintrich) ILS (Vermunt) LERQ (Minnaert) SPQ (Bigg) LASSI (Weinstein) Example item: I ask myself questions to make sure I know the material I have been studying. Likert-type or five-point scale From very true for me to not true for me at all, or from always to never.
Questionnaires: Pros: - Easy to administer to large groups; - Adequate internal consistency (reliability). Cons: - Limited range of fixed activities; - Hardly related to on-line measures (lack of concurrent validity); - Low prediction of Learning performance (lack of external validity); - Sensitive to social desirable answering; - Variable reference points. Conclusion: A quick and, sometimes, dirty measure that may reflect what students think they (will) do.
Concurrent (on-line): Most commonly used: 1) Protocol analysis (qualitative) 2) Protocol analysis (quantitative) 3) Systematical observation (quantitative) Either through audio- or videotapes, and rated or judged according to a clearly defined scoring scheme. Less frequently used: 4) Log-file registration 5) Eye-movement registration Needs validation of registration of specific activities as representative of metacognitive skills.
Concurrent (on-line): Prot.An. & Syst.Obs. Pros: - Registration of metacognitive skills in action (construct validity); - Usually adequate to high interrater reliability (internal consistency); - Coherence amongst concurrent methods is high (concurrent validity); - High prediction of learning performance (external validity). Cons: - Highly labor-intensive, thus reducing the number of students to be assessed; - Not all (mental) activities can be assessed (tip-of-the-iceberg phenomenon = threat to construct validity); - Mental activities sometimes have to be inferred from cognitive activities (threat to construct validity). Conclusion: Laborious, but more valid assessment method.
Concurrent (on-line): logfiles & eye movem. Pros: - Logfiles are easy to analyze with computers; - Both may represent metacognitive skills in action (construct validity); - Both methods may covary with other on-line methods (convergent validity); - A validated selection of observations is predictive of learning performance (external validity). Cons: - Eye-movement registration needs to be done individually (labor intensive); - Both methods assess a limited number of overt meta skills (threat to construct validity); - Both methods need to be validated against other on-line measures (convergent validity). Conclusion: Suitable for assessing a limited set of metacognitive skills if validated against other on-line measures.
Retrospective: - Adapted Questionnaires on metacognitive skills specific to task performance; - Structured interview addressing specific metacognitive activities after task performance; - Stimulated recall where a student reports on his/her activities while reviewing a video tape afterwards.
Retrospective: Pros: - For questionnaires, see before; - Interview allows for some flexibility in addressing specific activities; - Stimulated recall provides cues for the student to recollect what he/she actually did (construct validity). Cons: - For questionnaires, see before; - For interviews, the interviewer may steer the process of reflection (construct validity); - For all retrospective methods students have to reconstruct their previous metacognitive skills (threat to construct validity); - Retrospective methods hardly correspond to concurrent, on-line methods (lack of convergent validity) - Both questionnaires and interviews are low to moderate learning-performance predictors (external validity).
Multi-method: Plead for using multi-method designs for achieving convergent validity: 1) When introducing a new measurement method for metacognitive skills, e.g. through logfile-analyses in a particular task domain, start with a task analysis in order to determine relevant metacognitive activities in a specific learning environment. 2) Verify whether the selected (logfile-) measures represent metacognitive skills by validating them against another on-line assessment method (e.g. protocol analyses, systematical observation) => multi-method research. 3) External validity: the new assessment method should have a similar predictive value for learning performance, relative to other online assessment methods.
Protocol Measures Log.kwant Log.kwalitat Orientation: - deliberately rereading assignments, + + Systematical Orderliness - orderly sequence of actions, + - avoidance of unsystematical events (varying two variables), + - initial & final temperature, + + - put the burner on, + + - # blocks manipulated. + + Accuracy - adequate note taking (in scratchpad), + - avoidance of sloppy calculation (if answers are typed in). + Evaluation - none. Elaboration and Reflection - only if conclusions are entered into scratchpad. +