Assessing Motivation and Self-Regulation in Learning within a Predictive Design: Incorporating Systematic Elements of Change

Transcription

1 Educ Psychol Rev (2006) 18: DOI /s ORIGINAL ARTICLE Assessing Motivation and Self-Regulation in Learning within a Predictive Design: Incorporating Systematic Elements of Change Markku Niemivirta Published online: 20 September 2006 # Springer Science + Business Media, Inc Abstract This article comments on the target paper by Vollmeyer and Rheinberg. As the emphasis of this special issue is on the assessment of self-regulation in learning, my commentary will focus on the various aspects of the research design, measures, and analyses. Keywords Self-regulated learning. Motivation. Assessment. Structural equation modeling. Analysis of change The series of studies described by Vollmeyer and Rheinberg illustrates an impressive body of research. As their view on self-regulated learning, or self-regulation in learning, is primarily motivational, the focus of their work is on factors that direct and sustain the actions the students carry out during certain study or learning episodes. Within their cognitive-motivational framework, Vollmeyer and Rheinberg propose a process model that specifies components related to the different phases of performance or learning activity. These include initial motivation, situation-specific motivational and cognitive mediators, and performance outcomes. Vollmeyer and Rheinberg argue that students enter a particular performance or learning situation with a certain motivational mind-set, which again influences the way the students approach the task. Certain situation-specific factors then mediate the impact of initial motivation on the actual performance. Vollemeyer and Rheinberg s model is very appealing and welcome in the field. Recent research on motivation heavily relying on trait-like constructs and other non-contextual factors has somewhat failed to explain or accurately describe the processes that underlie the observed relationships between general motivation and performance. Thus, one of the merits of Vollemeyer and Rheinberg s work is that not only do they emphasize situationspecific processes, but also explicitly link them with more general motivational forces. The analytical approach for testing the assumptions derived from the model rests on the implementation of longitudinal designs and the use of regression models. Variables included in the studies reflect participants motivation and performance before, during, and after the M. Niemivirta (*) Research Center for educational Psychology, University of Helsinki, Helsinki, Finland markku.niemivirta@helsinki.fi

2 256 Educ Psychol Rev (2006) 18: given learning episode or performance situation. The time span of these episodes varies from short experiment-like tasks to university courses lasting several weeks. As the goal of this special issue is to discuss different methodological approaches to the study of self-regulated learning, my commentary will focus on some aspects of the designs and methods Vollemeyer and Rheinberg have adopted. Basically, Vollemeyer and Rheinberg have utilized two types of research designs within their process-oriented framework. One that consist of a relatively controlled experimental setting with a focus on short-term processes, and another that consists of a rather open setting with a focus on long-term processes. Despite these elementary differences, both designs include more or less identical variables. As Vollemeyer and Rheinberg explain, they have focused on constructs they consider particularly relevant within the cognitive-motivational framework. As regards to initial motivation, these include participants estimates of the probability of success, anxiety, interest, and challenge, which all represent variants of common indicators of motivation. The choice of these particular constructs stems from a historical overview of the field, and is thus theoretically driven and well founded. With respect to the experimental setting (computer simulation and a multimedia system), the design is relatively coherent in that all within setting variables are operationalized on the same level of measurement, that is, as situation-specific processes. My only concern with this is that some of the variables measured are ground and bound to the nature of the task itself. For example, time on task and strategy use are operationalized in a way that represents the nature of the task rather than the actual phenomena that are considered relevant. Time on task describes how long the participants work on each stage of the system, but tells us little about what sort of processes they actually spent the time on. Strategy use instead is strictly operationalized as an index describing the strategy of approaching the task in terms of basic operations (i.e., how to manipulate the input variables), and thus does not reflect those regulating and controlling strategies that the participants are likely to use in order to ensure they progress in an appropriate manner. Although difficult to implement in practice, I would welcome a more sophisticated examination of the process variables, particularly because the focus of the investigation is on self-regulation in learning and performance. Although Vollmeyer and Rheinberg s second study context, the classroom setting, includes an important addition to the experimental design the assessment of flow the overall design somewhat lacks the coherence of the lab-type setting. This is mainly due to the different levels or units of measurement included in the design. In the first study reported (Engeser, Rheinberg, Vollmeyer, & Bischoff, 2005, Study 1), competence in French and motivation to participate in the class were assessed before the course, and performance at the end of the course. Mediating variables, interest and challenge and flow, were assessed before a class and during the same class, respectively. In the second study (Engeser et al., 2005, Study 2), math grades in school and intention to study were assessed at the beginning of the semester, indices of initial motivation and flow before and during a particular task, respectively, few weeks before the end of the course. Finally, scores on final exam were collected. Although the findings from both studies generally support the predictions, the fact that the findings come from linking measures of general self-evaluations, situationspecific experiences, and relatively distant performance indices results in theoretically questionable interpretations. For example, how should we understand the finding showing how situation-specific experience of flow during one particular task influences exam performance three weeks later? Although one could argue that such a snap-shot estimate of flow represents a good average of each participants flow experiences over time, this would be purely speculative and tell us little about how flow experiences truly influence performance during an ongoing situation. The point here is, that this design seems to tap some general

3 Educ Psychol Rev (2006) 18: tendencies of students that more or less sustain over the whole course rather than those truly situational processes that the concepts and constructs employed here are supposed to reflect. Another issue relevant for both designs is the relation between measures assessed before the task (or course) and during the task (or course). The authors claim that motivational state reflects the participants motivation during the task, and thus somewhat overlaps with the construct of initial motivation. However, motivational state is operationalized as the participants estimates of understanding what to do next and how much fun they are having. Thus, it would seem that the construct incorporates motivational (i.e., enjoyment) and cognitive (i.e., feeling of understanding; see Efklides, 2002) elements. Although these processes have proven to be relevant factors reflecting task-specific engagement, one does wonder why the online measures are not designed to match variables reflecting initial motivation. Even from a theoretical point of view, an explicit focus on the situational variants of challenge, interest, anxiety, and estimates of success probability would be especially relevant. This clearly has not gone unnoticed by the authors (see their discussion on page 16); they even refer to a study examining the issue empirically (Vollmeyer & Rheinberg, 2003). However, since this limitation represents, in my opinion, the most important issue to be focused on in future work, I will provide some more detailed comments on it. In my view, the major limitation of the present work is the lack of comparable repeated measures over time. Currently, the design adopted does not include any systematic elements of change. Although the purpose is to study processes, the actual analyses focus on predictive effects and, to some extent, stability of dependencies between measures over time. In the studies reported, there are only few remarks referring to change in motivation, and even then, change does not seem to reflect fluctuations in the mean level of given variables (see pages 5 and 12). Of the several types of changes over time than can be addressed, gamma change (i.e., change in the meaning and/or structure of the construct over time), alpha change (i.e., change in the mean level of the construct over time), and normative stability (i.e., stability of individual ranks on the construct over time) would be of specific interest here. Measurement invariance (i.e., no gamma change) and sufficient normative stability are prerequisites for validly assessing mean level changes over time. Although it is quite unlikely to detect any major structural changes in given constructs within this sort of experimental setting, already Vollmeyer and Rheinberg s own findings seem to suggest that the explicit testing of such possibility would be important. When discussing the problems associated with including repeated task concurrent measures in the design, the authors mention noticing how the learners were less differentiated (italics added) on the motivational factors compared to their answers on initial motivation. Since less differentiated evaluation of one s own motivation while working on a task implies structural changes, the issue of whether this is a true psychological phenomenon (thus reflecting gamma change) or merely a methodological artifact is an issue that should be taken seriously. The more important concern here is, however, the analysis of mean level changes. As pointed out, Vollemeyer and Rheinberg s research, in the current form, has focused on predictive effects, not on different forms of change. With three or more repeated measures it would be possible to employ sophisticated and powerful methods for analyzing change over time. For example, within the latent growth curve modeling framework it would be possible to assess various types of trajectories of change as well as the predictors and consequences of those trajectories. Such an analytical approach would provide excellent possibilities for assessing the dynamic processes taking place while the participants proceed with the task. The added value of this sort of approach is illustrated in a recent study that not only was

4 258 Educ Psychol Rev (2006) 18: inspired by the work of Vollemeyer and Rheinberg, but also followed a similar experimental setting (Tapola & Niemivirta, 2003). The focus of this study was to examine trajectories of change in participants interest and self-efficacy while they were working on the linear systems simulation developed by Vollemeyer and Rheinberg. Using latent growth curve modeling, the study managed to reveal trajectories of change in participants motivation during the task, interrelations between different components of change, and predictive effects of instructional setting, prior competence, and gender on different parameters of change. As an example of the multifaceted findings, it was found that, on average, participants selfefficacy increased as a function of progress in the task. Increase in self-efficacy was also associated with an increase in interest, and both self-efficacy and interest were related to the degree of learning during the beginning of the task, even after controlling for competence. In simple terms, it appeared that the on-line experience of learning during the task was associated with mutual changes in self-efficacy and interest. These findings of complex interrelations between different elements of the task and changes in motivation would not have been detected without repeated measures and compatible analytical tools. Since indices of change over time would seem especially relevant when assessing the dynamics of selfregulation in learning, approaches similar to this would seem quite advantageous. My last comment before summing up may seem trivial, but as a fundamentally important element of valid assessment, it does raise some reservations about the accuracy of the findings from Vollmeyer and Rheinverg s studies. As mentioned, much of the work reported has used regression models, and especially path models, for describing the results. Although the authors have used structural equation modeling as a more powerful alternative to simple series of regression models, they have done so in a somewhat simplistic manner. For example, they collapsed several scales into one global measure and used composite scores as single indicators in the autoregressive models. With such small samples, these sort of compensating procedures are sometimes necessary, but even then, the indicators in the models should not be treated as error-free (i.e., without measurement error) as it seems to be the case here. Structural equation modeling techniques permit the introduction of fixed error variances into the models, which again provides more accurate and valid estimates (e.g., Kline, 1998). Naturally, the more plausible option would be to increase the sample size, and then use multiple indicators for even more proper analyses. Despite the critical tone of my comments above, I do find the work of Vollmeyer and Rheinberg rich and fascinating, and it indeed provides a significant contribution to the field. Their effort to extract detailed pieces of information on how students operate in various performance situations is an excellent example of a research program that systematically builds on prior findings and thus provides cumulative knowledge on the subject. Designwise, their basic approach is a clever one, and allows subtle variations and control procedures if needed. From the perspective of self-regulated learning, the original, more lab-oriented design, as a well-structured and controlled research context, has its merits, but I do find their later approach even more appropriate. Clearly, their work on the classroom-oriented approach has only begun recently, and thus incorporates some essential limitations, but from the perspective of ecological validity, I would really like to see this design to be developed further. As a suggestion, the design could expand the current scheme by including repeated measures, which would allow for tracing changes and fluctuations in motivation over time. Also, linking situation-specific measures of flow and motivation more tightly with concurring performances would be of crucial importance. Finally, indices of performance could also be broader in scope, including, for example, scores from formal tests and exams as well as selfevaluated or observed estimates of progress, understanding, and commitment. The implementation of such additions would obviously be very challenging, but in my opinion

5 Educ Psychol Rev (2006) 18: also a necessity in order to better capture the dynamic nature of self-regulation in learning and performance. References Efklides, A. (2002). The systemic nature of metacognitive experiences: Feelings, judgments, and their interrelations. In M. Izaute, P. Chambres, & P.-J. Marescaux (Eds.) Metacognition: Process, function, and use (pp ). Dordrecht, The Netherlands: Kluwer. Engeser, S., Rheinberg, F., Vollmeyer, R., & Bischoff, J. (2005). Motivation, Flow-Erleben und Lernleistung in universitären Lernsettings. Zeitschrift für Pädagogische Psychologie, 19, Kline, R. B. (1998). Principles and practice of structural equation modeling. New York: Guilford. Tapola, A., & Niemivirta, M. (2003). Students motivational appraisals and performance in a complex problem solving task. Paper presented at the 10th European Conference for Research on Learning and Instruction, Padova, Italy. Vollmeyer, R., & Rheinberg, F. (2003). Aktuelle Motivation und Motivation im Verlauf. In J. Stiensmeier- Pelster, & F. Rheinberg (Eds.), Diagnostik von Motivation und Selbstkonzept, (pp ). Göttingen, Germany: Hogrefe.