Institutionen för datavetenskap Department of Computer and Information Science

Transcription

1 Institutionen för datavetenskap Department of Computer and Information Science Final thesis A Gameful Quest to Make Second Language Acquisition Fun by Erik Olov Mårten Andersson LIU IDA/LITH EX A 15/055 SE Linköpings universitet SE Linköping, Sweden Linköpings universitet Linköping

2

3 Linköping University Department of Computer and Information Science Final Thesis A Gameful Quest to Make Second Language Acquisition Fun by Erik Olov Mårten Andersson LIU-IDA/LITH-EX-A--15/055--SE Supervisor: Erik Berglund Examiner: Johan Åberg

4

5 På svenska Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under en längre tid från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida In English The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: Erik Olov Mårten Andersson

6 Abstract This study explores the application of principles from game design, and the motivational psychology it is founded upon, to the domain of second language acquisition. A gameful design process based on playtesting and analysis with design lenses is adapted and used to iteratively design and develop a system for conversation practice with the goal of creating a motivating and engaging experience. The results indicate effectiveness of the process, but generalizing the results would require further research with bigger sample sizes and studies with varied core activities. iii

7

8 Acknowledgements I am deeply grateful to the people who contributed to this study. In particular, I am indebted to those who participated in the playtesting sessions, providing essential feedback and helping refine the system. Thank you Koya Sato, Tori, Qiao, Tatu Tolonen, Naoto Kanda and Midori Otani. Thanks also to Ross Campbell for the discussion of educational aspects related to this study. Your help made me think more clearly about roles, topics and content in learners discussions. Finally, my gratitude to Erik Berglund for supervising the study, nudging me in the right direction and helping me focus on the important parts.

9 Contents List of Figures 1 1 Introduction Motivation Purpose Problem Description Delimitations and Scope Contribution Literature Review Motivation Introduction Types of Motivation Self-Determination Theory (SDT) Flow Feedback Feedback levels Negative and Positive Feedback Games What Defines a Game? Playfulness and Gamefulness Learning in Games Playcentric Design Design Lenses The MDA Framework: Mechanics, Dynamics and Aesthetics Evaluation Methods Method Process Tools Evaluation v

10 CONTENTS CONTENTS 4 Process Strategy and Research Constraints and Requirements User Needs, Motivations and Hurdles Iterative Process st Iteration nd Iteration rd Iteration th Iteration th Iteration th Iteration Results Process Evaluation Discussion Results Method Referenced sources Wider Context Conclusion 45 References 46 vi

11 List of Figures 2.1 Iterative process flow (Fullerton, 2008) Summary of minimal contents of a playtest session (Fullerton, 2008, pp ) The lens of intrinsic skill atoms (Deterding, 2015) Main steps of the gameful design model (Deterding, 2015) MDA components Adapted gameful design process Sketches Game finished (one team) Game finished (two teams) First early prototype Second prototype Sketch of corrections Another alternative representation of corrections Levels in third prototype Feedback tool Correction Feedback page Improved feedback page Points remaining to reach the next level

12 Chapter 1 Introduction 1.1 Motivation Achieving fluency in a second language requires extensive practice. Passive skills such as reading and listening can be honed without necessarily needing a teacher but in order to improve speaking and writing skills, input and feedback are indispensable. However, most teachers do not have the time for quantitative one-on-one practice with students, and so these skills tend to be neglected. In other words, there is a mismatch between the need for students to practice and the teachers available to guide that practice. But instead of only practising with teachers, students can also improve their language skills by interacting with native speakers. Depending on location and language it may be difficult to meet speakers of second languages, but the advent of the Internet has lowered the barriers significantly. Other hurdles remain. For instance, the average native speaker does not have a background in pedagogy and is not able to explain detailed grammar points of their language in a systematic manner. Connecting with people from other countries online is easy, but merely connecting and chatting is not good enough for learning purposes. A system that aims to help learners should be designed with care, giving both student and native speaker the right tools to facilitate learning. Important aspects of such a system will likely include how to give helpful feedback when correcting errors and educating users about it, as well as ways to display and encourage progress. Perhaps even more importantly, in order for people to actually use a system, they must be motivated to do so. This point seems to be what many educational systems fail to realize, rendering them ineffective. 2

13 1.2. PURPOSE CHAPTER 1. INTRODUCTION 1.2 Purpose The purpose of this study is to explore and give indications as to how to create motivating computer applications for conversation practice by drawing on knowledge and practices from motivational psychology and game design. 1.3 Problem Description Finding opportunities to practice a language one is learning is not easy, and even when it is, such an opportunity may not be motivating. Past experiences may have resulted in too many failures and negative feelings, or perhaps practice have simply been too easy to be engaging. Good games excel at motivating and engaging people, so how can games inspire the design of a system for language practice? How can second language conversation practice be made more motivating by being situated in a game-like setting? How can a motivating system for language practice be effectively developed? 1.4 Delimitations and Scope This research project spans a range of fields, but in order to allow for a reasonably focused approach the main focus is on how game design can inspire motivation in the setting of online conversation practice. For the purposes of this thesis, the type of conversation is limited to the written format. Aspects relating the quality of learning have only to some extent been investigated. It is assumed that written conversation practice is indeed beneficial to language acquisition and proficiency, but its efficiency per se is not the target of this thesis. However, feedback is central to progress, learning and motivation and was therefore treated in the literature review. Furthermore, learning in the context of games has been briefly treated. 1.5 Contribution This research project is important because little effort has been made to improve the experience of second language conversation practice and make such tools available. There is a large body of research about IT-assisted practice but not in the context of conversation practice with native speakers. Furthermore, this type of practice system can be made widely available with little effort, helping many people improve their second language skills. 3

14 Chapter 2 Literature Review 2.1 The Psychology of Motivation Introduction Two fundamental questions of study in relation to motivation are the questions of what causes behavior and why behavior varies in intensity (Reeve, 2009, pp. 5-8). Behavior varies between different people, but it also varies for the same person depending on occasion and circumstances. What motivates one person today may not have the same effect tomorrow. Motivation theories attempt to explain the reasons for energy (strength) and direction (some particular goal) of behavior. As a starting point for exploring the field of motivation, I will use the following definition: The study of motivation concerns those processes that give behavior its energy and direction (Reeve, 2009, p. 8). The sources of motivation can be divided into external events and internal motives (Reeve, 2009, pp. 9-10). Internal motives comprise needs, cognitions and emotions while external events are sources of motivation that stem from social, cultural or environmental factors. Needs are both biological (e.g. hunger, thirst) and psychological (e.g. competence, belongingness) and are essential to the individual s growth and well-being. Cognitions comprise sources of motivation that relate to the indivual s way of thinking, e.g. goals, plans and beliefs. Emotions organize different aspects of experience to let individuals react to import events. For example, a sense of threat makes us feel afraid and want to escape. Motivation is a private experience that cannot be observed by another person, but instead motivation can be inferred (Reeve, 2009, pp ). There are two ways to infer information about a person s motivation: observing how motivation manifests itself through the person s behavior or observing events that are known to result in some particular motivational 4

15 2.1. MOTIVATION CHAPTER 2. LITERATURE REVIEW state. The first method is more commonly applicable than the second, and can include self-reported information about motivation as well as different types of observation of behavior or physiology Types of Motivation Many people think of motivation as something that one either does not have or that one has a certain degree of (Reeve, 2009, p. 16). However, contrary to this unitary concept it has been suggested that there exist different types of motivation. For example, Deci and Ryan (2000) have made a distinction between intrinsic and extrinsic motivation. Intrinsically motivated behaviors are those that are freely engaged out of interest without the necessity of separable consequences, and, to be maintained, they require satisfaction of the needs for autonomy and competence (Deci and Ryan, 2000, p. 233). In other words, it is not enough for an intrinsically motivating activity to be interesting irrespective of consequences, the activity additionally has to satisfy the individual s need to feel competent and self-determined. Extrinsic motivation, on the other hand, refers to behavior that somebody engages in because it will lead to some separable outcome (Ryan and Deci, 2000a, p. 55). Research has shown that the type of motivation (i.e. extrinsic or intrinsic) can have a great impact on performance and quality of experience. The type of motivation can also be seen as different depending on whether, for example, an individual is trying to reach a goal or avoid failure (Reeve, 2009, pp ). Motivation can also comprise different types of emotions, a person motivated by anger may not behave the same as way a person motivated by fear Self-Determination Theory (SDT) Self-Determination Theory is a theory of human motivation that investigates the innate psychological needs and growth tendencies of individuals (Ryan and Deci, 2000b). SDT identifies three fundamental psychological needs that are present in individuals across all cultures: people need to feel competent in negotiating their external and internal environments; they need to experience relatedness to other people and groups; and they need to feel autonomy or self-determination with respect to their own behaviors and lives (Deci and Ryan, 2012, p. 87). People whose motivation is intrinsic rather than extrinsic typically show better performance, persistence and creativity as well as feeling more interest, excitement and confidence in relation to the task at hand (Ryan and Deci, 2000b). Since the facilitation and maintenance of intrinsic motivation has can be readily disrupted by outside factors, SDT tries to explain 5

16 2.1. MOTIVATION CHAPTER 2. LITERATURE REVIEW which factors undermine or facilitate it and how it varies depending on circumstances. It has been argued and shown in studies that feelings of competence (such as when receiving positive performance feedback) facilitate intrinsic motivation, but only if the individual experiences her behavior as self-determined (the need for autonomy) (Ryan and Deci, 2000b). Research has also revealed that tangible rewards as well as threats, deadlines and other external measures undermine intrinsic motivation because they make the individual feel less autonomy. It should be noted, however, that different types of rewards (e.g. expected/unexpected, rewards that depend on task performance, rewards that merely depend on engagement with a task etc.) may have different effects on intrinsic motivation (see Deci et al., 1999). Other studies have shown that greater support for student autonomy in learning tend to lead to greater intrinsic motivation, curiosity and desire to take on challenges (Ryan and Deci, 2000b, p. 71). SDT recognizes that there are degrees of motivation and has put forward a model describing a continuum of motivation (Ryan and Deci, 2000b). We have already looked at intrinsic motivation, which is characterized by interest, enjoyment and other positive processes. Extrinsic motivation, in contrast, can range from a degree where the individual integrates the external motives and may involve a feeling of choice, whereas further down the continuum the motives are to a greater degree external and forcing as opposed to self-determined. The third innate psychological need in SDT, relatedness, comes with a few qualifications. Baumeister and Leary (1995) found that merely being related to other individuals but not interacting frequently is not enough to satisfy the need for belongingness. Rather, people need frequent positive interactions with a small number of people. Adding more relationships may have little consequence to an individual if the basic needs are already met. Further, people need interactions within relationships to happen in a stable, long-term manner in a context of care and concern. Baumeister and Leary (1995) have characterized this need to belong as a strong desire to form and maintain enduring interpersonal attachments Flow It has been observed that an individuals enjoyment of a range of tasks can be characterized by the relation between the task s difficulty and personal skill (Reeve, 2009, pp ). For an optimal experience to occur, the task has to be challenging enough and the individual has to feel highly skilled in carrying out that task, and this level of experience is called flow. High levels of personal skill and task challenge lead to a flow experience whereas low levels of both or an imbalance between personal skill and task challenge tend to lead to anxiety (very challenging task/low personal skill) or boredom (easy task/high skill). Illustrating this, it has been found that people more frequently enjoy challenging work or studies than unchallenging 6

17 2.2. FEEDBACK CHAPTER 2. LITERATURE REVIEW leisure activities. The level of enjoyment had from different levels of challenge varies between indidividuals: some enjoy low levels of challenge while others enjoy being overchallenged (Reeve, 2009, pp ). However, optimal challenge better nurtures the psychological need for competence than the enjoyment had from easy success. There are some qualifications for optimal challenge and flow to be experienced. The individual needs to feel that she is challenged, which happens when she receives performance feedback (Reeve, 2009, pp ). Further, there must be a certain level of failure tolerance in the challenging task, or else the fear of failure may cause the individual to avoid the task. Typically failure and success are equally probably in a setting of optimal challenge. 2.2 Feedback Feedback is information provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one s performance or understanding (Hattie and Timperley, 2007, p. 81). Feedback is among the most powerful influences on student learning. In order to allow for a better understanding of feedback, Hattie and Timperley (2007) conceptualizes feedback as a continuum of instruction and feedback. At one end the feedback is clearly separable from instruction, but as we approach the other end of the continuum the integration of feedback with instruction gradually increases. Hattie and Timperley (2007) identify three major questions that feedback should address and relate in order help reducing the difference between what the learner aims to understand and what the learner currently understands. These three questions are Where am I going?, How am I going? and Where to next?. These questions work best when addressed in relation to each other, so that it becomes clear what needs to be done or changed in order to reach the goal from the current position. Hattie and Timperley (2007) note that How am I going? is often answered by assessment or testing, but the focus should be on providing useful information in relation to a goal to the student (and teacher). Feedback related to whether a student has attained the pursued goals or not is the answer to Where am I going? and helps adjust effort and strategy. The answer to Where to next? connects the other two questions by providing intermediate goals or suggesting further possibilites for learning Feedback levels Hattie and Timperley (2007) claim that there are four major levels of feedback and that its effectiveness is directly affected depending on which level the feedback is aimed at. The four levels are described below (see Hattie and Timperley, 2007, pp ). 7

18 2.2. FEEDBACK CHAPTER 2. LITERATURE REVIEW Feedback about the task or product (FT). This includes feedback on whether the work is correct or incorrect as well as directions regarding what other information may be needed to complete a task. This level of feedback is often called corrective feedback. Feedback about the process (FP). FP is the type of feedback that relates to the underlying processes involved in carrying out a task. What strategies students use to detect their own errors is one area that FP could be given in relation to. Feedback about self-regulation (FR). Self-regulation is the process of controlling and directing oneself in order to reach one s goals. FR is related to the elements of this process. Feedback about the self as a person (FS). FS refers to commending or praising the person itself. FT can be an effective type of feedback, but if mixed with FS its effect weakens (Hattie and Timperley, 2007, 91-92). Further, if a student lacks information FT is less powerful than when a student has made an incorrect interpretation. The efficacy of FT is also determined by how complex the feedback is and how complex the task is, as well as whether the task is performed individually or in groups. Less complex tasks tend to benefit more from FT, and simple FT tends to be more effective than complex FT. Hattie and Timperley (2007) stated that there is considerable evidence that giving feedback in the form of written comments is more effective than giving marks or grades. FP, Hattie and Timperley (2007) noted, appears more effective than FT when it comes to enhancing deeper learning. However, the combination of process feedback and task feedback can be effective. Process-related feedback can also be effective when given as cues that help students use strategies for carrying out tasks and guide the student s information search. Feedback about self-regulation can help students assess their own performance and strategies in carrying out tasks (Hattie and Timperley, 2007). This self-assessment works best when students know when to ask for feedback from others (for example teachers or peers). Self-regulation also includes the ability and willingness to independently look for further information when interpreting feedback the skill of seeking help from others. The effectiveness of these skills and abilities are also influenced by the degree to which students try to get hints from others as opposed to complete answers. Hattie and Timperley (2007) also brought up the issue of attribution of success and failure in relation to self-regulation. Students failure to firmly attribute success or failure to a cause can lead to uncertainty and confusion, making unclear evaluative feedback have a negative impact on learning. Too much feedback focused on a particular level may lead to less positive or negative effects. For example, if a student receives too much FT then she may spend too much energy on overcoming the details in the FT 8

19 2.3. GAMES CHAPTER 2. LITERATURE REVIEW while missing the bigger goal and the strategies to achieve it (Hattie and Timperley, 2007, p. 91). The last type of feedback, feedback about the self or (often) praise, is rarely effective and should be avoided since it often distracts from the task, although it can positively influence the student if the praise is directed to some aspect of how the task were performed (Hattie and Timperley, 2007) Negative and Positive Feedback Although negative feedback can be more effective than positive feedback in some circumstances, it can be rejected or ignored if it contradicts students own perceptions, and worse, it can have a negative influence on motivation (Hattie and Timperley, 2007). Positive feedback, on the other hand, more likely leads to increased motivation and interest in the task. Selfefficacious students tend to be more receptive to negative feedback and less self-efficacious students tend to react negatively to negative feedback, but in some cases responses are heavily influenced by the presentation of the feedback. 2.3 Games What Defines a Game? Bernard Suits argued at length about what games are in The Grasshopper Suits (2014). The short version of the definition is that playing a game is the voluntary attempt to overcome unnecessary obstacles (Suits, 2014, p. 43). There are four components in the explicit definition: prelusory goal (a specific state of affairs, lusory means, constitutive rules (which prohibit more efficient means to reach the goal in favor of less efficient means) and lusory attitude (acceptance of the rules just to enable the activity). Another, perhaps more practical, definition somewhat similarly states four defining traits of games: rules, goal, feedback system and voluntary participation (McGonigal, 2011, pp ). The goal is what the game s players work to achieve in the game and the rules limit the ways a player can reach the goal. Rules help creativity and strategic thinking by removing the obvious ways to reach a goal, requiring exploration or more skill to carry out a particular task. The feedback system is what indicates to the player how well they are doing with regard to reaching the goal of the game. Some games have points, others have levels and progress bars. Voluntary participation, finally, means that any player willingly accepts and enters into the framework of the game. Because games are voluntary, challenging and stressful tasks or activities can be experienced as safe and pleasurable. Although they share some aspects, the two definitions can be used to illuminate game components from different angles. Note that Suits definition includes the notion that the means used to reach some goal must be 9

20 2.3. GAMES CHAPTER 2. LITERATURE REVIEW inefficient (constrained by the rules) unnecessary obstacles. This can help distinguish activities that have game-like elements and actual games Playfulness and Gamefulness There are a number of closely related terms for game-like system depending on the extent to which certain game or play features are employed. Deterding et al. (2011, p. 13) situated these in relation to two variables: gaming/playing and whole/part. The relations are represented by the table below. Gaming Playing Whole system (Serious) games Toys Parts Gameful design Playful design Table 2.1: Matrix of Game/Play Terminology Pure gaming refers to rule-bound, goal-oriented play whereas pure playing refers to free-form play (Deterding et al., 2011, p. 11). If only parts of a system creates a gaming/playing experience, the design is gameful or playful, whereas if the whole system is permeated by gaming or playing it s simply called a game or a toy, respectively. Serious games (and other games that have other motives than entertainment) are games characterized by a full game experience but where the intended goal is not merely entertainment but instead, often, education (Deterding, 2015). Studies have found that learning games become more effective if the learning task is part of a fictional world, such that there is real relation between what the player does and what happens in the game. Translating a word in order to fire a catapult exemplifies a task where there is no sensible relation between the task and the game outcome (Deterding, 2015). In contrast, an endogenous game has at its core challenges that require development of the skills desired from pedagogical viewpoint, also making the gameplay better from a motivational viewpoint. This type of challenge is called an intrinsically integrated challenge, and these have been proven more effective both motivationally and educationally. Deterding (2015) argues that the notion of intrinsic integration is useful also when creating a gameful design, but there is a marked difference in that the design has to support the activity that is the main purpose of the system, and in addition improve the experience through gameful design. If, for example, challenges are tacked onto the user activity it will likely result in frustration; instead the designer has to find the inherent challenges in the task, aspects of using the system that cannot be removed through better design. Deterding (2015) gives as examples that finding how and where to input information in an online form is not an inherent challenge, but writing a persuasive headline is. Ideally the inherent challenge is skill-based, so that competence improvement is possible. Put simply, gameful design is about 10

21 2.3. GAMES CHAPTER 2. LITERATURE REVIEW identifying inherent challenges and improving the ways in which a user goes through and experiences them, while game design is about creating the challenges themselves Learning in Games When putting an educational goal in a game or integrating it in a gameful experience, there are opportunities and dangers. The main opportunity, is to make education more motivating, but historically many educational software projects have tried to use games as extrinsic motivation while adding some educational elements on the side (see for example Jacob Habgood and Ainsworth, 2011). However, educational games have been proven effective if they incorporate the educational objectives as part of the core gameplay and the game world, such that those elements are intrinsically integrated (Jacob Habgood and Ainsworth, 2011). This is the same principle as the one outlined above for gameful settings that do not necessarily involve educational motives. Another criticism voiced against learning in games is that the learning merely extends to learning how to play the game, as opposed to acquiring knowledge from games that can be applied in other settings (Gee, 2005, p. 34). Against this, Gee argues that a subject without its associated activity is merely trivia, and that learning comes from and with doing. Some papers on dialogue games have been published (see for example Ravenscroft and Matheson, 2002), but I have not been able to find studies that satisfactorily address the aspects of motivation and how that relates to the game experience. For the purposes of this paper a learning system that does not motivate its intended users to use it, has little value. (Gee, 2005) noted a number of good learning principles that games use in practice. A number of these principles contribute to a player s feeling of agency, the sense that they are in control over what they are doing and creating. Related to this is the principle of production, stating that the player does not just passively consume, but actively participate in creating the experience or the world they participate in. The following list summarizes the idea of some of the other principles. Risk Taking: Players can take risks in a safe setting, failing means another chance to try Well-Ordered problems: Students learn better if problems are arranged so that earlier problems build skills to use for solving later problems Challenge and Consolidation: Players are offered challenges of suitable difficulty, and are then allowed to consolidate the skills before proceeding to harder challenges Just-in-Time: Information or tools are offered when needed, in the right context 11

22 2.3. GAMES CHAPTER 2. LITERATURE REVIEW Situated Meanings: Games offer words and other information in context, giving them situated meaning Explore, Think Laterally, Rethink Goals: In contrast to reaching a goal as quickly and efficiently and possible, games encourage exploration and lateral thinking Cross-Functional Teams: Players often master their own specialties, making them especially valuable to the group Performance before Competence: With the help of game tools and other players, players who are not yet competent can still perform Playcentric Design Fullerton (2008) has put forward an approach to game design called playcentric design. In this design scheme, the player experience is central, and playtesting is carried out throughout the steps of the design process in order to evaluate and refine the game. The process laid out by Fullerton starts by specifying the desired user experience and then brainstorming game concepts and mechanics that may be able to result in the previously specified user experience. These concepts are then tested with players. The subsequent steps in the process include physical prototyping, software prototyping, design documentation and aspects I will ignore for present purposes, related to the funding, production and release of the game. Both stages of prototyping involve playtesting, and it is important to the process that playtesting is done as part of an iterative process (Fullerton, 2008, pp ). The iterative process in this context means a loop from generating ideas, formalizing them, to their testing and then evaluation of the results. If the evaluation shows problems with the design, the designer goes back to the idea stage and starts over, and this procedure is repeated until the desired user experience is achieved. Fullerton (2008, ch. 7) emphasizes the importance of physical prototyping in order to test out ideas as soon as possible and save time in the long run, while avoiding being distracted by programming or other aspects that are not important to the high-level concepts in the beginning. It can be noted, however, that the examples brought up in the chapter on prototyping are based around a game world, characters or cards. Could the method of physical prototyping still be useful in the context of an inherently computerbased activity, such as written real-time dialogue? There may be instances where certain parts of a system, such as the interface, can be prototyped with pen and paper, while certain forms of interaction may be more difficult to prototype well without a digital environment. Additionally, a physical prototype may be doable in some cases but not justifiable in terms of cost and effort compared to using a simple digital prototype. Playtesting and iteration is described by Fullerton (2008) as a narrowing process where in the beginning (in the concept phase) it is possible to make 12

23 2.3. GAMES CHAPTER 2. LITERATURE REVIEW Figure 2.1: Iterative process flow (Fullerton, 2008) changes to the basic gameplay, and as the game design process progresses, opportunities to make changes decrease and gradually become limited to more superficial features. The people who test the game, the playtesters, can be the game designer, family and friends and strangers. However, it is explained that different stages of the game has different requirements for playtesters. In the beginning, it may be enough to have the core principles tested by the game design team, whereas it is especially important for the quality of feedback to have people who do not know the design team playtest (Fullerton, 2008, p. 251). When it comes to actual playtest sessions, Fullerton stresses that the playtesters should receive little or only minimal information before playing the game. This way, you gain a fresh perspective. Fullerton (2008, pp ) describes a minimun script to follow for each playtesting session, summarized below. Fullerton (2008) lists a number of playtesting methods, in addition to the previously mentioned playtest with follow-up interview, including the following: group testing, standardized feedback forms, data hooks that record usage data, and open discussion. Feedback forms and data hooks have the advantage that larger quantities of data can be gathered and compared (if there are enough users), while other methods generate more qualitative data. Fullerton notes the tendency for great difference between group answers and individual answers and that group dynamics are generally bad for evaluation but helpful for generating ideas. 13

24 2.3. GAMES CHAPTER 2. LITERATURE REVIEW A Playtest Session Introduction: Very short introduction of you and your project Warm-up Discussion: Short discussion to find out a little more about the playtesters and their game preferences Play Session: This is where the actual play takes place. Playtesters are asked to play the game for a while, not long enough to make them tired, and the designer observes how they interact with the game. Playtesters should be encouraged to speak their mind about what they are doing during the play. Discussion of Game Experience: With a set of questions depending on the nature of the game, carry out short interviews with the testers to find what they thought of the game s challenge, difficulty, appeal and so on. Wrap-up Figure 2.2: Summary of minimal contents of a playtest session (Fullerton, 2008, pp ) Design Lenses Design lenses phrase a single design perspective that is both inspiring and guiding the analysis of games, and although design lenses were first developed as an aid for game design they have later been adopted for interaction design as well (Deterding, 2014, p. 319). Lenses provide more general guidance compared to design patterns, which prescribe a particular and proven solution to a specific, often domain-bound type of problem. There are three components to each design lens: a name that is easy to remember, a statement of a design principle and the reason behind it and finally, a number of focusing questions that help the game designer adopt the lens perspective (Deterding, 2014, p. 319). The concept of design lenses was first developed by Jesse Schell (see Schell, 2008) in the book The Art of Game Design (Deterding, 2014, p. 319). The book contains a hundred different design lenses, spanning many aspects of game-making and providing many useful starting points, but for applications that are closer to gameful design or educational/serious games as in the case of this thesis, a number of the lenses are not directly applicable. To give an idea of what some of the applicable lenses provide, here is short list of selected lenses with partial descriptions from Schell (2008): Surprise (lens #2): Helps reason about surprises by asking questions about how they are related to rules, artwork and other aspects of the game. 14

25 2.3. GAMES CHAPTER 2. LITERATURE REVIEW Endogenous Value (lens #5): Helps reason about what is valuable to players in the game, how to enhance those aspects and how the game s value relates to the player s motivations The Player (lens #16): Helps reason about how players behave, what they like, expect and more, putting the the designer in the perspective of the player Flow (lens #18): Helps reason about aspects that can lead to experience of flow Needs (lens #19): Helps reason about how a game fulfills basic human needs and how to do it better Challenge (lens #31): Helps reason about what the game s challenges are, their difficulty, variety and change throughout the game Competition (lens #36): Helps reason about competition in the game, based on measurement of player skill, how players of different levels can play the game and why people want to win the game Balance (lens #47): Does the game feel right? This lens focuses on the big picture and helps the designer step back to get a feeling of it. Visible Progress (lens #49): Helps reason about what progress in the game means, if the amount of progress the game allows for is appropriate and how to make progress visible Although this is a small subset of all the lenses Schell suggested, we can see that several of these, such as #2, #5, #19, #31 and #49, can be directly linked to SDT because of how they relate to the fundamental psychological needs formulated by SDT. In cases where the experience being designed is not a full game, design lenses may not provide a clear hierarchy or starting point. Deterding (2015) proposes that a different type of lens can solve this problem when designing gameful experiences: the lens of skill atoms. This lens provides support for analysis in terms of the smallest elements of a game that cannot be broken down further without losing their inherent game characteristics. A skill atom as described by Deterding (2015) is a feedback loop between the user and the system. There are actions and objects which the user interacts with, these events are processed by the system and its rules resulting in some change of the system that is shown as feedback to the user. These components are organized around the central challenge, and additionally there are the goals (often stated explicitly by the system) and motivation of the user. These components are called skill atom particles. In order to formulate the complete lens, Deterding (2015) further incorporates the idea of intrinsic integration, the idea that the learning content or material is embodied in the game world and part of the core gameplay. 15

26 2.3. GAMES CHAPTER 2. LITERATURE REVIEW In other words, the challenge has to be intrinsic to the activity. Thus, the final lens put forward by Deterding (2015, section 4.3) is as follows: The Lens of Intrinsic Skill Atoms In pursuing her needs, any user s activity entails certain inherent, skill-based challenges. A gameful system supports the user s needs by both directly facilitating their attainment, removing all not skill-based challenges, and by restructuring the inherent challenges into nested, interlinked feedback loops of goals, actions, objects, rules, and feedback that afford motivating experiences, competence in specific. What motivations (might) energize and direct the activity? What challenges are inherent in the activity? (What challenges can be removed through automation or improving usability? What challenges remain that the user can learn to get better at?) How does your system articulate these inherent challenges in goals? (How might it articulate them to connect to the user s needs and motivations?) What actions does your system offer the user to achieve these goals? What are the objects the user can act on to achieve these goals? What rules does your system articulate that determine what actions are allowable and what system changes and feedback they result in? What feedback does your system provide whether the user s actions were successful, and how much progress the user has made towards her goals? (How might you make this feedback clear, immediate, actionable, speaking to the user s needs and motiva- tions, affording a sense of competence?) Figure 2.3: The lens of intrinsic skill atoms (Deterding, 2015) Deterding (2015) further puts forward a specific method in form of a number of specific steps for gameful design, where the lens of intrinsic skill atoms has a central role. Deterding notes that the method has not been formally evaluated, but it has been developed and iteratively refined in connection with practical use in 19 design projects and workshops. As a useful point of reference, I reproduce the innovating mode of the model here. See Deterding (2015, section 4.4) for further details. It is worth pointing out that this method assumes that there is already a particular user activity that the new design is centered around, whereas when designing a game the core activities themselves are also created. Also worth noting is that this gameful design method somewhat overlaps with the playcentric design method. Both methods highlight the importance 16

27 2.3. GAMES CHAPTER 2. LITERATURE REVIEW 1. Strategy (a) Define target outcome and metrics (b) Define target users, context, activities (c) Identify constraints and requirements 2. Research (a) Translate user activities into behavior chains (optional) (b) Identify user needs, motivations, hurdles (c) Determine gameful design fit 3. Synthesis (a) Formulate activity, challenge, motivation triplets for opportune activities/behaviors 4. Ideation (a) Brainstorm ideas using innovation stems (b) Prioritize ideas (c) Storyboard concepts (d) Evaluate and refine concept using design lenses (optional) 5. Iterative Prototyping (sequence repeated until result is satisfactory) (a) Build prototype (b) Playtest (c) Analyze playtest results (d) Ideate promising design changes Figure 2.4: Main steps of the gameful design model (Deterding, 2015) of iterative prototyping and playtesting in order to gauge and respond to the emergent dynamics of the system. Since the gameful design method presumes an already existing non-game activity, however, the method additionally prescribes ways of handling that activity such that gameful design can be integrated into the system. Deterding (2015) notes that this method focuses intentionally on satisfying the need for competence through skill-based challenges, and that this is only one of a number of ways to create enjoyable experiences in gameplay. 17

28 2.3. GAMES CHAPTER 2. LITERATURE REVIEW The MDA Framework: Mechanics, Dynamics and Aesthetics MDA is a framework for understanding and analyzing games (Hunicke et al., 2004). The framework is intended to guide game design by illuminating how different aspects of games come together to create the end-user experience. In MDA, the content of a game is seen as the result of the interaction between the player and the game. This highlighs the idea that the players experience is the result of a complex process that is influenced by everything from a game s data representation and algorithms to its mechanisms of interaction. Figure 2.5: MDA components MDA conceptualizes games as a series of components. Rules (and their processing) lead to a system, which enables experience of fun (Hunicke et al., 2004). In game design terms, these components correspond to mechanics, dynamics and aesthetics. Mechanics refers to a game s fundamental setup and constraints, in terms of data structures and algorithms. When played, the game s mechanics and players interaction with the system and each other gives rise to the dynamics of the game. Finally, the aesthetics describe desired emotional responses among players when they play the game. These emotions manifest different ways in which a game is fun, but since there is a multitude of ways to experience and describe fun, Hunicke et al. (2004) gives the following incomplete vocabulary to help describe fun experiences in games: Sensation (sense-pleasure) Fantasy (make-believe) Narrative (drama) Challenge Fellowship (social framework) Discovery (uncharted territory) Expression (self-discovery) 18

29 2.3. GAMES CHAPTER 2. LITERATURE REVIEW Submission (pastime) Starting from descriptions of desirable of aesthetics in terms like these, the designer can see more clearly what kind of dynamics support the desired aesthetics, and in turn, how to lay out the mechanics of the game to enable the appropriate dynamics. MDA thus provides a way of using different levels of abstraction to reason about games and game experiences. The framework supports an iterative approach to game design (Hunicke et al., 2004). This means that tuning and improving are integrated parts of the development process, which allows for reacting and adjusting to the emergent dynamics and aesthetics that result from a particular set of mechanics. The suggested vocabulary for describing fun does not explain why certain types of experiences are fun, but against the background of motivational psychology (see section 2.1) one can reason about how and why certain elements can result in these experiences. Challenge and fellowship can be directly associated with the psychological needs for competence and relatedness, respectively. The other items can in some cases be indirectly tied to these explanations Evaluation Methods In addition to using design lenses or playtesting for analysis of games or gameful experiences, there are a number of heuristic evaluation methods that have been found useful. One of them is the PLAY list of game usability heuristics (Desurvire and Wiberg, 2009). PLAY was created to help developers of games in the genres of real-time strategy, first-person shooters and action adventure. The heuristics differentiate between good and bad games in these categories, as judged by players in the study. An attempt to find heuristics that are useful in relation to serious games in particular is reflected in a study by Jerzak and Rebelo (2014), where a few different heuristics sets are compared. The authors point out that for serious games these are all lacking in some respects, such as learning content, social interactions player concentration and responses. We can find items in these heuristics sets that are relevant for evaluating serious games and gameful systems, but the coverage is far from complete. 19

30 Chapter 3 Method This chapter details the development and design process and the evaluation of the final system. 3.1 Process The gameful design process set forward by Deterding (see section 2.3.5) does not perfectly match the circumstances of this thesis, particularly because it assumes that there is already a user activity and that the objective is to facilitate and make the activity more interesting as opposed to defining and shaping the activity itself. This thesis concerns the application of a development method to the problem domain of conversation practice in a broad sense, so the activity did not have an well-defined form at the outset. Reflecting this, I adapted Deterding s gameful design process so that it does not require pre-defined user activities or contexts. In line with this, the synthesis step was dropped entirely. Furthermore, in order to simplify the process, some steps were merged and some removed. Both in MDA and playcentric design, the experience (or aesthetics) is the starting point. We do not know which exact mechanics will create a certain experience, so this is a hypothetical starting point that is tested and refined as part of the process. The analysis of ideas and prototypes were guided by using design lenses as well as the MDA framework to better understand how different aspects of the system are connected to other aspects. Design patterns were not employed due to their being more context-bound, whereas lenses allow for general ways of view one s design. In addition to using the above methods for analysis, the models from motivational psychology are related to during development, in order to anchor the practical principles from game development. The playtesting sessions were in the majority of cases carried out with the tester at a remote location. In-system testing was carried out in Chi- 20

31 3.1. PROCESS CHAPTER 3. METHOD Adapted Gameful Design Process 1. Strategy (a) Define target users, target outcome, metrics and desired experience. In line with playcentric design, deciding on a desired experience was added as one of the strategic considerations. (b) Identify constraints and requirements. This step is only partially applicable, since there are no business requirements. 2. Research (a) Identify user needs, motivations, hurdles 3. Ideation (a) Brainstorm and prioritize ideas. (When going back to ideation, use design lenses to evaluate design and generate ideas.) (b) (Storyboard concepts) (c) Evaluate and refine concept using design lenses 4. Iterative Prototyping (sequence repeated until result is satisfactory) (a) Build prototype (b) Playtest. As described in figure (c) Analyze playtest results (d) Ideate promising design changes Figure 3.1: Adapted gameful design process nese, English, Japanese and Swedish, with a total of six testers (excluding myself). In each playtesting session I played one of the two roles available. Testers were selected from the group of my acquaintances or second-degree connections that seemed to fit into the system s target group; there was no random selection Tools Concept sketches were made using pen and paper and sometimes Inkscape and GIMP, but all interactive prototypes were programmed using standard web technologies: HyperText Markup Language (HTML), Cascading Style Sheets (CSS) and Javascript. The Meteor Javascript framework was used in order to make development faster, mainly by providing reactive computations and communication, database handling and view rendering. Reactivity 21

32 3.2. EVALUATION CHAPTER 3. METHOD (see Meteor Development Group, 2015) meant real-time conversation over the network could be easily implemented without having to meticulously write communication and synchronization code. The system was deployed on a publicly accessible web server in order to allow for playtesting with remote testers. Most of the testing sessions were carried out with playtesters in remote locations, and follow-up conversation and interviews were conducted in written format. Screen sharing was not used, but in cases of technical issues (mostly related to browser compatibility) screenshots were taken and shared. 3.2 Evaluation Playtesting has been carried out throughout the process in order to evaluate and improve the system. As a final evaluation of the resulting system, one last round of playtesting was carried out. For this last round, more thorough interviews were carried out with each participating playtester. The interviews did not follow a standardized script, but as far as applicable I tried to gather the playtesters opinions on at least the below points. These points focus on competence (see 2.1) and related concepts, as that was the motivational factor most targeted and discussed in the development process. Competence and challenge Was the game repetitive or boring? How difficult was the game (as learner/mentor)? Was it challenging enough? Could you feel your progress? Feedback and marks Were the progress indicators helpful? Was the feedback you received clear? Was the feedback you received useful? Were marks fair? Overall feelings and comments What were the best and the worst aspects of the system? Was it fun to use? Interesting? In the evaluation round, all playtesters had participated in at least one previous round of playtesting. Note that not all testers tested all aspects of the system, and so not all points were applicable in all playtest sessions. 22

33 Chapter 4 Process 4.1 Strategy and Research The intended users of the conversation practice system are those learners of a language who have reached at least a basic level of conversational proficiency. Except for this basic constraint, the system is intended to be useful to virtually all levels of learners. Outcome and experience are, unlike in most games, different. The desired outcome from a player s use of the system is that player s increased competence in communication in a particular language. In terms of experience, the main objective is to make players feel competent (fulfilling the psychological need to feel competent) and their increase in competence, made possible by a sufficiently challenging and supporting environment. A secondary goal in terms of experience is social interaction, or the sense of relatedness. Meaningful interaction with other human players will satisfy the psychological need for relatedness as well as, hopefully, make the system less predictable, more surprising and in the end interesting Constraints and Requirements Because I am only one person with limited resources, the process and end result will necessarily be limited in the amount of effort that can be spent on reaching the goals of the project. It would for example be infeasible to attempt to develop an 3D immersive game world as a setting for the activities in question. Rather, I set out to create a system that is minimal in technical requirements as well as graphical expression, enabling focus on core mechanics that create the desired results User Needs, Motivations and Hurdles The user need being addressed is improving conversation skills in some particular language. There are numerous motivations for learning a second lan- 23

34 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS guage: friends or family speak it, moving to another country, work, study, travel and culture among others. Thus there is great diversity in motivations, but also in learners, considering differences in culture and behavior between learners. Targeting all different motivations in one system may be difficult, but conceivably it could be done for example by dividing a system into sections where content or context is appropriate for a particular motivation. Finally, common hurdles in language learning include lack of resources, such as time to study or money to spend on courses or instruction. From another perspective, one may lack the contacts or access to native speakers to practice with. More generally, there is the lack of motivation in the sense of something that actually makes somebody want to study. If somebody s motivation to learn a language is the work benefits that would result, that may not be a strong enough motive to make the person feel like studying at a given time. Hurdles in learning may also be related to one s study environment or the structure of one s studies, for example whether one has set clear goals or not. 4.2 Iterative Process st Iteration Brainstorming generated a number of ideas, a few of which are listed below. Most of the ideas that surfaced were existing analog games. 1. Picture response game. Teams of two work independently to find the best description of a picture shown. A judge assigns scores to the descriptions, the team with the highest score wins. 2. Guess the word. One player decides a word and writes it down. The other player asks binary questions until he/she can guess the correct answer. 3. Keyword sentences. A number of keywords are given, and these are used by players to form sentences. 4. Pictionary. A player gets a word and then has to draw a picture representing that word so that her teammates can guess the word. 5. Umigame no suupu. Japanese game in which one player knows a story, and the other participants have to figure out the story by asking the first player binary questions. Most of these games and activities do not necessarily encourage conversation and therefore are not good fits. The first item does, though, and may have enough variation (depending on the pictures) to make it interesting. To make it simpler to implement and analyze, I decided to strip the competitive element from the first prototypes. 24

35 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Figure 4.1: Sketches Figure 4.2: Game finished (one team) The figures show early sketches of the picture description game. I implemented a simple prototype based on the one-team model, but without scoring, just to find out whether the core activity is something that people could actually consider participating in. This prototype is shown in figure 4.4, with the conversation and image blurred. Although the prototype s functionality was limited to conversation about a picture, a brief playtesting session indicated that the concept itself is worth developing further and that the graphical design made the functions of different elements unclear. For a more functional prototype it should be clear especially how and when the box for supplying a final description of 25

36 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Figure 4.3: Game finished (two teams) the picture is to be used. No scoring or other types of feedback separated from the conversation itself were available in this prototype nd Iteration The second prototype introduced some new features as well as being overall more usable. Specifically, (1) users could now comment on individual messages in the chat (such as to provide corrections or ask questions regarding the language of a specific message), and (2) upon completing a game session the final description can be scored and commented on as seen in 4.5 on the left-hand side. Also, I had set up server deployments so that the application could now be tested online. The playtesting was carried out more thoroughly in this iteration since more aspects of the game were usable. The testers were mostly positive about the experience and the usefulness of this type of game. The main problems experienced, except for bugs, were related to lack of clarity in how to use the application. For example, there was no indication as to how long picture descriptions were required or accepted. Also, there was no way to know whether the other participant was actually viewing the game screen or 26

37 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Figure 4.4: First early prototype Figure 4.5: Second prototype had already left the game or closed the browser window. One tester noted that the scoring was unclear since it didn t show the maximum possible score, just the score awarded. Most of the issues found in this round of playtesting can be addressed relatively easily. The issue of not knowing how much is expected of the answer can be fixed by indicating the length of the answer, perhaps in the 27

38 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS form of a character count, also indicating in real-time how many characters remain. The scoring can be improved by adding outlined boxes or stars to the filled ones, indicating the maximum score. Analysis and Ideation The perhaps most difficult issue is the one of indicating the other participant s presence. Deactivating the chat input field would probably be desirable, so that users cannot send messages when the other participant has left the game and cannot see the messages, but this alone would not be a clear indicator. Showing a green circle next to a name or username is a fairly common way of indicating presence online and should therefore be easy to interpret. Since chat messages are aligned to the left or right depending on user, it could be appropriate to display the usernames and their presence status above the chat window with a similar alignment as the messages. From a psychological perspective, as outlined in 2.1.3, the presence of other people should be made clear, but also the system should encourage interactions with a small number of people instead of randomly assigning partners. These considerations will be expanded on in a future iteration. Figure 4.6: Sketch of corrections Competence and development can to some extent be indicated by the scoring/assessment, but progress is better shown as the change of results over time. As a first step in progress indication, there should be a way to see one s past performance. As a start, this can be a summary of past games, showing scores for each game and linking to the full conversation. The lens of visible progress asks if there is enough progress and how much of 28

39 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Figure 4.7: Another alternative representation of corrections the game s progress that is visible, and with this change only a small part of the player s progress can be seen. There can be plenty of actual progress in terms of increased proficiency using a language, and that is to some extent indicated by a score for each answer. Another common form of indicator of progress is the player s current level. Levels can be implemented as different picture categories, so that if further levels are unlocked (e.g. by attaining a certain level of experience), new categories become available. Another benefit of a level system, if the levels the player has not reached are shown as locked with their content hidden, is that there will be an added element of curiosity and working towards unlocking a new level. From the perspective of Schell s Lens of Curiosity (Schell, 2008, ch. 3), we can think of the environment as putting questions in the player s mind, and if the player cares about those questions his/her curiosity will be aroused rd Iteration In this iteration, the major change was the introduction of a level system with different picture categories belonging to different levels. The idea was well received by the testers, but the actual process of unlocking and playing categories from different levels has not yet been tested. One tester noted that it wasn t clear how to unlock a particular level, and indeed it was only possible to see the requirements for a level by first clicking the padlock icon. A few minor changes were made this iteration. The scoring/assessment was made clearer by adding indicators showing the maximum possible score, and past played games were made viewable from a history view accessible from the left-hand menu. Analysis and Ideation The most important realization from this round of testing was that new players have no clear way of knowing what is expected from them both in 29

40 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Figure 4.8: Levels in third prototype terms of mentoring and coming up with a picture description as a learner. Giving clear indications of what is expected is important not only to avoid trouble and discomfort, but also because mismatched expectations may create dissatisfaction with feedback and the help received from mentors. This can be described as an issue with the system s mechanics. The mechanics allowed for different strategies of interaction with little guidance as to what was the desired mode of interaction. As a learner, one might (1) choose to first write a description of the picture and then get feedback from the mentor, (2) first discuss the picture with the mentor and then write about it, or (3) mix the two previous strategies. Thus, the dynamics will vary greatly depending on players. Seen in the light of the lens of limited choice (Deterding, 2015), this could be an example of providing too much choice. How could the number of strategies be limited? The system could enforce a particular sequence of actions, like first discussing the picture and then typing the actual description into the answer box th Iteration The issue with choice in how to play was addressed in this iteration by adding a slightly different type of mode, independent mode. In this mode, writing the response to the picture is the final step and is carried out by the learner alone when the discussion time has run out. This mode was made available in addition to the original mode. 30

41 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Two testers tried and compared the two modes, and although more testing is warranted in order to draw a firm conclusion, both indicated a preference for the independent mode. One tester noted that it was easier to focus and also that it felt more like he was able to fully use his language ability when writing the picture description in the independent mode. Analysis and Ideation The new independent mode was a positive introduction at this stage but needed further testing to validate. Ignoring relatively minor and straightforward usability improvements, the most important elements going forward are those aimed at (1) helping players feel their increase in competence and (2) improving the social context and interactions. These are both important aspects of spurring motivation, as seen in section 2.1. (1) will be dealt with here. Currently indicators of progress are limited to the display of a player s experience points and how many levels the player has unlocked. Furthermore, these indicators are related only to how many games a player has played, not the outcome of the games. While the exact scoring rules could be adjusted in countless ways, the relation between experience points and performance could be introduced by increasing a player s experience points whenever the player s game result is scored favorably. Those points must make a tangible difference while the judgements implicit in scoring need to be reasonably fair. These considerations are illuminated by Schell s lens of judgment (Schell, 2008, ch. 9). Another issue is keeping the scores balanced. The experience points gained can be seen as a reward, and using Schell s lens of reward, we can consider whether the reward is balanced in relation to the action that led to it (Schell, 2008, ch. 11). Is it actually understood as a reward, or do players not care? Answers to these questions may reveal themselves more clearly over time, but will be explored during playtesting. A clear relation between performing well and experience points can be indicated to the user as a notification. Section 2.2 noted that positive feedback typically is a better motivator, accordingly only positive feedback shall be indicated as notifications. Negative feedback can be useful, but I will avoid connecting such feedback to the scoring mechanism since scoring can have unpredictable effects on the dynamics of the system even without that added complexity. Interim goals or suggested actions are other components that can help a player s progress and motivation [see Deterding s lenses interim goals/ next best actions]. These could also take the form of challenges. When it comes to improving one s language skills, there are often hurdles when mastering grammar constructs, word usage, idioms and so on. Overcoming these types of hurdles could make good, bite-sized interim goals, provided that the relevant hurdles for each individual players can be discovered. Designing and implementing algorithms to find and mark a wide variety of errors or problems in a player s texts would be exceedingly complicated for 31

42 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS the purposes of this thesis project. Instead, mentors could be asked to find and mark the problems they find. Mentors cannot be expected to always accurately find and mark every problem, but an acceptable level of accuracy is probably attainable. Also, the system could help mentors by providing suggestions when available. The priorities for next iteration are as follows (minor changes and fixes excluded). Implementation of experience increases as a function of the scoring of picture descriptions Marking problems and displaying them th Iteration In this iteration, a tool for marking problems or errors was implemented as outlined above. The tool allows mentors to mark parts of text written by a learner, and comment or explain what was good or bad about it. Feedback on a specific text snippet was indicated by a highlight, and clicking the highlight would show the feedback. All feedback were also made available from the system s main screen. Figure 4.9: Feedback tool Playtesting revealed some usability issues with the feedback tool, but the overall concept worked well. The feedback page had different reactions, one tester thought it was a great improvement and worked well overall while another tester found it unclear and not particularly useful. In particular, the different types of feedback were not easy to distinguish from one another. The other main improvement in this iteration, bonus experience points based on the scoring of picture descriptions, had an interesting reaction from one tester. After receiving high marks on a very short and fairly simple 32

43 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Figure 4.10: Correction Figure 4.11: Feedback page text, the tester jokingly made some remark about the near-native level of his writing, playing on the difference between his actual skill level and the skill level that the scoring indicated. Analysis and Ideation Scoring a text according to a set of scoring rubrics corresponds to feedback about the task (FT) as described in Simple FT tends be more effective than complex FT, but the complexity of the task also correlates 33

44 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS with FT efficacy. Furthermore, avoiding negative feedback may be desirable depending on the person and the skill level a person has reached. In sum, there are a range of factors that determine the efficacy of this type of feedback, and a thorough evaluation is beyond the scope of this thesis. However, as imposing scoring on all users may risk demotivating some, this feature should be optional. Regarding the feedback shown in the Feedback page, the different types of items shall be more clearly separated. This will be done by putting clear labels on each item. Note that there are feedback types that can be seen as negative ( corrections ) and positive ( praise ) and that the user can choose which of these to display, which may help mitigate negative effects on motivation of negative feedback. Further, to help mentors give better feedback and to help learners receive better feedback, it should be possible to indicate which feedback is helpful. This can be done with some simple actions, and for next iteration I ll try out star and move to trash actions, where starring an item will send a notification to the mentor who gave the feedback. Another issue is the accuracy and precision of scoring. Considering that the people who carry out scoring will have no training, some common set of guiding criteria is needed. Such criteria, if phrased well, would mitigate the problem of wildly varying scoring assessments. For the sake of having a functional set of criteria to work with, I created a temporary, simplified set of guidelines partially based on the IELTS 9-band scale for writing. 1 Looking back at the lens of intrinsic skill atoms (2.3.5), we can find more questions to think about and address. One question is how the user s current progress towards some goal is presented (there is some overlap with the game design lens of visible progress in the same section), and at present the system only shows what level a user has attained (albeit indirectly) and experience gained. Since unlocking the next level can be seen as intermediate goals, one type of meaningful progress would be how far on the way to unlocking the next level a player has come and how much remains. A simple progress bar can display this th Iteration The redesigned feedback page worked better this time when testing with users, it was considered more clear and useful, taking the starring functionality into account. Nobody made any specific comments about the removal of some superfluous content (showing incorrect text selections and corrections a second time). A next level progress bar was implemented, but its effects were not thoroughly tested. One user, when specifically asked about it, thought it was motivating and helpful. 1 Band descriptors accessed from ( ) 34

45 4.2. ITERATIVE PROCESS CHAPTER 4. PROCESS Figure 4.12: Improved feedback page Figure 4.13: Points remaining to reach the next level The detailed scoring and criteria introduced had a somewhat negative reaction. It was seen as too strict, and it was suggested by one tester that there could be different criteria depending on the users current language skill, so as to avoid demotivating a beginner by handing out a low score. Analysis and Ideation The scoring turned out to be problematic but is quite important to do well. Scoring that seems arbitrary is not meaningful and will be less motivating or even demotivating, and as noted in section scores and grades 35