A mobile gamified learning environment for improving student learning skills.

A mobile gamified learning environment for improving student learning skills. Group 5 Leave it blank Leave it blank Leave it blank Leave it blank Leave it blank ABSTRACT Information technology is becoming a major key instrument to measure the overall capabilities of students. In multiple Dutch secondary schools monitoring systems have conquered an influential position. Through those systems teachers are able to track students grades individually. Although these systems are well implemented, they cannot manage custom learning objectives, which may consist of more guidance in planning and learning. For this study, a system was developed which may help both students and teachers to make learning and planning more fun, interesting and interactive. Various experiments were conducted to reveal the viability of this common objective. Author Keywords Mobile technology, gamified learning, students. INTRODUCTION The quality of education in the Netherlands is deteriorating. Students opt more frequently for planning and learning guidance, which is available in specialized classes, usually outside school [4]. Secondly, the use of smartphones during class results in students losing their concentration. Teachers often do not know how to tackle this problem efficiently [4]. All in all, learning and planning can be made more fun, interesting and interactive for both students and teachers. Learning can be distinguished into formal learning and informal learning. Formal learning combines highstatus, propositional knowledge with learning processes centered upon teaching or instruction, and is located within specialist educational institutions, such as schools, colleges or universities [1]. Informal learning concerns everyday social practises and everyday knowledge, and is seen as taking place outside educational institutions [1]. In this study, mobile technology is being used to create a positive impact on the control of and interaction with the curriculum. Furthermore, education is a vital part of our society. Making learning more fun could improve the quality of education as higher grades could be achieved. Moreover, the curriculum will unconsciously be understood better as topics are not learned shortly before an exam and forgotten soon after. In other words, topics will remain longer in students long-term memory making them more intelligent. On behalf of reaching the specific goal of letting students control and interact better with the study material, an application was developed in which students, before each class, answer a certain amount of questions of the curriculum of each class. With this application, teachers have the possibility to obligate students in answering the questions before each class in order to properly and timely prepare students for that specific class. The advantages for students are the possibility to earn bonus points, which, for instance, could be useful for exams. In a previous study, Hwang and Chang [3] investigated this problem and developed a mobile learning environment, which enabled participants to learn in an interactive manner. This study showed that the participants who used the mobile learning application scored significantly better on learning tasks than the control group. The essence of mobile learning is accessing information and knowledge anywhere, anytime from devices that learners are used to carry everywhere with them [11]. According to Gikas and Grand [2] student participants described many advantages in which the mobile devices assisted in their learning. These advantages are accessing information quickly, communication and content collaboration, variety of ways to learn, and situated learning. According to Kafiulo [5] mobile phones were the most accessible technological tools at the school and at the college. Students, pre- service teachers and college instructors favoured the use of mobile phones for learning [5]. Soderqvist, Hardell, Carlberg and Mild [9] found that 83,8% of the children in their study, between the age of 7 and 14 uses a mobile phone. Girls generally reported more frequent use than boys [9]. Su and Cheng [10] demonstrated that incorporating mobile and gamification technologies into botanical learning process could achieve a better learning achievement than using nongamified mobile learning and traditional instruction. Based on the information mentioned in earlier research this research investigates the opportunities to build a complete gamified mobile learning app, which can be implemented in the curriculum and can be used by both students and teachers. In the following paragraphs the data gathering process will be described, followed by the results of the evaluations. Finally, the discussion and conclusion contain a reflection and explanation of the results. 1

USER RESEARCH Summary To reach the goal of letting students control and interact better with study material, their input was necessary. This was achieved by interviewing students in the first grade on a high school pre-university course. It is interesting to research this age group because these students are still unschooled in planning and did not have a lot of training in the field of independency. The purpose of this research phase was to gain insight in how the intended target group struggles with the problem and how it can possibly be solved. First of all, students were asked to give answers in an interview. The questions researched how and how long the students work on their homework. This method was used in various similar studies and appeared to be valid. Secondly, the students worked on a mind map containing their thoughts on homework. Through the mind mapping method an answer was given to the question how students feel about making their homework. Moreover, this method was used to get a possible solution on how to influence the sentiments of the students. Although it was not necessary for the students to motivate their answers, they provided feedback to each other, which was helpful to understand how they feel about making homework. Finally, the answers yielded some practical insights, which have been used for developing products in another phase in this study. The experiment was conducted in a first grade class of a Dutch secondary school consisting of 24 students. This amount of participants provided this research with a sufficient level of diversity regarding different points of view such as differences between sexes and cultural diversity. Figure 1: User research mind mapping Method The goal in this research was to stimulate first grade high school students with planning. This was achieved by researching students in the first grade of pre-university course on high school. Since those students are forced to be independent, it is believed that the app could help those students by guiding them. The students answered questions in the app about the curriculum of their course on a regular basis. Those questions in the application are asked in an entertaining manner, also called Gamified Learning Approach [10]. Through this approach the students learn to unconsciously prepare for the next class in an alternative way. The experiment was conducted in a first grade class of a Dutch secondary school consisting of 24 students, as previously mentioned. This amount of participants provides this research with a sufficient level of diversity regarding different points of view such as differences between sexes and cultural diversity. In the Netherlands, first grade students are around 12 till 14 years old. It was interesting to research this age group because the students are still unschooled in scheduling and did not have much training in the field of independency. Older students are expected to have developed these skills already and could thus reject this app. Furthermore, first grade students are not forced to be independent and are guided by their teacher. The key elements which could aid in achieving the goal of this research, was to conduct interviews with students and let them make a mind map with their thoughts on homework. The interview method has been chosen because the students can be asked directly on the topic of this project. In similar studies various researchers asked questions to students too, but in those cases the scientists used a questionnaire [3;9], which was combined with interviews in order to investigate certain subjects more thoroughly [5;7;8;10]. The second method, mind mapping, has been chosen because the students could think about homework and related topics in a group in order to give additional thoughts in a playful manner. Regarding the available amount of time for the experiment a focus group was conducted in five groups of four students where each group was supervised and interviewed by one group member, one of the researchers. The limitation of students per group resulted in the possibility to ask more questions for more thorough information. The second method, mind mapping, was conducted in the same groups. Through the mind mapping method an answer was given to the question how students felt about making their homework. Also, this method aided in getting a possible solution on how to influence the sentiments of the students, because they could give possible solutions. Four A6 size papers were used on which the students could brainstorm about this topic. Although it was not necessary for the students to motivate their answers, they could give feedback to each other, which was helpful to understand how they feel about making homework. Finally, the answers gave insights in what possible solutions are and provide a solid base for future work in this research. Results Students According to the interviews, almost every student dislikes making homework; 2

Almost a third of the participants get distracted when making homework. Mostly because of their smartphones or by playing games. Some listen to music while making homework; Almost a third does not start on time with their homework. They schedule more time for a course if it is difficult; The interviews show that almost every student does not divide the homework over the week. Making homework one day before the lecture happens frequently but some students do their homework in or before the weekend; Everyone has a smartphone, which can run mobile applications; Furthermore, all phones have a mobile internet connection; The majority of students would like to use smartphones for making homework; The majority of the class does not use a calendar, because all the homework is noted in a digital school application called 'Magister' Students provided plenty and creative ideas in the mind mapping session considering the topic homework, such as listening to music while making homework and too much reading/writing. The mind mapping session can be seen in Figure 3: Mind mapping. Another mind mapping outcome was that students prefer to make homework during class or on the computer Teacher It is very difficult to understand what a specific group or student finds difficult about their homework; Scheduling homework and learning is a high priority on Zwijsen College. However, there are always students who find this difficult; I (teacher) try to stimulate those students during the study-counselling class. We discuss possible scheduling problems and we try to give them supporting tools; Each student is listed in a personal file. These files are saved in one system so we (teachers) can see which student makes his or her homework frequently, and who does not. One particular group of students usually tries to avoid making their homework on a regular basis. The complete interview findings can be found in Appendix A Results user research interview DESIGN In the design phase the results of the user research phase were used to distinguish four possible solutions for the problem. The suggestions of the students were pinned down on a blackboard and were categorized into four possible solutions. The four solutions, as can be seen in Figure 1: Four Possible Solutions, are: The Smartest Student, Reminder, Gamification and Blackboard Questions. The Smartest Student is an app which will provide students with an interactive and fun way to let them start sooner with their homework. Teachers can add their own questions and analyse which questions were difficult for the students. The Reminder app is a notification app, which gives students a notification about their homework. The app asks the student if his homework has already been made. The Gamification app is a game which provides questions if a student was not able to reach the next level of the game. Finally, the Blackboard Questions app is a simple app that will list questions made by the teachers. Figure 2: Brainstorm Four Possible Solutions The Smartest student, from now on called Brain-e, was considered most relevant for further development because it can easily be inserted in the curriculum, it makes homework more fun for students and they absorb the curriculum better through the Gamified Learning Approach (Su & Cheng, 2013). The e stands for both education and electronic, which indicate the e-learning aspect of the app. Brain-e was based on the successful Dutch application of the De Slimste Mens (The Smartest Human). In this program, users have to answer several questions about, for example, history in different question structures. Users gain points with correct answers and users are able to compete with friends via a ranking. The Brain-e app has the same functionalities as De Slimste Mens, but is especially designed as a mobile learning environment. To make Brain-e accessible for teachers, an online tool has been developed, which will be integrated in the already present educational administration software Magister. This tool enables both teachers and students in selecting a platform most accessible for them. Especially for teachers an online tool is convenient since they have to insert the questions including the answers, which is done easier on a desktop PC. Furthermore, teachers can see an (statistical) overview (both online and within the app) of which students did make the assignments, which students did not and with which questions students experienced difficulties. Subsequently, teachers can send these students a (personalized) notification to remind them in making their 3

homework. Brain-e makes the classes more interactive and enables the use of the, often already present, smartphones of the students. PROTOTYPING Figure 3: Brain-e Sketches Low-fi Prototypes Prior to the actual sketching of the prototype two flowcharts (Appendix B Flowcharts) were made to get a good overview of the needed screen. Afterwards, the sketching tool Balsamiq Mockups has been used for the design of the low-fi prototypes for both the student application as well as the teacher portal. Subsequently, the screenshots were printed and cut into the form of an iphone (Students) or web browser (Teachers) and the intermediate users tests were conducted in two user groups. For the first user group a high school in Tilburg was visited on April 16 th, 2014. Six male first class students were approached randomly for participating in the intermediate user tests. The user tests for the second user group were conducted with two lecturers of the Zwijsen College. Different assignments (Appendix C User test assignments) were used during the user tests to analyse the two applications and get useful feedback for the rest of the design process. The paper prototype tests indicated the following feedback. Firstly, the student participants described the app as clear and fun to use. Secondly, the most important improvements of the student application were that the menus could be more distinct and clear by using visual elements. Furthermore, the menu structure should be changed and include the option to make a new group or add a class or friend. The participants of the teacher portal user test also indicated a broad range of improvements, which are focused on unclear button names, unclear flows through the portal and hard to find functionalities. The detailed improvements can be found in Appendix D Complete list of improvements after low-fi test. Despite of the broad range of improvements the users stated that the path to add new questions is very structured and it is, therefore, easy to upload new questions. Finally, all the findings are improved during the design process of the high-fi prototypes. Hi-fi Prototypes On grounds of the indicated improvements during the paper prototype tests the high-fi prototypes were built. The final high-fi prototypes for the student application were designed in Adobe Photoshop and subsequently placed in Flinto. The clickable prototype of the app can be found here. As with the mobile application, the teacher portal low-fi prototypes were tested by several users (teachers). The improvements that could be made are improved in the highfi prototype, which is designed by using Bootstrap and is, therefore, fully responsive. The clickable prototype of the teachers portal can be found here. Design principles The Norman Design Principles [6] were followed during the design process of the high-fi prototype. Furthermore, the following interface and interaction design principles are used: The use of cultural idioms. For example, the back button is placed on the upper left and the next buttons are always placed on the bottom rights. The principle of affordances is used in the design to make use of clear buttons by using clear names, colours and symbols. The principal of constraints is implemented to solely show the buttons, which are needed on a specific screen. Finally, the following visual design principles are implemented: Simplicity: Based on the constraints principle simplicity is used in the visual design. Only the important visuals are used. For example, shadows and other not useful design elements are not implemented. This creates a clear and simple look. Contrast is implemented through the use of blue and white as the most important colours. Buttons are blue and placed on a white background, which provides a good contrast between the background and the button. Also the text is coloured blue on a white background or vice versa. This ensures a good readability. White space is used to group parts of the app together. If a button or text is not part of a group a bigger white space is used to separate this from the group. Balance: The design of the app and the website is balanced by using enough white space and placing 4

the most important buttons on the bottom of the screen, separated from the main content. Alignment: All the screens are designed on the same grid. Buttons, text and titles are placed on the same location on each screen. Consistency: Both in the app and in the teachers backend the same colours are used. Furthermore, fonts and buttons are consistent in both applications. EVALUATION Figure 4: Brain-e hi-fi prototype Method In the evaluation phase of this study, several users have tested the mobile app and the teacher portal high-fi prototypes. Three teachers (two males, one female) and seven first year students (all males) participated in this evaluation. The teachers age varied from 24 to 54 and they have been working at Zwijsen College in Veghel and Kandinsky College in Nijmegen for at least two years. The students age varied from 11 to 14 years and they all studied at the Theresia lyceum in Tilburg. The aforementioned group differs from the group in the evaluation phase of the low-fi prototypes. Considering the high-fi user testing, no pre-test questionnaire was used. However, participants in the user tests were asked whether they had any knowledge of the application The Smartest Human, which entails a similar quiz-based procedure. If a participant had any knowledge of this application, they were given a brief explanation that the current application had a similar quiz-based procedure. Subsequently, the purpose and goal of the current application were explained to the participants along with the high-fi user test tasks. If the participants did not have any knowledge of The Smartest Human application, the purpose and goal of the current application were explained along with the high-fi user test tasks. The aforementioned explanation also contained a brief explanation considering the quiz-based procedure of the current application. The participants for the mobile application, seven first grade students, were selected randomly at the high school. At the start of the user test these students were given a short explanation about the project and the tasks during the user test. Afterwards, the users had the possibility to ask questions, upon which they were asked to fill out the consent form and the actual user test started. During the student user tests, the participants got three tasks. The first task entailed making a question for a course, in this case the history class. Secondly, the participants were asked to create a group and view their ranking on the leaderboard. Subsequently, the users had the possibility to click freely through the application. Finally, the students had to fill out a short survey (Appendix E short survey hi-fi user test students) about their views on the application. In contrast to the students, three teachers, who were selected randomly as well, were asked to conduct another experiment with other tasks. First, the teachers were asked to insert a question for a course. Subsequently, they had to view the sufficient grades per class within the course and determine the questions the students experienced to be most difficult. Thereafter, the participants could click on their own initiative through the teacher portal. Finally, the teachers had to fill out a short survey (Appendix F short survey hi-fi user test teachers) about their views on the site. Open questions were used in order to gain more in depth answers from the participants. One question could be seen as guiding as it stated: What did you like less in the app?. In this evaluation the usability (ease of use) and the attractiveness of the Brain-e app were measured globally. The usability was measured by giving both students and teachers two specific tasks. The two tasks for students were to make a homework assignment by answering questions and to create a group and to view their ranking on the leaderboard. The tasks for teachers were to add and send questions via the website to the mobile application and to view the statistical overview of one particular course and to add a class. To measure the attractiveness of Brain-e a short questionnaire was developed. Eventually, the questionnaire could also be used to collect general feedback. Results The user tests of the mobile application and the desktop web application resulted in several findings and, therefore, in enhancements of Brain-e. The results are, respectively, divided in a student and a teacher section. Both are subdivided in negative and positive comments and remarks. First of all, students stated that some buttons in the mobile app did not work well. Secondly, they did not notice the button for creating a new group, because it was located at the footer of the app. Furthermore, the participants stated it was hard for them to work with the app due to the English language the application was in ( It was hard to work with the app due to the English language ). Despite of the negative results, the students were positive about some aspects as well. According to them, the app was easy to use, the functionalities of buttons were clear and the overview of the app was globally good. Moreover, they told that they prefered to make questions in the app over making homework in a book and, therefore, they thought the app would make preparation for exams easier. Finally, the students said that the app was inspiring ( The app looks good and I would love to work with it ) and that it looks 5

like a game ( It was fun that I was able to see the results from friends or other people in my class ). On the other hand, teachers presented some feedback on actions on the site. They suggested adding tool tips ( It could be handy to add tool tips to buttons so users can see what every option stands for ) and clarifying the interface ( Using the app for the first time, it is necessary to search for the functions to use them ). Besides that, they wanted some other concrete functionality. For instance, they want to select all classes when they create a new course. Positive feedback entailed that the buttons are clear and well labelled. Furthermore, the teachers were content with the interface ( Clear and Metro like interface ). Finally, the teachers stated that the teacher dashboard gives the opportunity to directly see which materials the student mastered. DISCUSSION The results of the teachers portal displays that the portal is clear and useful. Nevertheless, participants stated that it is necessary to search for the functions of various buttons. Therefore, more research on the unclear buttons is necessary to solve this problem and provide an easy to use portal for the teachers. Students stated that they would use the app to make their homework due to the ease of use and attractive interface and game elements. Therefore, the app is in line with the stated goal in this study to design an useful app for students, which will make homework more fun. The results of this study are in line with the study of Traxer [11], which stated that the possibility to access information and knowledge anywhere and anytime from devices that learners are used to and they regard as friendly and personal are the essence of mobile learning. Although Gikas and Grand [2] stated that student participants described many advantages in which the mobile devices assisted in their learning, the participants within this study stated that they would use the app both in class as well as while making their homework. Finally, teachers often do not know how the tackle the possible efficiency of the use of smartphones during class [4]. This problem would be solved by using Brain-e, due to the fact that teachers can prepare questions, which can be made within Brain-e during the class. Although the students and teachers had confirmed that the app and the portal are useful for them it was not possible to implement all the screens within the prototypes. Secondly, not all the functionalities of the actual app and portal were developed. In addition, the user tests of the teachers were not conducted with teachers in a school environment. Furthermore, the measurements of the hi-fi user test were only qualitative. Future work could focus more on the prototypes to develop completely functional prototypes, which could be tested with the users. Moreover, measurements of the user tests could be both qualitative as quantitative to measure the specific improvements between the low- and the hi-fi prototype. The qualitative measurements remain important because participants of the user tests could come up with specific issues, which they experience during the user test and, therefore, has to be improved. Finally, the feedback from the participants of the hi-fi user tests is not implemented in later prototypes within this study. It is important for future studies to implement this feedback in the final design phase. CONCLUSION This research investigates the opportunities to build a gamified mobile learning application, which can be implemented in the curriculum of first grade high school students to, subsequently, be used by students and teachers. The results of the teacher s low-fi and hi-fi user tests presented the desire for improvement of the design and several functionalities of the Brain-e s dashboard. However, teachers appreciate the overall clear interface and the statistical possibilities of the application, which aid them in tracking the study results of their students. The students considered Brain-e to be useful, easy to use and helpful in understanding the curriculum better, but require the improvement of some functionalities and the translation of Brain-e into Dutch. The importance of Brain-e lies in the deterioration of the Dutch educational system. As starting students experience difficulties in timely planning and making their homework the use of a fun study guidance application could aid them in solving this problem. Secondly, the use of smartphones during class results in students losing their concentration. Teachers often do not know how to tackle this problem efficiently. All in all, learning and planning can be made more fun, interesting and interactive for both students and teachers. REFERENCES 1. Colley, H., Hodkinson, P., & Malcom, J. (2003). Informality and formality in learning: a report for the Learning and Skills Research Centre. Retrieved from http://kar.kent.ac.uk/ 2. Gikas, J., & Grand, M.M. (2013). Mobile computing devices in higher education: Student perspectives on learning with cellphones, smartphones & social media. The Internet and Higher Education, 19, 18-26. 3. Hwang, G.J., & Chang, H.F. (2011). A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students. Computers and Education, 56(4), 1023-1031. 4. Inspectie van het onderwijs. (2012e). Krimpbestendige onderwijskwaliteit. Een regio-onderzoek in Zuid- Nederland naar de gevolgen van krimp voor de kwaliteit van het basisonderwijs. Retrieved from https://zoek.officielebekendmakingen.nl/blg-229956.pdf 5. Kafyulilo, A. (2012). Access, use and perceptions of teachers and students towards mobile phones as a tool for teaching and learning in Tanzania. Education and Information Technologies, 19(1), 115-127. 6

6. Norman, D. A. (1983). Design rules based on analyses of human error. Communications of the ACM, 26(4), 254-258. 7. Post, T., & van der Molen, J. H. W. (2014). Effects of company visits on Dutch primary school children s attitudes toward technical professions. International Journal of Technology and Design Education, 1-25. 8. Sharples, M., Corlett, D., & Westmancott, O. (2002). The design and implementation of a mobile learning resource. Personal and Ubiquitous Computing, 6(3), 220-234. 9. Soderqvist, F., Hardell, L., Carlberg, M., & Mild, K.H. (2007). Ownership and use of wireless telephones: a population-based study of Swedish children aged 7-14 years. BMC Public Health, 7(1), 105. 10. Su, C. H., & Cheng, C.H. (2013). A mobile game-based insect learning system for improving the learning achievements. Social and Behavioral Sciences, 103, 42-50. 11. Traxer, J. (2007). Defining, discussing and evaluating mobile learning: The movie finger writes and having writ. The international Review of Research in Open and Distance Learning, 8(2). 7

APPENDIX A RESULTS USER RESEARCH INTERVIEW Students: Yes No Total Is het maken van huiswerk leuk? Do you enjoy making your homework? 3 21 24 Word jij afgeleid tijdens het maken van huiswerk? Are you being distracted while making homework? 15 9 24 Begin jij op tijd met het maken van huiswerk? Do you start on time making your homework? 8 16 24 Heb je een mobiele telefoon? Do you own a smartphone? 24 0 24 Zou je een mobiele applicatie willen gebruiken voor het maken van je huiswerk? Would you consider having a mobile application for making your homework? 21 3 24 Maak je je huiswerk verspreid over de week? Do you divide your homework over the week? 1 23 24 Interview Lecturer Do you consider yourself being aware of what your students find difficult? It is very hard to define and would also take too much time. We (teachers) know, based on our own experience, what the difficulties are for students. We try to adjust the lecture on the difficult topics. It is quite difficult to understand what a specific group or student finds difficult. Does it often happen that students avoid making their homework? Each student is being listed in a personal file. These files are saved in one system so we (teachers) can see which student makes homework frequently, and who does not. Usually, one particular group of students tries to avoid making their homework on a regular basis. What do you consider to be a solution for students who avoid making homework? The particular student is addressed personally and when it happens to often, there will be small consequences. Do you consider all students to be prepared for examinations? Scheduling homework and learning is a high priority on Zwijsen College. From the start, students are helped and stimulated to schedule their homework. However, there are always a couple of students who find this difficult. How do you try to stimulate those failing students? I try to stimulate those students during the study-counselling class. We discuss possible scheduling problems and we try to give them supporting tools. 8

APPENDIX B FLOWCHARTS Flowchart teacher portal Login / Dashboard Add.. Add Dashboard Quiz Course 1: X Course 2: Y Course Person Course 1: X - Dashboard Quiz Course 1: X - Quiz Add Quiz Prev. Quiz Stat. Course 1: X -Stat. overview all classes Other Quiz Select View Stat. Last Quiz(#2) Send personal notification Add Quest ions Select Prev. Quiz Course 1 : X - View Quiz Edit es Select Upload Quiz + notify Add Select Notify 9

10 Flowchart student application Login Question overview Profile Edit profile Edit photo Questions course A Question 1 Questions course B Question 10 Question 1 Question 10 Time Courses Student leaderboard Personal Score Make new group group X friends Add friend Search Search individual friends class Add class groupy Select course

APPENDIX C USERTEST ASSIGNMENTS Scenario students Scenario 1 Make questions 1. Assignment: Answer questions for the History class 2. Select Questions from 3. Select History Course from Course overview 4. Read given explanation on type of question 5. Answer questions 6. Let user explain question to test understanding 7. Read time Scenario 2 Create group and view ranking on leader board 1. Assignment: Create a leader board group with 1 friend and view your ranking 2. Select Student leader board from 3. Select Group X for individual friends 4. Select Search individual friends 5. Read friends 6. Select Search friends à Select Search friends, click on option bar to enter username 7. Enter Jan Mik 8. Select Search 9. Add friend 10. Read Group X Scenario teachers Scenario 1 Insert question for History course 1. Assignment: Insert question for History (course X) 2. Login 3. Select course X 4. Select Quizzes 5. Select Add Quiz (delete png 5 for test) 6. Insert question data including Quiz name, Q1, A1-A3, Explanation 7. Save Quiz 8. Select Send to mobile Scenario 2 View sufficient grades per class within History course and determine most difficult question 1. Assignment: View sufficient grades per class within History course and determine most difficult question and send notifications to those students who did not made their homework. 2. Login 3. Select Course X 4. Select Quizzes 5. Select previous Quizzes 11

APPENDIX D COMPLETE LIST OF IMPROVEMENTS AFTER LOW-FI TEST Findings Students Good points Four participants praised the clear overview of the app. Two participants described the app as nice and fun to use. All students participated fully in the intermediate user tests. Points for improvement According to one participant the buttons of the main menu (as can be seen in Figure 3: Menu) could be more distinct and clear. Figure 3: Menu Figure 4: Leaderboard One participant claimed that placing the Leaderboard button underneath Profile would make more sense as his first glance was directed at Profile. However, the participant quickly corrected himself, as he was too hasty in mentioning his point for improvement. The most important point for improvement entails the structure and content of the menu screen, which is displayed in Figure 4: Leaderboard. The menu structure should be changed and include the option to make a new group or add a class or friend. Findings Teachers Good points The path to add new questions is very structured. It is easy to upload new questions. Points for improvement 12

If a lecturer of one course has multiple classes in different educational years or educational levels (HAVO/VWO), how does he know to which of those classes he has to send the quiz. There should be one more pages to select a class before sending the questions to the mobile application. (Figure 5: Add Questions) In Figure 5: Add Questions it is not clear that Q means Question and the A means Answers. If you have never seen this overview before it is hard to understand that the explanation section is meant for an explanation of the right answer. Figure 5: Add Questions It is confusing that the statistics button does not look the same as the other two buttons. The participant thought that the statistics figure was just an overview. It would be better if the statistics button looks the same as Quizzes and es as can be seen in Figure 6: Select Statistics. Figure 6: Select Statistics The schedule of Figure 7: Questions mentions classes instead of students. It makes more sense if a teacher is able to send a notification to one particular student instead of a whole class (Figure 7: Questions). 13

In Figure 7: Questions it is not logic that a class has to be selected again in the notification section. It would make more sense if you can select the students to whom you want to send a notification about a particular question. Figure 7: Questions In Figure 8: Dashboard it was not clear if a participant had to add a class through es or Add. Perhaps it makes more sense if the dashboard in Figure 8: Dashboard only contains the four course buttons and one button to add a class. At this stage there are too many buttons to choose if you want to add a class. Should you select es or Add? Figure 8: Dashboard In Figure 9: Add it was hard to find the Add class button. One participant suggested to make a pop-up menu after you had selected all the right students to add the class. In general, it makes no sense to select a class student by student because the system knows which student is in which class. If there are special cases, teachers are able to adjust class settings in Magister. 14

Figure 9: Add 15

APPENDIX E SHORT SURVEY HI-FI USERTEST STUDENTS Leeftijd: Geslacht: Man / Vrouw Vond je de app makkelijk in gebruik? Zo ja, waarom wel? Zo nee, waarom niet? Wat vond je goed aan de app? Wat vond je minder goed aan de app? Zou je de app willen gebruiken voor school? Ja / Nee Waarom wel of waarom niet? 16

APPENDIX F SHORT SURVEY HI-FI USERTEST TEACHERS Leeftijd: Geslacht: Man / Vrouw Vond je de website makkelijk in gebruik? Zo ja, waarom wel? Zo nee, waarom niet? Wat vond je goed aan de website? Wat vond je minder goed aan de website? Zou je de website willen gebruiken? Ja / Nee Waarom wel of waarom niet? 17