An investigation of the relationship between online activity on Studi.se and academic grades of newly arrived immigrant students

Transcription

1 EXAMENSARBETE INOM TECHNOLOGY, GRUNDNIVÅ, 15 HP STOCKHOLM, SVERIGE 2017 An investigation of the relationship between online activity on Studi.se and academic grades of newly arrived immigrant students An application of educational data mining AKASH MENON NAHIDA ISLAM KTH SCHOOL OF INFORMATION AND COMMUNICATION TECHNOLOGY

2

3 Abstract This study attempts to analyze the impact of an online educational resource on academic performances among newly arrived immigrant students in Sweden between the grade six to nine in the Swedish school system. The study focuses on the web based educational resource called Studi.se made by Komplementskolan AB. The aim of the study was to investigate the relationship between academic performance and using Studi.se. Another purpose was to see what other factors that can impact academic performances. The study made use of the data mining process, Cross Industry Standard for Data Mining (CRISP-DM), to understand and prepare the data and then create a regression model that is evaluated. The regression model tries predict the dependent variable of grade based on the independent variables of Studi.se activity, gender and years in Swedish schools. The used data set includes the grades in mathematics, physics, chemistry, biology and religion of newly arrived students in Sweden from six municipalities that have access to Studi.se. The data used also includes metrics of the student s activity on Studi.se. The results show negative correlation between grade and gender of the student across all subjects. In this report, the negative correlation means that female students perform better than male students. Furthermore, there was a positive correlation between number of years a student has been in the same school and their academic grade. The study could not conclude a statistically significant relationship between the activity on Studi.se and the students academic grade. Additional explanatory independent variables are needed to make a predictive model as well as investigating alternative regression models other than multiple linear regression. In the sample, a majority of the students have little or no activity on Studi.se despite having free access to the resource through the municipality. keywords: Educational data mining (EDM), data mining (DM), Cross Industry Standard for Data Mining (CRISP-DM), Statistical analysis, Multiple linear regression, null-hypothesis, and level of significance.

4

5 Sammanfattning Denna studie analyserar inverkan som digitala läromedel har på skolbetyg bland nyanlända elever i Sverige mellan årskurs sex och nio i det svenska skolsystemet. Studien fokuserar på den webbaserade pedagogisk resursen Studi.se, gjord av Komplementskolan AB. Målet med studien var att undersöka relationen mellan skolresultat och användandet av Studi.se. Ett annat syfte var att undersöka vad för andra faktorer som kan påverka skolresultat. Studien använder sig av datautvinningsprocessen, Cross Industry Standard for Datamining (CRISP-DM), för att förstå, förbereda och analysera datan i form av en regressionsmodell som sedan evalueras. Datasamlingen som används innehåller bland annat skolbetyg i ämnena matematik, fysik, kemi, biologi och religion från sex kommuner som har tillgång till Studi.se. Aktivitet hos eleverna från dessa kommuner på Studi.se hemsidan användes också för studien. Resultaten visar en negativ korrelation mellan betyg och kön hos eleverna i alla ämnena. Den negativa korrelationen betyder i denna rapport att tjejer får bättre betyg i genomsnitt än killar hos urvalet av nyanlända från de sex kommunerna. Dessutom fanns det en positiv korrelation mellan antal år en elev varit i skolan alternativt i svenska skolsystemet och deras betyg. Studien kunde inte säkerställa ett statistisk signifikant resultat mellan aktivitet på Studi.se och elevernas skolresultat. Ett flertal förklarande oberoende variabler behövs för att kunna skapa en prognastisk modell för skolresultat samt bör en undersökning på alternativa regressions modeller förutom linjär multipel regression göras. I studiens urval av nyanlända elever från kommunerna, har majoriteten inte använt eller knappt använt Studi.se även om dessa kommuner haft tillgång till denna resurs. Nyckelord: Utbildningsdatautvinning, Datautvinning, CRISP-DM, Statistisk analys, Multipel linjär regression, nollhypotes och signifikansnivå.

6

7 Statement of Collaboration This bachelor thesis project was done by Akash Menon and Nahida Islam students of the program information and communication technology at KTH. The report was written with close collaboration with both authors writing multiple subsections in each chapter with exception of the Background which was mostly written by Nahida. Reviewing the report was done by both authors. Nahida did most of the data preparation of the data received from the municipalities while Akash handled most of the data preparation of the data from Studi.se. Most of the R scripts written as part of the data preparation, processing, regression and graphic plots generation was done by Akash and can be given upon request. The analysis was done jointly by both authors.

8

9 Contents 1 Introduction About Studi.se Research area Hypothesis Purpose Benefits and ethical aspects Delimitations Outline (Disposition) Background Big Data Learning Analytics Data Mining CRISP-DM model Data mining process by Oracle Educational Data Mining R programming language Statistical analysis Multiple Linear regression Important terms for the regression Pearson correlation coefficient Relevance to this project Methods Step one: Business understanding Step two: Data understanding Step three: Data preparation Step four and five: Modeling and evaluation Step six: Knowledge Deployment Result Analysis of mathematics grades and performance Summary of the Results Discussion and Conclusion Data reliability Discussion of Results Discussion of General hypothesis Conclusion Reflection Future work and ideas 51 7 References 53

10 Appendices 57 A Results on physics 57 B Results on chemistry 63 C Results on biology 69 D Results on religion 75

11 Chapter 1 1 Introduction Internet has been a platform that has gained tremendous growth constantly in the last couple of years. Twenty years ago the Internet platform might not have been widely used by the general public. In recent years, internet usage among individuals has reached over 90% in Sweden according to the International Telecommunication Union (ITU) [1]. This has led to the increased accessibility of different ICT solutions. Among the new technologies that are gaining traction due to the spread of the Internet are digital platforms for education. Leaving the old system behind, these digital platforms allow greater flexibility in education by using a mixture of audio and video cues that can be accessed at the pace and location that fits the individual in question. Studi.se is a digital educational platform that has a goal to make education easier for students and particularly newly immigrated students in Sweden. According to Migrationsinfo, around 16% of Sweden s population were born abroad in 2015 [2]. Education of newly arrived students is of very high importance as they have the potential to constitute a significant part of the future development of Sweden. Usage of Internet on a daily basis produce vast quantity of data that is created by individual users. This data sometimes contains traces of a user such as view history in YouTube. By analyzing these data one can predict which type of person the user is. In the same way there are tremendous numbers of user data produced in online educational material which remains not used. This data can be analyzed to improve the education system and to predict if such educational material can have any positive or negative impact on student s school grades. 1.1 About Studi.se Studi.se is a digital educational material that exists in the form of a website. It is owned by the Komplementskolan AB. Studi.se had been producing online educational material such as videos and exercises for pupils since In the fall 2013 they decided to introduce a project called The Pilot Project (Pilotprojektet). The aim of the project was to translate videos on chemistry, biology, and religion to Arabic. This was appreciated by students and teachers from twelve schools [3]. Because of the positive feedback, another project called The Language Project (Språkprojektet), had been introduced with the help of Vinnova in 2015 [4]. This project is managed in co-operation with researchers and municipalities across Sweden. As part of the language project greater emphasis is put on multilingual aspects by developing a multilingual digital educational platform. This platform will help to welcome the newly immigrated pupils to the Swedish language as well as other subject in schools. Studi.se is associated with 29 participated municipalities. They have animated videos on six different languages such as Swedish, English, Arabic, Somali, Tigrinya and Dari. The total number of translated films and quizzes are Page 9 of 80

12 Chapter 1 Table 1: Subscription types and associated supported features Features / Subscription Trial Individual School Språkprojektet Number of movies 10 Unlimited Unlimited Unlimited Number of quizzes 10 Unlimited Unlimited Unlimited Teacher accounts No No Unlimited Unlimited Homework functionality No No Full access Full access Multilingual All supported languages Swedish & English Swedish & English All supported languages Research support No No No Full support respectively 512 and 504 [5]. The Studi.se platform is offered in trial form and three different types of subscriptions. The three different subscriptions [6] are individual licences, school licenses and all-inclusive license which is offered to schools that partake in The Language Project as can be seen in Table 1. The largest number of users consist of students and teachers from schools in municipalities that partake in The Language Project and have full access to all the features that the platform offers. While signing up for the platform the user selects their role as either student or teacher. They fill in their full name, , desired password and if applicable associated school can also be given. Studi.se contains video lectures and quizzes on different topics covering grade seven to grade nine. When Studi.se first started to release the video lectures, there used to be a teacher standing at a whiteboard and explaining the topic. To engage a user and keep their attention they animated the lecture videos. The videos are between two to five minutes in length and seldom longer in order to retain the students attention. A lot of videos are dubbed and have subtitles in up to six different languages. A user can choose any of the six languages as they prefer. Users can choose the language of audio and subtitles independently on demand. Some of the topics have a written supportive text explanation under the lecture video. These texts are in Swedish and there is no function for translating them in any another language. Every video has a quiz with questions on the topic. The quiz questions are divided into three levels. Aspects of gamification is used in the quizzes in which the user has three hearts. Each time a user answers incorrectly they lose a heart. Answering incorrectly three times ends the quiz with a message that encourages the user to watch the related video again. Each completed quiz level awards the student with points. Watching a video until completion also awards points. The homework functionality that is presented in Table 1 is offered to the subscription level denoted schools and The Language Project. Only a teacher Page 10 of 80

13 Chapter 1 can utilize this functionality by distributing homeworks to the whole class or to a specific student. A homework consists of videos and/or related quizzes. 1.2 Research area While the Studi.se educational material is supposed to be beneficial for all students, the multilingual features of it give it potential to be an even more important resource for newly arrived students. Newly arrived students do not always have the sufficient language proficiency in Swedish to be able learn new concepts through it. The schools participating in the The Language Project have provided student grade data of newly arrived students to Studi.se which is why this paper primarily focuses on newly immigrated students. According to The Swedish national school law (Skollagen) [7], a student should not be called newly arrived if he/she has been in the school for more than four years in Sweden. This report contains one exception. The exception is there were a few students that were received from the municipalities have been in Sweden more the four years as well. They are also included in the study. SIRIS, the National Agency for Education s online information system on results and quality [8], provides pass rates of different grades. In Table 2 the national pass rates of grade 6 in mathematics, physics, chemistry, biology and religion [9] are presented. Table 2: pass rates of grade 6 Percentage (%) with grade A-E Subject Year 2013/2014 Year 2014/2015 Year 2015/2016 Mathematics Physics Chemistry Biology Religion Unfortunately the authors could not find any statistics on the pass rates from further back. In Table 2 it is noticeable that the pass rates in year 2013/2014 were better than the pass rates from 2015/2016. Page 11 of 80

14 Chapter 1 In Table 3 the national pass rates of grade 6 in mathematics, physics, chemistry, biology and religion [10] are presented. Table 3: pass rates of grade 9 Percentage (%) with grade A-E Subject Year 2012/2013 Year 2013/2014 Year 2014/2015 Year 2015/2016 Mathematics ,7 89, Physics ,4 92, Chemistry , Biology , Religion ,3 93, In Table 3 it is noticeable that the pass rates in year 2012/2013 were better than the pass rates in 2015/2016. This patterns shows a decrease in pass rates across subjects among Swedish students where the subset newly arrived immigrant students is included. This pattern could be turned around if an external help such as online educational material can be implemented in the school system. That is why it is important to find out how effective online educational materials are for students. A first step to answer this question is to analyze what correlation there is between activity of a student on online educational material and their achieved academic grade. 1.3 Hypothesis While doing an experiment it is hard to claim that a prediction is always true. That is why it is essential to put one or several hypothesis. The authors have set some general and specific hypotheses before performing the study. One general hypothesis is that the longer an immigrated student has been in a school the better result they have compared to the first year of their school. It is a common fact that the longer you have been in a country the more you learn about the country and as well as the language. This means there could be a correlation between the time period a student have been in Sweden and their academic grade. The second general hypothesis the authors set up is that using an online educational material can help a student to achieve higher academic grade at school. Online educational material is meant to be a quick help for students. For instance a student is doing his/her mathematics homework at home and he/she needs help with fraction he/she could easily search for an online material to get help instead of waiting the whole day and ask his/her teacher about the Page 12 of 80

15 Chapter 1 problem next day. Because it is a quicker way to get help one might think or believe online educational material makes the education easier for the students which in turn helps the students to achieve higher academic grades. Studies shows that female students do have better performance and academic grades at school than male students [11]. So the third general hypothesis is that we expect female students on average have better school results than their male counterparts. The authors have adapted these general hypothesis to five specific hypothesis with the attention to the online educational material Studi.se and the student s academic grade received from six municipalities. In mathematical language the terms null-hypothesis and alternative hypothesis are often used. Null-hypothesis is a short form of nullifiable hypothesis [12]. This is usually the opposite to that one wants to prove because it is the statement one wants to nullify or reject. The alternative hypothesis is often the statement one wants to accept in place of the rejected null-hypothesis [13]. Failure to reject the null-hypothesis means we can not accept our alternative hypothesis. The five null-hypotheses ( H 0 ) for this report can be described as following: H 0a : There is no relationship between the number of videos watched on Studi.se and academic grade received from the school. H 0b : There is no relationship between the number of quizzes completed on Studi.se and academic grade received from the school. H 0c : There is no relationship between the gender of a student and academic grade received from the school. H 0d : There is no relationship between number of years in school and academic grade received from the school. H 0e : There is no relationship between the set of predictors (Videos, Quizzes, Gender and Years in School) and academic grade received from the school In mathematical terms: H 0 : β 1,2,... = 0 which means there is no relationship between the variables. If the probability, also called the p-value, is less than the level of statistical significance α the H 0 can be rejected. α is set to 0.05 for this study. To summarize everything if p <α the H 0 can be rejected. The alternative hypotheses can be stated as following: H 1a : There is a relationship between the number of videos watched on Studi.se and academic grade received from the school. H 1b : There is a relationship between the number of quizzes completed on Studi.se and academic grade received from the school. H 1c : There is a relationship between the gender of a student and academic grade received from the school. Page 13 of 80

16 Chapter 1 H 1d : There is a relationship between number of years in school and academic grade received from the school. H 1e : There is a relationship between the set of predictors (Videos, Quizzes, Gender and Years in School) and academic grade received from the school In mathematical terms: H 1 : β 1,2,... 0 which means there is a relationship between the variables. If p <α the H 1 can be accepted. Table 4 describes the expected relationship between the independent variables and the dependent variable from the results according to the theory. Table 4: Summary of the independent variables expected relationship Independent variable Expected relationship Explanation Videos Positive More watched videos on Studi.se are associated with higher academic grade. Quizzes Positive More completed quizzes on Studi.se are associated with higher academic grade. Gender (male = 1, female = 0) Negative Female has better grade than male Years in school Positive The longer a student have been in a school the higher grade he/she has. 1.4 Purpose The idea behind the project is to measure the efficiency of the online educational material Studi.se by testing the null-hypothesis discussed in the previous section (section 1.3). The reason to choose this path to measure the efficiency is to utilize educational data mining. The goal is to present the correlation with the help of regression models. 1.5 Benefits and ethical aspects The main beneficiaries from this project would be the schools in the municipalities with access to Studi.se and Studi.se who would get quantifiable results showing the effects from using the platform. While signing up in the website the user need to specify their address, name and school name. Studi.se does not store national identification numbers or addresses of their users unlike some school IT systems or typical learning management systems. Page 14 of 80

17 Chapter 1 There are risks with the project concerning integrity when processing information containing personal identifiable information. Personal identifiable information includes data such as names and schools. Any results that are published have personally identifiable information redacted or replaced. If for instance data of a group of less than ten individuals from a certain geographic location is presented it is either redacted or aggregated into a larger group such that it is not possible to identify the individuals in real life. 1.6 Delimitations To execute the thesis project successfully some delimitations were set. section describes the established delimitations. This Online educational material: The project has the main focus on the platform called Studi.se Student Grades: The student grade data is from only six of the municipalities taking part in The Language Project and only includes individual grade data on newly arrived students. Studi.se data: The Studi.se data that was used ranges from August 2012 until February Grades: Using grades as an indicator for academic performance is not a fully objective indicator. The grades are set by the subject teachers in schools according to guidelines in the Swedish curricula. Sometimes the grade vary from teacher to teacher for instance according to their understanding of the Swedish curricula. A more objective indicator for academic performance would be national exams results. Very few schools had reported national exams results and thus solely grades could be used as an indicator for academic performances. 1.7 Outline (Disposition) The first part of the report is an introductory part. This part consists of a short description about the company and their website and the problem. The Second part (section 2) of the report describes some theoretical background that are interesting for the project. The Third section (section 3) is about the methods that has been used through the whole project. In the forth section (section 4) there is a presentation of the results that has been produced. The fifth part (section 5) conducts a discussion and finalize a conclusion of the results and discussions. Lastly the sixth part (section 6) describes ideas for future work on similar issues. Page 15 of 80

18

19 Chapter 2 2 Background The following section describes and presents background and theory related to the thesis. The section ends with a description on which parts are relevant for this study. 2.1 Big Data Everyday while using internet we leave digital traces in a form of a large volume of data in various websites. These data can be both structured and unstructured and are called Big Data [14]. Our activities in digital education create in the same way a vast amount of data (often terabytes or petabytes). Because of the size and volume of Big Data one might need a cluster of computers for computing and processing the data [15]. The purpose of using Big Data in education is to be able to improve student results, their learning experience and reduce the number of dropouts [16]. Studi.se has also a big number of data generated from their website. The volume is not as big as terabytes but still it is quite big to work with. The generated data consist of activities that every user have performed since August 2012 to February There are three types of activities a user can perform in Studi.se. These are watching videos, doing quizzes or doing homework given by teachers. Every time a user clicks on a video or quiz an entry is created in the database. Homework entries are created when teachers assign the homework to students. One can count these entries by an appropriate programming language to get the frequencies. 2.2 Learning Analytics Learning Analytics (LA) concerns an area dealing with collecting and analyzing data about and produced by students in the different educational contexts [15]. Stakeholders involved in most LA setting are students, teachers, institutions and researchers [17]. A policy brief from United Nations Educational, Scientific and Cultural Organization (UNESCO) [18] describes most common methods and concepts used in LA. These are learning management system (LMS) Analytics Dashboards, Predictive Analytics, Adaptive LA, Social Network Analytics, and Discourse Analytics. LMS Analytic Dashboards concerns the visualization aspect of LA and aims to provide stakeholders with graphs, tables or other form of visualization of students learning data. Predictive Analytics involves analysis of usage patterns and data to identify students that are doing well or are at risk of failing certain learning outcomes. Adaptive LA focuses on creating a dynamic model of a learners understanding of a topic. The aim here is to provide educational material based on this model to the learner. Social Network Analytics looks at the structure of interactions and relationships between people and groups using a certain LMS in order to identify behavioral patterns that can be beneficial for Page 17 of 80

20 Chapter 2 learning. Discourse Analytics is a LA in which written text and conversation is evaluated. It looks at how language can shape learning. Typically, a LMS offer a platform for communication and educational material. Studi.se is not a full-fledged LMS as it only provides videos and quizzes as educational content. Teachers do not have any authorship over the educational environment. It offers limited communication between teachers and students. Studi.se can instead be viewed as a educational material rather than platform. 2.3 Data Mining Data mining is the practice of automatically searching large stores of data to discover patterns and trends that go beyond simple analysis. Data mining uses sophisticated mathematical algorithms to segment the data and evaluate the probability of future events., according to Oracle [19]. Data mining makes it easier to address questions that are hard to answer with simple queries. This hardship occurs because of the large size of data that is generated. Data mining is widely used to discover patterns automatically, predict likely outcomes, create actionable information, and focus on huge databases. Data mining applications have been used widely in fighting against terrorism, business trends, health care and insurance, and web and semantic web [20]. Data mining goals can be categorized [21] as: Prediction: focuses on some variables in the data set to predict future values of other variables of interest. This type of data mining produces a model of the system described by the given data. Description: focuses on finding human-readable patters describing the data. This type of data mining produces new information based on the input data. The author Kantardzic of Data Mining Concepts, Models, Methods, and Algorithms [21] claims that the predictive and descriptive spectrum creates different types of data mining methods such as: Classification: This technique classifies data into several predefined classes. This can be helpful while executing predictive data mining. Regression: This is a method that has its core from statistical field. To achieve predictive result the data is mapped into a real-value prediction variable. Clustering: This is used to identify a finite set of clusters to describe the data. This method belongs to descriptive data mining. Summarization: The technique belongs to descriptive data mining that involves methods for finding a description for a data set. Dependency Modeling: Describes the significant dependencies between different variables of the data set. Page 18 of 80

21 Chapter 2 Change and Deviation Detection: describes the significant changes in the data set. There were two data mining process models the authors came across during the pilot study. One is the Cross Industry Standard Process for Data Mining (CRISP-DM) and the other one is suggested by Oracle documentation on Data mining concepts. Both data mining process models are generalized processes which are easy to implement as a base of a data mining project CRISP-DM model The Cross Industry Standard Process for Data Mining is a data mining process model that widely used within the data mining field [22]. The aim of CRISP-DM model is to make easy, reliable, less costly, manageable and faster data mining projects. Figure 1 shows the five different phases of the CRISP-DM model. Business Understanding Data Understanding Data Preparation Data Deployment Modeling Evaluation Figure 1: CRISP-DM process model overview Below the different steps of the Cross Industry Standard Process for data mining process are described in details. [22]. Step one: Business understanding The initial step to begin a project is to understand what is the goal with this business, what needs to be accomplished. It is important to uncover the different objectives and requirements of the project. The requirements include Page 19 of 80

22 Chapter 2 a summary over which data are needed, which attributes from the database are interesting, the different variables that are important to execute the project and other relevant factors. There should occur a discussion on what risks and contingencies that can occur during the project. Lastly, this knowledge needs to be converted into the data mining view. A goal should be set that is desired to be achieved through the data mining process. Step two: Data Understanding The second phase summarize the understanding of the data that were received. First one need to collect the data that are interesting for the project. These data can be collected from a database or handed over as another format. Data exploration, finding subsets, finding patterns can give one a good insight over the data one will work with. After understanding the data, the data preparation phase begins. Step three: Data preparation Data gathering, cleaning and preparing used to be the biggest part of such project. The data often needs to be processed and formatted into a uniform template. Data that is generated from databases needs to be cleaned and formatted according to the relevance of the project as well. One might need or want to extract, add or delete some data as it is required for the research. The preparation phase can be done several times to achieve the desired data set. This step should get full attention because the rest of the project depends on how well you have gathered your data to implement data mining. There are different tools in the market to ease the process. The most common tool is MS Excel which can be problematic when it comes to Big Data. Other prevailing tools for handling data processing and manipulation are IBM s SPSS and the open source programming language R. Step four: Modeling This phase comprises of building a new model or selecting an already existing model. Some studies on different types of data mining models and algorithms is necessary. After selecting a preferable model or algorithm the data might require data transformation to fit into the selected model. In the first stage of model building one might want to work with a small set of data. It is often noticed that the model that was chosen might need more data cleaning and preparation. That is why it is an iterative process which requires returning back to the previous step for further data preparation. Step five: Evaluation The constructed or chosen model is evaluated. The evaluation should answer how well the model satisfies the requirements described in the step one. According to the answer one might be satisfied with the model or change it to achieve the desired result. Page 20 of 80

23 Chapter 2 Step six: Deployment The last phase is to deploy the knowledge that were gained in a presentable way so that it is understandable for the customer. There are different ways to deploy the knowledge and it depends on what the customer requires. It can be done by generating a report or can even be done by implementing the model in their system Data mining process by Oracle The process there was recommended by Oracle [19] consisted of four stages: Problem definition, Data gathering and preparation, Model building and evaluation and Knowledge deployment. Figure 2 shows the four stages of the data mining process. Problem Definition Data Gathering and Preparation Model Building and Evaluation Knowledge Deployment Figure 2: Data mining process suggested by Oracle The stages of the data mining process suggested by Oracle follows the same general pattern as the CRISP-DM model. The CRISP-DM reference model breaks the Problem Definition step into the Business Understanding and Data Understanding steps. Model building and Evaluation is broken down into two separate steps in the CRISP-DM model as can be seen in figure 1 as opposed to the Oracle model in Figure 2. In general, the two processes for data mining are roughly similar. Page 21 of 80

24 Chapter Educational Data Mining Educational Data mining is an emerging field within Data mining. This field got popularity in recent years because of the growth of educational software and internet usage for educational purposes. This produces a big amount of data which is also known as Big Data. These data can be used to reflect student s learning [23]. Educational Data Mining (EDM) is a field that is used for analyzing educational data in order to research educational issues. To resolve such issues this field takes advantages of statistical, machine-learning, artificial intelligence, information technology, database management system and data-mining algorithms [24]. Educational Data mining inherits methods and techniques from Data Mining and can therefore be used in this field as well. EDM objective can be divided into two objectives as mentioned by Jindal and Borah [25]: 1. Academic Objectives: This objective can be oriented by different fields for instance person oriented, department/institutions oriented or domain oriented. 2. Administrative Objectives: This can be administrator oriented. 2.5 R programming language Among the most common tool for data preparation and processing within the context of data science is the R programming language according to a poll conducted in 2016 on the data science blog KDnuggets [26]. R is a open source programming language which has extensive libraries for statistical tools and graphical techniques. Among the relevant R features for this project, are the regression models, the graphical tools for plot generation and the tools for data preparation. Data can be loaded and manipulated from a variety of different sources in R. The two most relevant ones being commaseparated value (CSV) files and MS Excel files in this project. 2.6 Statistical analysis Statistics is a way of analyzing collected data to support statements about a population. A statistical analysis is often based on probability [27]. Statistics have been used widely in the field of science for various types of analytic researches. A statistical survey often consists of four parts; planning, data gathering, processing and presentation [28]. A common statistical method for modelling is Regression analysis. Regression analysis attempts to explain possible relationships between variables through the use of constructed mathematical models [29]. Page 22 of 80

25 Chapter Multiple Linear regression The most common and widely used regression is linear regression which is used to model the assumed linear relationship between a dependent variable y and one or more explanatory variables X i, i = 0, 1, 2, 3. One simple equation of linear regression model would be y = β 0 + β 1 X 1 + β 2 X 2 + β 3 X 3 + ε (1) where β 0 is called the intercept term, β i, i = 0, 1, 2, 3 are unknown parameters and ε is the error value that represents the difference between the observed value and the predicted value from the regression equation [30]. Multiple linear regression is a type of linear regression with one dependent variable y and two or more explanatory variables X. The equation for multiple linear regression would be n y = β 0 + β k X k + ε (2) k=1 A linear regression has five assumptions that must be fulfilled in order to execute a regression on the data set. The assumptions[31] are listed below: 1. Linear relationship: It is important that the independent and dependent variables has a linear relationship. This is can be tested by scatter plots. 2. Multivariate normality: All variables must be multivariate normal. If the data is not normally distributed a non linear transformation can be useful to fix the problem. 3. No or little multicollinearity: All independent variables must be independent from each other. 4. No auto-correlation: The residuals must be independent from each other. 5. Homoscedasticity: The error terms along the regression are equal Important terms for the regression There are some terms that one should be familiar with while executing regression in R-environment. Below comes a list of some terms with definition that are used often: Residuals: gives the unexplained variation in the regression model. It is calculated by observed y value minus the calculated y value based on the regression model [32]. Coefficients: Standard deviations, t-value and probability for the null hypothesis provides estimates for the model s coefficients [32]. Multiple R-squared: Describes how many points fall on the regression line. This is also called the coefficient of determination [32]. Page 23 of 80

26 Chapter 2 Adjusted R-squared: adjusts for the number of terms in a model [32]. F-statistic: yields F significance and p value of the F test which states whether the regression is a statistically significant (p < 0.05 at 95% confidence interval). p-value: probability for a statistical model. T-test: yields a t-value and p-value that is the check for statistical significance of each beta coefficient in the regression equation. Residuals vs Fitted: A scatter plot of residuals on the y axis and fitted value on the x axis. It shows if the residuals have non-linear patterns. There is no non-linear relationships between the variables if the residuals are equally spread around a horizontal line and has no distinct patterns [33]. Figure 3 is simulated from example data set in R called women. This shows a parabola pattern and is not equally spread around the horizontal line. Residuals vs Fitted 15 Residuals Fitted values lm(weight ~ height) Figure 3: Residuals vs Fitted Normal Q-Q: A plot to show if residuals are normally distributed. If it is normally distributed the residuals will follow the straight dashed line [33]. Figure 4 is simulated from the same example data set as previous. This Page 24 of 80

27 Chapter 2 figure shows that the residuals do not follow the straight dashed line so well. Normal Q Q 15 Standardized residuals Theoretical Quantiles lm(weight ~ height) Figure 4: Normal Q-Q plot Scale-location: Known also as Spread-Location plot and use the squareroot of Standardized residuals. This plot provides a picture over if the residuals are spread equally along the predictor ranges. It can be used to check the assumption of homoscedasticity. A horizontal line with randomly spread points is not what to expect. It is good to have a plot where the points are equally spread around the horizontal line [33]. Figure 5 is simulated from the same example data set as previous. The Scale-location plot shows that the points are randomly spread. Page 25 of 80

28 Chapter 2 Scale Location 15 Standardized residuals Fitted values lm(weight ~ height) Figure 5: Scale-location Residuals vs Leverage: Leverage shows how much each data point impacts the regression. If points are far from the centroid they have greater leverage, the fewer points nearby the greater leverage. Leverage presents both the distance from the centroid and the isolation of a point. Cook s distance can also be seen in the plot. This describes how much the regression would change if a a point was deleted. The expected result is to see the smooth line (in red) close to the horizontal dashed line (in gray) and no point should have large Cook s distance than 0.5 [34]. Figure 6 is simulated from the same example data set as previous. TÍn this figure the smooth line (in red) is not close to the dashed line (in gray). The Cook s distance is 1. Page 26 of 80

29 Chapter 2 Residuals vs Leverage 1 15 Standardized residuals Cook's distance Leverage lm(weight ~ height) Figure 6: Residuals vs Leverage Pearson correlation coefficient The Pearson correlation coefficient also known as product moment correlation coefficient is a method for measuring the degree of linear association between two variables. The coefficient is represented by r. It is measured in the range from -1 through 0 to 1 with no units. If the coefficient is positive it indicates that there is a positive correlation. If the coefficient is 0 it means there is no correlation. If the correlation is negative it indicates that there is a negative correlation. The coefficient r can be calculated [35] as following: r xy = n( x i y i ) ( x i )( y i ) ([n x 2 i ( x i ) 2 ][n y 2 i ( y i ) 2 ]) (3) 2.7 Relevance to this project Everything that was discussed in the Background section (section 2) was indirectly relevant for the project. The most relevant aspects are Big Data, Data Mining and the Data Mining processes, Educational Data Mining and Statistical Analysis. The core of the data analysis depends of statistical tools and theory. Page 27 of 80

30 Chapter 2 The CRISP-DM process model was adopted for this study and most data preparation was done in the R programming language and MS Excel. Most of the data analysis and graphs were produced also in R as it has extensive built-in libraries adequate for data and statistical analysis. Page 28 of 80

31 Chapter 3 3 Methods To execute this project the authors followed the Cross Industry Standard Process for data mining process [22] which was presented in the previous section (section 2). The reason behind choosing the CRISP-DM model that it is one of the most common models that has been used in data mining field [36]. Below, there is a description about how the authors used CRISP-DM model to acheive the results. 3.1 Step one: Business understanding In the beginning of the project a discussion about the problem took place. The formulation that was given by the supervisor from Studi.se to the authors was as following: Investigate to which extent we can use computer traces left by a student in a digital teaching component with preserved personal integrity. We particularly want to study to what extent these traces can be helpful to understand student s learning and understanding. The next step was to break this formulation into its components to have a better understanding on what the company wants and define each part. The study then began with defining the words from the formulation. After getting help from the supervisor by having a discussion the authors could construct the meaning of the two most important terms from the problem formulation. Computer traces: The authors needed to know what is computer traces, which type of traces are they and what to do with these traces. Computer traces are event logs caused by user s activity on any online material. These activities can be how many times a user clicked on a button, how long they have watched a video or how many time they have logged in to the website. Preserved personal integrity: Why is this important to preserve the personal integrity in this case? Studi.se does not require any national identity number while signing up to their website. One of the requirement that was stated by the client was to not mention any school name and student name in the report. The stakeholders, Studi.se and the municipalities wanted to evaluate whether their learning resource had a meaningful positive impact on academic performances. After understanding the problem the authors discussed with supervisor and a new problem formulation was suggested.the formulation of the problem in a quantifiable and measurable form can thus be as following: Investigate to which extent we can use computer traces to predict a correlation between high activity of a student on an online educational material and their academic grade. Page 29 of 80

32 Chapter 3 The result was suggested by the supervisor to be presented in the form a regression with a mathematical equation and statistical measures to illustrate and denote the relationship. During this period the authors have discussed the different requirements, variables and risks that can be met as it was a part of the CRISP-DM model. A list of the requirements and variables were created from the problem formulation. To be able to go ahead with the project the event logs of user activity (computer traces), academic grades (received from six municipalities) and knowledge of regression were required. The authors decided to estimate the level of user activity with the help of two variables, how many videos a student has watched and how many quizzes a student has completed. Another variable, number of homework done could be interesting in this study. A homework consists of a video and/or quizzes which would be a subset of how many videos a student has watched and how many quizzes a student has done. Beside that, there were very few schools that used the homework functionality of Studi.se. That is why the number of homework done was not used as a variable in this study. Another way to estimate the level of user activity could be number of times a student has logged in to Studi.se. This considered as an ineffective way as a student can log in and do nothing. The two main learning activities on the site are watching videos and doing quizzes related to the contents of these videos thus these are the two most relevant variables for measuring user activity. The relevant variables that were accessible at this stage were different metrics of activity, grade in subjects and years in school. The variables that were decided to be used in the project were: 1. y - Grade in subject of student 2. X 1 - Number of videos watched by student 3. X 2 - Number of quizzes completed by student 4. X 3 - Gender of the student. 5. X 4 - Number of years the student has been in the school Academic grades and the number of years in school were available in the data sent from the municipality and the the number of videos watched and quizzes completed by the students were available from studi.se. Even though the problem formulation had the focus on activity on Studi.se the authors decided to add two extra explanatory variables, gender (estimated) and numbers of years in school. The reason behind this is to be able to make a better regression model. Gender for instance, should matter as girls tend to perform better in school than boys [11]. This variable should display a similar pattern in the regression.the authors wanted to also investigate whether being in Sweden for a longer period affects a students grade or not, which the number of years in the school variable gives a indication of. Page 30 of 80

33 Chapter 3 Several risks could be encountered during this project. The authors might not receive all the data in time which would lead to they have to work with small data set. The data that were received might not result to any significant results because of the quality. This step gave the authors a list on the variables that will be involved in further studies and an insight on the risks that can be met. 3.2 Step two: Data understanding While the step one was ongoing, the author received a number of files that contained user activities in Studi.se. Some example of activities are watching video, doing quiz and doing home works. In Table 5 the file name and their contents are described in details. Table 5: Different data files Data file name Size (MB) Number of Rows Description Student Activity This file contains entry ID, lesson ID, subject name, entry date and time, school type and municipality name. Quizzes This file contains entry ID, user ID, name of the user, role, school, school ID, municipality name, entry date and time, subject name and difficulty level. Videos 51, This file contains entry ID, user ID, name of the user, role, school, school ID, municipality name, entry date and time and subject name. With the help of the IT-department of Studi.se the authors figured out what every column in the raw data of activity means. Below one can read definition of different columns in the files. 1. Entry ID: every entry in the log has a unique ID. 2. Lesson ID: every lesson in Studi.se has an ID that differs from other lesson ID. 3. User ID: every user of the website has a unique ID which can identify a specific user. 4. Role: in the website two types of roles can be found. Teacher and student. Both teachers and students can open an account in Studi.se. Page 31 of 80