An Evaluation of Desmos for Use in Physical Science Courses

An Evaluation of Desmos for Use in Physical Science Courses Submitted to John Thompson, PhD Boise State University ABSTRACT Prepared by Marta R. Stoeckel- Rogers EdTech 505: Evaluation for Educational Technologists

2 Desmos in Physical Science Table of Contents Summary... 3 Program Description... 3 Program Objectives... 4 Program Components... 4 Evaluation Method... 4 Participants... 4 Procedures... 4 Data Sources... 5 Results... 5 Linear Motion Assessment... 5 Classroom Observations... 6 Student Survey... 6 Discussion... 9 Linear Motion Assessment... 9 Classroom Observations... 9 Student Survey... 9 Overall Conclusions... 10 Project Cost... 10 Appendix A: Linear Motion Assessment... 11 Appendix B: Student Survey... 14 Appendix C: Open-Ended Survey Item Responses... 16 Notes on Interpretation and Analysis of Open-Ended Survey Items... 16 Engineering & Physical Science Responses... 17 Physics Responses... 18

Desmos in Physical Science 3 Summary In preparation for a bring your own device (BYOD) program scheduled to begin in the spring of 2015, the use of Desmos (desmos.com), a web-based tool for creating and analyzing graphs, was piloted in physical science courses. The courses that utilized Desmos are largely lab-based, meaning that students regularly complete basic data analysis focused on the creation and analysis of graphs, in order to learn new concepts in science. The goals of the program were to use Desmos to help students to generate science content knowledge, to teach students to identify and describe trends in data, to assist students in finding the equation for a line of best fit on a graph. The program was also attempting to identify a tool for creating and analyzing graphs that students would find easy to use. The program evaluation included several tools. First, students took both a pre-test and a post-test over content taught using data analysis activities built around the use of Desmos. Second, classroom observations were conducted to determine whether students were able to successfully complete tasks using Desmos, identify particular challenges students encounter when using Desmos, and collect information on student attitudes towards Desmos. Finally, students completed a survey regarding their perceptions of Desmos. Each of these data sources were used in two physical science courses. Engineering & Physical Science is a one-trimester course required of all 9 th grade students. Physics is a year-long elective targeted at 12 th grade students interested in pursuing college to study a STEM field. The evaluation reached favorable conclusions regarding the use of Desmos in physical science courses. Students acceptable growth on the test of science content, indicating that they were able to generate science content knowledge based on data analysis completed using Desmos. Classroom observations indicate that students were able to successfully create graphs, identify and describe trends, and determine equations for a line of best fit from within Desmos. A combination of classroom observations and the results of the student survey suggest that students find Desmos to be accessible and easy to use. Based on this data, Desmos should be used in the physical science curricula in the future with some modifications to the program to address issues identified in the evaluation. Program Description During the fall of 2014, Desmos, an online graphing calculator intended for educational uses, was piloted for use in physical science courses at Tartan High School. Tartan is located in a suburb of St. Paul, Minnesota and serves a diverse population of approximately 1200 students. During the pilot, Desmos was used by students in Engineering & Physical Science, a one-trimester course required of all 9 th grade students, and Physics, a year-long elective targeted at 12 th grade students interested in future study in STEM fields. Both courses are taught using lab-based methods, meaning students must frequently engage in data analysis, including graphing and graph interpretation. Students in Engineering & Physical Science were required to use Desmos for all graphs normally a part of the course while students in Physics were required to use Desmos for the first three weeks of the course, then given their choice of different graphing tools. In the spring of 2015, Tartan will begin a formal BYOD program, leading many of the school s science teachers to consider options for producing graphs digitally. Desmos was selected for a pilot and evaluation because it works on a wide variety of

4 Desmos in Physical Science devices. During the pilot, students used a mix of school-owned and student-owned devices, ranging from smartphones to desktop computers, to create and analyze graphs in Desmos. This evaluation covers the use of Desmos in a single, 12-week long trimester. If the program is deemed promising, the program will continue with any necessary modifications as a part of Tartan s BYOD program. If the evaluation finds significant issues with the use of Desmos, it will be abandoned after the fall trimester and other tools will be examined. Program Objectives This program is intended to support students in developing graph interpretation skills, a key component of both scientific literacy and the scientific process. This program is also intended to support the development of science content knowledge through handson, student-centered strategies. 1. Students will be able to generate science content knowledge through data analysis, including graph interpretation. 2. Students will be able to recognize and describe trends in data. 3. Students will be able to determine the equation for a line of best fit. 4. Students will consider the graphing tool easy to use. Program Components The use of Desmos is intended to supplement the larger program of instructional practices used in physical science courses at Tartan. These courses are heavily lab-based, meaning that students must frequently interpret data and produce graphs. This need led to numerous opportunities scattered throughout the trimester for students to use Desmos to create and analyze graphs. Students were provided with instruction over the basic functions, including entering a data table to produce a scatterplot and producing a line of best fit, as part of the first lessons where Desmos was used. Students used a variety of tools to access Desmos. On several occasions, a cart of Windows netbooks was brought into the classroom for students to run Desmos from. Students were also given the option of connecting their own devices to the classroom wifi to access Desmos. While a few students had their own laptops or tablets, most used smartphones to access Desmos. Evaluation Method Participants All participants in this evaluation came from two groups. 30 students were enrolled in Engineering & Physical Science, a required one-trimester course for all 9 th grade students. 115 students were enrolled in Physics, a year-long elective targeted at 12 th grade students intending to pursue college education in a STEM field. These students were selected by virtue of having been randomly assigned to an instructor interested in using Desmos during the fall term. Procedures The evaluation was conducted during the first term in which Desmos was utilized in Physics and Engineering & Physical Science, meaning both the instructor and the students were learning to use the tool. Students in both courses were provided with instruction on using a graph from a series of data points and finding the

Desmos in Physical Science 5 equation for a line of best fit for the data. Students in Engineering & Physical Science were required to use Desmos for all scatter plots produced during the course. Students in Physics were required to use Desmos for graphs of lab data associated with instruction over linear motion, then allowed to use the graphing tool of their choice for the remainder of the course. Data Sources The first data source is the linear motion assessment, included in appendix A, given to all students enrolled in Physics. The assessment was given as a pre-test during the first week of the program, then repeated as part of a post-test during week four of the program. This assessment includes several questions which explicitly require students to use graph interpretation skills. The remaining questions address science content knowledge developed using lab exercises emphasizing graph analysis. During these lab exercises, students were required to use Desmos to produce graphs which were then used to develop a student understanding of key science concepts. This tool primarily assessed the first two objectives, namely students ability to generate science content knowledge via data analysis and students ability to recognize and describe trends in data. A score of 60%, or 8/13, on this assessment is considered mastery by the teachers who developed the assessment. Next, classroom observations were conducted while students used Desmos through out the program. The evaluator first tracked whether students were able to successfully describe a trend in experimental data and determine the equation for a line of best fit when using Desmos, as described in two of the program objectives. The observations also attempted to assess whether students found Desmos to be easy to use by watching how much support students required beyond the initial instruction and how many students selected to use Desmos when given a choice. The final data source was the student survey included in appendix B. This survey was delivered to all students in Physics and Engineering & Physical Science to record their attitudes regarding how well the program met its objectives. Students completed the survey during the school day to encourage a high response rate. A random selection of just over half of the students in these courses completed the survey. The survey was given during week 12 of the program. Results Linear Motion Assessment Table 1 summarizes the results of the linear motion assessment used in Physics. Results are given for both the pre-test and the post-test as percentages of a total score. The gain represents the increase individual students achieved from the pre-test to the post-test. Note that all statistics increased from the pre-test to the post-test. One of the most important pieces of data from this table is the median gain of 30.8%, indicating acceptable growth in student understanding of the content over the course of instruction. Table 1: Linear Motion Assessment Results MEDIAN MAXIMUM MINIMUM RANGE PRE-TEST 46.2% 69.2% 0.0% 69.2% POST-TEST 76.9% 100% 61.5% 38.5% GAIN 30.8% 84.6% 15.4% 69.2%

6 Desmos in Physical Science Histograms of student scores for the pre-test and the post-test are shown in figure 1. Scores to the right of the orange bar achieved the benchmark for mastery while scores to the left fell short. Note that on the pre-test, only 19 out of 113 students scored at or above mastery. On the post-test, however, all students scored at or above mastery. Figure 1: Linear Motion Assessment Scores Linear Motion Assessment Results Number of Students 35 30 25 20 15 10 5 0 Scores Below Mastery Scores at or Above Mastery 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Score (out of 13) Pre-Test Post-Test Classroom Observations When using Desmos, students were able to consistently create a graph based on experimental data and describe any trends in the data in accordance with lesson objectives. In addition, students were able to determine the equation for a line of best fit. During most lessons, students required minimal support beyond the initial instruction over the use of Desmos. In an experiment involving particularly small values, students did need assistance in setting an appropriate range to display. Most of the technical support provided by the instructor was related to Internet connectivity and other issues not directly related to Desmos. When 12 th grade students were provided the opportunity to select a tool for creating graphs, 72% of lab groups opted to use Desmos. Each group typically had several reasons for selecting Desmos. Nearly all groups cited the ease of use of Desmos, the ability to save and share graphs, and the ability to use Desmos on any internetenabled device, including a smartphone. Student Survey The student survey contained six Likert scale items where a 1 indicated strongly disagree and a 5 indicated strongly agree. Items were worded such that agreement indicated a positive attitude towards the use of Desmos. Table 2 contains the text of these items along with the identifiers used in the graphs of student responses.

Desmos in Physical Science 7 Table 2: Student Survey Question Summary Q1 It is easy to plot data in Desmos. Q2 Creating a graph in Desmos helps me to understand what my data means. Q3 It is easy to find the equation for a line using Desmos. Q4 When I use Desmos, I can focus more on what the graph means than on making the graph. Q5 Desmos is a helpful tool for my science class. Q6 Desmos should continue to be used in science. Figure 2 contains a histogram the total number of students who selected each response for the Likert scale items. Most students selected either neutral or agree for all of these items, suggesting an overall neutral to positive attitude towards Desmos. Figure 2: Number of Responses by Question Number of Responses by Question Number of Responses 30 25 20 15 10 5 0 Q1 Q2 Q3 Q4 Q5 Q6 Strongly Disagree (1) Disagree (2) Neutral (3) Agree (4) Strongly Agree (5) Figure 3 provides the mean response for each Likert scale item. Mean responses are provided for students in each course, as well as for the overall population included in the evaluation. The mean responses range from 3.27 to 4, reinforcing the overall weakly positive attitude towards Desmos seen in figure 2. It is worth noting that students enrolled in Engineering & Physical Science tended to display a slightly more positive attitude towards Desmos than students enrolled in Physics.

8 Desmos in Physical Science Figure 3: Mean Responses by Question and Course Mean Responses by Question and Course Q1 Q2 Q3 Q4 Q5 Engineering & Physical Science Physics All Students Q6 0.00 1.00 2.00 3.00 4.00 5.00 Mean Response Figure 4 summarizes responses to the open-ended survey items by categorizing each response as primarily positive or primarily negative towards the use of Desmos. In this section of the survey, students demonstrated a much more strongly positive attitude towards Desmos than in the Likert scale items with 82% of all respondents demonstrating an overall positive attitude in the open-ended questions. As seen in table 3 and figure 3, students enrolled in Engineering & Physical Science again showed a slightly more positive attitude towards Desmos than students enrolled in Physics. Figure 4: Open-Ended Response Summary 100.00 Open-Ended Response Summary Percent of Responses 80.00 60.00 40.00 20.00 Primarily Primarily Negative 0.00 Engineering & Physical Science Physics Overall

Desmos in Physical Science 9 Discussion This evaluation was intended to determine the effectiveness of Desmos in physical science courses at Tartan High School. The goal is to determine whether Desmos will be an appropriate complement to the BYOD program Tartan is scheduled to implement in spring 2015. The program has four identified objectives: 1. Students will be able to generate science content knowledge through data analysis, including graph interpretation. 2. Students will be able to recognize and describe trends in data. 3. Students will be able to determine the equation for a line of best fit. 4. Students will consider the graphing tool easy to use. Based on the pre-test and post-test administered to students enrolled in Physics, classroom observations of students using Desmos, and the results of a survey completed by students, the program is meeting its intended objectives and should be continued with some modifications. Detailed discussion based on each data source follows. Linear Motion Assessment The scores on the linear motion assessment suggest that program successfully met objective 1 in that students were able to demonstrate a mastery of science content knowledge taught through lab exercises utilizing Desmos. The program also met objective 2 since on the assessment students were able to describe trends in the graphs included in the assessment. As indicated by figure 1 in the results section, only 19 out of 113 students achieved a score recognized as mastery on the pre-test, while all students scored at or above the cutoff for mastery on the post-test. In addition, the median score increased from 46.2% on the pre-test to 76.9% on the post-test with the median student s score increasing 30.8% during the unit. These increases fall within the acceptable range for the Physics course. Classroom Observations The classroom observations were intended to assess objectives 2, 3, and 4. During these observations, all students were able to identify and articulate key patterns in experimental data after producing a graph using Desmos, providing additional data that the program has met objective 2. These observations also provide evidence that the program has met objective 3 in that all students were able to successfully produce a line of best fit and find the equation for that line when using Desmos. The observations also provide evidence that the program has met objective 4 in that students needed minimal support using Desmos beyond the initial instruction provided. Student Survey While the student survey includes elements of all program objectives, it most directly addresses student perceptions on objective 4, providing data on whether students perceive Desmos as easy to use. The survey suggests that students are overall positive, though not enthusiastic, toward the use of Desmos in their science classes. The response count in figure 2 shows that for all Likert scale items, most students selected either neutral or agree where agreement indicates a positive attitude towards Desmos. The mean responses in figure 3 are consistently above neutral. There is room for students to develop stronger agreement with the statements, but the program is on the right track with regard to objective 4.

1 0 Desmos in Physical Science The open-ended questions included in the survey provide stronger evidence that the program has met objective 4. Figure 4 includes what fraction of the open-ended responses demonstrate an overall positive or negative attitude toward Desmos with 82% of the responses falling into the positive category. When asked what students like about using Desmos, many students described the tool as easy or quick and easy. A few students described Desmos as confusing when asked what they do not like about using it, but the responses also suggest that additional instruction may remedy the confusion. It is worth note that students enrolled in the 9 th grade course, Engineering & Physical Science, tended to demonstrate more positive attitudes towards Desmos on both the Likert scale and open-ended items than the students enrolled in Physics, a 12 th grade course; these differences can be seen in figures 3 and 4. Based on the open-ended responses, this may be in part due to what students consider an alternative for creating and analyzing graphs. The students in Engineering & Physical Science frequently compared Desmos to creating graphs by hand while the students in Physics frequently compared Desmos to using a TI-Nspire graphing calculator, a tool that few 9 th grade students have. Overall Conclusions Given the positive results described above, the use of Desmos in Tartan s physical science courses should continue. There is sufficient evidence that all four objectives are currently being met by the program. There are, however, opportunities for improvements. On the open-ended survey items, several students commented that they found Desmos confusing. Given the number of students who described Desmos as easy to use, additional instruction on the use of Desmos may not be an appropriate solution. However, students could easily be provided with resources including brief screencasts and illustrated how-to guides describing how to perform common tasks within Desmos. Another crucial theme from the open-ended portion of the student survey is the importance of the device used to run Desmos. During this program, most students used either personal smartphones or school-owned Windows netbooks. Many students commented that it was difficult to type or use many of the advanced features from a smartphone. A number of students also found that the small screen size of a smartphone made it difficult to get a clear view of the graph. When using netbooks, many students were frustrated by the long start-up time and the Internet connectivity issues frequently encountered with these particular devices. When Tartan s BYOD program is formally implemented, students should have access to personal Chromebooks, tablets, or laptops which can be expected to significantly improve the student experience using Desmos. Project Cost The primary cost in this evaluation was the daily rate for Marta Stoeckel, the principal evaluator. This includes five days of classroom observations in addition to the development and analysis of other data sources. The primary operational cost was photocopies of the student survey and content knowledge assessments. ITEM COST Marta Stoeckel: 10 days at $500/day $5,000 Operational Costs (photocopies, office supplies, etc) $100 Total Costs $5,100

Appendix A: Linear Motion Assessment Desmos in Physical Science 1 1

1 2 Desmos in Physical Science

Desmos in Physical Science 1 3

1 4 Desmos in Physical Science Appendix B: Student Survey Student Survey: Desmos in Physics & Physical Science 1. It is easy to plot data in Desmos. Mark only one oval. 1 2 3 4 5 Strongly Disagree Strongly Agree 2. It is easy to find the equation for a line using Desmos. Mark only one oval. 1 2 3 4 5 Strongly Disagree Strongly Agree 3. Creating a graph in Desmos helps me to understand what my data means. Mark only one oval. 1 2 3 4 5 Strongly Disagree Strongly Agree 4. When I use Desmos, I can focus more on what the graph means than on making the graph. Mark only one oval. 1 2 3 4 5 Strongly Disagree Strongly Agree 5. Desmos is a helpful tool for my science class. Mark only one oval. 1 2 3 4 5 Strongly Disagree Strongly Agree

Desmos in Physical Science 1 5 6. Desmos should continue to be used in science. Mark only one oval. 1 2 3 4 5 Strongly Disagree Strongly Agree 7. What do you like about using 8. What do you dislike about using 9. Do you have any other thoughts in the use of Desmos in physics and physical science? Powered by

1 6 Desmos in Physical Science Appendix C: Open-Ended Survey Item Responses Notes on Interpretation and Analysis of Open-Ended Survey Items The following pages contain student responses to the open-ended survey items in a table format. Each row represents the responses provided by a single student and responses have been separated by the course students are enrolled in. Blank cells indicate that the student did not respond to the item. Each response was rated by the evaluator as primarily positive or primarily negative towards Desmos itself. Since this evaluation is concerned with Desmos itself, not the technology used to access it, this rating was based only on comments about Desmos as much as possible. In ambiguous cases, responses to the Likert scale items were considered to identify the student s general attitude as positive or negative.

Desmos in Physical Science 1 7 Engineering & Physical Science Responses Four surveys did not contain responses to these items. Those surveys were not included in the analysis. Q7: What do you like about using Q8: What do you dislike about using Q9: Do you have any other thoughts on the use of Desmos in physics and physical science? Evaluator s Rating Its easy Nothing I like it Its a easy way to put things into a clean graph Its difficult at first No I like messing around with it I don't really use it I prefer to make graphs by hand It's quicker than making one yourself It was easier to find the slopes and graphing Easy It makes the graph quick It takes a lot less time using Desmos than making your own It's quick and easy It makes the graph for me, so I don't have to take a while figuring out how to graph the lines Nothing No Nothing Nope Negative Some people don't know how to do it No At first, it was kinda confusing to use No Hard to find some No things Nothing Nope It gets kind of confusing sometimes The netbooks are really slow It should be continued being used.

1 8 Desmos in Physical Science Physics Responses Twelve surveys did not contain responses to these items. Those surveys were not included in the analysis. Q7: What do you like about using Its fun to learn what different graphs mean It's different, useful when you don't have a graphing calculator When it actually works, I can zoom in on data It is helpful when a calculator is not available It is similar to Excel; easy to use Its simple to read and create them Its relatively easy once its used a couple times and its easier to highlight and move data compared to Microsoft Excel Q8: What do you dislike about using Q9: Do you have any other thoughts on the use of Desmos in physics and physical science? Evaluator s Rating It's time consuming Nope It's weird to use at first I never understood how to use it and I find it really confusing to graph things It is sometimes hard to access Certain things they don't have compared to Excel or maybe I just don't know Sometimes its difficult with extra settings Its confusing at first to get the equation out and typing in data was somewhat tricky too nope I actually prefer using Google graphs, I think it's way easier, but I don't think you should eliminate Desmos because everyone prefers different things No N/A It could only be beneficial, not counter beneficial None really Negative Negative Negative

Q7: What do you like about using It really helps me to visualize things I like the visual of it because I am a visual learner You just input numbers and a graph is made You can use it anywhere, you don't need the unreliable net books The graph is very detailed More accurate than other programs I like that Desmos allows us to see the graphs and equations for realtime data Its simple and I can do it on my phone Fast, easy, accurate Q8: What do you dislike about using Nothing much. Maybe associating the picture with velocity/acceleration graphs is a little difficult. Its kind of difficult to know how to do everything on it On a mobile device it is very confusing Its a bit confusing Its complicated to make Hard to find what you're looking for I have not found much dislike other than a few complications w/ the interface Its small print so a little challenging to type and navigate Hard to get perfect line of best fit Desmos in Physical Science 1 9 Q9: Do you have any other thoughts on the use of Desmos in physics and physical science? I think it would be a great tool for the future. Nope Much easier on a computer than on a mobile device N/A N/A Nope Nope Evaluator s Rating Negative Negative

2 0 Desmos in Physical Science Q7: What do you like about using I like that I'm able to find about everything I need when I do it correctly Lots of features, has everything you need mathematically Allows me to interpret graphs in different ways It is an easy tool to use It makes a graph It shows you a picture of your data It is extremely easy to make the graph and it is easy to understand It is a way that is simple if you know how to use it, but if you don't have instruction on what to do it isn't very self-explanatory Q8: What do you dislike about using Making sure that I do it correctly because if its wrong all the work I did along with it was wrong It can be confusing at times I can do it just as easy on my calculator Its on the computer, the name is weird, and its overly complicated N/A Its a pain to use it when I have a graphing calculator already Q9: Do you have any other thoughts on the use of Desmos in physics and physical science? No No Great tool to introduce students to No A TI-Nspire CAS is a far superior item for this level of physics than Desmos It goes faster than on Google Docs, so I like it better, it should be used more often Evaluator s Rating Negative Negative Negative

Q7: What do you like about using I like that you are able to create multiple lines on the same plane It helps to understand what is happening with the data It gives us more time to work on other parts of what we are doing It is a fairly easy tool to make accurate graphs quickly Lots of functions to use Not really because the graph has to be extremely precise I like the convenience and accuracy as opposed to hand drawing them The graphs are accurate and easy to interpret Q8: What do you dislike about using Using the iphone app things are hidden in menus Its confusing to use sometimes It is sometimes hard for me to understand Sometimes it would not let me enter data if I used it on my phone Can be difficult to find the equation the first time or 2 you use it Desmos in Physical Science 2 1 Q9: Do you have any other thoughts on the use of Desmos in physics and physical science? We should spend more time on how to use Desmos exactly Desmos should be used but it should have walk throughs on how to use it Evaluator s Rating Negative The app No That it always ends up in a decimal The fact that I have trouble creating graphs in the first place The numbers get mixed up and can be hard to put in equations Nope I have no other thoughts

2 2 Desmos in Physical Science Q7: What do you like about using Makes it easy to interpret the data and understand what it means It is generally user friendly I don't find it helpful Its handy and better than other things Its a good visual aid It is faster and more accurate It makes everything easy Q8: What do you dislike about using Can be hard to make depending on the circumstances I was unfamiliar with the program at first, so it took a little while to get the hang of it It's too much work, I rather use formulas At first is hard to figure out how to type the stuff in Have to use internet, easier to use a graphing calculator Q9: Do you have any other thoughts on the use of Desmos in physics and physical science? It should continue to be used! It should continue to be used No No N/A Evaluator s Rating Negative Negative