Evaluating Gaggle Use at Naples Elementary School By Jeff Wood Evaluation Educational Technology 505 Section 4173 Department of Educational Technology Boise State University Date Submitted Dec. 8, 2010 1
Table of Contents Learning Reflection Page 3 Executive Summary Page 4 Purpose of the Evaluation Page 5 Background Information Page 7 Description of Evaluation Design Page 10 Results Page 13 Discussion of the Results Page 17 Conclusions & Recommendations Page 19 Appendices Page 20 2
Learning Reflection Evaluation was an unknown area for me. Now that I have started studying evaluation I truly realize its value. I realize it is a sign of ignorance on my part, but I never thought much about how programs were evaluated. In my own personal and professional life I made informal evaluations based on hunches, feelings and emotions. That always worked for me because my work did not affect many other people. Now I teach, and my work affects hundreds of people. Over the course of years it may affect thousands. A hunch or feeling, in this case, is no longer valid. I need to know with more certainty about the programs and technology that I use. How successful are they? Since starting this course, I now try to think how I can evaluate many aspects of my life. Are my parenting decisions effective? Have we chosen the right music school for my children? Did I choose the correct software for our home computer? This list could go on and on. The benefit of this evaluation course to me will obviously be in my professional life. Even if I never become the educational technologist at our school, evaluation will have an impact on my teaching. Even if done informally, I will look more closely at programs and software that I use. It will be difficult for me to go back to using a hunch to evaluate the effectiveness of a program, software, or strategy. If I do become the educational technologist, the value of this course will become much more valuable. It is more likely that I will be in a position to perform official evaluations ordered by the school administration or district. Even though I am not an expert, I am happy that I know enough to perform a basic evaluation. It will help richen my life as a professional educator. Because I entered the teaching profession recently, I absorbed from the professional world around me that more technology meant more learning. It was a given that if technology was involved in the learning process, more learning took place. I never questioned this. As a result I embraced technology wherever I found it. It was later a shock to me that technology did not always have the desired results. Sometimes the latest technology does not provide the best results. Now I realize that any technology, or program, needs to be evaluated to understand it well enough to make intelligent decisions about its use. This course has helped me stop being blinded by technology. I am prepared that as a teacher I may come across a situation where the best technological solution to a learning situation may be a pencil and paper. Fortunately, I now know enough about evaluation to be able to use it to help me make that decision to choose between paper and pencil and a more technological solution. I have learned that a systematic approach is fundamental in a successfully evaluation. Some situations will not provide perfect conditions for an evaluation, but an analysis needs to be made and strategy chosen based on what is possible. The quality of the evaluation results will then change from situation to situation. It is only normal, and reflects an imperfect world. 3
Executive Summary This evaluation of Gaggle use at Naples Elementary School has been conducted to help make administrative decisions about its use and management. Gaggle consists of a suite of online communication tools meant to facilitate the integration of 21 st century into the academic lives of students. It has many applications available and is very adaptable to almost any classroom, granted there are computers with an internet connection available. However, Gaggle has been in place for many years now at Naples Elementary School (NES), and its use appears to be limited. This intention of this evaluation was to try to better understand Gaggle use at NES. Through the use of surveys, interviews, and system data, the evaluation was able to get a better understanding of the extent of Gaggle use amongst the eligible population at NES and possible reasons why the system was not utilized to a greater degree. The evaluation revealed that Gaggle use is not as high as it could be. The Gaggle system is set up for only the 4 th, 5 th, and 6 th grade teachers and students. 45% of this population even uses the system at all. The teachers at these grade levels who are using the system are using it for only the most basic of applications (email and file storage). Once the evaluation looked at who was using the more sophisticated applications with a potentially higher impact on learning (blogs, discussion boards, social networking), the numbers of teachers and students using them dropped significantly. Only 18% of all teachers in the target population surveyed used these advanced features. The evaluation also set out to understand the reasons for the lack of higher Gaggle use at NES. The reasons, discovered by the evaluation, are varied, but some are related. Some reasons that teachers gave for not using Gaggle were the lack of educational value in using Gaggle, already too much to fit into the school day, a lack of familiarity with Gaggle, and a lack of training in the use of the system. Another complaint was the cumbersome and slow teacher interface necessary for the teacher s management of student entries made in the system. The evaluation has made some recommendations based on the data gathered. Training on specific Gaggle use and management is the first suggestion, but the second is on technology integration in general terms. It would be beneficial for teachers to understand that technology can support learning in many ways, and that this integration does not necessarily result in more work for the teacher. 4
Purpose of the Evaluation What is the purpose of this evaluation? This evaluation came about as a reaction to perceived observations about the Gaggle communication program at Naples Elementary School. It is an online tool that could potentially be very valuable to teachers and students. In fact, the goal of the program is to integrate 21 st century skills into the academic lives of its users; however, there have been concerns that the system is not being used. In a time when funding and resources are very tight, it is important that Naples Elementary School take account of the programs it is funding in order to make wise decisions concerning such programs. For this reason Gaggle has become the object of this evaluation. What are the driving questions of the evaluation? In order to better understand Gaggle use at Naples Elementary School (NES) this evaluation sought to answer very few, but pertinent questions Of the 4 th, 5 th, and 6 th grade teachers and students who have access to Gaggle at NES, how many of them are using the system? It is important to know how extensively the system is being used by intended user group. Of those who are using the system, how are they using it? Gaggle has many diverse applications which can all enhance student learning, but which ones are being utilized and how. Are students just using the secure email to send messages to friends? Or is the system being utilized in a more educational way. What potential barriers are keeping teachers from using this set of tools? If teachers (and their students) are not using the system, this evaluation seeks to understand the possible reasons. The hope is to determine if attitude, perception, technological limitations or resources, lack of training, or any other cause could be the source of its lack of use. This evaluation may be able to help decisions be made regarding increased program training or discontinuation of the program. Who will the results impact the most? Considering that only certain grade-levels have access to this online tool, the results of this evaluation will not be pertinent to everyone at NES. Depending on how the school and 5
district act upon the results, the teachers and students of the appropriate grade levels may be affected. Otherwise, the results will be of most interest to the school s educational technologists, administrators, and district technology director. The results of this evaluation could affect the future funding of the program or possibly interventions to encourage is proper and expanded use. 6
Background Information Naples Elementary School is a Department of Defense Education Activity (DoDEA) school based in a town close to Naples, Italy. DoDEA schools don t come under the jurisdiction of any state, but are rather managed and funded directly from the federal government. These schools are spread all over the world, but are primarily located where American communities are involved in supporting American military presence abroad. These schools are divided into geographic zones, or districts. NES is located in the Mediterranean District. What was the origin of the program? Having students communicate with others on the internet has always had its risks. Some of the risks are that either students will send or receive inappropriate language. As a result, it was not safe to have students be involved in sending and receiving email, creating and posting blogs, and participating in online discussion forums. This program (Gaggle) was developed to integrate these and other 21 st century skills and technology into the academic lives of students and their teachers, all in a safe and protected environment. This safety is created with software that filters all communication and posts for suspect language and activity. Anything suspect gets forwarded to local system administrators, before it makes it to its destination. The system administrator can then intervene accordingly. What are the goals of the program? Within the safe environment created by Gaggle, students are able to interact with other students through the use of blogs, email, discussion boards, chat rooms, online digital storage, online document management and storage, online personal calendars, and online social networking. The goal of Gaggle to provide a safe online environment, which students and teachers may use to integrate their learning with 21 st century skills. What did previous programs look like? In 2010 a new version of the program was introduced. Prior to this most recent version, Gaggle was a bit more cumbersome and awkward to utilize. The older version also did not have as many online tools available. It seems that the developer of Gaggle put effort into improving its ease-of-use. Who are the people involved with the program? The school not only has all grade levels from kindergarten to 6th grade, but also includes Sure Start and Pre-School Child Development (PSCD). The total school population is approximately 900 students, but the number of students able to access the Gaggle program is only a portion of that total. Fourth, fifth, and sixth grade students do have 7
access to the program. That part of the student population is approximately 350 students. Those students are divided into 15 classrooms with approximately 20-25 students per classroom. There are five classrooms at each of these grade levels. What are the characteristics of the program? As described earlier, Gaggle is a suite of safe, online learning tools designed to engage students (and teachers) in a 21 st Century style of learning. Gaggle is set up so that there is a local system administrator for each school. In the case of NES, it is one of the school s two educational technologists. She establishes all of the student accounts and helps the students create and maintain their login information. This also includes keeping passwords secure. The system administrator is the immediate recipient of any suspect activity on Gaggle. She then informs other relevant professionals at the school, depending on the situation. Being that Gaggle is an application that is completely online, any computer with internet access may be used to access the system. When any computer at NES opens Internet Explorer (the default browser) the home page is set to the school s internet page. On that page a link to Gaggle is found. Students or teachers are then prompted to login. Students and teachers who find themselves off campus may easily find the same login page. Once logged into the system the user will find a page very similar to the one pictured below. The Gaggle package itself includes a variety of tools, most of which were already briefly introduced. The suite includes: 8
Email This email service has all the addresses of anyone in the school already loaded into the system. Homework Dropbox- Students can hand in work to their teachers online. Message Boards Students and teachers can hold online discussions. Zoho Docs & Spreadsheets This tool is very similar to Google Docs. It allows creation, storage and collaboration of online documents. Social Learning Wall This is a social networking application based on the idea of Facebook. Integrated Calendars Users can maintain an online calendar accessible from almost anywhere. Blogs This blog is just like any other. Who views the blog may be limited to just the teacher, made available to the whole school, the wider DoDEA population, or the general public. Digital Lockers Users have online storage space. It may also be used for simple storage of documents or for bringing documents to and from school. Chat Students can take part in chat discussions. Gaggle Tube This tool provides users with access to filtered video. At the time of this evaluation, it was not yet active at NES. 9
Description of Evaluation Design In order for administrators to make decisions about the future of Gaggle at NES, which is the goal of this evaluation, three driving questions have been identified. This evaluation attempts to answer them. 1. Of the 4 th, 5 th, and 6 th grade teachers and students who have access to Gaggle at NES, how many of them are using the system? 2. Of those who are using the system, how are they using it? 3. What potential barriers are keeping teachers from using this set of tools? Evaluation Model Used Based on the type of questions asked it was decided to use the decision-making model. This type of model is most appropriate in situations where the evaluation is employed to make decisions regarding the future use of the program. This is certainly the case of this evaluation. Data Collection Tools Although both qualitative and quantitative data were gathered to answer the questions asked by this evaluation, most of the data gathered was qualitative. Statistics on Gaggle use were gathered from the Gaggle system administrator at NES, but most of the other data was gathered through the use of interviews, observations, and surveys. How Background Data Was Gathered In order to understand the setting in which Gaggle is being used, a series of initial questions were asked of teachers. All 15 of the 4 th, 5 th, and 6 th grade teachers were given the survey. The survey was conducted electronically, using the forms tool within Google Docs. A copy of the survey utilized is included in the appendices at the end of this report. Although the survey was used to support all area of this evaluation, five questions were asked to get a better idea of the idea of the attitudes the target population of teachers has toward technology in general and their present use of it in the classroom. As a result, teachers were asked the following questions in the survey: Do you consider yourself a teacher who embraces new technology? Do you use a digital projector with your students? Do you use a Smartboard in your classroom? Do you use laptops with your students? How often do your students log into and use computers in your classroom? Those answering the questions were asked to respond with yes or no, or by selecting a number on a scale from one to five. 10
How Data Was Gathered for Question #1 Of the 4 th, 5 th, and 6 th grade teachers and students who have access to Gaggle at NES, how many of them are using the system? In order to establish how much of the target population is using Gaggle, two methods were employed to gather the appropriate data. First, the Gaggle system administrator was contacted to establish what type of data could be gathered about Gaggle use at NES. It was hoped that data on frequency of use and type of use could be established, but it was quickly confirmed that only very limited data is collected electronically on user activity. In fact, the only data from the system administrator s Gaggle account that could be obtained was how many student accounts were active. Active means that the account has been created and may be accessed by the user. There are certain questions included in the survey that attempt to gather data that relate to how many of the target population at NES use Gaggle. In regards to the first question, the survey did not try to discover how often users access the system. Its goal was to just discover if users have ever accessed it or not. Those survey questions relating to this first question are listed below: Have you ever logged into Gaggle? Have you ever used Gaggle with your students? How Data Was Gathered for Question #2 Of those who are using the system, how are they using it? The NES Gaggle system administrator was also asked about her observations of Gaggle use. Although those observations are considered useful, it was necessary to gather data from other sources. This is because the system administrator tends to only witnesses what happens in the computer lab and will not witness what is happening other places on campus or off-campus. Once again, the electronic survey was the used to gather data about Gaggle users at NES. This was deemed to be a more valuable data source than the interview with the system administrator. This series of questions was designed to gauge the extent to which teachers, and therefore students, were using Gaggle. It s most obvious application is email, which requires very little teacher input for students to use. There are, however, many more useful applications. This part of the evaluation is searching to see if they are being used by those who already use Gaggle. The questions asked are as follows: If you have used Gaggle with your students, did you have them use the student email? If you have used Gaggle with your students, did you have them use the digital lockers? If you have used Gaggle with your students, did you have them use the discussion boards? 11
If you have used Gaggle with your students, did you have them use the blogs? If you have used Gaggle with your students, did you have them use the social wall? How Data Was Gathered for Question #3 What potential barriers are keeping teachers from using this set of tools? In the case of answering this question, the interview with the Gaggle system administrator was more useful. She has been helping teachers work with Gaggle for several years now and has been observing the system in action during that time. She gave valuable input into answering this third question. Once again, the electronic survey was the used to gather data about Gaggle users at NES, and this series of questions was designed to establish the reasons that teachers may not be embracing this technology. There were some questions that needed to be answered on a scale of one to five, but feedback was also requested in the form of a short answer. The questions for this section are as follows: How would you describe Gaggle's educational value? What are your feelings about Gaggle s ease of use? How comfortable would you be instructing your students to perform tasks within Gaggle? What is your level of familiarity with Gaggle? And the short answer questions for this section are: If you have never used Gaggle, please describe what is stopping you? Do you have any additional thoughts about Gaggle that you would like to share with me? Although the data for this last question would not necessarily be unique to this section of the evaluation, it was left in this section because it is most suited to answering this third question. 12
Results Table I survey results of general teacher technology usage in classroom Do you consider yourself a teacher who embraces new technology? Do you use a digital projector with your students? 4 5 3 1 2 3 4 5 very much so not at all 9 1 1 1 2 3 4 5 very frequently never Do you use a Smartboard in your classroom? Do you use laptops with your students? 6 3 1 1 1 2 3 4 5 very frequently never 2 2 3 3 1 2 3 4 5 very frequently never 1 How often do your students log into and use computers in your classroom? 6 4 1 1 2 3 4 5 very frequently never 13
Table II survey results of general teacher usage of Gaggle at NES Have you ever logged into Gaggle? Have you ever used Gaggle with your students? no 45% yes 55% no 55% yes 45% Table III Specifics of Gaggle use at NES If you have used Gaggle with your students, did you have them use the student email? If you have used Gaggle with your students, did you have them use the digital lockers? not used Gaggle with my students 55% yes 45% not used Gaggle with my students 55% yes 45% If you have used Gaggle with your students, did you have them use the blogs? If you have used Gaggle with your students, did you have them use the discussion boards? have not used Gaggle with my students 55% no 27% yes 18% have not used Gaggle with my students 55% no 27% yes 18% 14
If you have used Gaggle with your students, did you have them use the social wall? not used Gaggle with my students 55% no 27% yes 18% Table IV Possible barriers to Gaggle use at NES How would you describe Gaggle's educational value? What are your feelings about Gaggle s ease of use? 3 2 1 1 1 2 3 4 5 strong value weak value 5 2 1 3 1 2 3 4 5 very easy to use difficult to use How comfortable would you be instructing your students to perform tasks within Gaggle? What is your level of familiarity with Gaggle? 2 3 1 1 1 2 3 4 5 very familiar not at all familiar 4 4 2 2 1 2 1 2 3 4 5 very familiar not at all familiar Table V Possible barriers to Gaggle use at NES 15
Survey Question If you have never used Gaggle, please describe what is stopping you? Do you have any additional thoughts about Gaggle that you would like to share with me? Teacher Responses Do not know the program at all and do not know how to implement its use with math. Yes, since our computer time is on Friday it is often a workday, training day or holiday. With that said, one of the sessions was used to log in and to take an SRI. Not interested. Time is taken up with core subjects. Taking the time to make myself familiar with it It s seems like something else I have to monitor. I find that it is a valuable tool for the students. They need exposure to 21 st century ways of communicating. The drawback is that for the teacher the system is cumbersome to organize and also to review student submissions. It takes much longer than reviewing student work completed on a piece of paper. There are too many clicks and wait time between those clicks. I would like to learn how it works so perhaps I could use it. I try to embrace all of the technology that our school has to offer. I think that students may benefit from using technology as it relates to academics, researching, reading, discussing. Familiarity with technology is required to succeed in today s world and students need to be able to use and apply technology in its various forms. Have been told this is just students emailing each other. What s the point? Table VI Results of Interview with local Gaggle system administrator Question Notes from Responses Of the 4 th, 5 th, and 6 th grade teachers and students who have access to Gaggle at NES, how many of them are using the if they are being actively used. system? Of those who are using the system, how are they using it? What potential barriers are keeping teachers from using this set of tools? The only existing records gathered by the Gaggle system show that all students in 4 th, 5 th, and 6 th grade have accounts. It does not show Amongst those three grade levels, five teachers seem to be using the system in at least some way. From her point of view, there is only limited use of Gaggle. It offers so much. Teachers don t see the value in using Gaggle They are not comfortable with using Gaggle 16
Discussion of the Results Validity of Results The data gathered did help gain a better understanding of Gaggle use at NES. Data was collected using the planned process. An electronic survey was sent out all 15 teachers of the target population. 73% of them completed the survey, at least in part. That percentage of participation gives validity to the results, although it would have been more effective to have had an even higher response. The only other authority from whom data could have been gathered was the local Gaggle system administrator. An interview was conducted with her, which provided further insight into Gaggle use at NES. It was also considered to interview the Gaggle representative at the district level, or higher, but that would not have given local information from NES. It was decided that such information would be irrelevant. Another source of data would have been the students, but they are not the protagonists in this sense. It is the teachers that need to initiate the academic use of the system. For this reason the evaluation centered on input from the teachers capable of implementing Gaggle use locally. For the reasons mentioned above, it is believed that the results obtained are valid, and therefore can reliably be used to make administrative decisions regarding the use of this program at NES. All three questions outlined at the beginning of the evaluation were answered. Below is a discussion of the results for each question to be answered. Of the 4 th, 5 th, and 6 th grade teachers and students who have access to Gaggle at NES, how many of them are using the system? It appears that only slightly more than half of the surveyed population of teachers has ever logged into the system, and slightly less than half actually use the system. So it appears that usage of the Gaggle system at NES is significantly below what it could be. Although it has not been a part of the survey, it was learned from speaking with some of the remaining teachers that did not complete the survey, that they do not use the Gaggle system either. This tends to point to an even lower usage of the Gaggle system than learned through the data collection process for this evaluation. Of those who are using the system, how are they using it? After establishing that Gaggle use is less than 50% among teachers of the target population, the evaluation was able to establish a better picture of what type of Gaggle use was occurring. Keeping in mind that Gaggle has a number of tools (nine +) available to integrate into a student s academic life, this evaluation only tried to establish if only a few of the those were being used. The results show that email and file storage tend to be the most utilized. These are the easiest Gaggle tools for teachers to manage. In fact these two tools show the highest teacher usage amongst those who use Gaggle. 100% of teachers using Gaggle in their 17
classrooms use the email and file storage tools. Once blog, discussion board, and social network use was examined, the use of these tools went way down. For these three tools Gaggle use goes down to 40% of amongst teachers who use Gaggle in their classrooms. What potential barriers are keeping teachers from using this set of tools? This last question is the logical extension of the other two. The evaluation tries to understand why the system is underused. The interview of the system administrator and the teacher survey proved valuable to understanding the answer to this last question. Both these data sources pointed in similar directions. The responses given from both these sources indicate that teachers are not familiar with the program, do not have time for it, see it as extra work for themselves, or think it is a waste of time. Another complaint is that it is slow and cumbersome for teachers to manage the system for students. This helps explain the results found for the first two driving questions of this evaluation. Costs-Benefit Summary One aspect of the program that could have been considered is a comparison of costs and benefits. Unfortunately, it was not possible to establish the cost to NES for the purchase of the program. It was purchased at a much higher level than the school or even district level. It appears that a world-wide purchase was made for all DoDEA schools. At the time of the printing of this report, there was no more precise information. This means that the only costs taken on so far by NES are the manpower hours needed to create approximately 350 student accounts and place the link to the site on the school s website. This would mean from the local point-of-view, whatever gains in 21 st century skills by the students may be, it is worth the NES investment. 18
Conclusions & Recommendations Immediate Conclusions Some teachers do not understand the benefits of using Gaggle. They see it as something else they have to fit into the day, something else they have to manage, or something that does not have academic value. Training and support on the value of Gaggle, as well as the importance of technology standards needs to be given to teachers. Training and support on how to integrate Gaggle into the curriculum needs to be given to the staff. It needs to be taught that if integrated properly, it shouldn t result in extra work, but extra student engagement. There is a valid concern that Gaggle can be slow and cumbersome for the teacher to grade and check student work submitted on Gaggle. Gaggle should be contacted to communicate these concerns. Long-Range Planning for Part 2, you will make suggestions as to how the evaluation will have a bearing on long-range planning or development. Considering that some teachers don t seem to understand that technology integration into the curriculum doesn t mean more work, more effort should be placed into long-term training goals that aim to teach technology integration in general. Gaggle could be one solution included in that training scheme. Efforts should be made to influence Gaggle to streamline the teacher interface. If Gaggle is cumbersome, then even the most technologically integrated of teachers will shy away from its use. Evaluation Insights This evaluation was an opportunity to objectively view Gaggle and look at how it is perceived in an educational setting. The first reaction was to find fault with any negative attitude facing the use of Gaggle. However, certain concerns are legitimate. If teachers have not had the training necessary, how are they expected to use a new system full of online tools? As well, it is understood the problems created by a slow and cumbersome teacher interface. If it takes much longer to grade an assignment submitted online, where is the incentive to take assignments in this way? It was also anticipated to have 100% of teachers submit the survey. With only fifteen of them, it seemed reasonable to expect 100% submission. That did not happen. Another problem is that the survey needed to be modified after it was already sent out. It would have been too disruptive to recall it and start again. Already there was not 100% submission by all teachers. It would have likely been worse had the survey been recalled. It would have been better to review it further before sending it out to the target population. It is also would have been worth considering student feedback. This might have helped establish what types of activities and tools students use, and what they get out of the experience. However, that may have provided information that doesn t necessarily coincide with the goals of the evaluation. 19
Appendices There is no required formatting for the appendices. Types of things that you might include in the appendices are surveys, examples of actual data you collect, artifacts created by participants, pertinent background documents, and so on. The thing to keep in mind is that the appendices section should be highly organized. Survey 20