VIRTUAL VIRGINIA BEACH e-learning PROGRAM: COMPREHENSIVE EVALUATION

VIRTUAL VIRGINIA BEACH e-learning PROGRAM: COMPREHENSIVE EVALUATION VIRGINIA BEACH CITY PUBLIC SCHOOLS Department of Planning, Innovation, and Accountability Office of Research and Evaluation 2512 George Mason Drive Virginia Beach, VA 23456 January 2014 2014 All Rights Reserved i

VIRTUAL VIRGINIA BEACH e-learning PROGRAM: COMPREHENSIVE EVALUATION EXECUTIVE SUMMARY AUTHORS: Shawn L. Dickerson, M.S., Research Specialist, Lead Evaluator Heidi L. Janicki, Ph.D., Coordinator of Research and Evaluation Department of Planning, Innovation, and Accountability OTHER CONTACT PERSON: Donald E. Robertson, Jr., Ph.D., Assistant Superintendent Department of Planning, Innovation, and Accountability On August 21, 2012, the School Board approved the Virtual Virginia Beach e-learning (VVBe) Program Evaluation Readiness Report including the program goals and objectives, the evaluation plan, and the recommendations concerning the evaluation of the program. The recommended evaluation plan was to conduct a comprehensive evaluation during the 2012-2013 school year, in which program implementation, student academic outcomes, stakeholders perceptions of the program, and program cost or savings would be examined. This evaluation report is based on the School Board-approved plan and focuses on the operational components of the program, characteristics of students who participated in the program, progress made toward meeting the goals and measurable objectives, stakeholders perceptions, and the cost or savings of the program to the school division. The evaluation was based on quantitative and qualitative data that were collected through surveys, reviews of program documents, course enrollment and withdrawal records, and student course grades and standardized test performance data from the Virginia Beach City Public Schools (VBCPS) data warehouse. SUMMARY OF EVALUATION FINDINGS Evaluation Question 1: What were the components of program implementation? Since the inception of the VVBe Program in 2004, a total of 17 courses have been offered. During 2012-2013, 12 online courses were offered through the program in the areas of English, mathematics, science, history, government, and physical education. A total of 5 sections of 3 different courses were offered during the regular school year, and a total of 36 sections of 12 different courses were offered during the summer. Across all years, VVBe course enrollments have continually increased from 98 enrollments in 2004-2005 to 992 during 2012-2013. Health & PE I & II have had the highest number of students (1,944) enrolled compared to all other VVBe courses. The majority of students indicated that they learned about the VVBe Program through the school counselor (46%) and that this staff member assisted the students in deciding if an online course would be an appropriate option to pursue (53%). Parents were most likely to indicate that they learned about the VVBe Program through their child (41%) and that the Distance Learning webpage was a resource utilized for additional information (58%). Additionally, 92 percent of VVBe students agreed that the resources available were effective in helping them to understand the program.

During the 2012-2013 regular school year, ninth- and tenth-grade students could enroll in one online course that could be counted as part of the standard course load or as an additional seventh class, and eleventh- and twelfth-grade students could enroll in up to two online courses. Additionally, any interested middle school student could enroll in a single online course during the summer term, but only after obtaining approval. Students interested in taking an online course during the regular school year are required to contact their guidance counselor and register during the normal annual course selection process. Conversely, the registration process for online summer courses is independent and accomplished online. Both VVBe students and parents agreed that the registration processes were clear and efficient (93%). Once enrolled students participate in an orientation process which aims to assist the students in understanding the course demands, expectations, policies, procedures, and the online course delivery platform, Desire2Learn. Based on survey responses, students indicated that they understood the demands and expectations of the online courses (91%), understood the policies and procedures (92%), and understood the course delivery system (89%). Additionally, 82 percent of students indicated that the delivery platform was user-friendly, and 89 percent of teachers agreed that Desire2Learn was user-friendly for the students. Students indicated that the online courses were well organized (81%) and that the materials (88%) and assignments (82%) were user-friendly. All VVBe teachers are exposed to a rigorous training process. These teachers must complete a seven-week online pedagogy course and be evaluated by the course facilitator. On the teacher survey, 100 percent of teachers indicated that they were satisfied with this process. As of the summer 2013 term, the majority of the VVBe teachers had taught 2 to 3 online courses. Students enrolled in the VVBe Program are provided additional supports to assist them if technical problems arise with the delivery platform or the student has questions related to the course content. For students, 89 percent indicated that they received this technical assistance and answers to course content within 24 hours. Students (95%) also indicated that the most effective means of communication with the course teacher was verbally using the telephone and electronically. Teachers (100%) also agreed that these methods were the most effective in improving student outcomes. However, only 32 percent of students indicated that they received this support. Evaluation Question 2: What were the characteristics of students taking VVBe courses during the 2012-2013 school year? During 2012-2013, there were more female (60%) VVBe students compared to males (40%). With regard to ethnicity, the majority of VVBe students were Caucasian (67%) followed by African American students (12%). Of the VVBe students, 21 percent were economically disadvantaged, 34 percent were identified as gifted, and 5 percent were identified for special education services.

Of VVBe students, current ninth graders (38%) were the most commonly represented grade level for all students followed by eighth graders who were enrolled in the summer and preparing to enter ninth grade (23%) and eleventh graders (19%). Most students indicated that the 2012-2013 VVBe course was the first online course they had enrolled in while in school (64%), and 85 percent indicated that they were only enrolled in one online course at the time of the survey administration. Additionally, the majority of students chose to take an online course in order to make room for elective courses in their schedule (48%), have the opportunity to enroll in more than the standard course load of six courses (30%), and balance academic and extracurricular activities (30%). Additionally, 29 percent of students were utilizing the online courses as a means to meet graduation requirements. With regard to the characteristics needed to be successful in an online course, VVBe students indicated that they were able to adequately manage and direct their own learning (93%), were confident in their ability to use technology (93%), were able to use time effectively (87%), and were motivated or self-starters (85%). Evaluation Question 3: What progress is being made toward meeting the goals and objectives for the VVBe Program? Goal 1: VVBe will meet the needs of high school students by providing opportunities for students to meet graduation requirements, recover credits, take extra courses, and/or balance academic and extracurricular activities. The majority (98%) of VVBe students and parents indicated that the program allowed students to meet or exceed graduation requirements, and all (100%) of the VVBe teachers and school administrators agreed to this survey item. Based on students who graduated during the regular or summer term during 2012-2013, only 10 percent of these students completed an online VVBe course at any point while enrolled in school. Student survey responses indicated that 78 percent felt that the program provided them with the opportunity to recover credits for courses they previously failed. Course enrollment data demonstrated that very few students (2%) recovered a previously failed credit through an online offering even though the opportunity was available. The majority (95%) of VVBe students indicated on the student survey that the program provided them with the ability to enroll in more than six courses. Similarly based on actual course enrollment records, 98 percent of students who had VVBe experience enrolled in more than six courses. The majority of students who responded to the 2013 School Climate Survey agreed that VVBe courses were available (68%), but only 14 percent of these students indicated that they had online course experience. Based on a review of VVBe course offerings and state diploma requirements, available courses align with all English, science, and Health & PE standard credit requirements for a Standard Diploma or all English and Health & PE standard credit requirements required for

an Advanced Diploma through the VVBe Program. Successful completion of a virtual course is also now required by the state in order to obtain a Standard or Advanced Diploma. Goal 2: VVBe courses will demonstrate quality of content and delivery, and their interactive nature will meet or exceed that of face-to-face instruction. Teachers (92%), parents (91%), and students (89%) agreed that the Desire2Learn platform operated reliably. Moreover, the majority of teachers (89%) and students (57%) indicated that the platform operated reliably 90 to 100 percent of the time. Records provided by the Desire2Learn vendor also substantiated these survey results with only 1 percent of downtime across nine months. A large percentage of VVBe teachers indicated that the course materials were engaging (100%), interactive (100%), and interesting (96%) compared to the survey responses from the VVBe students (72%, 85%, and 64%, respectively). Additionally, 71 percent of students indicated that the VVBe course materials were engaging the same extent or to a greater extent compared to the face-to-face materials they have experienced. Seventy-seven percent of students agreed that the VVBe courses were about the same or greater with regard to how challenging the courses were compared to other face-to-face courses. When comparing student-to-teacher interactions in VVBe courses to those of face-to-face courses, students (65%) and teachers (35%) indicated that in general the VVBe courses involved less interaction. Both students (75%) and teachers (46%) also felt the student-to-student interaction in VVBe courses was generally less compared to that experienced in face-to-face courses. Goal 3: Courses offered through VVBe will be at or near capacity, and students will remain enrolled through the completion of each online course. During the regular 2012-2013 school year, course capacities at each of the four enrollment bands ranged from 64 to 100 percent with four of the five sections considered to be at or near capacity (80%). Of the 36 summer sections, 89 percent of the sections were at, near, or over capacity. Additionally, 14 percent of the summer courses which had a capacity of 28 students were above the capacity limit. Based on course enrollment records provided by the Distance Learning Team, 96 percent of students who were enrolled as of the first day of class persisted in the VVBe courses following the last date to withdraw. Of the online students who remained enrolled following the designated drop date, only 6 percent received a failing grade and 94 percent successfully completed the online course for credit. Goal 4: Students will be successful in VVBe courses and will gain skills that prepare them for future endeavors. End-of-course grades extracted from the division s student data warehouse suggest that completion rates for students in online courses that are also offered as face-to-face courses are similar overall with roughly 94 percent of students successfully passing both delivery formats.

Students in online Health & PE II (100%), Health & PE I (98%), VA & US History (97%), English 11 (95%), English 9 (94%), and Biology (92%) outperformed students in the face-to-face courses. For these courses, the online pass rates ranged from 1 percent to 7 percentage points higher compared to face-to-face. Students in the face-to-face English 12 (97%), Geometry Honors (98%), VA & US Government (98%), Oceanography (93%), Earth Science (93%), and English 10 (90%) outperformed the online students with pass rates ranging from 1 to 16 percent higher. Of the six standardized assessments administered to students in online courses that are administered in the face-to-face courses, students in face-to-face courses outperformed VVBe students on four of the six assessments by 4 to 24 percent. Online students outperformed the face-to-face students on the English 11 reading and writing assessments by 9 to 10 percent. The overall SOL pass rate for online students and face-to-face students were similar and only differed by two percentage points. Based on survey responses, students agreed that the VVBe Program has allowed them to develop time management skills (83%), demonstrate creative and innovative thinking skills (76%), and helped to prepare them for future educational (83%) and career (74%) endeavors. Evaluation Question 4: What are stakeholders perceptions of the program? Stakeholders overwhelmingly expressed their satisfaction with the VVBe Program through survey responses with satisfaction levels ranging from 88 to 100 percent. Parents (97%), teachers (96%), students (94%), and administrators (87%) also indicated that the courses offered through the program met the needs of students. Of the students who were not graduating at the time of the survey and would be eligible to take another online course, 70 percent indicated that they would be interested in enrolling in another VVBe course. Additionally, 87 percent of parents with eligible students indicated that they would encourage their child to participate in another VVBe course. Parents (92%) and students (82%) also indicated that they would recommend a VVBe course to others. Based on the most common open-ended comments provided by students and parents on the surveys, both groups indicated that they were generally pleased with the program and their experience (22 and 37 percent, respectively). Although these groups expressed positive perceptions about the program, students (37%) and parents (19%) also commonly indicated that the courses and online requirements were too demanding and that it was difficult to meet course assignment deadlines, noted mostly with regard to the condensed format during the summer. Twenty-seven percent of the teachers suggested in their open-ended survey comments that the students and parents did not understand the requirements and level of commitment necessary for an online course. The teachers also mentioned potential technological improvements were needed (e.g., ipad compatibility, hiding course content, and monitoring student time through Desire2Learn). Evaluation Question 5: What is the cost or savings of the program to the school division during 2012-2013? The total direct cost for the 2012-2013 regular school year and the 2013 summer term was roughly $297,325.

The majority of the program cost was related to staffing ($255,382) which included salaries and benefits ($17,019) for instructional staff ($217,160), a summer school guidance counselor ($5,310), and the instructor payment for the online pedagogy professional development ($15,893). During the summer of 2013, a total of $243,150 was collected through student tuition payments for VVBe courses. Tuition for each course ranged from $75 to $300 based on students free and reduced lunch status eligibility. After taking into account the tuition received, the VVBe Program cost the school division $54,175 and was able to serve 891 individual students who had 992 course enrollments in 2012-2013. RECOMMENDATIONS The information obtained from the VVBe Program evaluation led to the formulation of the following recommendations. A rationale is presented for each recommendation. Recommendation 1: Expand the number of online offerings through the VVBe Program. (Responsible Departments: Department of School Leadership, Department of Teaching and Learning, Department of Technology) Rationale: With only 68 percent of students indicating that the school division does a good job at making online courses available, this recommendation is based on the overwhelmingly positive student and parent perceptions towards the program. A high percentage of students indicated that they were able to meet or exceed graduation requirements (98%), enroll in more courses (95%), and have their needs met (94%) through the VVBe course offerings. Additionally, parents expressed even higher agreement levels (97% to 98%) to these same items for the VVBe Program. Based on teacher and parent survey responses, the VVBe courses also allow students to demonstrate characteristics aligned with the division s Strategic Plan, Compass to 2015. Specifically, 92 percent of teachers and 87 percent of parents agreed that the courses allowed the students to demonstrate creative and innovative thinking skills. Seventy-six percent of students also agreed to this survey item. Furthermore, parents felt that the VVBe Program helped prepare their child for future educational (92%) and career endeavors (87%), and the majority of students also agreed to these items with 83 percent and 74 percent, respectively. With regard to student academic outcomes, online students (94%) experienced success at the same rate as their face-to-face peers (94%) based on end-of-course grades in similar courses. As a result of participating in the online courses, VVBe students (98%) were able to enroll in more than the standard course load compared to students without VVBe experience (50%). In addition to the stakeholder perception data and the student outcome data, with the new Virginia Department of Education s (VDOE) virtual course diploma requirement, the demand to enroll in online courses will increase. During the 2012-2013 regular school year, there were three online course subjects offered, and the capacity for these courses allowed for only 105 students to enroll. Based on these courses, four out of the five class sections were at or near capacity. During the summer of 2013, 89 percent of the courses operated at, near, or above capacity, including five courses that were above capacity. Moreover, 83 percent of school administrators who provided an open-ended survey response also indicated that an increase in online course offerings is needed in order to make the courses more available to students. With

5,825 1 students entering the ninth grade who will potentially be affected by the new diploma requirement, the school division will need to make these courses more accessible. Recommendation 2: Ensure that VVBe students are provided opportunities for student-to-student and student-to-teacher interactions. (Responsible Departments: Department of Teaching and Learning, Department of Technology) Rationale: This recommendation is being made based on the program objective which states that the interactions in the online courses will meet or exceed those of face-to-face courses. When students were asked to compare the perceived amount of student-to-student interactions, 75 percent indicated these were less than those in face-to-face courses, and 25 percent indicated the amount of interactions were the same or greater. Also, when students were asked about the student-to-teacher interaction levels in VVBe courses compared to face-to-face courses, the majority (65%) indicated that the amount of interaction in online courses was less. Of students, 28 percent indicated that the level of interaction was the same compared to face-to-face courses, and 8 percent indicated that it was greater. The VVBe program has a support structure in place in order to assist in improving student academic outcomes. These supports provided by the teachers include telephone conversations, electronic communication, face-to-face tutorials, and online tutorials. When asked if students participated in any of these interactions, only 32 percent indicated they participated in the telephone support followed by 35 percent participating in face-to-face tutorials, and 36 percent participating in online tutorials. The majority of students (96%) indicated that they participated in electronic communications with the course instructor. These supports have the ability to improve perceived interactions and students should have adequate opportunities to participate in these interactions. 1 Source: VDOE Educational Information Management 2013-2014 Fall Verification Report (dated October 17, 2013).