Summative Evaluation Report

Transcription

1 Summative Evaluation Report st Century Community Learning Centers The School Board of Highlands County Prepared By: Crystal Taylor, M.B.A., Ed.S. CEO/Lead Evaluator

2 This report was prepared for in regards to their st Century Community Learning Center (CCLC) grant. The writing contained in this evaluation may not be reproduced, transmitted, published or broadcast without prior written permission of D & C Education Company, LLC dba EduMatrix. Any questions regarding this evaluation may be directed to Crystal Taylor, CEO of EduMatrix.

3 Table of Contents 1.0 Overview and History Student Characteristics Total Student Enrollment and Attendance Student Demographics Program Operations Summer Operation School Year Operation Staff Characteristics Staff Demographics Student-to-Staff Ratio Staff Training Staff Turnover Certified Teachers Objectives and Outcomes Objectives and Activities Data Collection Methods Measures and Data Collected Data Collection Timeline Data Quality Continuous Assessment Student Inclusion Data Analysis and Results: Progress Toward and Achievement of Summative Evaluation Report

4 Objectives Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Other Findings Reading Report Card Data Math Report Card Data Science Report Card Data FCAT Reading Data FCAT Math Data Parent Survey Data Teacher Survey Data Student Survey Data Student Success Snapshot Overall Findings for Each Objective Summative Evaluation Report

5 5.6.1 Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Objective Progress Towards Sustainability Lessons Learned and Recommendations Summative Evaluation Report

6 1.0 OVERVIEW AND HISTORY Introduction, the Subgrantee, received funding to operate the 21 st Century Community Learning Center (CCLC) grant project for the project year (August 1, 2014 July 31, 2015), which is the first year of the five year funding period that was initially awarded by the FDOE. This report will include the results of the Summative Evaluation that was completed for the st CCLC project year. The Summative Evaluation is considered the year-end report and informs the center and other stakeholders about the progress that has been made during the project year, while also driving decisionmaking regarding program improvement and sustainability. Student characteristics, program operations, staff characteristics, objectives and outcomes, and the sustainability plan will be detailed in the report along with a variety of tables and charts that visually represent relevant program data. 21 st Century Community Learning Centers 21 st Century Community Learning Centers (CCLC) are funded programs that have to operate in accordance to what is stated in the Elementary and Secondary Education Act (ESEA) as amended through Title IV, Part B. According to the RFA for Discretionary Continuation Programs, Florida Department of Education and the Bureau of Family and Community Outreach, the purpose of the program funding is to develop and create community learning centers that provide academic and enrichment opportunities during non-school hours for children that attend low performing high poverty schools, as well as their families. This program helps students meet state and local standards in core subjects, including Reading and Math. It also offers enrichment activities to students that complement the academic program offerings and provides literacy and education services to the families of children in the program. Schools (Kindergarten-12 th grade) that are eligible for Title I School-Wide Program Services, or that have at least 40% of students receive free or reduced lunch, are eligible for the 21 st CCLC program. In the ESEA Section 4205(b), 21 st CCLC programs must be designed and developed using the Principles of Effectiveness. This section states that for a program or activity developed pursuant to this part to meet the Principles of Effectiveness, such program or activity shall Summative Evaluation Report

7 (A) Be based upon an assessment of objective data regarding the need for before and after school programs (including during summer recess periods) and activities in the schools and communities; (B) Be based upon an established set of performance measures aimed at ensuring the availability of high quality academic enrichment opportunities; and (C) Be based upon scientifically-based research, if appropriate, that provides evidence that the program or activity will help students meet the state and local student academic achievement standards. (Florida Department of Education, Bureau of Family and Community Outreach, RFA for Discretionary Continuation Programs, Project Year ) Approved activities for a 21 st CCLC program are identified in the ESEA Section 4025(a) which states, Each eligible entity that receives an award under this part may use the award funds to carry out a broad array of before and after school activities (including during summer recess periods) that advance student academic achievement, including (1) remedial education activities, academic enrichment learning programs, providing additional assistance to students to allow the students to improve their academic achievement (2) math and science education activities (3) arts and music education activities (4) entrepreneurial education programs (5) tutoring and mentoring programs (6) after school activities for LEP students that emphasize language skills and academic achievement (7) recreation activities (8) telecommunications and technology education programs (9) expanded library service hours (10) program that promotes parental involvement and family literacy (11) program that provides assistance to students who have been truant, suspended, or expelled to allow them to improve their academic achievement (12) drug and violence prevention programs, counseling programs, and character education programs. (Florida Department of Education, Bureau of Family and Community Outreach, RFA for Discretionary Continuation Programs, Project Year ) Program Description 21 st CCLC program was operated at four school sites: Avon Park Middle School, Lake Placid Middle School, Hill-Gustat Middle School, and Sebring Middle School. Lake Placid Middle and Avon Park were required to serve 25 students each, and Hill-Gustat Middle and Sebring Middle were required to serve 50 students each. The 21 st CCLC Summative Evaluation Report

8 program at each site operated Monday through Thursday from 3:00pm-5:45pm for 6 th -8 th graders, except for any school holidays. The after school program operated for a total of 77 days at each site. The purpose of the 21 st CCLC program was to help students improve in core academic subject areas and a variety of enrichment areas by engaging them in educational activities, targeted remediation, tutoring and enrichment activities. The School Board of Highlands County purchased the MindWords Curriculum for all four 21 st CCLC sites for the project year. This curriculum has been a great benefit to the program because of the content depth and ease of use. It allows students to be more engaged by participating in student-driven hands-on projects. There are multiple lessons that encourage students to create, explore, investigate, predict, discover cause and effect, critically think, build and analyze deeply in the areas of Science, Technology, Engineering, Math, Language Arts, Art Education, Reading and more. The MindWorks curriculum is designed to address different student age groups and levels of ability including Limited English Proficiency (LEP) and those with disabilities. All lessons incorporate a variety of content areas as well as provide differentiated instruction and scaffolding so that students are able to learn at their own pace, which has helped them tremendously. 21 st CCLC teachers like the curriculum because the teacher s guide is helpful and detailed. In addition, MindWorks provides all the teacher lesson plans so teachers have more time to implement and prepare for existing plans. Avon Park Middle, Lake Placid Middle, Hill- Gustat Middle and Sebring Middle all use the same MindWorks curriculum and cover the same content. This allows for collaboration between the sites since the teachers at all sites are implementing the same lessons. Here are some examples of the MindWorks lessons implemented during the year at all four sites: Storm Chasers Students learned about weather and storms. Re Wind Students learned about historical things while integrating other core subject areas into their hands-on activities. Summative Evaluation Report

9 Art in Action Students learned about art and learned about color and texture in various art pieces. Creature Houses Students built houses for squirrels. Bring On the Books Students built bridges. In addition, students built airplanes, musical instruments and other objects. In addition, the MindWorks curriculum that was used integrates the personal enrichment activities into the academic PBL activities so that students are unaware of the difference. Throughout the year, the External Evaluator conducted site visits at each site to evaluate the program and collect qualitative data for reporting. After each site visit, the External Evaluator provided a written report to the site that detailed the results of the site visit and provided recommendations for improvement. During the site visits, the External Evaluator was able to witness the program in action and observe students learning and participating in engaging activities. For parent involvement, each of the sites held parent workshops and/or events that parents could attend. Even though parent attendance was very low, the program continued to offer the workshops in hopes of more parents attending as the year progressed. This is an item that will need to be improved for the upcoming project year. Program Evaluation Formative evaluations provide the program with feedback from both formal (i.e., surveys, objective performance measurements, progress monitoring reporting) and informal (i.e., program and classroom visits, meetings, conference calls) assessments throughout the year. This type of feedback promotes continuous growth and improvement. The summative evaluation reflects the program s overall annual performance, recommendations for program improvement, and reporting outcomes. Quantitative and qualitative data were both analyzed using a mixed methods research design. Various types of data was collected to provide a thorough understanding about program findings versus quantitative or qualitative data alone. Summative Evaluation Report

10 Mixed methods data analysis was included in the collection and analysis of independent and dependent variables and the relationships between these variables was defined. Triangulation through the mixed methods research design approach allowed for a complete and strong evaluation of quantitative and qualitative program data. Program evaluation best practices was utilized throughout the evaluation process and included defining patterns in collected evaluation data, disaggregating program data, analyzing site visit results and providing feedback (qualitative), establishing academic and enrichment connections (quantitative data) and using evaluation results to make recommendations. This led to relevant recommendations for program improvement regarding data collection efforts, completeness of data and connecting program implementation to grant outcomes. Data was collected at various times throughout the year for evaluation. The data source used for evaluation was the objective assessment (report card grades, test scores, etc.) that is tied to each grant objective. Data was analyzed using data collection and statistical software for test statistics, and to conduct significance tests, correlations, standard deviation and an inductive and deductive analysis. Collected data results were disaggregated as follows: number of days for student attendance, student enrollment by age, gender, ethnicity, grade level, school, free/reduced lunch status, program academic activities, number of family member activities/workshops offered and attended, description of family member offerings, frequency of attendance and number of family members that took the assessments. Evaluation activities were scheduled around the program schedule and did not interfere with program offerings. All qualitative and quantitative data collection methods were discussed in advance to ensure this. As part of the evaluation plan an outcomes and implementation analysis was conducted to assess program impact. Student recruitment strategies, program activities, student retention, participation, program operational plans and sustainability are all factors that are considered during the implementation analysis. Formative evaluation results were used to inform the program how it was progressing at mid-year. Summative evaluation results were used to develop this report and inform the program, stakeholders and community members about the overall performance of the program (from beginning to end of year). Summative Evaluation Report

11 2.0 STUDENT CHARACTERISTICS This section provides information about the students served by the program, including demographics, enrollment, and daily attendance. 2.1 Total Student Enrollment and Attendance Table 1 below details the number of total enrolled and regularly participating students that attended the program during the 2014 summer program only, the school year only and for both summer and school year. Since the program s first year was the school year, there will be not data included in the table for the 2014 summer program because there was no 2014 summer program that was funded by the current grant. Table 1. Student Enrollment: Total and Regularly Participating Students for Summer 2014 and School Year Site Name Summer Only Total Enrolled Attending (at least one day) School Year Only Both Summer AND School Year Total Regularly Participating Enrollment (30 days or more) Summer Only School Year Only Both Summer AND School Year Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School Note. Unduplicated counts shown. Students attending/enrolled in both operation periods are only reported under Summer AND School Year. Only Summer + Only School Year + Summer AND School Year = Total. Total Summative Evaluation Report

12 2.2 Student Demographics Tables 2-9 below detail the number of total enrolled and regularly participating students that attended the program during the school year and includes information about their gender, age range, limited English proficiency status, disability status, race/ethnicity, grade level and free or reduced lunch status. If the program did not have the information for any of these specific areas or the parent or target school did not provide this information to the program, those student numbers are included in the tables below under the DK (Don t Know or Could Not Be Determined) columns. Table 2. Student Demographics for Total Participating Students (All Students Served) and Regularly Participating Students. Site Name Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School Total Participating Students Regularly Participating Students Gender Age Gender Male Female DK* Range Male Female DK Age Range *DK = Don t Know/Could Not Be Determined. Table 3. Students with Special Needs: Total Participating Students. Site Name Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School Limited English Proficient Identified with Disability Yes No DK* Yes No DK Summative Evaluation Report

13 *DK = Don t Know/Could Not Be Determined. Table 4. Students with Special Needs: Regularly Participating Students. Site Name Limited English Proficient Identified with Disability Yes No DK* Yes No DK Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School *DK = Don t Know/Could Not Be Determined. Table 5. Student Race and Ethnicity: Total and Regularly Participating Students. Total Participating Students Regularly Participating Students Site Name American Indian/ Alaska Native Asian/ Pacific Islander Black or African American Hispanic or Latino White or Caucasian American Unknown** American Indian/ Alaska Native Asian/ Pacific Islander Black or African American Hispanic or Latino White or Caucasian American Unknown Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School * Ethnicity categories are non-exclusive; students can be identified under multiple ethnicities. ** Unknown = Racial/ethnic group is unknown or cannot be verified. Summative Evaluation Report

14 Table 6. Student Grade for Total Participating Students. Grade In School* PK K Total Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School * Grade levels are exclusive, as students can only be in one grade level. The total number of students where grade level is unknown are not indicated, but can be derived from this table. Table 7. Student Grade for Regularly Participating Students. Grade In School* PK K Total Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School * Grade levels are exclusive, as students can only be in one grade level. The total number of students where grade level is unknown are not indicated, but can be derived from this table. Table 8. Free/Reduced Lunch Status of Total Participating Students. Site Name Free or Reduced-Price Lunch Yes No DK Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School *DK = Don t Know/Could Not Be Determined. Summative Evaluation Report

15 Table 9. Free/Reduced Lunch Status of Regularly Participating Students. Site Name Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School Free or Reduced-Price Lunch Yes No DK *DK = Don t Know/Could Not Be Determined. Summative Evaluation Report

16 3.0 PROGRAM OPERATIONS The program operations information is detailed in sections 3.1 and 3.2 below which includes 2014 summer operations data and school year operations data. Since the program s first year was the school year, there will be not data included in the table for the 2014 summer program because there was no 2014 summer program that was funded by the current grant. 3.1 Summer Operation There was no 2014 summer program since the program s first year in the current grant cycle was the school year. Table 10. Summer 2014 Operation. Site Name Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School Total number of weeks THIS site was open: Typical number of days per week THIS site was open: Typical number of hours per week THIS site was open on: WEEKDAYS WEEKDAY EVENINGS WEEKENDS N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Summative Evaluation Report

17 3.2 School Year Operation The school year operations information is detailed in table 11 below which includes information about the total number of weeks, days and days per week the sites were open, the typical number of hours per week the sites were open, and the total number of days each site operated. Table 11. School Year Operation. Site Name Avon Park Middle School Hill Gustat Middle School Lake Placid Middle School Sebring Middle School Total # weeks THIS site was open: Total # days THIS site was open: Typical # days per week THIS site was open: Typical # hours per week THIS site was open: Before School During School After School Weekends / Holidays Total # days THIS site operated: Before School During School After School Weekends/ Holidays Summative Evaluation Report

18 4.0 STAFF CHARACTERISTICS This section provides information on the composition of staff at each site including staff demographics, ratio of staff to students, staff quality (training and certifications), and turnover. 4.1 Staff Demographics The following tables detail the staffing type for each site and includes information about whether those staff were paid or considered volunteers for the school year program. Table 12. Regular Staff by Paid and Volunteer Status (Avon Park Middle School) Staff Type* Summer of School Year Paid 1 Volunteer Paid 1 Volunteer School day teachers (former and substitute) N/A N/A 4 0 Center administrators and coordinators N/A N/A 0 0 Youth development workers and non-school day staff with college degree or higher N/A N/A 0 0 Other non-teaching school day staff N/A N/A 3 0 Parents N/A N/A 0 0 College students N/A N/A 0 0 High school students N/A N/A 0 0 Other community members N/A N/A 0 0 Other non-school day staff with some or no college N/A N/A 0 0 Other** N/A N/A For all staff categories except Other, report only staff paid with 21 st CCLC funds. * These categories represent the regular responsibilities of program staff during the regular school day. ** Use this category if data do not fit in specific categories provided Summative Evaluation Report

19 Table 13. Regular Staff by Paid and Volunteer Status (Sebring Middle School) Staff Type* Summer of School Year Paid 1 Volunteer Paid 1 Volunteer School day teachers (former and substitute) N/A N/A 5 0 Center administrators and coordinators N/A N/A 0 0 Youth development workers and non-school day staff with college degree or higher N/A N/A 0 0 Other non-teaching school day staff N/A N/A 3 0 Parents N/A N/A 0 0 College students N/A N/A 0 0 High school students N/A N/A 0 0 Other community members N/A N/A 0 0 Other non-school day staff with some or no college N/A N/A 0 0 Other** N/A N/A For all staff categories except Other, report only staff paid with 21 st CCLC funds. * These categories represent the regular responsibilities of program staff during the regular school day. ** Use this category if data do not fit in specific categories provided Table 14. Regular Staff by Paid and Volunteer Status (Lake Placid Middle School) Staff Type* Summer of School Year Paid 1 Volunteer Paid 1 Volunteer School day teachers (former and substitute) N/A N/A 10 0 Center administrators and coordinators N/A N/A 0 0 Youth development workers and non-school day staff with college degree or higher N/A N/A 0 0 Other non-teaching school day staff N/A N/A 2 0 Parents N/A N/A 0 0 College students N/A N/A 0 0 High school students N/A N/A 0 0 Other community members N/A N/A 0 0 Other non-school day staff with some or no college N/A N/A 0 0 Other** N/A N/A For all staff categories except Other, report only staff paid with 21 st CCLC funds. Summative Evaluation Report

20 * These categories represent the regular responsibilities of program staff during the regular school day. ** Use this category if data do not fit in specific categories provided Table 15. Regular Staff by Paid and Volunteer Status (Hill-Gustat Middle School) Staff Type* Summer of School Year Paid 1 Volunteer Paid 1 Volunteer School day teachers (former and substitute) N/A N/A 9 0 Center administrators and coordinators N/A N/A 0 0 Youth development workers and non-school day staff with college degree or higher N/A N/A 0 0 Other non-teaching school day staff N/A N/A 0 0 Parents N/A N/A 0 0 College students N/A N/A 0 0 High school students N/A N/A 0 0 Other community members N/A N/A 0 0 Other non-school day staff with some or no college N/A N/A 0 0 Other** N/A N/A For all staff categories except Other, report only staff paid with 21 st CCLC funds. * These categories represent the regular responsibilities of program staff during the regular school day. ** Use this category if data do not fit in specific categories provided 4.2 Student-to-Staff Ratio The student-to-staff ratio at each site was 10:1 for academic enrichment and 20:1 for personal enrichment activities. 4.3 Staff Training All 21 st CCLC staff received professional development training during the project year including a session at the beginning of the program and end of the program. Teachers also received a training regarding grant evaluation where they learned about what the grant Summative Evaluation Report

21 objectives were and what they needed to do to help meet each objective by the end of the project year. Meeting agendas and sign-in sheets were submitted to the 21 st CCLC state office during the program year as one of the required monthly deliverables. 4.4 Staff Turnover Avon Park Middle turnover: None Sebring Middle turnover: One teacher did not finish year due to low student attendance, however, it was not due to the teacher s performance. Lake Placid Middle turnover: One change in staff during the year due to sporting events or time constraints, however, it was not due to the teacher s performance. Hill-Gustat Middle turnover: One teacher left the program due to time restraints, however, it was not due to the teacher s performance. 4.5 Certified Teachers There was a total of 28 certified teachers that worked for the School Board of Highlands County s 21 st CCLC program during the school year. All teachers were certified and non-certified staff did not present academic lessons without a certified teacher in the room, as proposed in grant. Summative Evaluation Report

22 5.0 OBJECTIVES and OUTCOMES This section provides information on program objectives, how those objectives are measured, data analysis methods, progress toward objectives, and finding implications and recommendations. 5.1 Objectives and Activities There were eighteen grant objectives that the program used to guide the teacher instruction and parent involvement initiatives during the school year. In the table below, each objective is listed with the corresponding description of activities for each objective. Table 16. Objectives and Description of Activities Objective Objective 1: 70% of the regularly participating students will maintain or improve their reading comprehension skills as demonstrated by pre, mid & post assessment scores on FAIR and report card grades Description of Activities Students participated in reading activities that targeted comprehension, fluency, predicting, paraphrasing, vocabulary, identifying text structure, main idea, context clues and other reading standards to improve their knowledge in reading. Objective 2: 70% of the regularly participating students will maintain or improve their reading comprehension skills as demonstrated by pre, mid & post assessment scores on FAIR and report card grades Students participated in reading activities that targeted comprehension, fluency, predicting, paraphrasing, vocabulary, identifying text structure, main idea, context clues and other reading standards to improve their knowledge in reading. Objective 3: 70% of the regularly participating students will maintain or improve their math skills as demonstrated by pre, mid & post Students participated in math activities that targeted operations, algebraic thinking, number and operations in base ten and fractions, geometry, measurement and Summative Evaluation Report

23 assessment scores on Math Performance Matters and report card grades other math standards to improve their knowledge in math. Objective 4: 70% of the regularly participating students will maintain or improve their math skills as demonstrated by pre, mid & post assessment scores on Math Performance Matters and report card grades Students participated in math activities that targeted operations, algebraic thinking, number and operations in base ten and fractions, geometry, measurement and other math standards to improve their knowledge in math. Objective 5: 70% of the regularly participating students will maintain or improve their science skills as demonstrated by pre, mid & post assessment scores on Science Performance Matters and report card grades Students participated in science activities that targeted earth, physical and life science standards such as investigations, experiments, computations, collecting data, and testing hypotheses to improve their knowledge in science. Objective 6: 70% of the regularly participating students will maintain or improve their science skills as demonstrated by pre, mid & post assessment scores on Science Performance Matters and report card grades Students participated in science activities that targeted earth, physical and life science standards such as investigations, experiments, computations, collecting data, and testing hypotheses to improve their knowledge in science. Objective 7: 80% of the regularly participating students will increase knowledge of technology programs and procedures as demonstrated by pre, mid & post assessment scores on TechLiteracy test and projects throughout the year Students utilized computers in the computer lab and completed technology activities using educational technological software where they learned about various aspects of technology programming and other programs. Objective 8: 80% of the regularly participating students will increase Students utilized computers in the computer lab and completed technology activities using educational Summative Evaluation Report

24 knowledge of technology programs and procedures as demonstrated by pre, mid & post assessment scores on TechLiteracy test and projects throughout the year technological software where they learned about various aspects of technology programming and other programs. Objective 9: 70% of family members from regularly participating students who attend at least 1 literacy event will demonstrate growth on the monthly assessments Literacy workshops were offered to the parents. Assessments were to be used to help ensure that parents were retaining the information that was taught. Objective 10: 25% of family members from regularly participating students who participate in the Active Parenting program offered through the year will demonstrate growth on pre and post assessments. The Active Parenting program was a program targeted at parents to help increase their knowledge on various parenting strategies to best help their children. Assessments were to be used to help ensure that parents were retaining the information that was taught. Objective 11: 40% of family members from regularly participating students will attend at least one 21st CCLC sponsored event, offered monthly and related to family requested topics including curriculum, homework help, homeschool communication, and drug and gang awareness as measured by adult sign in sheets through the year The family events were to include information about curriculum used with their children, homework help, home/school communication, and drug and gang awareness. Objective 12: 60% of family members from regularly participating students will increase their participation in school sponsored events as measured by monthly sign in sheets distributed quarterly Parents were encouraged to increase their participation in school sponsored events, which would be measured by sign in sheets for all parent events. Summative Evaluation Report

25 Objective 13: 80% of the regularly participating students will maintain or improve of behavior related to physical fitness as measured by Healthy Lifestyle Choices by pre, mid & post assessment as well as increased Likert style surveys throughout the year Students participated in healthy lifestyle choices activities that targeted learning about the physical needs of the body, exercise, healthy eating and more to improve their knowledge of healthy lifestyle choices. Objective 14: 80% of the regularly participating students will maintain or improve of behavior related to physical fitness as measured by Healthy Lifestyle Choices by pre, mid & post assessment as well as increased Likert style surveys throughout the year Students participated in healthy lifestyle choices activities that targeted learning about the physical needs of the body, exercise, healthy eating and more to improve their knowledge of healthy lifestyle choices. Objective 15: 80% of the regularly participating students will increase knowledge of art and culture as demonstrated by Likert styled surveys and performance measures related to art, guitar and cultural elements throughout the year Students participated in art and culture activities related to art, guitar and cultural elements throughout the year. Objective 16: 80% of the regularly participating students will increase knowledge of art and culture as demonstrated by Likert styled surveys and performance measures related to art, guitar and cultural elements throughout the year Students participated in art and culture activities related to art, guitar and cultural elements throughout the year. Objective 17: 80% of the regularly participating students will maintain or improve behavior related to physical fitness and social/emotional Students participated in character building activities where they learned about social skills. They also participated in physical fitness activities including kickball and other similar sports which helped to improve their knowledge of physical fitness. Summative Evaluation Report

26 choices on a Likert scale to transfer into classroom. Objective 18: 70% of the teacher reponses will agree or strongly agree that student behavior has improved from the time they were enrolled in the program. Regular school day teachers of the 21 st CCLC students were able to take a teacher survey where they answered questions about student behavior. 5.2 Data Collection Methods The method of data collection was based on the actual data that was being collected for each objective. The program staff collected the required data throughout the year and submitted the data to the External Evaluator for evaluation. A variety of methods were used to collect the data including obtaining district and program level student assessment information, standardized state assessment data, surveying students, parents and teachers, and the collection of parent attendance information from program workshops Measures and Data Collected: Student, parent and teacher data was collected and assessed for the program. Student data was obtained from FAIR and Performance Matters assessments given by the school district for reading, math and science content areas. Student assessments for technology, physical fitness, healthy lifestyle choices and art and culture were collected, as well as, parent workshop sign in/out sheets and student, parent and teacher surveys. Report card grades and 2014 Florida Comprehensive Assessment Test (FCAT) scores were also obtained from the district and evaluated Data Collection Timeline: Assessments were administered and data was collected by the program throughout the year. Baseline data was collected close to the start of the program, mid-year data was collected in December 2014/January 2015, and end-ofyear data was collected in May/June If additional data needed to be collected due to students being absent on the day that assessments were being given, they were able to take the assessment upon their return which may have occurred outside of the aforementioned time periods. Summative Evaluation Report

27 5.2.3 Data Quality: Data quality used in this evaluation received an average rating. The program collected most of the appropriate data that was required by the grant for each objective, but was also missing a significant amount of data. Missing data cannot be evaluated and therefore leads to not meeting grant objectives that are tied to the missing data. A high standard for data quality and collection will need to be set at program start for the upcoming project year. Program leadership will need to play an integral role in helping to ensure that the expectations are set and met. The manner in which data was collected would be considered sufficient for evaluation purposes, as well. However, more data needed to be collected in order to receive a high rating in data quality overall Continuous Assessment: The program was advised to continuously assess students throughout the year to help students learn the targeted concepts and retain more information, which can lead to better performance on assessments. Teachers performed comprehension checks and reviews for students during the year to help them meet each student-related grant objective, however not all grant objectives were met. Monitoring quarterly report card grades, progress reports and administering short quizzes or mini-assessments on a regular basis are all forms of continuous assessments for students that help teachers stay aware of how students are performing on a more regular basis. It is recommended that the program place additional emphasis in this area to help students perform at an even higher academic level and show greater academic gains by the end of the program year Student Inclusion: Students were not excluded from the student-related objective assessments and parents were not excluded from the parent-related objective assessments. All students and parents had equal opportunity to complete their individual assessments during the project year. Students, parents and teachers were also not excluded from being invited to take the program surveys at the end of the year. If any students left the program during the year, they would not have been able to take any assessments that were administered during the time in which they were absent. In addition, if students did not have two time points worth of data for an objective, their data was not able to be evaluated for that particular objective because progress could not be evaluated with only one score. Summative Evaluation Report

28 5.3 Data Analysis and Results: Progress Toward and Achievement of Objectives Objective 1: 70% of the regularly participating students will maintain or improve their reading comprehension skills as demonstrated by pre, mid & post assessment scores on FAIR and report card grades Objective Assessment Plan: 70% of the regularly participating students will maintain or improve reading comprehension as measured by State Monitoring Assessment Tool from Quarter 1 to Quarter 4. Success Criterion: Maintain scores within the 50 th percentile (or higher) or improve reading comprehension percentile score by at least 1 percent 57% of students with sufficient data indicated improvement in reading comprehension skills as measured by Pre-Post Assessment during the current year. The success criterion was found using the pre and post-test scores, attained at the beginning and end of the year respectively. The success criterion required that the students maintain their scores within at least the 50 th percentile, or improve their scores by at least 1 percent over the course of the year. In order to calculate a criterion, the Pre and Post scores were used to measure improvement for each student, and whether each student maintained a score at or above the 50 percentile. 129 students had sufficient data to measure the criterion, of which 73 students met the criterion. Please refer to the table below for statistics related to this objective. Table 17: Highlands Reading Comprehension Score Statistics Total Min Max M SD Students* Pre-Test Scores Post-Test Scores Difference or Change in Score *Indicates total number of students with usable data, i.e. students with both preliminary and end-ofyear data Summative Evaluation Report

29 Across all sites, students improved reading comprehension assessment scores by an average of 2.9 points over the course of the year. The standard deviation (SD) of change in score is 22.8, which is indicative of the representation of the overall changes of score within the data set based on the calculated mean (M). A lower scale value standard deviation in relation to the mean is representative of a lower deviation from the calculated average of difference in test scores across the entire data set. A higher scale value standard deviation represents a high deviation of differences in test scores from the calculated average throughout a larger sample of the overall data set. Chart 1: Highlands Change in Reading Comprehension Score Comparison Avon Park Hill Gustat Lake Placid Sebring Minimum Maximum -5.5 Mean SD A statistical comparison of the sites reveals that students at Sebring improved their scores by an average of 11.8 points, which was the highest average throughout all sites. The differences in score from pre to post tests at Lake Placid Middle School maintained the highest variance from the mean by a narrow margin with a calculated standard deviation (SD) of Avon Park Middle School achieved the lowest standard deviation of Table 18: Highlands Student Reading Comprehension Improvement (% of students) Score Score Score Increased Met Goal Decreased Maintained Avon Park 46% 19% 35% 46% Hill Gustat 34% 13% 51% 55% Lake Placid 26% 13% 61% 65% Sebring 33% 6% 61% 61% All Sites 35% 13% 52% 57% Summative Evaluation Report

30 5.3.2 Objective 2: 70% of the regularly participating students will maintain or improve their reading comprehension skills as demonstrated by pre, mid & post assessment scores on FAIR and report card grades Objective Assessment Plan: 70% of the regularly participating students will maintain or improve reading comprehension as measured by Grades from Quarter 1 or Quarter 4. Success Criterion: Maintain C or above or improve reading grades by at least 1 percent 46% of the students with sufficient data indicated improvement in reading comprehension as measured by grades from Quarter 1 to Quarter 4. The success criterion was found using the Q1 and Q4 report card grades attained at the beginning and end of the year respectively. The success criterion required that the students improve their grades by at least one percent, or maintain the same letter grade from Q1 to Q4 at a C or above. 125 students had sufficient data to measure the criterion; of those, 57 students met the criterion. Please refer to the table below for statistics related to this objective. Table 19: Highlands Reading Report Card Student Improvement (% of students) Total Students Improved Grade Maintained C+ Met Goal (%) (%) Grade (%) Avon Park 25 40% 0% 40% Hill Gustat 46 48% 11% 59% Lake Placid 27 33% 4% 37% Sebring 27 37% 0% 37% All Sites % 5% 46% Summative Evaluation Report

31 Chart 2: Highlands Reading Report Card Grade Improvement Degreased Grade Maintained Grade Increased Grade th Grade 7th Grade 8th Grade Objective 3: 70% of the regularly participating students will maintain or improve their math skills as demonstrated by pre, mid & post assessment scores on Math Performance Matters and report card grades Objective Assessment Plan: 70% of the regularly participating students will maintain or improve mathematics skills as measured by State Monitoring Assessment Tool from Quarter 1 to Quarter 4. Success Criterion: Maintain scores within the 50 th percentile (or higher) or improve math skills percentile score by at least 1 percent 43% of students with sufficient data indicated improvement in math skills as measured by Pre-Post Assessment during the current year. The success criterion was found using the pre and post-test scores, attained at the beginning and end of the year respectively. The success criterion required that the students maintain their scores within at least the 50 th percentile, or improve their scores by at least 1 percent over the course of the year. In order to calculate a criterion, the Pre and Post scores were used Summative Evaluation Report

32 to measure improvement for each student, and whether each student maintained a score at or above the 50 percentile. 67 students had sufficient data to measure the criterion, of which 29 students met the criterion. Please refer to the table below for statistics related to this objective. Table 20: Highlands Math Skills Score Statistics Total Min Max M SD Students* Pre-Test Scores Post-Test Scores Difference or Change in Score *Indicates total number of students with usable data, i.e. students with both preliminary and end-ofyear data Across all sites, student math skill assessment scores decreased by an average of -1.2 points over the course of the year. The standard deviation (SD) of change in score is 13.4, which is indicative of the representation of the overall changes of score within the data set based on the calculated mean (M). A lower scale value standard deviation in relation to the mean is representative of a lower deviation from the calculated average of difference in test scores across the entire data set. A higher scale value standard deviation represents a high deviation of differences in test scores from the calculated average throughout a larger sample of the overall data set. Chart 3: Highlands Change in Math Skills Assessment Score Comparison Avon Park Hill Gustat Lake Placid Sebring Minimum Maximum Mean SD Summative Evaluation Report

33 A statistical comparison of the sites reveals that students at Lake Placid improved their scores by an average of 1.9 points, which was the highest average throughout all sites. The differences in score from pre to post tests at Sebring Middle School maintained the highest variance from the mean with a calculated standard deviation (SD) of 16. Avon Park Middle School achieved the lowest standard deviation of 9.1. Table 21: Highlands Student Math Skills Assessment Score Improvement (% of students) Score Score Score Increased Met Goal Decreased Maintained Avon Park 87% 0% 13% 13% Hill Gustat 59% 0% 41% 41% Lake Placid 31% 0% 69% 69% Sebring 33% 8% 58% 58% All Sites 55% 1% 43% 43% Objective 4: 70% of the regularly participating students will maintain or improve their math skills as demonstrated by pre, mid & post assessment scores on Math Performance Matters and report card grades Objective Assessment Plan: 70% of the regularly participating students will maintain or improve mathematics skills as measured by Grades from Quarter 1 or Quarter 4. Success Criterion: Maintain C or above or improve math grades by at least 1 percent 60% of the students with sufficient data indicated improvement in mathematics skills as measured by grades from Quarter 1 to Quarter 4. The success criterion was found using the Q1 and Q4 report card grades attained at the beginning and end of the year respectively. The success criterion required that the students improve their grades by at least one percent, or maintain the same letter grade from Q1 to Q4 at a C or above. 135 students had sufficient data to measure the criterion; of those, 81 students met the criterion. Please refer to the table below for statistics related to this objective. Summative Evaluation Report

34 Table 22: Highlands Math Report Card Student Improvement (% of students) Total Students Improved Grade Maintained C+ Met Goal (%) (%) Grade (%) Avon Park 25 56% 4% 60% Hill Gustat 49 51% 6% 57% Lake Placid 27 70% 7% 78% Sebring 34 38% 12% 50% All Sites % 7% 60% Chart 4: Highlands Math Report Card Grade Improvement Decreased Grade Maintained Grade Increased Grade 5 0 6th Grade 7th Grade 8th Grade Objective 5: 70% of the regularly participating students will maintain or improve their science skills as demonstrated by pre, mid & post assessment scores on Science Performance Matters and report card grades Objective Assessment Plan: 70% of the regularly participating students will maintain or improve science skills as measured by State Monitoring Assessment Tool from Quarter 1 to Quarter 4. Summative Evaluation Report

35 Success Criterion: Maintain scores within the 50 th percentile (or higher) or improve science skills percentile score by at least 1 percent 56% of students with sufficient data indicated improvement in science skills as measured by Pre-Post Assessment during the current year. The success criterion was found using the pre and post-test scores, attained at the beginning and end of the year respectively. The success criterion required that the students maintain their scores within at least the 50 th percentile, or improve their scores by at least 1 percent over the course of the year. In order to calculate a criterion, the Pre and Post scores were used to measure improvement for each student, and whether each student maintained a score at or above the 50 percentile. 55 students had sufficient data to measure the criterion, of which 31 students met the criterion. Please refer to the table below for statistics related to this objective. Table 23: Highlands Science Skills Score Statistics Total Min Max M SD Students* Pre-Test Scores Post-Test Scores Difference or Change in Score *Indicates total number of students with usable data, i.e. students with both preliminary and end-ofyear data Across all sites, students improved science skill assessment scores by an average of 4.8 points over the course of the year. The standard deviation (SD) of change in score is 11.3, which is indicative of the representation of the overall changes of score within the data set based on the calculated mean (M). A lower scale value standard deviation in relation to the mean is representative of a lower deviation from the calculated average of difference in test scores across the entire data set. A higher scale value standard deviation represents a high deviation of differences in test scores from the calculated average throughout a larger sample of the overall data set. Summative Evaluation Report

36 Chart 5: Highlands Change in Science Skills Assessment Score Comparison Avon Park Hill Gustat Lake Placid Sebring Minimum Maximum Mean SD A statistical comparison of the sites reveals that students at Lake Placid improved their scores by an average of 7.6 points, which was the highest average throughout all sites. The differences in score from pre to post tests at Lake Placid maintained the highest variance from the mean with a calculated standard deviation (SD) of Avon Park Middle School achieved the lowest standard deviation of 8.0. Table 24: Highlands Student Science Skills Assessment Score Improvement (% of students) Score Score Score Increased Met Goal Decreased Maintained Avon Park 30% 0% 70% 70% Hill Gustat 33% 11% 56% 56% Lake Placid 33% 0% 67% 67% Sebring 44% 11$ 44% 44% All Sites 36% 7% 56% 56% Summative Evaluation Report

37 5.3.6 Objective 6: 70% of the regularly participating students will maintain or improve their science skills as demonstrated by pre, mid & post assessment scores on Science Performance Matters and report card grades Objective Assessment Plan: 70% of the regularly participating students will maintain or improve science skills as measured by Grades from Quarter 1 or Quarter 4. Success Criterion: Maintain C or above or improve science grades by at least 1 percent 46% of the students with sufficient data indicated improvement in science skills as measured by grades from Quarter 1 to Quarter 4. The success criterion was found using the Q1 and Q4 report card grades attained at the beginning and end of the year respectively. The success criterion required that the students improve their grades by at least one percent, or maintain the same letter grade from Q1 to Q4 at a C or above. 135 students had sufficient data to measure the criterion; of those, 62 students met the criterion. Please refer to the table below for statistics related to this objective. Table 25: Highlands Science Report Card Student Improvement (% of students) Total Students Improved Grade Maintained C+ Met Goal (%) (%) Grade (%) Avon Park 25 28% 4% 32% Hill Gustat 49 49% 4% 53% Lake Placid 27 48% 0% 48% Sebring 34 41% 3% 44% All Sites % 3% 46% Summative Evaluation Report

38 Chart 6: Highlands Science Report Card Grade Improvement Decreased Grade Maintained Grade Increased Grade th Grade 7th Grade 8th Grade Objective 7: 80% of the regularly participating students will increase knowledge of technology programs and procedures as demonstrated by pre, mid & post assessment scores on TechLiteracy test and projects throughout the year Objective Assessment Plan: 80% of the regularly participating students will increase computer technology as measured by Local Standardized Assessment from Quarter 1 to Quarter 4. Success Criterion: Maintain or improve EasyTech Literacy score by at least 1 percent 50% of students with sufficient data indicated improvement in technology skills as measured by Pre-Post Assessment during the current year. The success criterion was found using the pre and post-test scores, attained at the beginning and end of the year respectively. The success criterion required that the students either maintain or improve their scores by at least 1 percent over the course of the year. 32 students had sufficient data to measure the criterion, of which 16 students met the criterion. Please refer to the table below for statistics related to this objective. Summative Evaluation Report

39 Table 26: Highlands Technology Skills Score Statistics Total Min Max M SD Students* Pre-Test Scores Post-Test Scores Difference or Change in Score *Indicates total number of students with usable data, i.e. students with both preliminary and end-ofyear data Across all sites, students improved science skill assessment scores by an average of 4.4 points over the course of the year. The standard deviation (SD) of change in score is 19.7, which is indicative of the representation of the overall changes of score within the data set based on the calculated mean (M). A lower scale value standard deviation in relation to the mean is representative of a lower deviation from the calculated average of difference in test scores across the entire data set. A higher scale value standard deviation represents a high deviation of differences in test scores from the calculated average throughout a larger sample of the overall data set. Chart 7: Highlands Change in Technology Skills Assessment Score Comparison Avon Park Hill Gustat Lake Placid Sebring Minimum Maximum -3.1 Mean SD A statistical comparison of the sites reveals that students at Lake Placid improved their scores by an average of 19.3 points, which was the highest average throughout all sites. The differences in score from Summative Evaluation Report

40 pre to post tests at Hill Gustat maintained the highest variance from the mean with a calculated standard deviation (SD) of Avon Park Middle School achieved the lowest standard deviation of 6.0. Table 27: Highlands Student Technology Skills Assessment Score Improvement (% of students) Score Score Score Increased Met Goal Decreased Maintained Avon Park 57% 14% 29% 29% Hill Gustat 38% 6% 56% 56% Lake Placid 25% 0% 75% 75% Sebring 40% 20% 20% 20% All Sites 41% 9% 50% 50% Objective 8: 80% of the regularly participating students will increase knowledge of technology programs and procedures as demonstrated by pre, mid & post assessment scores on TechLiteracy test and projects throughout the year There was not a sufficient amount of data available to evaluate this objective. The program was not able to purchase technology resources until January 2015 for all four sites. This meant that much of the information that students were supposed to learn for the projects was not able to be learned over the course of a year, but instead, needed to be learned in a very short time. This was further complicated by the fact that the computer labs at all four sites became unavailable when the schools set them up for the new state testing requirements so that other students in the school could complete their state standardized tests. When students were allowed back into the computer labs at each site, they were able to use the EasyTech Literacy software, but were not able to compete the projects anticipated in grant objectives. Next year, it is recommended that the program purchase all materials sooner so that students are able to learn the information needed over the course of the year and complete the necessary projects if they are indeed a part of the program next year Objective 9: 70% of family members from regularly participating students who attend at least 1 literacy evemt will demonstrate growth on the monthly assessments Objective Assessment Plan: 70% of the parents of regularly participating students who attend at least 1 literacy event will demonstrate increased knowledge as measured by Authentic Assessment from Quarter 1 to Quarter 4. Summative Evaluation Report

41 Success Criterion: Parents who score 70% or higher Objective 10: 25% of family members from regularly participating students who participate in the Active Parenting program offered through the year will demonstrate growth on pre and post assessments. Objective Assessment Plan: 25% of the parents of regularly participating students will demonstrate parents increasing knowledge as measured by Pre-Post Assessment/Survey from Quarter 1 to Quarter 4. Success Criterion: Parents who score 70% or higher Objective 11: 40% of family members from regularly participating students will attend at least one 21st CCLC sponsored event, offered monthly and related to family requested topics including curriculum, homework help, home-school communication, and drug and gang awareness as measured by adult sign in sheets through the year Objective Assessment Plan: 40% of the parents of regularly participating students will demonstrate attendance in family activities as measured by Sign-In/Out Sheets from Quarter 1 to Quarter 4. Success Criterion: 40% attendance at events during the year Summative Evaluation Report

42 Objective 12: 60% of family members from regularly participating students will increase their participation in school sponsored events as measured by monthly sign in sheets distributed quarterly Objective Assessment Plan: 60% of the parents of regularly participating students will demonstrate attendance in family activities as measured by Sign-In/Out Sheets from Quarter 1 to Quarter 4. Success Criterion: 60% attendance at events during the year Objective 13: 80% of the regularly participating students will maintain or improve of behavior related to physical fitness as measured by Healthy Lifestyle Choices by pre, mid & post assessment as well as increased Likert style surveys throughout the year Objective Assessment Plan: 80% of the regularly participating students will increase knowledge of healthy lifestyle choices as measured by Authentic Assessment from Quarter 1 to Quarter 4. Success Criterion: Maintain if above mean score or improve grade by at least 2 percent 67% of students with sufficient data indicated improvement in knowledge of healthy lifestyle choices as measured by Pre-Post Assessment during the current year. The success criterion was found using the pre and post-test scores, attained at the beginning and end of the year respectively. The success criterion required that the students improve their scores by at least 2 percent over the course of the year, or maintain a score above the mean score. 57 students had sufficient data to measure the criterion, of which 38 students met the criterion. Please refer to the table below for statistics related to this objective. Summative Evaluation Report

43 Table 28: Highlands Technology Skills Score Statistics Total Min Max M SD Students* Pre-Test Scores Post-Test Scores Difference or Change in Score *Indicates total number of students with usable data, i.e. students with both preliminary and end-ofyear data Across all sites, students improved science skill assessment scores by an average of 5.8 points over the course of the year. The standard deviation (SD) of change in score is 13.4, which is indicative of the representation of the overall changes of score within the data set based on the calculated mean (M). A lower scale value standard deviation in relation to the mean is representative of a lower deviation from the calculated average of difference in test scores across the entire data set. A higher scale value standard deviation represents a high deviation of differences in test scores from the calculated average throughout a larger sample of the overall data set. Chart 8: Highlands Change in Healthy Lifestyle Choices Assessment Score Comparison Avon Park Hill Gustat Lake Placid Minimum Maximum Mean SD A statistical comparison of the sites reveals that students at Lake Placid improved their scores by an average of 13.8 points, which was the highest average throughout all sites. The differences in score from pre to post tests at Lake Placid maintained the highest variance from the mean with a calculated Summative Evaluation Report

44 standard deviation (SD) of Hill Gustat Middle School achieved the lowest standard deviation of There were no students will sufficient data from the Sebring Middle School site. Table 29: Highlands Student Healthy Lifestyle Choices Assessment Score Improvement (% of students) Score Score Score Increased Met Goal Decreased Maintained Avon Park 30% 4% 65% 65% Hill Gustat 25% 13% 63% 63% Lake Placid 10% 10% 80% 80% All Sites 29% 9% 67% 67% Objective 14: 80% of the regularly participating students will maintain or improve of behavior related to physical fitness as measured by Healthy Lifestyle Choices by pre, mid & post assessment as well as increased Likert style surveys throughout the year Objective Assessment Plan: 80% of the regularly participating students will increase knowledge of healthy lifestyle choices as measured by Perceptual Survey from Quarter 1 to Quarter 4. Success Criterion: Students who score 70% or higher on the Likert scale pre/post assessment 59% of students with sufficient data indicated improvement in knowledge of healthy lifestyle choices as measured by Pre-Post Assessment during the current year. The success criterion was found using the post-test scores of the perceptual healthy choices survey, attained at the end of the year. The success criterion required that the students achieve a score of 70 or higher. 80 students had sufficient data to measure the criterion, of which 47 students met the criterion. Please refer to the table below for statistics related to this objective. Summative Evaluation Report

45 Table 30: Highlands Perceptual Healthy Lifestyle Choices Survey Statistics Total Min Max M SD Students* Post-Test Scores *Indicates total number of students with usable data, i.e. students with both preliminary and end-ofyear data Across all sites, students achieved an average of 70.4% on the healthy lifestyle choices perceptual survey. The standard deviation (SD) of the post-test scores is 11.5, which is indicative of the representation of the variance in score within the data set based on the calculated mean (M). A lower scale value standard deviation in relation to the mean is representative of a lower deviation from the calculated average of test scores across the entire data set. A higher scale value standard deviation represents a high deviation in test scores from the calculated average throughout a larger sample of the overall data set. Chart 9: Highlands Healthy Lifestyle Choices Perceptual Survey Comparison Avon Park Hill Gustat Lake Placid Sebring Minimum Maximum Mean SD A statistical comparison of the sites reveals that students at Sebring Middle School achieved an average post-test score of 72.7 points, which was the highest average throughout all sites. The post-test scores at Hill Gustat maintained the highest variance from the mean with a calculated standard deviation (SD) of Sebring Middle School achieved the lowest standard deviation of 7.3. Summative Evaluation Report

46 Chart 10: Highlands Student Healthy Lifestyle Choices Perceptual Survey Achieved Success Criterion 13% 21% 28% Avon Park Hill Gustat Lake Placid Sebring 38% Objective 15: 80% of the regularly participating students will increase knowledge of art and culture as demonstrated by Likert styled surveys and performance measures related to art, guitar and cultural elements throughout the year Objective Assessment Plan: 80% of the regularly participating students will increase knowledge of visual/performing arts as measured by Perceptual Survey from Quarter 1 to Quarter 4. Success Criterion: Students who score 70% or higher on the Likert scale pre/post assessment 7% of students with sufficient data indicated increase in knowledge of art and culture as measured by Pre-Post Assessment during the current year. The success criterion was found using the pre and posttest scores of the perceptual art survey, attained at the end of the year. The success criterion required that the students achieve a score of 70 or higher. 70 students had sufficient data to measure the criterion, of which 5 students met the criterion. Please refer to the table below for statistics related to this objective. Summative Evaluation Report

47 Table 31: Highlands Perceptual Art and Culture Survey Statistics Total Min Max M SD Students* Post-Test Scores *Indicates total number of students with usable data, i.e. students with both preliminary and end-ofyear data Across all sites, students achieved an average of 40.2% on the art and culture perceptual survey. The standard deviation (SD) of the post-test scores is 15.7, which is indicative of the representation of the variance in score within the data set based on the calculated mean (M). A lower scale value standard deviation in relation to the mean is representative of a lower deviation from the calculated average of test scores across the entire data set. A higher scale value standard deviation represents a high deviation in test scores from the calculated average throughout a larger sample of the overall data set. Chart 11: Highlands Student Art and Culture Perceptual Survey Achieved Success Criterion 40% 60% Avon Park Hill Gustat Summative Evaluation Report

48 Objective 16: 80% of the regularly participating students will increase knowledge of art and culture as demonstrated by Likert styled surveys and performance measures related to art, guitar and cultural elements throughout the year by completing 2 of the 4 projects. There was not a sufficient amount of data available to evaluate this objective. This year, although the program began operations when they were notified they had received the grant, funding was not distributed until the new year. The program implemented the MindWorks Resources at all four sites which offered a very hands on approach to learning, and many projects through the units were implemented during the project year. With the MindWorks curriculum, the Site Coordinators implemented the six clubs with projects completed for each one, but did not finish the end of unit project. As this was not what the grant indicated as a criterion of success, the program does not have a measure for the grant objective. Next year, the program will work towards completing all projects and meeting the related grant objectives Objective 17: 80% of the regularly participating students will maintain or improve behavior related to physical fitness and social/emotional choices on a Likert scale to transfer into classroom. Objective Assessment Plan: 80% of the regularly participating students will demonstrate positive behavior as measured by school/district records. Success Criterion: Students with less than 2 referrals or those who have reduced the number of referrals by 1 over the school year 93% of students received less than 2 referrals or decreased the number of referrals by at least 1 over the course of the year. Of 138 students across all sites, 116 earned less than 2 referrals over the course of the year, and 13 decreased the number of referrals received. 129 met the success criterion. Summative Evaluation Report

49 Chart 12: Highlands Student Referrals (Less than 2 or Decreased) Achieved Success Criterion 26% 19% 18% 37% Avon Park Hill Gustat Lake Placid Sebring Objective 18: 70% of the teacher reponses will agree or strongly agree that student behavior has improved from the time they were enrolled in the program. Objective Assessment Plan: 80% of the regularly participating students will demonstrate positive behavior as measured by teacher perceptual survey. Success Criterion: 70% of the teacher responses will agree or strongly agree that student behavior has improved from the time they were enrolled in the program. 60% of the teachers who participated in teacher surveys agreed or strongly agreed that student behavior improved. Of 124 teacher responses, 75 met the criterion. Summative Evaluation Report

50 5.4 Other Findings Findings from the student, family member, and teacher end-of-year surveys, as well as detailed information about student performance on the 2014 FCAT are included below. Information about student report card grades for each grade level have also been included in this section Reading Report Card Data The tables below reflect the reading report card data for all regularly participating students in the 21 st CCLC program located at each site. Report card letter grade improvement was attained by measuring the degree to which students changed their individual letter grades in the reading subject area between the first and fourth quarters of the school year. A total of 125 students reported enough data to determine letter grade improvements. Of those students, 18% increased their grade by one letter grade, and 2% increased by two letter grades. Chart 13: Avon Park Reading Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 8th Grade Summative Evaluation Report

51 Chart 14: Hill Gustat Reading Report Card Improvements (by Grade Level) Increased by th Grade 7th Grade Chart 15: Lake Placid Reading Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

52 Chart 16: Sebring Reading Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade Chart 17: All Sites Reading Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

53 5.4.2 Math Report Card Data The tables below reflect the math report card data for all regularly participating students in the 21 st CCLC program located at each site. Report card letter grade improvement was attained by measuring the degree to which students changed their individual letter grades in the mathematics subject area between the first and fourth quarters of the school year. A total of 135 students reported enough data to determine letter grade improvements. Of those students, 28% increased their grade by one letter grade, 6% increased by two letter grades, and 1% increased by three letter grades. Chart 18: Avon Park Mathematics Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

54 Chart 19: Hill Gustat Mathematics Report Card Improvements (by Grade Level) Increased by 1 Increased by 2 Increased by th Grade 7th Grade 8th Grade Chart 20: Lake Placid Mathematics Report Card Improvements (by Grade Level) Increased by 1 Increased by 2 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

55 Chart 21: Sebring Mathematics Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Chart 22: All Sites Mathematics Report Card Improvements (by Grade Level) Increased by 1 Increased by 2 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

56 5.4.3 Science Report Card Data The tables below reflect the science report card data for all regularly participating students in the 21 st CCLC program located at each site. Report card letter grade improvement was attained by measuring the degree to which students changed their individual letter grades in the science subject area between the first and fourth quarters of the school year. A total of 135 students reported enough data to determine letter grade improvements. Of those students, 19% increased their grade by one letter grade, and 4% increased by two letter grades. Chart 23: Avon Park Science Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

57 Chart 24: Hill Gustat Science Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Chart 25: Lake Placid Science Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

58 Chart 26: Sebring Science Report Card Improvements (by Grade Level) Increased by th Grade 7th Grade 8th Grade Chart 27: All Sites Science Report Card Improvements (by Grade Level) Increased by 1 Increased by th Grade 7th Grade 8th Grade Summative Evaluation Report

59 FCAT Reading Data The tables below reflect the 2014 FCAT reading performance of all regularly participating students in the 21 st CCLC program located at each site. Table 32: FCAT Reading Tests Taken by Grade Avon Park Hill Gustat Lake Placid Sebring Grade Grade Grade All Grades Students completed 125 FCAT Reading tests, of which 66 students were enrolled in Grade 6, 37 students were enrolled in Grade 7, and 22 students were enrolled in Grade 8. All students achieved an average score of 2.2, and scores ranged from 1 to 5. Chart 28: All Sites Completed Reading FCAT Assessments (by Grade Level) 18% 6th Grade 29% 53% 7th Grade 8th Grade Summative Evaluation Report

60 Chart 29: Avon Park Completed Reading FCAT Assessments (by Grade Level) 26% 13% 61% 6th Grade 7th Grade 8th Grade Chart 30: Hill Gustat Completed Reading FCAT Assessments (by Grade Level) 15% 27% 58% 6th Grade 7th Grade 8th Grade Summative Evaluation Report

61 Chart 31: Lake Placid Completed Reading FCAT Assessments (by Grade Level) 21% 50% 6th Grade 7th Grade 8th Grade 29% Chart 32: Sebring Completed Reading FCAT Assessments (by Grade Level) 12% 46% 42% 6th Grade 7th Grade 8th Grade Summative Evaluation Report

62 Chart 33: All Sites FCAT Reading Scores by Grade Level th Grade 7th Grade 8th Grade Chart 34: All Sites FCAT Reading Scores by Grade Level th Grade 7th Grade 8th Grade Summative Evaluation Report

63 Chart 35: All Sites FCAT Reading Scores by Grade Level th Grade 7th Grade 8th Grade Chart 36: All Sites FCAT Reading Scores by Grade Level th Grade 7th Grade 8th Grade Summative Evaluation Report

64 Chart 37: All Sites FCAT Reading Scores by Grade Level th Grade 7th Grade 8th Grade Summative Evaluation Report

65 FCAT Math Data The tables below reflect the 2014 FCAT math performance of all regularly participating students in the 21 st CCLC program located at each site. Students completed 126 FCAT Mathematics tests, of which 66 students were enrolled in Grade 6, 38 students were enrolled in Grade 7, and 22 students were enrolled in Grade 8. All students achieved an average score of 2.3, and scores ranged from 1 to 5. Chart 38: All Sites Completed Math FCAT Assessments (by Grade Level) 18% 30% 52% Grade 6 Grade 7 Grade 8 Summative Evaluation Report

66 Chart 39: Avon Park Completed Math FCAT Assessments (by Grade Level) 26% 13% 61% 6th Grade 7th Grade 8th Grade Chart 40: Hill Gustat Completed Math FCAT Assessments (by Grade Level) 15% 27% 58% 6th Grade 7th Grade 8th Grade Summative Evaluation Report

67 Chart 41: Lake Placid Completed Math FCAT Assessments (by Grade Level) 20% 48% 6th Grade 7th Grade 8th Grade 32% Chart 42: Sebring Completed Math FCAT Assessments (by Grade Level) 12% 46% 42% 6th Grade 7th Grade 8th Grade Summative Evaluation Report

68 Chart 43: All Sites FCAT Math Scores by Grade Level th Grade 7th Grade 8th Grade Chart 44: Avon Park FCAT Math Scores by Grade Level th Grade 7th Grade 8th Grade Summative Evaluation Report

69 Chart 45: Hill Gustat FCAT Math Scores by Grade Level th Grade 7th Grade 8th Grade Chart 46: Lake Placid FCAT Math Scores by Grade Level th Grade 7th Grade 8th Grade Summative Evaluation Report

70 Chart 47: Sebring FCAT Math Scores by Grade Level th Grade 7th Grade 8th Grade Summative Evaluation Report

71 5.4.6 Parent Survey Data The parent survey results are detailed in the table below for all sites combined. The survey questions are listed as a reference and parents were able to answer each question indicating if they were very satisfied, satisfied, not sure, unsatisfied, very unsatisfied or not applicable. Table 33: Parent Survey Results Very Satisfied (%) Satisfied (%) Not Sure (%) Unsatisfied (%) Very Unsatisfied (%) Not Applicable (%) How satisfied are you with this after school program as a whole? Staff warmth and friendliness Ability of staff to work with my child Ability of staff to relate to me as a parent Variety of activities offered to my child Program meals and snacks Safety of the program environment Staff reached out to me as a parent My child s happiness with the program Helping me become more involved with my child s education Child s improvement in homework completion Child s improvement in getting along with others Child s improvement staying out of trouble Child s improvement of appreciation of different people or cultures Would you recommend this program to other parents? Yes 100% No 0% Maybe 0% Summative Evaluation Report

72 Would you sign your children up for this program again? Yes 100% No - 0% Maybe 0% If this after school program was no longer available, my child would be: Home alone 37% Cared for by a parent 26% Cared for by a sibling 10% Cared for by another relative 0% Cared for by a friend or neighbor 5% Participating in a different after school program 5% Other 11% Summative Evaluation Report

73 5.4.7 Teacher Survey Data The teacher (regular school day teacher) survey results are detailed in the table below for all sites combined. The survey items are listed as a reference and teachers were able to indicate whether the student in question made a significant improvement, moderate improvement, slight improvement, did not need to improve, slight decline, significant decline and moderate decline. Table 34: Regular School Day Teacher Survey Results Significant Improvement (%) Moderate Improvement (%) Slight Improvement (%) No Change (%) Did Not Need to Improve (%) Slight Decline (%) No Data Reported (%) Significant Decline (%) Moderate Decline (%) Turning in homework on time Completing homework to your satisfaction Participating in class Volunteering Attending class regularly Being attentive in class Behaving in class Academic performance Coming to school motivated to learn Getting along well with other students Summative Evaluation Report

74 5.4.8 Student Survey Data The student (21 st CCLC regular participating students) survey results are detailed in the table below for all sites combined. The survey questions are listed as a reference and students were able to indicate whether they definitely, somewhat or did not at all agree with what the question was pertaining to and/or asking about. If students did not answer the question at all, that information is also included. Table 35: Student Survey Results Did Not Answer (%) Definitely (%) Somewhat (%) Not At All (%) Do you like the after school program? Did your after school program have adults who care about you? Did you feel safe during your after school program? Did your after school program help you get along with others? Did your after school program help you understand that following rules are important? Did your after school program help you solve problems in a positive way? Did your after school program help you with your homework? Did your after school program help you improve your grades? Percentage of male students that completed the survey: 49% Percentage of female students that completed the survey: 51% Grade levels of students that completed the survey: 6 th 43% 7 th 27% 8 th 30% Summative Evaluation Report

75 5.5 Student Success Snapshot As a Site Coordinator for the Lake Placid Middle School 21 st CCLC Program I want to write about a young man named Jesus. Jesus was 13 years old when he came to 6 th grade. His mother was very concerned because he had serious difficulties with reading and math. Even when both parents wanted to help him they were not fluent in English and had not finished school themselves. I recommended the 21 st CCLC Program as an option to get the help he needed. The parents registered Jesus in the program and he was able to get the help and support that he needed to do the work. At the end of 6 th grade the family moved back to Mexico and returned for Jesus s 8 th grade year. Jesus registered again in the program and attended when he was not playing sports for the school. He is very aware of his academic needs and never wanted to get behind. At the end of his 8 th grade year he was very excited that he was promoted to 9 th grade. Jesus is an example of the students whose parents must work long hours and want their children to be safe after school. He is also an example of many students that have difficulties with subject areas that have parents that want to help them, but they do not have the academic knowledge themselves. Lastly, he is an example of how the 21 st CCLC Program partners with parents and the school to provide the help that students need to succeed. - Written By: Lake Placid Middle School Site Coordinator Arianna was chosen to be in this year s 21 st CCLC After School Program because her participation and attendance were excellent in our program during her 6 th grade year. Arianna was in both Intensive Math and Intensive Reading classes due to her chronically below average scores on the Florida Comprehensive Assessment Test. Arianna s parents felt that her participation in the program during her 6 th grade year was an instrumental factor in both her personal and academic success. Her parents were fearful that her academic performance may decline because 7 th grade can be a very difficult year both academically and emotionally. Summative Evaluation Report

76 Her parents hoped that through our 21 st CCLC program, Arianna would continue to be successful in school because it allowed her the academic reinforcement she required and the enrichment to apply what she had learned. The 21 st CCLC After School program also allowed Arianna to spend time in a safe and productive environment as opposed to going home after school to her neighborhood influences. As anticipated, Arianna was challenged academically during her 7 th grade year. Her grades often fell below the passing mark. Through the 21 st CCLC program we were able to communicate with her teachers to ensure her work and test preparation were accurate. Arianna earned passing marks in all her classes for the year. Standardized test scores for this year are not yet available. During the school year through the 21 st CCLC After School Program, Arianna has participated in the GO Girls Program and the Positive Action Club. She also participated in the 5th grade Orientation by presenting information about the 21 st CCLC After School Program to a group of over 250 students and teachers. - Written By: Sebring Middle School Site Coordinator 5.6 Overall Findings for Each Objective This section provides information on the program s progress toward and achievement of each objective Objective 1: 70% of the regularly participating students will maintain or improve their reading comprehension skills as demonstrated by pre, mid & post assessment scores on FAIR and report card grades Findings: Objective not met. Total students with sufficient data % met the stated success criterion Objective Status Meaningful Progress Summative Evaluation Report

77 5.3.2 Objective 2: 70% of the regularly participating students will maintain or improve their reading comprehension skills as demonstrated by pre, mid & post assessment scores on FAIR and report card grades Findings: Objective not met. Total students with sufficient data % met the stated success criterion Objective Status Some Progress Objective 3: 70% of the regularly participating students will maintain or improve their math skills as demonstrated by pre, mid & post assessment scores on Math Performance Matters and report card grades Findings: Objective not met. Total students with sufficient data 67 43% met the stated success criterion Objective Status Some Progress Objective 4: 70% of the regularly participating students will maintain or improve their math skills as demonstrated by pre, mid & post assessment scores on Math Performance Matters and report card grades Findings: Objective not met. Total students with sufficient data 135 Summative Evaluation Report

78 60% met the stated success criterion Objective Status Approaching Benchmark Objective 5: 70% of the regularly participating students will maintain or improve their science skills as demonstrated by pre, mid & post assessment scores on Science Performance Matters and report card grades Findings: Objective not met. Total students with sufficient data 55 56% met the stated success criterion Objective Status Meaningful Progress Objective 6: 70% of the regularly participating students will maintain or improve their science skills as demonstrated by pre, mid & post assessment scores on Science Performance Matters and report card grades Findings: Objective not met. Total students with sufficient data % met the stated success criterion Objective Status Some Progress Summative Evaluation Report

79 5.3.7 Objective 7: 80% of the regularly participating students will increase knowledge of technology programs and procedures as demonstrated by pre, mid & post assessment scores on TechLiteracy test and projects throughout the year Findings: Objective not met. Total students with sufficient data 32 50% met the stated success criterion Objective Status Some Progress Objective 8: 80% of the regularly participating students will increase knowledge of technology programs and procedures as demonstrated by pre, mid & post assessment scores on TechLiteracy test and projects throughout the year Findings: Objective not met. Total students with sufficient data 0 0% met the stated success criterion Objective Status No Progress Objective 9: 70% of family members from regularly participating students who attend at least 1 literacy evemt will demonstrate growth on the monthly assessments Findings: Objective not met. Total parents with sufficient data 2 0% met the stated success criterion Summative Evaluation Report

80 Objective Status None Objective 10: 25% of family members from regularly participating students who participate in the Active Parenting program offered through the year will demonstrate growth on pre and post assessments. Findings: Objective not met. Total parents with sufficient data 6 0% met the stated success criterion Objective Status None Objective 11: 40% of family members from regularly participating students will attend at least one 21st CCLC sponsored event, offered monthly and related to family requested topics including curriculum, homework help, home-school communication, and drug and gang awareness as measured by adult sign in sheets through the year Findings: Objective not met. Total parents with sufficient data 138 5% met the stated success criterion Objective Status Limited Progress Summative Evaluation Report

81 Objective 12: 60% of family members from regularly participating students will increase their participation in school sponsored events as measured by monthly sign in sheets distributed quarterly Findings: Objective not met. Total parents with sufficient data 138 5% met the stated success criterion Objective Status Limited Progress Objective 13: 80% of the regularly participating students will maintain or improve of behavior related to physical fitness as measured by Healthy Lifestyle Choices by pre, mid & post assessment as well as increased Likert style surveys throughout the year Findings: Objective not met. Total students with sufficient data 57 67% met the stated success criterion Objective Status Meaningful Progress Objective 14: 80% of the regularly participating students will maintain or improve of behavior related to physical fitness as measured by Healthy Lifestyle Choices by pre, mid & post assessment as well as increased Likert style surveys throughout the year Findings: Objective not met. Summative Evaluation Report

82 Total students with sufficient data 80 59% met the stated success criterion Objective Status Meaningful Progress Objective 15: 80% of the regularly participating students will increase knowledge of art and culture as demonstrated by Likert styled surveys and performance measures related to art, guitar and cultural elements throughout the year Findings: Objective not met. Total students with sufficient data 70 7% met the stated success criterion Objective Status Limited Progress Objective 16: 80% of the regularly participating students will increase knowledge of art and culture as demonstrated by Likert styled surveys and performance measures related to art, guitar and cultural elements throughout the year by completing 2 of the 4 projects. Findings: Objective not met. Total students with sufficient data 0 0% met the stated success criterion Objective Status No Progress Summative Evaluation Report

83 Objective 17: 80% of the regularly participating students will maintain or improve behavior related to physical fitness and social/emotional choices on a Likert scale to transfer into classroom. Findings: Objective met. Total students with sufficient data % met the stated success criterion Objective Status Meets or Exceeds Benchmark Objective 18: 70% of the teacher reponses will agree or strongly agree that student behavior has improved from the time they were enrolled in the program. Findings: Objective not met. Total students with sufficient data % met the stated success criterion Objective Status Meaningful Progress Summative Evaluation Report

84 6.0 PROGRESS TOWARDS SUSTAINABILITY This section provides information on strategies employed to ensure the sustainability of 21 st CCLC program and enhance the quality of the services offered. Table 36: Partnerships and Sub-Contracts Agency Name Rural Health of Highlands County Tri-County Health Children s Advocacy Center Type of Organization Sub- Contractor (Yes/No) Estimated Value ($) of Contributions Estimated Value ($) of Sub-contract CBO No $4,500 N/A CBO No $1,000 N/A CBO No $1,000 N/A Type of Service Provided Character Education: Go Girls club which provides self-esteem and goal setting for girls middle school level. Heritage Keepers presentations focus on selfesteem and is an abstinence based program at middle school level. Drug Education which provides a drug prevention program at middle school level. Provides training as needed for staff on wellness for children. Works with Summative Evaluation Report

85 McDonald s FP No $100 N/A program on Parent Literacy Events. Provides gift cards to be used for incentives for Parent Literacy Events. Note: Value of subcontract must be ZERO if the agency is listed as No in the subcontractor column. Likewise, the value of the subcontract must be greater than ZERO if the agency is listed as Yes in the subcontractor column. Note: The USED s federal definition differentiates between a partner and a subcontractor. A subcontractor receives compensation (to some extent) for their services; partners do not. 7.0 LESSONS LEARNED AND RECOMMENDATIONS learned valuable information this year about what needs to be done next year to have a more high functioning 21 st CCLC program at Avon Park Middle, Sebring Middle, Hll-Gustat Middle and Lake Placid Middle schools. Due to receiving their funding well after the school year started, the program struggled all year with student attendance and having students truly commit to the program. Over the summer, parents had already made arrangements for their children after school, so when the program started later in the school year parents were resistant to changing their current arrangements and enrolling their children in the 21 st CCLC program. For the upcoming year, The School Board of Highlands County will explore recruitment opportunity and will market the program to incoming 5 th graders at their 6 th grade orientation. Parents will also receive informational materials about the program at the end of the school year so that they can plan accordingly. This will be done in an effort to boost attendance for the upcoming project year. In addition, the following recommendations have been made: School Administrators and the Program Director need to visit the sites during the time in which the 21 st CCLC is operating on a more regular basis to stay abreast of how the sites are doing individually and collectively. First-hand observations from program leadership Summative Evaluation Report

86 would prove to be beneficial. In addition, the presence of leadership conducting walkthroughs of the program sets the expectation high for all involved and could help improve program fidelity. Conduct more pre-planning meetings before the program begins so that lesson plans are already identified for each nine weeks before the first day of the program. It would be beneficial for all sites to implement similar or the same schedules, classes and lesson plans so that there is one uniform program since the same MindWorks curriculum is being used at each site. This would also allow the Site Coordinators to coordinate and collaborate together by sharing ideas and other relevant information that will help improve the program overall. Ensure that all students in the program at each site are tested for each objective and create a schedule that allows for this to take place. Create a more comprehensive plan for outdoor enrichment activities that are targeted to students in grades 6-8 specifically so that they are engaged during the activities. Conducting organized activities where the staff leads the activities and the students participate in the group activity that is being led by the staff are more engaging for the students and will yield more positive results. Review all grant objectives and their success criterion in each teacher meeting that is held throughout the year so that Site Coordinators are aware of what will need to be done to meet each objective. This year some of the objectives were not met because the data for those objectives were not collected. Prepare a detailed plan regarding parent events/workshops for each site before the upcoming project year begins. Align the plan with the parent-related grant objectives for the upcoming year so that those objectives can be met. Summative Evaluation Report

87 THIS EVALUATION REPORT HAS BEEN PREPARED BY: LISTENING REVIEWING ADVISING TRAINING MEASURING GOALS ACHIEVEMENT SUCCESS Grant Evaluation Services Professional Development Education Consulting (800) Summative Evaluation Report

88 "Tell me and I forget. Teach me and I remember.involve me and I learn." - Benjamin Franklin Summative Evaluation Report