Evaluation Toolkit and Data Collection Guide

Evaluation Toolkit and Data Collection Guide Youth Participatory Action Research Projects: Nutrition, Physical Activity, Empowerment August 2016

This material was originally produced in 2012 by the California Department of Public Health s (CDPH) Nutrition Education and Obesity Prevention Branch for its Youth Engagement Initiative (YEI) with funding from USDA SNAP-Ed, known in California as CalFresh. These institutions are equal opportunity providers and employers. CalFresh provides assistance to low-income households and can help buy nutritious food for better health. For CalFresh information, call 1-877-847-3663. For important nutrition information, visit http://www.cdph.ca.gov/programs/neopb/pages/default.aspx The initial draft of this toolkit was developed in collaboration with LPC Consulting Associates. A final version built upon and incorporated much of the work done for the draft version that was written by Tracey Weld of LPC Consulting Associates. Subsequent updates and additions were added by Metria Munyan and Barbara MkNelly of the Public Health Institute in 2014. This version was further edited by Sharon Sugerman of the Public Health Institute Center for Wellness and Nutrition to make it less California-specific so that it will be useful for programs conducting Y-PAR activities in other states as well. One data collection tool that is required and used only in California has been omitted as has the final section on submitting your data since neither will apply to the general user. The explicit CDPH program name, Youth Engagement Initiative, has been replaced with the more general term Youth Participatory Action Research (Y-PAR) in most instances, unless referring to something particular to YEI, such as the logic model it developed. The original forms developed for this manual were identified with the logo for the Network for a Healthy California. Logos have been omitted from the forms included in the appendices in this manual so that they can be used more widely. This project was supported by Regional Nutrition Education and Obesity Prevention Center of Excellence Initiative of the USDA National Institute of Food and Agriculture, grant number 2014-48757- 22607. Thank you to LPC Consulting Associates for their help in the development of this guide. Adapted from a collaboration project with the CDPH. Updated August 2016

TABLE OF CONTENTS Table of Contents 3 Introduction to the Youth Engagement Initiative Evaluation Toolkit 1 How Were the Data Collection Tools Developed?... 1 Why Fill Out Forms and Collect Data?... 1 Learning from What We Do: What is a Logic Model?... 2 Y-PAR Evaluation Data Collection Tools and Description... 5 Instructions for Using Y-par Data Collection Tools 6 Core vs. Required vs. Supplemental: What s the Difference?... 6 Instructions for Completing Core Forms... 6 Instruction for Completing Supplemental Forms... 16 Appendices A: Core Data Collection Tools... 19 Appendices B: Supplemental Data Collection Tools... 36

Youth Participatory Action Research Evaluation Toolkit and Data Collection Guide INTRODUCTION TO THE YOUTH ENGAGEMENT INITIATIVE EVALUATION TOOLKIT The purpose of this guide is to explain the evaluation and data collection process, provide instructions for the completion of the forms, and enable local project sites to tell the story of a Youth Participatory Action Research Project (Y-PAR) in their community. This guide and the associated data collection tools will help you systematically track and document the activities and outcomes of your Y-PAR projects. If all programs conducting within the same jurisdiction, such as a county or a state, follow the data collection methods using the tools in this toolkit, findings can be combined to convey a powerful picture of the activities in that area. Evaluating your activities will help you learn what works best to engage youth in obesity prevention and decide how to proceed in the future. The data you collect will provide valuable information at the local and state level for program planning, engaging new partners, soliciting support, and telling the story of your successes and challenges. This guide is intended for adult allies and project coordinators participating in Y-PAR at local project sites. Please read the guide carefully so you will feel comfortable completing the forms, collecting data, and documenting your work. How Were the Data Collection Tools Developed? Following the principles of community engagement, the data collection forms were created with the participation of youth and adult allies at California Department of Public Health Youth Engagement Initiative (YEI) project sites. All sites were invited to take part in surveys, webinars, and site visits in order to inform the development of the initiative s logic model and evaluation tools. Fourteen sites volunteered to complete an online survey. The Alameda County Office of Education s Project Eat and Del Norte Unified School District went the extra mile to participate in a logic model webinar and a site visit to test the draft data collection tools. As a result, the evaluation toolkit is more user-friendly and the logic model more accurately reflects the YEI theory of change. Additionally, the LPC Consulting Associates, Inc. research team conducted a review of the academic literature relevant to Y-PAR, youth engagement, PhotoVoice, and evaluation of efforts similar to YEI. This background research provided a context for the development of the logic model and evaluation tools. Why Fill Out Forms and Collect Data? One of the primary reasons to collect data for program evaluation is to learn whether or not the activities you re carrying out are having the intended effect. You will be able to track your results locally and use the information to make decisions about your Y-PAR project. The data can be used to create reports that help tell the story of your project and, if a number of projects in a jurisdictions are collection data, show how it is working across that area. PG. 1

Youth Participatory Action Research Evaluation Toolkit and Data Collection Guide Collecting and analyzing program data lets you learn what changes are happening for youth after they become involved in Y-PAR, what impact youth are having in their schools and communities, and how Y-PAR can be improved. This information also allows you to explain to policy makers why funding for this program should continue and assure the public that their tax dollars are being wisely invested. Documenting the positive benefits and results of Y-PAR can also help you gain more support for your efforts at the local level. For example, information from the data collection forms can potentially persuade other stakeholders in your community to partner with you or provide additional funding for your Y-PAR project. Learning from What We Do: What is a Logic Model? Evaluation lets you to learn how you are making a difference with the programs and projects you undertake to improve health. The Stepping Stones Y-PAR model is guided by a theory of change a set of ideas about how and why we think a program will produce desired results. This theory of change can be thought of as a series of if-then relationships. For example, if we build the capacity of youth to conduct research about nutrition and share the findings with decision makers, then policies and practices will change such as better quality food in schools. A logic model is a one-page picture that displays the theory of change and shows the connection between project activities (what you re doing) and project outcomes (what you accomplish and your impact). A logic model not only helps explain what you re doing and why, it also guides program planning and evaluation. Additionally, it can be a very effective tool for communicating what you hope to achieve in order to gain community support for your efforts. The Statewide California YEI Logic Model illustrates the range of strategies and activities that its local project sites use in order to achieve short and medium-term outcomes, as well as long-term impacts. It provides a sense of what they are trying to achieve as a whole and helps people understand how YEI is designed to work. To create the Statewide YEI Logic Model, local project sites volunteered to provide input through surveys and webinars. Youth and adult allies described why they selected their research issues, what activities were planned or implemented, and what success would look like at the end of the project. This information was compiled and analyzed and informed the development of the Statewide logic model. Two versions of the logic model were created: version A distinguishes between community and individual-level change building upon a logic model approach used by a Gates Foundation high school reform program; and version B is better suited to an audience that would prefer a streamlined version. Local project sites are encouraged to adapt the Statewide Logic Model or create their own to enhance decision making and engagement of other community partners. The contents of this model can be adopted and revised as needed by other states, counties, school districts, and individual Y-PAR projects or programs. PG. 2

Youth Participatory Action Research Evaluation Toolkit and Data Collection Guide * Youth Engagement Initiative (YEI) is a California-specific name for Youth Participatory Research (Y-PAR). PG. 3

* Youth Engagement Initiative (YEI) is a California-specific name for Youth Participatory Research (Y-PAR). PG. 4

Y-PAR Evaluation Data Collection Tools and Description The following pages describe the data collection tools, along with how and when to use them. Y-PAR Evaluation Data Collection Tools Quick Guide Tool # Completed By Ally Youth 1 2 3 4 5 6 7 8 Core Data Collection Tools Information Form for Youth Researchers: All youth complete this form to participate in YEI. Data collected: Demographic data, baseline data, and optional screening questions. Sites may add permissions. Beverage and Snack Questionnaire 2: This survey measures changes in the frequency of consumption of a variety of food and beverage choices the student makes, separately, during school and out of school. Data collected: water, sugary and non-sugary beverages, a variety of both salty and sweet snacks, fruit, and vegetables. Baseline Survey (Form title is: Getting To Know You Youth Survey): This survey measures changes that youth experience before participating in Y-PAR. Data collected: Skills learned, knowledge gained, future advocacy plans, changes in behavior, and current activities and interests. Y-PAR Team Meeting Sign-in Sheet: Use at all youth-attended Y- PAR meetings. Data collected: Individual levels of participation, tracks Stepping Stones progress, and activities completed during meetings. Event and Activity Log: Completed after every activity or event. Data collected: Types (coded), audience (coded), description (open), # of researchers, youth, adults, school staff, and community members involved or reached by the activity or event. This form submitted to the person filling out the online Activity Tracking Form. Y-PAR Project Evaluation (Form title is: Feedback for Your Project Leader/Adult Ally): This survey assesses youth s experience with Y-PAR. Data collected: Opinions on youth engagement, Y-PAR fidelity and experience with adult ally. End of Year Youth Survey: This survey measures changes that youth experience after participating in YE-PAR. Data collected: Skills learned, knowledge gained, future advocacy plans, changes in behavior, and current activities and interests. Youth Engagement Team Annual Summary Report: Provides a quick snapshot/very brief description of each site s program. Data Start up When to complete On Going End of Year End of Project PG. 5

Tool # Completed By Ally Youth 9 Tool # Completed By Ally 10 Youth Core Data Collection Tools collected: Research question, # of youth involved, focus area, project description, evaluation/research method, achievements, challenges, next steps and technical assistance needs. Youth Engagement Final Team Report: Due at end of project. Provides an outline of items to include in a creative product to conclude the entire project. Data collected: Research methods, instruments, findings, outcomes, accomplishments, challenges, conclusions, and youth biographies. Supplemental Data Collection Tools Assessing Adults as Allies: This pre-assessment measures belief systems around working with young people Start up Start up When to complete On Going End of Year When to complete Ongoing End of year End of Project End of project 11 Adult Ally Pre/Post Survey: This survey measures changes in adults knowledge and skill after participating in Y-PAR. Data collected: Skills learned, knowledge gained, 12 Adult Ally Assessment of Participants: This post-assessment measures changes in youth observed by the adult ally. Data collected: Key skills gained from participation in Y-PAR. INSTRUCTIONS FOR USING Y-PAR DATA COLLECTION TOOLS Core vs. Required vs. Supplemental: What s the Difference? Core data collection tools ensure a thorough evaluation of your progress. Additionally, the information collected in the core forms can provide information to assist sites in their local project planning and sustainability efforts. Supplemental forms are optional. These additional data collection tools may enhance your evaluation and lead to a deeper understanding of the changes that result from the Y-PAR activities implemented at your site. Instructions for Completing Core Forms 1. Information Form for Youth Researchers WHO/WHEN: Completed by all youth and their parent/guardian prior to participating in YEI. PG. 6

WHY: The purpose of the form is to collect basic demographic information about youth researchers, learn about their interests and gain parent permission for their participation. Y- PAR sites may choose to add additional permissions (e.g. photo release) as needed. This registration form can also serve double duty as an application for prospective youth researchers. The first page is required and the second page is optional. The form includes optional interest questions to screen potential participants to determine if they a good fit for Y-PAR. Sites may elect to use these interest questions or tailor them to local needs. This baseline data will allow us to track how many youth participate in Y-PAR, their demographics and what changes may result in their activities and community involvement after participating in Y-PAR. HOW: Youth should take the form home to be completed in partnership with their parent(s) or guardian(s). The demographic information is important information to the funder, but students have the right to decline. Sometimes race and ethnicity questions are confusing and people do not complete this question correctly or feel they are not being represented. It may help us get accurate data if you help your students answer this two part question. The optional questions about interest in the project should be completed by youth on their own. Adult allies or other faculty/staff may consider responses to the interest questions as a means of determining which youth are selected to participate. PG. 7

2. Beverage and Snack Questionnaire 2: This survey measures changes in the frequency of consumption of a variety of food and beverage choices the student makes, separately, during school and out of school. WHO/WHEN: Youth anonymously complete this survey at the beginning and end of the project year. WHY: This survey asks youth to report how often, on a weekly or daily basis, they drink or eat water, a variety of sugary and non-sugary beverages, different types of salty snacks, sweets, fruits, and vegetables, before and after participating in Y-PAR. This data will help you quantify the effects or changes on youth as a result of their participation in the program. The collective results can help you make the case for program sustainability at both the local and state level. HOW: Youth anonymously complete the questionnaire, based on their personal history and project experience. PG. 8

3. Baseline Youth Survey (Getting To Know You Survey): This survey measures changes that youth experience before participating in Y-PAR. Data collected: Skills learned, knowledge gained, future advocacy plans, changes in behavior, and current activities and interests. WHO/WHEN: Youth anonymously complete this survey at the beginning of the project year. WHY: This survey asks youth to report on their feelings, behavior and experiences before participating in Y-PAR. It is a baseline survey that asks students what they are doing now. This data will help you quantify the effects or changes on youth as a result of their participation in the program. It acts as a check on the findings of the End of the Year Youth Survey when students are asked to remember what they were doing and thinking before they took part in Y-PAR. HOW: Youth anonymously complete the questionnaire, based on their personal experience. PG. 9

4. Y-PAR Team Meeting Sign-in Sheet WHO/WHEN: Completed by youth and adult allies at every Y-PAR meeting. This form should be completed every time the Y-PAR project has a regular project meeting (usually weekly). WHY: The form tracks the participation of youth in Y-PAR activities and progress toward the completion of the Stepping Stones from the Youth Engagement Training Binder. The data will be used to determine the number of students involved, the type of activities they participate in and which Stepping Stone is addressed. At the local level, the information collected on these forms also can help you determine how to allocate youth stipends based on attendance. This log also will help with the end of the year summary. HOW: Youth should sign-in using this form during the meeting. Adult allies should check the box indicating which Stepping Stone the group is working on and briefly describe the activity e.g. SS1 Getting Ready, SS2 Orientation and Teambuilding, SS3 Research/Evaluation Design, SS4 Skill Development, SS5 Data Collection and Documentation, SS6 Data Analysis, SS7 Final Report, Presentation & Celebration, SS8 Next Steps and ACTION! PG. 10

5. Event and Activity Log WHO/WHEN: Youth complete this form after every activity or event that Y-PAR project sponsors or attends, outside of regular team meetings. This form is used on an ongoing basis throughout the project year. WHY: Tracking other people s participation in events will help you understand the reach Y- PAR has into the wider community. Tracking the participation of youth researchers in additional events and activities will help you understand what it takes to develop skills in youth and create changes in the school or community. Keeping track of your process also enables you to tell the story of your efforts. This log also will help with the completion of the end of the year summary and reports. HOW: Youth should record the date the activity took place. In the activity type column, youth should record a number 0-7 to indicate the type of activity or event, as coded at the bottom of the form. In the Audience column, youth should enter a number 0-10 to indicate who participated in the event, as coded at the bottom of the form. You may record more than one number in each of these columns. The total number of events and participants should be calculated and entered in the corresponding boxes at the bottom of the page. PG. 11

6. Y-PAR Project Evaluation WHO/WHEN: Youth complete this survey at the end of the project year. WHY: This survey assesses youth s experience with Y-PAR. The questions are designed to help you understand how closely the principles of youth development and engagement are being followed. The data will help us understand youth s leadership experience. The information also can help adult allies assess their strengths and challenges in facilitating a youth-led project. HOW: Youth anonymously complete the questionnaire, based on their project experience. PG. 12

7. End of Year Youth Survey: This survey measures changes that youth experience after participating in Y-PAR. Data collected: Skills learned, knowledge gained, future advocacy plans, changes in behavior, and current activities and interests. WHO/WHEN: Youth anonymously complete this survey at the end of the project year. WHY: This survey asks youth to report on their feelings, behavior and experiences before and after participating in Y-PAR. It is a retrospective survey that asks students what they are doing now compared to what they remember they were doing before they were participating in Y- PAR. This data will help you quantify the effects or changes on youth as a result of their participation in the program. The collective results can help you make the case for program sustainability at both the local and state level. HOW: Youth anonymously complete the questionnaire, based on their project experience. PG. 13

8. Youth Engagement Team Annual Summary Report WHO/WHEN: Youth and adult allies complete this form at the end of the project year. WHY: Completing this form provides a description of each site s project and a summary of the year s activities and serves multiple purposes. At a jurisdiction level, e.g., state or county, the information collected will guide future program planning, for example by helping to determine if program refinements or additional technical assistance are needed. By documenting impacts, the data collected will help tell the story of Y-PAR both on the local level and across the state. Additionally, youth and adult allies can track project progress and accomplishments from year to year, leading to a deeper understanding of the Y-PAR project and process. The results may be shared with community members, decision makers or media. HOW: The completion of this form is intended to be a cooperative effort between youth and adult allies. We recommend reserving time at a weekly meeting to hold a discussion of the key questions. A youth researcher, youth team or the adult ally can take responsibility for entering the group responses for each question. PG. 14

9. Youth Engagement Final Team Report WHO/WHEN: Youth, assisted by adult allies, complete the final team report when the Y-PAR project is complete typically at the end of two years. WHY: Completing a final report is an important skill for young people to learn. It represents the culmination of their work and is a product that can be shared with other funders, partners, prospective employers, and/or included in college applications. HOW: Youth researchers should work together to decide what they want to produce and how to divide the work. An outline of final report components is provided as a guideline. Teams are encouraged to be creative in designing their final product. For example, the final report can be in a PowerPoint presentation, a video, or PhotoVoice format. The adult ally or other senior project staff should complete a facilitator statement by answering the questions contained in the outline. Research instruments, pictures, logos, and copies of promotional materials or publicity should be included. The examples below are from the California Department of Public Health s Youth Engagement Initiative. PG. 15

10. Assessing Adult as Allies Instruction for Completing Supplemental Forms WHO/WHEN: Adult allies complete this form at the beginning of the project or in the interview process for the adult ally position. WHY: Completing this form as a self-assessment or a pre-assessment allows you to understand the belief structures that exist with that adult about how youth should be involved and what they are capable of. HOW: Adults complete the questionnaire as a self-assessment or as a pre- assessment for the interview process for the adult ally position. PG. 16

11. Adult Ally Pre / Post Assessment WHO/WHEN: Adult allies complete this form at the beginning of the project and then again at the end of the project. WHY: This assessment will be used to develop future trainings and inform coaching of allies. HOW: Adults complete the questionnaire as a self-assessment at the beginning of the year or project and then again at the end of the year or project. PG. 17

12. Adult Ally Assessment of Participants WHO/WHEN: Adult allies complete this form at the end of the project year. WHY: This form allows adult allies to assess changes in youths skills and knowledge that occur after participating in Y-PAR. This data allows quantification of the collective impact of Y-PAR on youth across a multiple-site program or jurisdiction. The results can help make the case for sustainability of the program on both the local and State level. Just as important, this tool may be useful for youth in employment or college applications. HOW: Adult allies complete one form for each Y-PAR participant, if feasible. This tool is most useful for youth who are consistently involved in Y-PAR. You might ask youth at the beginning of the project if they would like to participate in this level of evaluation. For each item, adult allies should enter a score (4-0) that characterizes the level of mastery the participant has at the beginning of the program and at the end of the project year. The first number should be subtracted from the last and entered into the final column as a + or number. Total each column to gain an overall sense of the youth s progress during the year. PG. 18

APPENDICES A: CORE DATA COLLECTION TOOLS PG. 19

Appendix A: Information Form PG. 20

PG. 21

Appendix A: Beverage and Snack Questionnaire 2 PG. 22

Neuhouser ML, Lilley S, Lund A, and Johnson DB. Development and validation of a beverage and snack questionnaire for use in evaluation of school nutrition policies. J Am Diet Assoc 2009; 109 (9): 1587-1592. To access this tool contact: Nutrition Assessment Shared Resource of the Fred Hutchinson Cancer Research Center (https://sharedresources.fredhutch.org/documents/beverage-and-snack-questionnaire-2). For additional information, email nasr@fhcrc.org or call (206) 667-4161 or (800) 460-7270 (toll free). PG. 23

Appendix A: Getting To Know You Youth Survey PG. 24

PG. 25

Appendix A: Y-PAR Team Meeting Sign-in Sheet PG. 26

PG. 27

Appendix A: Event and Activity Log Youth Participatory Action Research PG. 28

Appendix A: Feedback for Your Project Leader/Adult Ally Youth Participatory Action Research PG. 29

Appendix A: End of Year Youth Survey Youth Participatory Action Research PG. 30

Youth Participatory Action Research PG. 31

Appendix A: Youth Engagement Team Annual Summary Report PG. 32

PG. 33

PG. 34

Appendix A: Youth Engagement Final Team Report Youth Participatory Action Research PG. 35

APPENDICES B: SUPPLEMENTAL DATA COLLECTION TOOLS PG. 36

Appendix B: Assessing Adults as Allies Youth Participatory Action Research PG. 37

Appendix B: Adult Ally Pre/Post Survey Youth Participatory Action Research PG. 38

Youth Participatory Action Research PG. 39

Appendix B: Adult Ally Assessment of Participants Youth Participatory Action Research PG. 40