Toward Empowerment Evaluation in Alameda County: Lessons Learned Innovative Approaches to Evaluation in Complex System of Care Communities: Using Empowerment Evaluation (Putting It Into Practice) Sonia Jain, Dr.P.H., Mansi Master, M.P.A., Alison Cohen, M.P.H., Margie Gutierrez-Padilla, LCSW, Liz S. Grossman, M.P.H., Stacy F. Johnson, M.S.W. March 4, 2013 Children s Mental Health Research and Policy Conference
Early Connections Is... A collaboration of Family members Community members Providers in a systems change effort designed to improve or promote wellness for young children and their families in Alameda County
Early Connections: Just the Facts A 6-year Federal grant funded by SAMHSA (Substance Abuse and Mental Health Services Administration) as part of the Children s Mental Health Services Initiative, 2009 2015 Currently in year 3 Alameda County, CA First 5 Highly diverse region with 1.5 million people, with a very active, experienced early childhood community of policymakers and practitioners 14 mental health providers who serve over 2,000 families with young children with mental health needs each year SAMHSA ACBHCS UACF
Early Connections Vision: Building Partnerships With Families Early Connections Vision All children birth to 5 with serious mental health needs countywide and their families: Are healthy and stable Have access to quality service and supports Receive services that are respectful and promote hope and resilience We envision a strength-based, coordinated, culturally responsive system of prevention. A system of care is a different way of doing business. It is a philosophy of supporting children and families. Family driven Youth guided Strength based Culturally and linguistically responsive Community based Collaborative across child-serving systems Data driven
Stages of System of Care Development and the Uses of Evaluation Outcomes Cost effectiveness Decisions about program adaptation Assess the problem and the need and use it to plan the solution Measure the process S
How did Early Connections put Empowerment Evaluation (EE) into practice? Inclusion, Democratic Participation, Community Ownership Built partnerships and close relationships with mental health providers To better understand providers evaluation and data needs, existing data capacity and buy-in for evaluation, variation in intake and assessment protocols Developed enrollment, recruitment, and consent materials in partnership with their staff
How did Early Connections put EE into practice? Social Justice, Capacity Building Hired 6 bicultural/bilingual field interviewers Many are family members as part of the evaluation team, embodying and practicing the familydriven model
How did Early Connections put EE into practice? Inclusion, Community Knowledge Constantly tried to engage system-level stakeholders, providers, family partners, and families In development and implementation of the evaluation design and methods, including Consent forms Recruitment charts Data collection tools Retention cards Incentives for families CQI processes Monthly Community Evaluation Action Team, participating in governance
How did Early Connections put EE into practice? Democratic Participation Organizational Learning Made our evaluation plans clear and transparent by Having open-ended conversations about indicators of success and results Using simple non-technical language Constantly tried to translate presentations, tools, reports, materials into simple community-friendly languages, including Chinese, Vietnamese and Spanish
How did Early Connections put EE into practice? Improvement, Community Ownership, Evidence-Based Strategies Worked closely and constantly to make ourselves and data more accessible, demystifying evaluation Doing an Evaluation 101, sharing the decision-making power with community and practitioners (not solely funding- and data-driven) Attending and collaborating with other Action Teams as a critical thinker Promoting results-based accountability
What do all EE principles have in common? Community Knowledge, Ownership Capacity Building Inclusion, Democratic Participation
Strategies to Engage Community Partners in Evaluation Specific strategies to strengthen collaborative relationships is a significant component of successful evaluation. Identify necessary collaborative relationships Define ways to strengthen buy-in Review potential barriers to collaboration Obtain community input Identify data allies across systems and in community Maintain local investment during implementation of the evaluation
Evaluation Process Use data for program improvement, policy, and sustainability Identify participants, users, and their expectations Prioritize objectives and indicators of interest Analysis and reporting Agree on methods, responsibilities, and timeline Tools development and data collection Reference: Participatory evaluation http://depts.washington.edu/ccph/ pdf_files/evaluation.pdf
Benefits of Using EE Results are more meaningful and useful Capacity building of interviewers, family members, staff to do research More family-driven evaluation Shared ownership of data Cultural shift toward data-driven decision making
Challenges More time consuming, labor and resource intensive There are varied audiences (community, family members, data folks, broad-based partners...) Program is constantly evolving goals, strategies, and activities changing; no set logic model and outcomes
Challenges Differential perspectives, experiences and organizational cultures re: role and value of evaluation and data Constantly trying to simplify and translate Difficult to discern where to have technical conversation and where to simplify Need additional resources to translate
Lessons Learned It s a dynamic process, relationship-based Find a middle ground (not completely participatory but not completely hands-off; sometimes EE approach is not feasible) Effective and open communication is key Use simple, easy-to-understand language, and translate materials Capacity to facilitate groups with diverse audiences is important
Lessons Learned Being on the same page is critical on an ongoing basis understand the program needs Have clear deliverables, expectations, and timelines (clarify on an ongoing basis, with core group or PD) Be proactive Lens of the researcher and program matters a lot take time to build trust, understanding, mutual respect...
Small Group Activity: Working Through Challenges Toward EE Pick one major challenge you currently are having related to evaluation What can you do differently to improve or move toward EE? Where do you need support? Report back and share lessons learned
Resources and References Empowerment Evaluation Blog: http://eevaluation.blogspot.com/ Getting to Outcomes : http://www.cmhilibrary.org/ Empowerment Evaluation Principles in Practice, Fetterman & Wandersman, Guilford Press, 2005.
Thank You! Sonia Jain, Dr.P.H. Lead Evaluator, Early Connections Senior Research Associate WestEd, Health and Human Development 300 Lakeside Drive 25th Floor Oakland, CA 94612 Tel: 510.302.4251 E-mail: sjain@wested.org Margie Gutierrez-Padilla, LCSW Early Connections 0 5 System of Care Coordinator Alameda County Behavioral Health Care Services 500 Davis Street, Suite 120 San Leandro, CA 94577 Tel: 510.481.4206 E-mail: mpadilla@acbhcs.org Liz S. Grossman, M.P.H. Manager ICF International 3 Corporate Square, NE, Suite 370 Atlanta, GA 30329 Tel: 952.303.6765 E-mail: liz.grossman@icfi.com Stacy F. Johnson, M.S.W. Senior Associate ICF International 3 Corporate Square, NE, Suite 370 Atlanta, GA 30329 Tel: 503.236.1626 E-mail: stacy.johnson@icfi.com