Parent Support Circles Los Angeles County Preliminary Evaluation Report Research & Evaluation Center Children s Institute, Inc. April 30, 2016
Table of Contents I. Program Summary 2 II. Program and Evaluation Background 3 III. Evaluation Goals and Purpose 4 IV. Evaluation Methods 5 V. Evaluation Questions and Preliminary Results 6 Chart A: Protective Factors Survey Results 8 Chart B: Satisfaction Survey Results 10 Chart C: Program Data Collection and Demographics Report 13 VI. Next Steps: Final Evaluation Reports 15
I. Program Summary Parent Support Circles of Los Angeles County (PSCLAC) is an innovative program to create clusters of parents of young children around the county who come together to share and learn. Funded by First 5 LA as part of their strategic planning efforts to help communities enhance protective factors, the program s goals are to increase social connections, knowledge of parenting, and knowledge of resources for participants. These voluntary, parent-led groups are based on the principle that people feel most supported in the company of others who share their experiences. Children s Institute, Inc. (CII) developed the program and oversees the work of the five community-based organizations (CBOs) who provide the support groups across LA County. Since the support groups are voluntary and designed for sharing, there are no specific curricula; instead, the groups are organized around topics of interest to communities. Based on interviews with parents around LA County, CII developed topic guides for the most requested subject matter. For example, self-care, nurturing infants and toddlers, children with special needs, and behavior challenges were themes that arose throughout the county. Each of the nine topics has guides available in English and Spanish that include information on various aspects of the topic, resources for further information, red flags, and tip sheets for distribution to participants. At the center of the model are the facilitators who are recruited by the CBOs. The CBOs look for natural helpers with lived experience as parents in communities of need. Facilitators attend an intensive 3-day training in the approach. Importantly, they also participate in regular reflective supervision sessions in which they can air their concerns and share their learnings. CBOs work with community partners to develop support groups and choose topics. The groups run for 8 to 12 weeks and include a minimum of five participants. The group facilitators and participants determine the frequency of the meetings and the length of the sessions, although most meet weekly for an hour to an hour and a half. Child care and light refreshments are provided. 2
II. Program and Evaluation Background In November 2010, the First 5 LA Commission approved funding for the Parent Support Circles Project, for a Lead Agency to work with five community-based organizations to establish 100 peer support groups for parents of children from the prenatal stage to age 5 in Los Angeles County communities where support, social cohesion, and connection to neighborhood peers are weak or lacking. The project aims to increase participants social connectedness; increase participants knowledge, confidence, and efficacy in parenting skills; and raise participants awareness and use of resources available to support them. In September 2011, the First 5 LA Commission approved additional funding for a Curriculum Developer/Training and Technical Assistance (TA) Provider, responsible for enhancing and modifying the implementation of the project by including the development of core competencies, training, and technical assistance. In 2013, Children s Institute, Inc. was selected as both the Lead Agency and Curriculum Developer/Training and TA Provider to implement the Parent Support Circles Project. Program evaluation responsibilities were added to CII s contract in October 2014; evaluation protocols were implemented in January 2015 across all peer support groups and have been supported by CII s Research and Evaluation Center (REC). Guided by evaluation consultant Dr. Bruce L. Baker, REC also manages data processing, analysis, and reporting. 3
III. Evaluation Goals and Purpose As described in the Evaluation Plan, the evaluation data are intended to guide program improvement and assess program impact on participants and facilitators; results are therefore shared by REC on a regular basis with CII management, CBO coordinators, group facilitators, First 5 LA collaborators, and other stakeholders. In order to support a data-informed program, participant satisfaction data collected at the midpoint session of groups have informed program improvement and led to programmatic changes for subsequent group sessions. Midpoint satisfaction data reports for individual groups are generated and sent to CBO coordinators within 7 days of the satisfaction surveys being submitted to REC; CBO coordinators can then review these results with each group s facilitators and make program improvements as needed. Examples of adjustments have included extending the duration of a group beyond the original 8 weeks, extending the length of group sessions, and improving facilitator communication skills and knowledge of resources. Additionally, process data are tracked and reported on a monthly basis in order to support the CBOs with participant recruitment and data collection goals. At monthly CBO meetings since January 2015, REC shares program-level reports with number of participants, number of completed and active groups, number of referrals, data collection totals and compliance, and participant demographics. The evaluation discussion at the monthly meetings has allowed staff to jointly address programming and evaluation challenges; for example, discussion of the ethnic breakdown among participants (predominantly Latino or Hispanic) reinforced CBO commitment to cultural sensitivity and inclusiveness in their recruitment efforts. CBO and REC staff also collaborated on finalizing the original satisfaction survey s wording (English and Spanish) in order to ensure participant comprehension. In order to assess the cumulative effects of the program and inform program improvement, pre- and post-data from the protective factors survey are analyzed and shared with CBOs and other stakeholders. Protective factors results in the first Preliminary Evaluation Report (October 2015) showed increases in Social Support and Knowledge of Parenting factors but a decrease in Concrete Support factors; these results helped to inform program improvement and reinforce the need to increase participant knowledge of community resources. Qualitative data collected via facilitator interviews will supplement participant outcome data in demonstrating program impact. Content analysis of facilitator interviews is in progress and will be included in the final PSCLAC Evaluation Report. 4
IV. Evaluation Methods Data collection protocols with the following outcome measures were implemented across all support groups, beginning in January 2015: i. Participant Protective Factors Survey 1 : The 15-item, self-report Protective Factors Survey (PFS) consists of four subscales: Social Support, Concrete Support, Nurturing and Attachment, and Knowledge of Parenting. At intake and the last session of the group, the PFS is administered to participants and collected by the facilitator or another CBO appointee, who may assist the participants with reading comprehension as necessary. Completed surveys are submitted to the CBO coordinators, who send them electronically to REC. REC staff enter participant responses into an internal data sheet; numerical identifiers are assigned to ensure participant confidentiality and track individual outcomes across all participant measures. ii. Participant Satisfaction Survey 2 : The 15-item, self-report Satisfaction Survey asks participants to rate group implementation, participation from group members, and group facilitators. The survey is administered at the midpoint session and the last session of each group. A CBO appointee (not the facilitator) may assist participants as necessary. The CBO appointee collects the completed surveys and sends them electronically to REC. REC staff enter participant responses in an internal data sheet; numerical identifiers are assigned to ensure participant confidentiality and track individual outcomes across all participant measures. iii. Facilitator Interviews: Based on a review of the literature, REC developed an interview protocol focusing on the major areas of impact on the facilitators from running groups; domains include personal enjoyment, professional development, and relationships with group members. As a sample, 12 facilitators were selected across several criteria (e.g., language, CBO affiliation, experience with facilitating at least two groups) and were interviewed in either English or Spanish based on their preference. Interviews have been translated, transcribed, and coded (via Dedoose coding software) in preparation for content analysis. CBO and REC staff enter process data (e.g., service delivery totals, data collection totals, and participant demographic information) into an online group tracking sheet, an online platform for participant registration, and REC s internal data sheets. 1 Developed by the FRIENDS National Resource Center for Community-Based Child Abuse Prevention, in partnership with the Center for the Study of Social Policy. 2 Developed by CII s Research and Evaluation Center and CBOs. 5
V. Evaluation Questions and Preliminary Results Participant Outcomes Results from the Protective Factors Survey (PFS) address the following evaluation questions: Does participants social connectedness increase? Is there statistically significant improvement between pre- and post-ratings for the Social Support items of the PFS? Does participants knowledge of parenting increase? Is there statistically significant improvement between pre- and post-ratings for the Nurturing and Attachment or Knowledge of Parenting items of the PFS? Does participants knowledge of community resources increase? Is there statistically significant improvement between pre- and post-ratings for the Concrete Support items of the PFS? Based on data collected between January 2015 and April 2016, preliminary results from the PFS (see Chart A, p. 8) are very encouraging. The PFS is organized into three composite scores: Social Support (three items), Concrete Support (three items), and Nurturing and Attachment (four items). In addition, there are five Knowledge of Parenting items that are considered separately. Items present a statement (e.g., When I am lonely there are several people I can talk to ) and ask for a response on the following scale: 1 (Strongly disagree), 2 (Mostly disagree), 3 (Slightly disagree), 4 (Neutral), 5 (Slightly agree), 6 (Mostly agree), or 7 (Strongly agree). Each subscale score ranges from 1 to 7and is calculated from the mean of the subscale items. At the pre assessment, many participants scored high on the composite scales, which would be expected, as PSCLAC is not a clinical program and participants were not referred in for existing problems. Despite this, there were meaningful and statistically significant pre- to post-group gains in Social Support, Nurturing and Attachment, and Knowledge of Parenting factors. Social Support is the domain most relevant to the PSCLAC group experience, and the gains here are gratifying. For all participants, the overall gains in the means of Social Support items (from 5.2 to 5.8) were highly statistically significant; t(180) = 4.20, p <.001. (A p-value below.001 indicates that something could happen by chance less than one time in a thousand.) The Social Support scores were examined another way; the change in means was considered for just those participants who initially expressed lower available support and for whom the groups could be of particular benefit. At pre, 36% of participants scored below the cutoff score of 5 (on average, below Slightly agree ); at post, only 17% scored below 5. For these participants who entered low in expressed social support, the gain following participation in the group (from 3.4 to 5.3) was highly significant; t(65) = 7.78, p <.001. 6
In the Nurturing and Attachment domain, the findings are also encouraging. For all participants there was a statistically significant gain from 6.1 to 6.4, despite relatively high scores to begin with and little room for change; t(172) = 2.66, p =.009. For those participants who initially scored below 5 (Slightly agree), there was a sizeable gain from 4.0 to 5.2; t(24) = 4.38, p <.001. The five items related to Knowledge of Parenting are not to be combined into a composite score but should be considered separately. On the two positive items ( I know how to help my child learn and I praise my child when he/she behaves well ), there were statistically significant increases when considering all participants (p <.001 and p =.042, respectively). The percentage of participants indicating Mostly agree or Strongly agree for these two items increased from 49% to 67% and from 66% to 75%, respectively. On the three negative items ( There are times when I don t know what to do as a parent, My child misbehaves just to upset me, and When I discipline my child I lose control ), participants as a whole did not change significantly. However, for the subgroup of participants who at least somewhat agreed with these negative items at the pre assessment, there were major decreases on all three items from the agree range to the disagree range (all with p <.001). Thus the support groups considerably affected Knowledge of Parenting factors for those participants who were most in need. In the Concrete Support domain, mean scores for all participants decreased slightly from 4.3 to 4.2 but not at a significant level; statistically speaking there was no change between pre and post assessments. The analysis for first Preliminary Evaluation Report showed a larger decrease between pre and post (from 4.6 to 4.1) for the Concrete Support items, indicating a potential need for the support groups to make programming adjustments and focus more on sharing community resources with participants. A final analysis of Concrete Support data will be conducted in June 2016 after all groups have finished and all PFS data has been collected. 7
Chart A. Protective Factors Survey Results PRE POST Matched Pairs = 181 5.2 5.8 6.1 6.4 5.1 5.7 5.3 5.2 5.9 6.2 5.6 5.7 4.3 4.2 4.0 4.3 Social Support Concrete Support Nurturing & Attachment Knowledge of Parenting - Know what to do as parent Knowledge of Parenting - Help child learn Knowledge of Parenting - Child misbehaves to upset me Knowledge of Parenting - Praise child for good behavior Knowledge of Parenting - Lose control when disciplining child 8
Results from the Participant Satisfaction Survey address the following evaluation questions: What is the participants level of satisfaction with the groups and the facilitators? Does the participants level of satisfaction improve between midpoint and postratings on the satisfaction survey? The satisfaction survey asks participants to express their agreement or disagreement with 15 statements across three categories: About the Group (e.g., number of meetings, length of group sessions), About Your Participation (e.g., information was valuable, would recommend group to others), and About the Facilitators (e.g., understood participants issues, knew information well). Response options for the first category are: 1 (Too small/short or not enough), 2 (Just right), and 3 (Too big/long/many/much). Response options for the second and third categories are: 1 (Definitely disagree), 2 (Disagree), 3 (Agree), and 4 (Definitely agree). As shown below (see Charts B1 and B2, p. 10-11), participants ratings at both midpoint and post assessment have been almost universally positive. Impressively, even with relatively high satisfaction levels to start with, there was statistically significant improvement in participation satisfaction between the midpoint and end of groups (from 44.3 to 45.1 out of a possible 50); t(172) = 2.03, p =.044. Participants completing the satisfaction survey at the last group session overwhelmingly endorsed the overall experience: 94% of participants agreed that they had changed their parenting as a result of the group, 98% of participants agreed that the facilitators cared about their issues and made it possible for them to talk about difficult topics, and 98% of participants agreed that they would recommend their group to others. On items about the group implementation (ratings 1-3), most participants responded with Just right to every statement. Exceptions were generally due to wanting more of the group experience, with 28% of participants indicating that there were not enough sessions and 19% indicating that sessions were too short. On items about the participation experience and the facilitators (ratings 1-4), over 94% of participants agreed or strongly agreed with every statement, indicating high overall satisfaction across various support groups, group topics, and facilitators. 9
Chart B1. Satisfaction Survey Results (by Means) MID POST About the Group Highest Rating = 2 Matched Pairs = 173 1.93 1.91 1.89 1.83 1.86 1.99 2.03 1.88 1.86 1.72 Group size Length of meetings Number of meetings Information presented Discussion time About Participation Highest Rating = 4 3.49 3.6 3.58 3.53 3.58 3.66 3.39 3.47 3.66 3.74 Respected by other members Had a chance to speak Information was valuable Changed my parenting Would recommend this group About the Facilitators Highest Rating = 4 3.55 3.65 3.62 3.69 3.55 3.62 3.58 3.68 3.56 3.66 Cared about our issues Made members comfortable Possible to talk about difficult topics Made sure everyone had chance to speak Knew how to find resources 10
Chart B2. Satisfaction Survey Results at Post (by Percentage) CBO All Groups 57 Participants 290 Collection Post Dates 1/1/15 to 4/1/16 Earliest Latest About the Group Too Small/Short or Not Enough Just Right Too Big/Long/Many/Much Group size 12 87 2 Length of meetings 17 83 Number of meetings 29 71 Information presented 3 91 6 Discussion time 16 82 2 About Participation Definitely Disagree Disagree Agree Definitely Agree Respected by other members 01 28 71 Had a chance to speak 1 31 68 Information was valuable 29 71 Changed my parenting 1 5 42 52 Would recommend this group 1 21 78 About the Facilitators Definitely Disagree Disagree Agree Definitely Agree Cared about our issues 2 29 69 Made members comfortable 1 26 73 Possible to talk about difficult topics 2 32 66 Everyone had a chance to speak 1 26 73 Knew how to find resources 2 30 69 11
Facilitator Outcomes Results from the facilitator interviews will address the following evaluation question: What are the facilitators perceived benefits and effects of facilitating groups (e.g., increasing personally-relevant knowledge, gaining social approval and social support, developing interpersonal competence, achieving professional growth, and acting as a community resource for others)? Content from 12 facilitator interviews has been transcribed, translated as needed, and coded across various domains including personal growth, social support, and professional growth. Feedback from facilitators appears to be generally positive with some suggestions for program improvement; further content analysis will be included in final evaluation report. Sample excerpts from interview transcripts: I ve loved being able to participate in it. I ve learned a lot. You start gaining confidence and that confidence makes you get more motivated, and it makes you get more involved with the community. I didn t think I had the confidence to be up there and to speak and to guide and to listen, which is one of the key skills I think I learned through the facilitating. Process Data Based on the April 2016 Program Data Collection Report and Participant Demographics Report (updated monthly since January 2015): i. Service delivery (see Charts C1 and C2, p. 13-14) Number of groups implemented: 71 completed, 13 active Number of group attendees (based on CBO estimates): 961 Number of referrals: 332 Number of participants with measures: 793 Number of matched pairs for measures: 205 for protective factors survey, 196 for satisfaction survey ii. Demographic information (see Chart C2, p. 14) Gender: 88% Female, 12% Male Age: Highest proportion (39%) of participants are in 30-39 range Ethnicity: Highest proportion (82%) of participants are Latino or Hispanic Individual CBO reports have been generated periodically to assist with boosting data collection compliance and share CBO-specific demographics and outcomes. 12
Chart C1. Program Data Collection Report Program Data Collection Progress Report January 2015 - April 2016 Data Collection Totals Matched Pairs 793 645 Protective Factors Survey PRE & POST 205 324 342 Satisfaction Survey MID & POST 196 TOTAL CLIENTS W/MEASURES PRE MID POST 71 COMPLETED GROUPS (2015-2016) 13 ACTIVE GROUPS (2015-2016) 961 CBO-ESTIMATED ATTENDEES (2015-2016) 82% COMPLIANCE RATE FOR MEAS. COMPLETION 13
Chart C2. Participant Demographics Report Participant Demographics January 2015 - April 2016 Gender 835 88% Age (Years) 372 39.2% 948 REGISTRANTS ENTERED IN WEBSITE 113 12% 26 2.7% 217 23.0% 172 18.1% 39 4.1% 20 2.1% Male Female 15-19 20-29 30-39 40-49 50-59 60-69 Black/Afro-Caribbean/African American East Asian/Asian American Ethnicity 23, 2.4% 57, 6.0% Divorced 16, 1.7% Marital Status Hawaiian/Pacific Islander 3, 0.3% Married 379, 40.0% Latino/Hispanic American 774, 81.6% Partnered 149, 15.7% Middle Eastern/Arab American Native American/Alaskan Native Non-Hispanic White/Euro-American 3, 0.3% 4, 0.4% 45, 4.7% Separated Single 40, 4.2% 155, 16.4% Other 21, 2.2% Widowed 7, 0.7% 56 INVOLVED WITH BEST START 19 INVOLVED WITH NEIGHBORHOOD ACTION COUNCIL 58 REFERRED BY HOSPITAL/CLINIC (WELCOME BABY) 274 REFERRED BY FRC/ COMMUNITY AGENCY 14
VI. Next Steps: Final Evaluation Reports In the last few months of PSCLAC s programming, REC will continue to track and report outcome and process data. The final Data Collection Reports and Participant Demographics Reports (program-level and CBO-level) will be shared in May 2016 at the last CBO meeting. Midpoint satisfaction reports will continue to be shared with individual groups and CBOs as needed. For the Final Evaluation Report due July 2016, REC will conduct a final analysis of all outcome data and matched pairs collected. With the largest possible sample of participant data, REC will examine additional variables (such as attendance) that could be associated with gains from the PSCLAC support groups. Attendance data is currently being processed and aggregated by evaluation consultant Bill Monro; preliminary results have already been shared at recent CBO meetings. As previously mentioned, REC will also conduct further content analysis of facilitator interviews across multiple domains in order to address evaluation questions about benefits to facilitating PSCLAC support groups. Final evaluation results will be shared with group facilitators, CBO coordinators, CII management, First 5 LA collaborators, and other various stakeholders. 15