United Way of Lancaster County Collective Impact Initiative. Year Two Evaluation Report

Transcription

1 United Way of Lancaster County Collective Impact Initiative August 2017

2 Table of Contents Executive Summary... 1 Overall Evaluation Processes from Year Two Collaboration with CI Directors... 3 Evaluation Tools... 3 Logic Model Refinement... 3 Data Collection Plan Design... 4 Presentation at Cohort Meeting... 5 Technical Assistance... 5 Collaborative Assessment Tool... 5 End-of-Year Evaluation Reporting... 6 Evaluation Outcomes Community Indicators Update... 7 Education... 8 Finance... 8 Health... 8 Collaborative Assessment Tool... 9 Breakdown of Results by Section: Changes from Fall to Spring... 9 Mutually Reinforcing Activities... 9 Continuous Communication Common Agenda Backbone Organization Network Analysis Appendices Appendix A: Community Indicators Update: Expanded Results... A1 Collaborative Assessment Tool: Expanded Results... A3

3 Executive Summary The at Franklin & Marshall College serves as external evaluator for the United Way of Lancaster County s Collective Impact (CI) initiative. During the second year of CI activities, the evaluation team continued to work closely with the CI Directors to refine and use the evaluation methodology to monitor impact. The evaluation team provided the following services during the second year of CI evaluation: Regular, ongoing collaboration with CI Directors for macro-level and partnership-level planning, monitoring, and evaluation Implementation and refinement of evaluation tools, including logic models, collaborative assessment surveys, evaluation dashboard, end-of-year evaluation reporting, and community indicators data Development of data collection plans for each partnership Logic model design and monitoring for each individual partnership Distribution and analysis of the fall and spring collaborative assessment survey Presentations at cohort meetings on evaluation topics Technical assistance, when requested and deemed appropriate, to individual partnerships on topics including survey design and self-sufficiency matrix training Ratings and recommendations delivered to CI Directors after review of each partnership s end-of-year reporting A few of the overall findings from the evaluation activities include: Based on results of the collaborative assessment tool at the end of year two, partnerships reported more positive ratings on all four main CI constructs measured: mutually reinforcing activities, continuous communication, common agenda, and support from backbone organization. Analysis of network connections, as reported in the collaborative assessment survey, reveals that the connectivity of community organizations within the CI framework has improved greatly during the first two years of CI work. Collective Impact outcomes and highlights: Bold Goal # 1 kindergarten readiness SAIL After receiving SAIL services, home-based child care providers increased their FCCERS scores an average of 2-3 points, making the average composite score a 5.05 (out of 7), which translates to a STAR 2 or STAR 3 childcare facility. FCCERS stands for Family Child Care Environmental Rating Scale and is used to assess components of quality such as health and safety, age-appropriate learning activities, program structure, and family engagement. Increased quality predicts better school readiness outcomes for children. Pinwheel Partners 87% of children (162 out of 186) scored on-target or above on the standardized Ages and Stages assessment, translating into school readiness. 100% of families (56 out of 56) reported an increased knowledge of positive parenting after participating in a series of family workshops. Plant the Seed of Learning 9 out of 16 school districts (Cocalico, Ephrata, Conestoga Valley, Elanco, Lampeter-Strasburg, SDoL, Manheim Township, Pequea Valley, and Warwick) are facilitating parenting sessions with LGH/Penn Medicine or WellSpan and early learning professionals. 71% of families surveyed report increased purposeful play/interactions and reading (at least five times per day) with their child after attending PTSL sessions. 1

4 Bold Goal #2 post-secondary credential attainment Integration Services for New Americans reduced the wait time for ESL classes for Lancaster County refugees and immigrants from 31 days to 7 days; improved basic English proficiency (using National Reporting Systems Standards) from 55% to 61%. Impacting Generations -100% (27/27)of the teen parents served by Impacting Generations graduated from high school - and are now working to support the young parents towards post-secondary goals Together Initiative Network (TIN) TIN found that many of their clients need remedial interventions before they are ready to take the GED for example, if a client is reading on a third-grade level. There are also preliminary steps / goals that are being tracked which create a pathway to a post-secondary credential. Their numbers are small due to this intensive work one adult learner being served earned her GED and four adult learners have increased their preparedness (which is measured after every 50 hours of class time) to take the GED test. Pathways Out of Poverty - Three leaders in the Pathways Out of Poverty partnership have enrolled in post-secondary programs. Moving Forward - Sixteen survivors of domestic violence are actively pursuing education and/or employment, and three participants have earned a post-secondary credential. Bold Goal #3 poverty reduction Lancaster County Coalition to End Homelessness Twenty-two clients being served by the Lancaster County Coalition to End Homelessness have increased their earned income from program entry to exit. 97% (310/325) of individuals placed in permanent housing do not return to homelessness within the next seven months. Elizabethtown Area HUB working to connect social service providers in the Elizabethtown area. 46 out of 50 agencies are now collaborating to provide streamlined services to residents. ECHOS (Elizabethtown Community Homelessness Outreach Services) is a direct result of the Etown HUB identifying and collaborating to meet a community need. Collective Impact the collective impact model that proved that framework under which United Way Partnerships must work has been identified as the best practice guiding the work of the Coalition to Combat Poverty. Bold Goal #4 connection to a medical home Let s Talk Lancaster improving access and treatment for behavioral health care needs, especially for low-income and minority residents As a result of this Partnership s efforts to increase early, consistent, and standardized screenings for depression, 113 medical professionals (LGH, Welsh Mountain, Water Street Ministries, and South East Lancaster Health Services) have been trained and are using the PHQ-9 (The PHQ-9 is a multipurpose instrument for screening, diagnosing, monitoring, and measuring the severity of depression). The result is earlier detection and referral to treatment. 211 has been an instrumental part of Let s Talk, Lancaster s efforts around increasing access to mental health services by ensuring comprehensive and complete referral information for behavioral health providers. Lancaster Medical Legal Partnership advancing the financial stability and health of Lancaster County by integrating civil legal aid and financial case management with an intensive, high-utilization health care treatment model The Lancaster Medical Legal Partnership reports that, for reporting year 7/1/16 to 3/31/17, after individuals received services from LMLP, the number Emergency Room visits and inpatient utilization dropped from 602 to 350. This resulted in a cost-savings of over $46,000,000. The first section of this report describes the evaluation activities undertaken in year two of the CI initiative. The second section of this report describes the results and findings from these various evaluation activities. 2

5 1) Collaboration with CI Directors Overall Evaluation Processes from Year Two The evaluation team worked with the CI Directors to develop a process for monitoring partnership activities, and our meeting structures and evaluation activities reflected this development. The evaluation team and CI Directors met in person on a monthly basis. The meetings followed an agenda which allowed for partnership updates, planning for upcoming cohort meetings, discussing outcomes and data measurement, and any other items as needed. Monthly meetings were supplemented by individual phone calls between the applicable CI Director and the evaluation team, as needed, to debrief on recently held individual partnership meetings. In addition, the team and CI Directors used a secure Google drive to easily share files and collaborate between meetings. 2) Evaluation Tools Below is a list of the primary tools used for evaluation. Logic Model: See Section 3 Data Collection Plan: See Section 4 Collaborative Assessment Tool: See Section 7 Dashboard: In year one, the evaluation team collaborated with the CI Directors to develop a dashboard to rate the attributes that best define a successful, synergistic, and thriving partnership. This dashboard was further refined in year two to reflect the attributes expected to be achieved during the second year, including progress toward measuring and achieving short-term outcomes. Dashboard ratings were based on conversations among CI Directors, the evaluation team, and partnership self-evaluation. End-of-Year Reporting: See Section 8 Community Indicators: The evaluation team compiled County-level data on the major indicators related to the Bold Goals. Data are provided in the results section of this report. 3) Logic Model Refinement The evaluation team had individual, 90-minute meetings with each partnership and their CI Director in July and August The purpose of the meetings was to discuss the progress made and to adapt the logic model as needed based on their learnings from year one. The evaluation team led the discussion using the following set of questions: For year two and moving forward, will your Anticipated Impacts remain the same, or have they changed? Has your Target Population stayed the same, or has it changed? We are going to discuss the action steps from your original logic model, including whether they have been completed and how to build on the momentum and continue the work into year two. o Review each Activity Has the activity been successfully completed (or is it an ongoing activity)? If no, what barriers have prevented successful completion? Does the activity s completion (or revision) relate to any other parts of the logic model? If so, what are those changes? Do any new activities need to be added? o Review each Output Do any outputs need to be revised, added, and/or removed? o Review each Short-Term Outcome Do any outcomes need to be revised, added, and/or removed? Will each outcome be measurable? If so, will data be available THIS YEAR? 3

6 o What must happen next to move toward the desired outcomes? (Add to the model) Summarize the direction and focus for year two as realized by revised logic model At the end of the meeting, the partnership established their year two priorities and activities in the context of their logic model discussion. Following the meeting, the evaluation team delivered the refined logic model to each group, via the CI Director, with the understanding that the partnership would adapt it as needed during the second year. 4) Data Collection Plan Design Beginning in August 2016, the evaluation team consulted with each partnership via their CI Director to develop a data collection plan. This entailed the partnerships identifying the following items for each of their short-term outcomes: the measurement tool used (e.g., standardized Kindergarten screener), the collection method (e.g., institutional data from school districts), the collection schedule (e.g., annually in the fall), and the person(s) responsible for collecting the data. Once the data collection plan was reviewed and finalized by the evaluation team, partnerships were charged with using their plan to collect and report data for year two. Figure 1 below shows the instructional template given to partnerships to use in creating their data collection plans. Figure 1. Data Collection Plan Template and Instructions 4

7 5) Presentation at Cohort Meeting On September 14, 2016, the evaluation team presented to the cohorts an overview of the findings from the first year of CI evaluation activities, with a focus on the collaborative mapping and community indicators. 6) Technical Assistance The evaluation team was invited to provide technical assistance when deemed appropriate by the CI Directors. The following list highlights these activities. Within the CaseWorthy information system being piloted by many CI partnerships is the self-sufficiency matrix, a tool which allows the tracking of individual level needs, goals, and progress. The evaluation team provided technical assistance to the United Way and Community Action Partnership by collaborating on the development of a training manual. The training manual is intended for providers administering the selfsufficiency matrix to their clients, and includes a script questionnaire to guide the provider through the process of collecting and tracking client outcomes in a standardized manner. The evaluation team helped develop a questionnaire for Pathways out of Poverty to measure the relationship between Leaders and their mentors. The team helped design a questionnaire to measure SAIL participants trust of providers. The team programmed the survey in the online platform, Qualtrics, which allowed the team to house and output the data for the partnership to use. The team helped refine a questionnaire to measure behavior change among Plant the Seed of Learning workshop participants; this survey was programmed in Qualtrics, and data was then output for the partnership to use. 7) Collaborative Assessment Tool In October 2016 and April 2017, the evaluation team administered an online survey of members of all CI partnerships. The purpose of this survey was to gather information directly from individuals, in a confidential context, related to the four key elements of collective impact: common agenda, mutually reinforcing activities, continuous communication, and backbone organization support. After the spring survey, the evaluation team delivered and discussed changes at the partnership level from fall to spring. After each survey, the evaluation team summarized results overall and at the partnership level, and delivered these results to the CI Directors. In addition, the team delivered a network map that displayed the connections between organizations in the spring. The figures display the connections between the partnership organizations based on results from the survey question: Which other organization(s) in your partnership do you go to for help and advice as you work on this project? 5

8 8) End-of-Year Evaluation Reporting In collaboration with the CI Directors, the evaluation team developed a framework for each partnership to provide their end-of-year reporting. The framework was delivered from the CI Directors to each partnership. The framework was: Logic Model/Data Collection Plan Due March 31, 2017 Provide current data collection plan, adding columns to report annual performance data o Baseline data, at a minimum, should be reported. Use the column titled Reporting Period 1 Baseline and fill in the period you are reporting on (e.g., Jan 1 Dec 31, 2015) o Additional data beyond baseline should be reported, if available Use the column titled Reporting Period 2 and fill in the period you are reporting on (e.g., Jan 1 Dec 31, 2016) Accountability Report Due April 21, 2017 One to two page narrative that covers the following: o Did we implement our work as planned and authorized? Describe your progress toward achieving the specific Bold Goals you proposed. Speak to your partnerships inputs, activities, processes, outputs, and quality standards you have used. o To what extent are we continuing to build strong, positive relationships with key stakeholders and partners? Speak to the ways partners are working together as a result of using the Collective Impact Model. o What were our major successes and challenges in year two? o What have we learned, and in what ways are we applying what we have learned to improve effectiveness? o To what extent and in what ways are we attaining desired and intended program outcomes and impacts? Speak to the short-term outcomes your partnership identified and what the data tells you about attaining these outcomes. o What plans do we have for year three? How will what we learned from the data impact our management, programming, and/or service delivery in year three? Budgetary summary and up-to-date compliance documents Due April 21, 2017 o Year two, quarter three budgetary summary o Year two, end-of-year projected budgetary summary o Year three proposed budget including remaining prior year funds o Current compliance documents: 501(c)3 determination letter W-9 Bureau of Charitable Organizations Certificate (if applicable) Financial audit and review Each partnership delivered their reporting via e-cimpact. The evaluation team reviewed all materials, summarized results, and assigned ratings for each partnership. The team then met with the CI Directors to discuss the results and assign final ratings to each partnership for the dashboard. A final summary was developed and then delivered to the CI Committee. 6

9 Evaluation Outcomes Community Indicators Update Trends in Selected Indicator Data for Lancaster County and Pennsylvania United Way and the evaluation team selected 12 indicators to assess changes in Lancaster County related to the United Way s Bold Goals. Table 1 and Figure 2 (below) show changes in these indicators since the start of the United Way CI initiative in The selected indicators fit into three categories: education, finance (economy), and health. Lancaster s rank among the 67 counties in the state is shown below each indicator in italics. Since 2014, Lancaster has made positive changes in rank in 3rd grade PSSA scores, monthly homeowner costs greater than 30% of income in units with a mortgage, poverty rates, and the ratio of dentists and mental health care providers. Lancaster s rank has declined in rates of children enrolled in high quality, publicallyfunded Pre-K, educational attainment, gross rent greater than 30% of income, poverty among children, uninsured individuals, and the ratio of primary care physicians. Table A-1 in Appendix A shows comparison data from Pennsylvania and Lancaster as well as Lancaster s rank among the 67 counties in the state. Table 1. Trends in Selected Indicators for United Way s Bold Goals and Impacts 7

10 Figure 2. Trends in Selected Indicators for United Way s Bold Goals and Impacts Percent of PA Counties Performing Better Education Lancaster has made positive changes to its relative ranking on 3 rd grade PSSA reading and math scores, and has gone down in ranking in every other education category. Note that although the math PSSA scores have declined in Lancaster County, math PSSA scores have also decreased statewide due to changes made in 2015 to the scoring of the PSSA. Educational attainment of individuals aged 25+ with a bachelor s degree or higher and rates of children enrolled in highquality publically-funded Pre-K have remained virtually static, but have gone down in ranking compared to other counties in the state. Finance Monthly costs exceeding 30% of household income for homeowners (with a mortgage) have gotten slightly better in Lancaster County, and although the poverty rate in Lancaster has improved in ranking compared to the rest of the state, the rates have gotten slightly worse. Gross rent greater than 30% of income and poverty among children have gotten worse in both rates and ranking in Lancaster County compared to the rest of the state. Health In Lancaster County, rates of uninsured individuals have slightly decreased since 2014, but is far behind the rest of the counties in the state virtually the worst performing county (ranked 64 out of 67 counties in PA). Rates of dentists and mental health care providers have slightly improved, while rates of primary care physicians have decreased since

11 Collaborative Assessment Tool The figures below show the results of the collaborative assessment tool, which was first administered in the fall of 2015, with subsequent assessments given in spring of 2016, fall of 2016, and in the spring of Significant differences between scores are noted with an asterisk. Table A-2 in Appendix A shows the mean scores for each question as well as the change from fall to spring. Breakdown of Results by Section Changes from Fall 2015 to Spring 2017 Mutually Reinforcing Activities This section shows the largest positive change from fall 2015 to spring 2017, specifically in areas such as clarity of strategies to carry out the goals and objectives of the partnership, systems for conflict resolution, the assignment of partner roles and responsibilities according to strengths, the flexibility of changing policies and regulations when necessary to increase the effectiveness of the partnership, and shared decision-making (Figure 3). Figure 3. Mutually Reinforcing Activities 9

12 Continuous Communication This section also shows the next largest positive change from fall 2015 to spring 2017, specifically in areas such as frequency of formal communication among partners and communication promoting cooperation among partners (Figure 4). Figure 4. Continuous Communication 10

13 Common Agenda The Common Agenda section showed virtually identical positive change as the previous section, specifically in areas such as members understanding the role of other members within the partnership as well as trust between partnerships and feelings of ownership in the operation of the partnership. The only negative change was in the view that members bring unique skills to address the partnership s needs (Figure 5). Figure 5. Common Agenda 11

14 Backbone Organization This section showed the smallest increase from fall 2015 to spring The most positive changes were in areas such as addressing conflicts when they arise, adequacy of in-kind support, the view that United Way is working to mobilize funding for the partnership, and the view that United Way establishes shared measurement practices for the partnership. Scores have decreased or remained virtually static in areas such as the view that United Way is responsive to the needs of the organization and the timeliness of communication with United Way (Figure 6). Figure 6. Backbone Organization 12

15 Network Analysis The initiative has been purposely designed to connect the resources and expertise of diverse community organizations to more effectively address community problems as identified by the four bold goals. This requires the development of an efficient network structure that facilitates communication and cooperation between organizations, ultimately creating a more effective and efficient way to reach their shared goals. The United Way CI process has significantly improved the connectivity of the community organizations within the network during its first two years. One of the most basic characteristics of any network is its density, which is the proportion of observed ties in a network to the maximum number of possible ties. High scores show a network that is more interconnected. The network density scores rose from.087 in fall 2015 to.154 in spring 2016 to.176 in spring 2017, which is a clear indication that the network of community organizations has grown more connections during the course of the year. Network centrality measures also indicate positive change. Centrality can be thought of as the position of a node (in this case each organization) within a network. Degree centrality represents the number of ties an organization has to the other organizations in the network. The average degree or number of ties of those organizations in the network rose from 17.5 (fall 2015) to 26.8 (spring 2016) to 30.9 (spring 2017). Another indicator of the network s changing inter-relationships can be seen in the number of relatively independent subgroups that exist within the network, which might be thought of as network communities. A network subgroup or community is described as a set of organizations that have more internal ties and fewer ties to other parts of the network. In fall 2015, we detected approximately 22 different communities based on their inter-relationships, indicating a relatively fragmented network. By spring 2016 we identified only 8 different network communities and by spring 2017 there were 7 different network communities. Figures 5 and 6 on the following pages display the connections between organizations participating in the United Way s Community Impact Initiative. Each organization is represented by a circle and each line represents a connection between organizations. The size of the circle indicates the number of close connections an organization has the larger the size, the more close connections the organization has. Network communities are identified by color and signify groups of organizations that are more frequently and closely connected to one another than they are to organizations in other parts of the network. Organizations that are more interconnected appear closer to the center of the figure and those that are less interconnected appear closer to the periphery. Data comes from the self-administered collaborative assessment survey of all community impact partners responding to the following question: Which other organization(s) in your partnership do you go to for help and advice as you work on this project? 13

16 Figure 7. Network Analysis, Fall

17 Figure 8. Network Analysis, Spring

18 Appendix A Appendix A Community Indicators Update Trends in Selected Indicator Data for Lancaster County and Pennsylvania United Way and the evaluation team selected 12 indicators to assess changes in Lancaster County related to the United Way s Bold Goals. The table below shows comparison data from Pennsylvania and Lancaster as well as Lancaster s rank among the 67 counties in the state. The selected indicators fit into three categories: education, finance (economy), and health. Lancaster s rank among the 67 counties in the state is shown on the far right of the table. Since 2014, Lancaster has made positive changes in rank in 3rd grade PSSA scores, selected monthly homeowner costs greater than 30% of income in units with a mortgage, poverty rates, and the ratio of dentists and mental health care providers. Lancaster has gone down in rank in rates of children enrolled in high quality, publically-funded Pre-K, educational attainment, gross rent greater than 30% of income, poverty among children, uninsured individuals, and the ratio of primary care physicians. 1

19 Appendix A Table A-1. Trends in Selected Indicators for United Way s Bold Goals and Impacts 2

20 Appendix A Collaborative Assessment Tool The tables below show the results of the collaborative assessment tool, which was first administered in the fall of 2015, with subsequent assessments given in spring of 2016, fall of 2016, and in the spring of Significant differences between scores are noted with an asterisk. The mean scores for each question as well as the change from fall to spring are shown in Table A-2. All but one item (5. Partnership members bring unique skills to address the partnership s needs) in all three sections (Common Agenda, Mutually Reinforcing Activities, Continuous Communication, and Backbone Organization) show positive changes from fall 2015 to spring The biggest improvements were made in the Mutually Reinforcing Activities section (mean total change=0.33) and the Common Agenda section (mean total change=0.26). The biggest improvements in individual items were 13. There is a system in place for resolving conflicts between the demands of partnering organizations and demands of this partnership (+0.64); 4. Partnership members understand the roles of all members (+0.49); and 11. Strategies to carry out the goals and objectives of this partnership are clear (+0.45). Table A-3 shows the percentage of responses in the 1-3 category (not at all, rarely, sometimes) and in the 4-5 category (often, always). 3

21 Appendix A Table A-2. Collaborative Assessment Tool Results, Mean Aggregate Changes 4

22 Appendix A Table A-3. Collaborative Assessment Tool Results 5