MSI MEASURING COLLEGE VALUE WEBINAR

Transcription

1 MSI MEASURING COLLEGE VALUE WEBINAR Introducing Three Options for Tracking Workforce Outcomes May 22, 2017

2

3 Introducing Matt Soldner Matt, who is a Principal Researcher at AIR, has joined our team to support your work. He began his career at the University of Maryland, Baltimore County (UMBC) where he worked in Residence Life. After earning his Ph.D. at the University of Maryland, he went to work for the U.S. Department of Education s National Center for Education Statistics. At AIR, Matt works with 2-year and 4-year colleges as they redesign developmental education, reform advising processes, and implement next-generation digital courseware. You can reach him at: (202) or msoldner@air.org

4 Purpose of Today s Webinar (1/2) Using the Workforce Outcomes Tool The required Workforce Outcomes Tool is used to establish baseline information about the workforce outcomes of your recent graduates. In Year 1, it can be used: To test assumptions made during the planning phase; As a source of new learning to inform action plan adjustments; To build buy-in among colleagues for the activities of the grant; and To build buy-in among students for engagement in workforce preparation. After the timeline of the grant, it can be used to evaluate changes made during the grant period.

5 Purpose of Today s Webinar (2/2) Provide an Overview of Three Options for the Required Tool An option based on the Gallup-Purdue Index An option based on information about graduates earnings after college An option based on your own survey (or other) data about alumni

6

7 University of X Survey of Alumni Project Implementation Overview and Timelines November 1, 2016

8 Copyright Standards This document contains proprietary research, copyrighted materials and literary property of Gallup, Inc. It is for the guidance of your organization only and is not to be copied, quoted, published or divulged to others outside your organization. All of Gallup, Inc. s content is protected by copyright. Neither the client nor the participants shall copy, modify, resell, reuse or distribute the program materials beyond the scope of what is agreed upon in writing by Gallup, Inc. Any violation of this Agreement shall be considered a breach of contract and misuse of Gallup, Inc. s intellectual property. This document is of great value to Gallup, Inc. Accordingly, international and domestic laws and penalties guaranteeing patent, copyright, trademark and trade secret protection safeguard the ideas, concepts and recommendations related within this document. No changes may be made to this document without the express written permission of Gallup, Inc. Gallup are trademarks of Gallup, Inc. All rights reserved. All other trademarks and copyrights are property of their respective owners. 8 Copyright 2016 Gallup, Inc. All rights reserved.

9 Agenda INTRODUCTIONS Gallup and University GALLUP-PURDUE INDEX NATIONAL STUDY ALUMNI SURVEY Gallup s Research on Great Jobs and Great Lives Methodology of National Survey Creation of the National Database of Undergraduate Alumni Project Introduction Survey Design and Preparation Analysis Reporting Timeline Next Steps 9

10 Gallup-Purdue Index National Survey 10 GPI NATIONAL SURVEY University ALUMNI SURVEY

11 Defining a Good Life : Gallup s Global Well-Being Research Gallup has conducted decades of global research surveying a representative sample of 98% of the world s population from 150 countries. Well-being is associated with numerous positive health and employment outcomes: Physical Medical and health expenditures, ER visits, absenteeism, etc. Purpose Factor analysis identified five interrelated and interdependent elements of well-being: Purpose Liking what you do each day and being motivated to achieve goals. Social Having supportive relationships and love in your life. Financial Managing your economic life to reduce stress and increase security. Community Liking where you live, feeling safe and having pride in your community. Community Social Physical Having good health and enough energy to get things done daily. Financial 11 GPI NATIONAL SURVEY University ALUMNI SURVEY

12 Gallup s Employee Engagement Measures A Good Job Defined Gallup has measured the engagement of more than 1.3 million employees globally from hundreds of organizations and companies. Workplace engagement is an important driver of overall well-being, surpassing other domains of well-being in predictive value. Gallup s Q 12 instrument measures an individual s emotional connection and investment in their job. 12 GPI NATIONAL SURVEY University ALUMNI SURVEY

13 Definitions of Employee Engagement ENGAGED EMPLOYEES work with passion and feel a profound connection to their company. They drive innovation and move the organization forward. ACTIVELY DISENGAGED EMPLOYEES aren t just unhappy at work; they re busy acting out their unhappiness. Every day, these workers undermine what their engaged coworkers accomplish. About 30-34% of U.S. workers are engaged About 50% of U.S. workers are not engaged 15-20% are actively disengaged NOT ENGAGED EMPLOYEES are essentially checked out. They re sleepwalking through their workday, putting time but not energy or passion into their work. 13 GPI NATIONAL SURVEY University ALUMNI SURVEY

14 How Can College Promote Lifelong Well-Being and Engagement? The Gallup-Purdue Index: Examining the Outcomes of College Graduates Annual survey of representative sample of about 30,000 U.S. college graduates (who have obtained a bachelor s degree), measuring: Overall Well-Being (Five Domains) Workplace Engagement (Q 12 ) College Experiences Affinity and Attachment to Alma Mater Where you go to college matters far less than how you go to college. 14 GPI NATIONAL SURVEY University ALUMNI SURVEY

15 Methodology Results for the Gallup-Purdue Index Survey are based on: MODE SAMPLE COMPOSITION SURVEY Web surveys in English only A random sample of respondents with a bachelor s degree or higher, aged 18 and older, with Internet access, living in 50 states and DC Well-Being, Employee Engagement, Undergraduate Collegiate Experiences, Demo SAMPLE SIZE 2014: N = 29, : N = 31, : N = 11,483 FIELDING PERIOD RECRUITMENT WEIGHTING 2014: 2/4/14 3/7/ : 12/16/14 6/29/ : 8/22/16 10/11/16 Via the Gallup Panel (2014 study only) and Daily Tracking (RDD); 60/40 cell/landline Data are weighted to match national demographics of gender, age, race, Hispanic ethnicity, education, and region. Weighting targets are based on the most recent Current Population Survey figures for the U.S. population holding a bachelor's degree or higher aged 18 and older 15 GPI NATIONAL SURVEY University ALUMNI SURVEY

16 The Gallup-Purdue Index Great Jobs, Great Lives 2014 Report 2015 Report 2016 Report 16 GPI NATIONAL SURVEY University ALUMNI SURVEY

17 The Undergraduate Experience SUPPORT, EXPERIENTIAL AND DEEP LEARNING I had at least one professor at [UNIVERSITY NAME] who made me excited about learning. % Strongly Agree % Strongly Agree 63 My professors at [UNIVERSITY NAME] cared about me as a person. 27 I had a mentor who encouraged me to pursue my goals and dreams. 22 I worked on a project that took a semester or more to complete. 32 I had an internship or job that allowed me to apply what I was learning in the classroom. I was extremely active in extracurricular activities and organizations while attending [UNIVERSITY NAME] All three 14 All three 6 ALL SIX = 3% GPI NATIONAL SURVEY University ALUMNI SURVEY

18 Six Critical Experiences in Undergraduate Life Graduates who experienced the following GPI outcomes measures had increased odds of lifelong well-being and workplace engagement: Emotional Support Having a professor who cares about you Having a professor who makes you excited about learning Having a mentor who encourages your goals and dreams 1.7X Workplace Engagement 1.4X Well-Being 1.7X Workplace Engagement 1.4X Well-Being 1.9X Workplace Engagement 1.4X Well-Being Experiential Learning Having an internship or job that applies what you re learning in class Working on a project that takes a semester or more to complete Being extremely active in extracurricular activities and organizations 1.8X Workplace Engagement 1.3X Well-Being 1.7X Workplace Engagement 1.2X Well-Being 1.6X Workplace Engagement 1.2X Well-Being 18 GPI NATIONAL SURVEY University ALUMNI SURVEY

19 Critical College Experiences Prepare Students for Life After College Almost half (47%) of graduates had zero or one of the six experiences in college Only 3% experienced all six Among graduates who had none of the experiences, 5% strongly agree their university prepared them well for life outside of college 85% of graduates who had all six experiences strongly agree their university prepared them well for life outside of college Prepared For Life Outside of College 19 GPI NATIONAL SURVEY University ALUMNI SURVEY

20 University Alumni Survey 20 GPI NATIONAL SURVEY University ALUMNI SURVEY

21 Alumni Survey Survey Overview Key Areas of Focus MODE LANGUAGE DURATION SURVEY LENGTH CONTENT SAMPLE SUPPORT Web-based English only 4-week field time Approximately 12 minutes GPI core questions for comparison to the national and up to 10 custom, close-ended questions. Invite up to 30,000 undergraduate alumni for which the university has valid address Gallup Support Desk available for those who have trouble accessing survey Overall Well-Being (Five Domains) Workplace Engagement College Experiences Affinity and Attachment to Alma Mater 21 GPI NATIONAL SURVEY University ALUMNI SURVEY

22 University Information Form Client Deliverable PURPOSE WHAT WILL BE REQUESTED Provide Gallup team with key information 1. The university name as it should appear on the survey 2. The contact information of individuals from the university who should have access to the secure server 3. The contact information of individuals from the university who should receive daily participation reports 4. The contact information of individuals from the university who will serve as the principal investigator for study on campus should alumni have questions for the school about the study 5. Documentation key information that may impact select core questions 6. * Documentation of the university s plans to seek internal IRB submission and approval 7. * The university s coverage for eligible alumni 22 GPI NATIONAL SURVEY University ALUMNI SURVEY

23 Institutional Review Board (IRB) Client Deliverable Part of the University Information Form Gallup will seek IRB approval for the survey Core undergraduate survey has been approved by Gallup IRB Any custom items, additional data appends or other changes are subject to Gallup s IRB approval Gallup s IRB can be the IRB of record for your university, or you can seek internal IRB approval. Please make note of your preferred method on the University Information Form. 23 GPI NATIONAL SURVEY University ALUMNI SURVEY

24 Who to bring into the study Client Deliverable Part of the University Information Form ELIGIBILITY Received an undergraduate degree from University Have a valid address on file with University WHO IS SELECTED? Have a valid address on file with University Year of Graduation from # Living # Records Coverage University Alumni 1949 or earlier % ,031 1,046 35% ,103 3,739 37% ,958 7,735 35% ,632 8,734 39% ,274 10,943 42% ,055 14,135 54% ,281 13,999 81% 24 GPI NATIONAL SURVEY University ALUMNI SURVEY

25 Sample list Client Deliverable University will provide alumni records with unique ID, first name, last name, address, degree type and graduation year required Unique ID* First Name Last Name Address* Graduation Year* Degree Type Sam Johnson 1993 Bachelor s University Major (optional) Industrial Engineering University Donor (optional) No Terry Larson terry_larson@domain.com 1977 Bachelor s Environmental Horticulture $15,000 University can provide additional variables in the list: Optional additional variables include administrative data (alumni assoc. membership, financial giving, scholarship recipient, etc.) Will determine once we determine exact research objectives All records should be transferred via Gallup s secure site Please review Sample Submission Checklist 25 GPI NATIONAL SURVEY University ALUMNI SURVEY

26 Survey Schedule University pre-note to alumni Reminder 1 University reminder Close survey (EOD reminder 5) WEEK 0 WEEK 1 WEEK 2 WEEK 3 WEEK 4 Gallup invitation Reminder 2 Reminder 4 Reminder 5 Reminder 3 Client Tasks Gallup Tasks 26 GPI NATIONAL SURVEY University ALUMNI SURVEY

27 Alumni Survey Communications Client Deliverable University Initial Communication to Alumni University will send initial communication to each alumni included in the sample file Gallup will provide an example Gallup will review proposed text After is sent, University will be responsible for sharing the list of bounce back addresses with Gallup before the survey invitations can be sent University will identify the name of an individual and /phone contact that alumni can contact with questions Advanced notice to this office/person is necessary 27 GPI NATIONAL SURVEY University ALUMNI SURVEY

28 Alumni Survey Communications Gallup will send invitations and reminders 1 Gallup invitation 5 Gallup reminders to non-respondents Participation reports available on a daily basis University will inform Gallup who should receive these University additional reminder (optional) Communication to all alumni thanking those who have participated and encouraging nonrespondents to participate Gallup will provide example Gallup will review proposed text 28 GPI NATIONAL SURVEY University ALUMNI SURVEY

29 Analysis Gallup will analyze University s results. We will also compare University s results to the GPI National Database a nationally representative survey of bachelor s degree holders conducted by Gallup and Purdue We will compare University s alumni outcomes to our GPI National Database University alumni survey responses, as part of this study, are never added to the GPI National Database GPI National Database University Survey Results This database is compiled from the national, random sample conducted by Gallup and Purdue. This serves as a benchmark for University alumni. The results from University alumni will serve to inform this specific research effort. They will not be merged with the national database following this examination. 29 GPI NATIONAL SURVEY University ALUMNI SURVEY

30 Analysis GPI National Database Comparison Groups University has the ability to look at choose who their alumni are compared to for analysis GPI National Database Two Custom Comparison Groups* Carnegie classification Includes graduates from: Public non-profits Private non-profits Private for-profits 2014: N = 29, : N = 31,121 Custom group of peer or aspirational institutions NCAA Division Consortium or association *NOTE: smaller groups will limit Gallup s ability to analyze the data due to smaller sample size. 30 GPI NATIONAL SURVEY University ALUMNI SURVEY

31 Reporting Alumni Report Up to 8-page scorecard report Outcomes and Comparisons Provided Employee Engagement Well-Being Alumni attachment Undergraduate experiences Custom measures 31 GPI NATIONAL SURVEY University SURVEY

32 Three Options for the Workforce Outcomes Index Tool Matt Soldner, Principal Researcher Copyright 2017 American Institutes for Research. All rights reserved.

33 The Value of Exploring Workforce Outcomes We need to equip our students with the skills and resources they need to succeed after graduation, so understanding the experience of recent graduates is critical. Currently enrolled students who see the workforce relevance of their program of study are more likely to be engaged and ultimately complete than their peers who do not. Empirical evidence of recent graduates workforce outcomes can be a powerful motivator for change! 33

34 What Tools Are Available for the Project? We would like you to consider three options, based on your institutional needs and circumstances. 1. A tool based on graduates responses to the GPI; 2. A tool based on graduates wage outcomes; and 3. A tool based on your institution s own data, like an alumni survey. 34

35 The GPI-Based Tool As Brandon explained, the GPI is a robust survey of graduates experiences. The GPI Workforce Outcomes tool helps you focus on a subset of those questions for recent graduates, and conduct comparisons between groups of students that are the focus of this work. For example, you might compare students in one or more groups you ve targeted for interventions versus all other students. 35

36 What Types of Questions Can the GPI Tool Answer? How long does it take for graduates to find jobs? How helpful are campus career services? Are graduates jobs related to their undergraduate major? How much are graduates working, and how much are they earning? Are graduates satisfied with their employment, and do they feel they can do their best? Are graduates satisfied with their undergraduate experience? Did graduates feel excited about learning, and did they feel cared for by faculty? Are students feeling financially secure? Are students living their best possible lives? 36

37 What Should I Be Considering When Thinking About the GPI-Based Tool? Possible pros: It is comprehensive. Robust measure across many dimensions. It includes even more data than used in the tool. Institution benefits by participating in the larger GPI the tool is an added benefit. Possible cons: It works better for larger programs and colleges. Student groups would ideally include 100 or more responding graduates for highest quality data. It is an additional financial investment. Benefits must be considered relative to costs of participation. 37

38 The Wage Outcomes-Based Tool Compares wage outcomes between two or more student groups based on their program of study. Focuses exclusively on wages, and contrasts those wages to the benchmark of your choice. Options include: State s median wage for BA holders; or Institution-based median wage measure. Importantly, requires state s participation in a project like LaunchMyCareer. 38

39 Not Familiar With Launch My Career? Currently active in Colorado, Tennessee, Texas, and (soon) Florida. launchmycareercolorado.org launchmycareertn.org launchmycareertx.org Details wages 1 year, 5 years, and 10 years after graduation by program within institutions. 39

40 Helps Users Understand Graduates Wage Outcomes Primary purpose is to help students think about the payoff to specific degrees. Importantly for our work, LMC also gives us a window into wage outcomes for up to 20 years after graduation. We use those wages as the inputs into the Workforce Outcome Tool. 40

41 What Should I Be Considering When Thinking About the Wage-Based Tool? Possible pros: Relevance. Focuses on an outcome that is particularly important to many graduates and policymakers. Ease of use. Easily understood data that are easy to access. Accuracy and objectivity. Data are not selfreported, so accurately represent what is known about graduates wage outcomes. Possible cons: Not available everywhere. Data are currently available in Colorado, Tennessee, Texas, and Florida. AIR may be able to assist in accessing data in other states, if we have worked with them in the past. More narrow focus. Does not present a picture of graduate outcomes that is as robust as the GPI-based tool. 41

42 What If I Might Want to Use My Existing Alumni Survey? If you have high-quality alumni survey data, and neither the GPI-based nor wage-based options fit your institution s needs, AIR is glad to create a customized tool to leverage it! 42

43 Questions About These Options? We know we ve given you a lot of information hopefully, it has helped you begin to think about next steps. Please reach out as questions arise! We love to talk data. Matt Soldner msoldner@air.org 43

44 Next Steps in Action Plan Identifying Which Tool You Plan to Use In your Year 1 Action Plan, you should identify which of the three options you intend to pursue: A GPI-based tool A wage-based tool A tool based on your own alumni (or other) data You don t have to make this decision alone! Matt is available to help you talk through which data might be most appropriate for your circumstances. Regardless of your choice, AIR will build an Excel-based tool that fits your institution s data, needs, and context.

45 Concluding Thoughts What other comments, questions, or concerns do you have? What other items should we discuss? Remember the following are due JUNE 19, 2017: Workforce tool decision (communicated with your Action Plan submission) Action Plan Proposed budget