An Integra*ve, Theory-Based, Mul*-Faceted Paradigm for STEM Program Evalua*on: Lessons from the Meyerhoff Scholars Program Kenneth I. Maton Department of Psychology, UMBC Presented at the University of Michigan, Ann Arbor, MI November 29, 2016
Meyerhoff Research: Integra*ve, Theory-Based, Mul*-faceted, Outcome Research Theory-based: Mechanisms of Program Influence Program Elements: Program Theory & Measure Development QualitaPve Inquiry & FormaPve Assessment DocumenPng Systemic and RadiaPng Influence
Meyerhoff Program Components 1. Financial Aid 2. Summer Bridge 3. Program Values 4. Advising/Counseling 5. On-campus & Summer Research Internships 6. Faculty involvement 7. Recruitment 8. Study Groups 9. Program Community 10. Tutoring 11. Administra*ve Support 12. Mentors 13. Open communica*on with Families
Outcome Research Examine the effect of MSP versus Declined Comparison Sample Previous Findings Undergraduate GPA, RetenPon Entry into STEM PhD Program Recent Findings: STEM PhD ComplePon
Previous Findings Compared to students who declined the Meyerhoff scholarship offer (comparison group), African American Meyerhoffs are: ü 2 #mes more likely to graduate with a STEM bachelor s degree ü 5 #mes more likely to pursue a STEM PhD
STEM PhD ComplePon Study: ParPcipants * * * **
STEM PhD ComplePon: Findings African American Meyerhoffs were 4.8 *mes more likely to complete a STEM PhD than comparison group. Furthermore, odds rapo analyses show that they are over 7 *mes more likely to complete a STEM PhD. Also, African American Meyerhoffs from more recent cohorts (1997-2004) were more likely to complete a STEM PhD ( 36.5% ) than those from earlier cohorts (1989-1996; 21.9%).
Top U.S. baccalaureate-origin ins*tu*ons of 2005-14 Black S&E doctorate recipients, by ins*tu*onal control, 2010 Carnegie classifica*on, and HBCU status
Carnegie ClassificaPon of UniversiPes AfAm Meyerhoffs Received PhD Degrees from
Theory-Based: Mechanisms of Program Influence Previous Findings Research Excitement (at college entry) Research Involvement (during college) Program Components (mean rapngs) Recent Study: Does perceived program benefit mediate the relapon between sense of community and science idenpty & research self-efficacy?
ParPcipants Meyerhoffs (N = 109) entering between 2010 & 2013 Gender: 57.8% Male; & 42.2% female Race/Ethnicity: 55 % URM Mean SAT scores: Math=696.2, Verbal=628.4, WriPng=632.9 Mean weighted high school GPA = 4.1
Measures* Sense of Community in MSP. Collected at end of 6-week Summer Bridge. Measured by 12-items adapted from the Sense of Community Index. Sample items: I can trust people in the program. ; I expect to be a part of the program for a long Pme. Perceived Program Benefit. Collected at end of year 1. Measured by 17-items that assess how useful each program element is to students. Financial aid, study groups, tutoring, academic advising, etc. *Scale scores = mean of all items. Measures have strong reliability espmates.
Measures* Science iden#ty. Measured by 5-items from Scien*fic Iden*ty Scale that assess how much being a scienpst is a part of student idenpty (Chemers, et al., 2010) Sample items: I have a strong sense of belonging to the community of scienpsts ; I feel like I belong in the field of science Research Self-Efficacy. Measured by 6-items from Scien*fic Self-Efficacy Scale that assess students ability to funcpon as a scienpst. Sample items: Generate a research quespon ; Use scienpfic literature and/or reports to guide research and develop theories. *Collected at end of year 2. Scale scores = mean of all items. Measures have strong reliability.
Findings: Science IdenPty Perceived Program Benefit (end of freshman year).38**.31** Psychological Sense of Community (end of Summer Bridge).12 (.24**) Science Iden*ty (end of sophomore year) Note: **p<.01
Findings: Research Self-Efficacy Perceived Program Benefit (end of freshman year).38**.33** Psychological Sense of Community (end of Summer Bridge).13 (.26**) Research Self- Efficacy (end of sophomore year) Note: **p<.01
Program Elements: Program Theory and Measurement Development Previous Research Theory Development Mean RaPngs of Elements Recent Research: Measure Development
Program Elements: Measure Development Set of interviews with Program Director and Asst. Director Meyerhoff Model (program theory) Intended Outcomes Development of Pilot Measure GeneraPon of inipal set of items Item review, cripque and refinement Pilot tespng with sample of students Psychometric Analyses Inclusion in End of Academic Year Survey
Program Values Element: Focus Program inculcates in students a commitment to Excellence Accountability Group success Community Leadership Professionalism Character and integrity Diversity in STEM Pursuit of a STEM PhD
Program Values Element: Intended Outcomes FormaPon of Meyerhoff idenpty Unified understanding of program intenpons InternalizaPon of values and value pursuit throughout career Applied passion for diversity in STEM fields Accountability among scholars AcPve parpcipapon in building support structures for scholars, through peer support Strengthened commitment to PhD pursuits
Sense of Community Element: Focus Family-like social and academic support system Live in the same residence hall first year Required to live on campus subsequent years. Staff regularly hold family meepngs IntenPonal creapon of a community of scholars Meyerhoff Family concept Living and learning community Regular Group meepngs ResidenPal requirement Parental involvement Meyerhoff Student Council
Sense of Community Element: Intended Outcomes Support structure for students of like-minded peers Enhanced sense of community (cohort; program) Avenues for leadership and influence AppreciaPon & pracpce working/succeeding together Enhanced academic and social integrapon Commitment to STEM career through community discourse Increased social and cultural networks
Scales: Reliabilities and Means Scale (# Items) Reliability Mean S.D. N Recruitment/ SelecPon Weekend (5).79 4.2.63 112 Summer Bridge (7).81 4.2.60 110 Financial support (6).85 4.1.73 112 Program Values (9).83 4.4.49 117 Advising (10).92 4.2.69 118 Research experience (7).83 4.2.62 93 Administrator Support (4).79 4.4.54 118 Program Community (14).92 4.2.59 118 Study Groups (7).88 4.4.55 117 Faculty Involvement (5).78 4.0.66 118
Scales: Intercorrelations Scale Min Max Mean Scale-Total Recruitment/ SelecPon Weekend (5).24.53.41.59 Summer Bridge (7).22.69.45.62 Financial support (6).08.40.25.34 Program Values (9).29.75.54.77 Advising (10).26.70.53.69 Study Groups (7).14.61.39.51 Program Community (14).33.70.56.84 Research experience (7).37.75.48.67 Faculty Involvement (5).27.53.40.56 Administrator Support (4).08.55.43.61
Qualitative Inquiry & Formative Assessment Interviews Focus Groups ObservaPon Surveys
Qualitative Inquiry: Selection Weekend They spend a lot of Bme planning [SelecBon Weekend] and it takes a lot of people and a lot of man hours to do it. And when you go, you can tell that they ve spent a lot of Bme and money to bring you here... and they re spending that much Bme on you. You know, that tells you something about how they see the students in the program.
Qualitative Inquiry: Summer Bridge Summer Bridge is where it all begins. It s like the saying that a chain is only as strong as its weakest link. That s Summer Bridge.
QualitaPve Inquiry: Staff and Peer Advising/Counseling The guidance that they give [is most helpful].if I were coming to UMBC and I wasn t a Meyerhoff scholar, I d have no idea where to go. I know I want to major in Biochemistry but I m not sure what s the best path to do that, what are the best classes to take, what are the best internships to do.the Meyerhoff program will help do that.
Qualita*ve Inquiry PROGRAM COMMUNITY: Family Mee*ng...You could tell [Dr. Hrabowski] and all the staff... just genuinely wanted to see you succeed. That they were going to try to do everything, as long as you did your best, they would do their best to just help you any way they could. And that was their main goal.
QualitaPve Inquiry: Research Experience/PresentaPons The Meyerhoff Program actually gave me an opportunity to do research....it was my first bench science experience....and I just fell in love with it. I just really loved how I had to think about everything and it just was very new for me. And that had a lot to do with considering the PhD. Every summer I did a new internship and I learned very quickly that internships get you around the country. [chuckling] So I had a joyful kind of thought about research, just azached to research... So I just had these recurrent reinforcements about research and life.
Documen*ng Systemic and Radia*ng Impact Prior Scholarship Inclusive excellence, changing campus culture and climate (theory of insptuponal culture change) Science ArPcle (undergraduate impact) Recent Scholarship Undergraduate Impact (updated) SynergisPc Effect on Program Development Beyond UMBC Meyerhoff AdaptaPon Program (HHMI)
Campus-Wide Impact: Undergraduate Number of African-American Undergraduates majoring in science and engineering increased more that 11.5-fold since 1985 Overall and S&E enrollments among La*no students grew 6.1 and 16.1-fold, respec*vely, since 1985 Number of Caucasian undergraduates majoring in S&E increased more than 3.1-fold since 1985 Avg. GPA of African-American students majoring in S&E increased from 2.7 in 1989 to 3.2 in 2014
1988 2000 2014 1988: Meyerhoff Program 1988: Honors College Meyerhoff model - 1992: HumaniPes Scholars 1994: HHMI Scholars 1995: Linehan Ar*st Scholars 1997: MARC Promoting broader institutional change across majors from BA through PhD Evaluation as integral 1999: Sondheim Public Affairs Scholars 2000: Living Learning CommuniPes 2000: CWIT Scholars 2002: First Year Seminars 2002: AGEP Promise 2003: ADVANCE 2004: PhD ComplePon Project 2004: IHU 2005: Chemistry Discovery Center 2008: Collegiate Success InsPtute 2008: Learning Resources Center 2008: LSAMP 2010: I-cubed@UMBC 2010: TRS 2010: CASTLE-Course redesign 2011: Transfer Success 2012: InnovaPon Fund 2014: STEM BUILD
Beyond UMBC: Meyerhoff Adapta*ons Schools that have used the Meyerhoff Scholars Program as a model for programs on their campuses include: Bates College Cornell University Duke University Louisiana State University Morehouse College University of Michigan Winston-Salem University Winthrop University
Meyerhoff Adapta*on Partnership Project (MAP) Alliance between HHMI & UNC-Chapel Hill Penn State UMBC Ra*onale To increase URM representa*on in STEM
MAP Assessment: Integrated Research Design Mul*-Level Implementa*on Assessment Yearly Student Academic Outcomes Program Component Implementa*on Value-Added Student Academic Outcomes Ins*tu*onal Impact on Science Culture and URM Success. Forma*ve Assessment and Feedback Modeling Analyses Partnership Assessment Consulta*on Integra*ve Report of Findings
Lessons Learned
Outcome/SummaPve EvaluaPon Mul*ple Comparison Samples UMBC, Declined, Na*onal (NSF) Mul*ple Outcome Measures GPA, STEM Major, PhD Entry, PhD Receipt Longitudinal Tracking over *me, Mul*ple cohorts, Funding Covariates Gender, Race/Ethnicity, SAT, HSGPA, Year of Entry
Theory-Based/Mechanisms Genera*ng Program Theory Mul*ple sources (literature, program) Itera*ve Mixed Methods Design (convergence) Surveys, Interviews, Focus groups, Observa*on Measurement Exis*ng measures, Newly developed measures Mul*ple Collaborators, Varied Backgrounds Applied, Theory-based, Mul*ple disciplines/perspec*ves
FormaPve/Process-Focused Importance Program Development and Improvement Mul*ple Methods Observa*on, Interviews, Focus Groups, Surveys Collabora*ve Process Protocol development, *ming, dissemina*on Means of Sharing (formal & informal) Reports, Presenta*ons, Lunch, Hallways
People and RelaPonships Trus*ng Rela*onships (UMBC-wide) Reciprocity Flexibility Interpersonal Skills Research Team (students of color)
Acknowledgements Current Funding Sources: Howard Hughes Medical InsPtute, NaPonal Instutes of Health, NaPonal Science FoundaPon Research Team Members: Lab Research ScienPst, Post- Doctoral Research Associate, Graduate and Undergraduate Students HHMI Assessment Teams at Penn State and UNC-Chapel Hill
References (Selected) Maton, K.I., Beason, T.S., Godsay, S., Sto. Domingo, M.R., Bailey, T.C., Shun, S., & Hrabowski III, F.A. (2016). Outcomes and processes in the Meyerhoff Scholars Program: STEM PhD complepon, sense of community, perceived program benefit, and research self-efficacy. CBE-Life Sciences EducaBon, 15 (3). Maton, K.I. Pollard, S.A., Weise, T.V.M., & Hrabowski, F.A. III. (2012). The Meyerhoff Scholars Program: A strengths-based, insptupon-wide approach to increasing diversity in STEM. Mt. Sinai Journal of Medicine, 79, 610-623. Maton, K.I., Hrabowski, F.A., Ozdemir, M. & Wimms, H. (2008). Enhancing representapon, retenpon and achievement of minority students in higher educapon: A social transformapon theory of change. In M. Shinn, & H. Yoshikawa, H. (Eds.), Toward posibve youth development: Transforming schools and community programs (pp. 115-132). New York: Oxford University Press.