Evaluating Progress NGA Center for Best Practices STEM Summit Jeni Corn, Ph.D. (jocorn@ncsu.edu) Friday Institute for Educational Innovations NC State University December 2011
SREB Health Sciences Computer Sciences Agri-Sciences Career Cluster STEM Science Technology Engineering & Math (STEM) Programs and Initiatives throughout K12 Education Career Academies Career and Technical Education PLTW Technology Engineering Design Ed. Math STEM Initiatives NC DPI Math/Science Partnership Science Turnaround Schools Instructional Technology NC STEM Community Collaborative NC New Schools Project Virtual Schools 21 st Century Community Learning Centers
NC STEM Education Strategic Plan April 2011 Lt. Governor s JOBS Commission Advisory Panel Review June 2011 Contract with New Schools Project August 2011 - Contract with MCNC NC STEM Community Collaborative September 2011 - JOBS Commission Endorsement November 2011 SBE Presentation and Endorsement March 2011 - Education Cabinet Initiate Strategic Plan October 2010 - Legislature Ed Oversight Department Presentation May 2010 - RttT Grant Application
NC STEM Attributes Integrated STEM curriculum, aligned with state, national, international and industry standards Project-based learning with integrated content across STEM subjects Connections to effective in- and out-of-school STEM programs Integration of technology and virtual learning Authentic assessment and exhibition of STEM skills Professional development on integrated STEM curriculum, community/industry partnerships and postsecondary education connections Outreach, support and focus on underserved, especially females, minorities, and economic disadvantaged On-going community and industry engagement A communicated STEM plan is adopted across education, communities and businesses STEM Work-based learning experiences, to increase interest and abilities in fields requiring STEM skills, for each student and teacher Business and community partnerships for mentorship, internship and other STEM opportunities that extend the classroom walls Connections with postsecondary education Alignment of student s career pathway with post-secondary STEM program(s) Credit completion at community colleges, colleges and/or universities
Convergence of Strategic STEM Evaluations at the Friday Institute RttT STEM Schools Evaluation GLF STEM Initiative Evaluation NCSU Maximizing the Impact of STEM Outreach (MISO)
STEM Evaluation Participants Project Funder LEAs Schools Teachers Students RttT STEM USED 18 20 700 9,000 GLF STEM GLF 43 200 1,200 32,000 MISO NSF --- --- 150 450 Total 61 220 2050 41,450
RttT STEM Evaluation Purpose Document the activities of the RttT STEM initiatives; Provide timely, formative data, analyses, and recommendations to help the STEM initiative teams improve their ongoing work; and Later in the project, provide summative evaluation results to determine whether the RttT STEM initiatives met their goals and to inform future policy and program decisions
GLF STEM Evaluation Purpose Work with STEM GLF grantees to design and implement an evaluation of STEM outcomes across projects Examine collective impact of GLF STEM grantees Build capacity of STEM GLF grantees to use data to make effective decisions for program improvement
MISO Research Study Purpose To creatively integrate longitudinal evaluation with innovation within NC State s K-12 STEM outreach programs, particularly those funded by NSF, to help ensure the breadth and depth of the future U.S. STEM workforce
STEM Theory of Action for Students awareness of STEM careers knowledge about STEM careers STEM motivation STEM engagement Improve STEM selfefficacy STEM knowledge & skills advanced STEM coursetaking postsecondary attendance completion of prereqs for STEM careers Ultimate Goal: More students get jobs in STEM fields
Questions How do students feel about learning STEM content and future career paths? How confident are teachers about teaching STEM related content? How do successfully do administrators lead STEM programs? Are STEM programs implementing with fidelity? How are STEM programs aligned with NC STEM Attributes? To what extent do teachers attain professional growth and remain in-field? How has student academic achievement changed over time? Did NC STEM high school students go on to post-secondary STEM program? Variables of Interest Surveys developed to measure student STEM Attitudes, Interest in STEM Careers, and 21 st Century Learning Skills; teacher confidence and efficacy in teaching STEM; administrators leadership for STEM Quality of implementation of the 12 NC STEM Attributes Existing NC education database that includes teachers education/licensure, experience, turnover; and students academic progress, attendance, course taking, growth and proficiency by academic subject, grade promotion and retention, transfer, dropout, and graduation status. Student applications and enrollments such as high school course grades/gpa, high school class rank, SAT scores, AP test scores, university enrollment, university matriculation, and university course grades/gpa. (All UNC and NCCCS Students) Data Sources 1. Common STEM Survey Tools (administrators, teachers, students) 2. STEM Implementation Rubric (LEA, school) 3. NC Education Administrative Data (LEA, school, administrators, teachers, students) 4. Student database maintained by enrollment management and student services Goal To determine the extent to which STEM programs impact long-term educational outcomes.
STEM Survey Development Survey development was guided by research and is derived in part from existing surveys Pilot surveys may be completed via a web-based survey tool Pilot phase of MISO surveys is in process: To establish validity and reliability of the survey instruments To eliminate or rework specific survey items
STEM Survey Development Teacher Surveys measure teacher confidence and efficacy in teaching STEM related content STEBI-Science Teaching Efficacy Belief Instrument (Enochs & Riggs, 1990) MTEBI-Math Teaching Efficacy Belief Instrument (Enochs, Smith, & Huinker, 2000) added engineering, technology, elementary Also developing an Administrator Survey
STEM Survey Development Student Survey measures STEM Attitudes, Interest in STEM Careers, and 21 st Century Learning Skills STEM Attitudes and basis for Career Exploration: adapted from an Evaluation of Women in Engineering (Erkut & Marx, 2005) Careers : adapted from the Occupational Outlook Handbook from the Bureau of Labor Statistics, http://www.bls.gov/oco/ 21st century Learning: adapted from NC Student Learning Conditions survey developed at the FI 6-12 student survey; 3-5 elementary student survey
NC STEM Implementation Rubric