An External Evaluation of the Maryland Mathematics Engineering and Science Achievement (MESA) Program

Transcription

1 An External Evaluation of the Maryland Mathematics Engineering and Science Achievement (MESA) Program The Center for Research and Reform in Education (CRRE) Johns Hopkins University Roisin P. Corcoran, Ph.D. Jane M. Eisinger Joseph M. Reilly Steven M. Ross, Ph.D. July 31, 2014

2 MESA Evaluation 2 An External Evaluation of the Maryland Mathematics Engineering and Science Achievement (MESA) Program Maryland MESA is a structured, grade 3 12, precollege program designed to prepare students for academic and professional careers in science, technology, engineering, and mathematics (STEM). The program is generally held weekly either before or after school for 1 2 hours, depending on the school. The program's stated goals are to (a) increase the number of engineers, scientists, mathematicians, and related professionals at technical and management levels, and (b) serve as a driving force in encouraging minorities and females to participate in these fields. Maryland MESA provides services and programs to over 2,100 students and 180 teachers from the following geographical areas: Baltimore City, Anne Arundel County, Baltimore County, Calvert County, Charles County, Howard County, Montgomery County, Prince George s County, and Talbot County. More than 24,000 Maryland students have been served since the program's inception in To achieve its goals, the MD MESA program offers the following educational outreach components: Advisor (teacher) professional development training STEM-based project activities Field trips and academic competitions Tutoring/study skills instruction Industry involvement/guest speakers and presenters Business collaboration/outreach Parental/guardian support College and career counselling Scholarships/incentives Study Design Evaluators from the Center for Research and Reform in Education (CRRE) of the Johns Hopkins University School of Education visited three MESA school clubs in the Howard County Public School District (one elementary, one middle, and one high school) for the purpose of observing the clubs activities and interviewing the clubs site leaders. In addition, students from multiple Maryland MESA clubs were asked to complete an online survey regarding their experiences in the clubs, their reasons for joining, and their interests in STEM. Evaluation Questions This study sought to address the following questions: 1. How effective is the MESA program at helping students develop an interest in STEM?

3 MESA Evaluation 3 2. What are the average self-reported grades (math, science, and computer science) for students in the MESA program? 3. What is MESA doing well, and what are some suggestions for program improvement?

4 MESA Evaluation 4 Participant Survey Overview Survey Findings The student program survey addressed student participation and self-reported achievement in STEM coursework, student interest in future STEM pursuits (including coursework, college focus, and career focus), and student perception of the MESA program. The survey was created by CRRE for the purpose of this study and administered online in the spring of Survey Population Participants were 77 MESA club members who attended six schools in Maryland: one high school, two middle schools, and three elementary schools.. MESA program coordinators selected the schools that participated in the survey activity. The mean age of participants was years. In terms of grade representation, 76.7% of participants were between fourth and seventh grades, and the remaining 23.3% of participants were distributed across grades The sample consisted of 47 boys (61.0%) and 30 girls (39.0%). In total, 49.4% of the participant population identified themselves as African American, 19.5% identified as White, and 19.5% as Asian Indian. Participants of Hispanic, Latino, or Spanish origin made up 11.7% of the survey sample, and participants of numerous other ethnicities were also represented, although at somewhat lower percentages. Of these participants, 75.3% indicated that MESA was the only school-based STEM program in which they were currently participating. Participants who responded (76.6%) to the question, what is the highest level of education completed by a parent or guardian they lived with most of the time, reported the following: 85.6% of participants reported having a parent who completed an associate s degree or higher 79.2% of participants reported having a parent who completed a bachelor s degree or higher 40.3% of participants reported having a parent who completed a master s degree or higher Participant Course Selection and Performance Participants were asked to respond to questions concerning their current course selection for math, science, and computer science. They were then asked follow-up questions concerning their reasons for selecting the courses they did, and their general performance in these classes. Overall, the majority of participants (68.8%) indicated that they participated in at least one STEM class during the school year that was categorized as honors, gifted/talented, advanced, International Baccalaureate, or AP. More detailed analysis of the item blocks corresponding to these questions are discussed for math, science, and computer science.

5 MESA Evaluation 5 Math Course Selection and Performance Although participants selected a wide variety of math courses, the courses selected most often were as follows: 26.0% of students selected Pre-Algebra, 26.0% selected other math course, and 14.3% selected Algebra I. As shown in Table 1, when participants were asked, Why are you taking this math course? the two options reflecting intrinsic interest: You really enjoy math, and You like to be challenged were the most common responses (26.0% and 19.5%, respectively). However, participants also frequently selected courses because of school course requirements. Specifically, 15.6% of participants indicated that they had no choice in the course they enrolled in, and 16.9% of participants indicated that the course was assigned to them. Thus, 32.5% of participants selected their courses primarily due to school assignment. Interestingly, these two selection patterns represent the two most frequently cited criteria in students enrolling in math courses. When asked What have most of your grades been in math this year? 57.1% of participants reported receiving mostly A grades; 90.9% of participants reported mostly receiving B grades or higher, and no participant reported mostly earning grades below a C. Table 1 Participant Reasons for Math Course Selection Reason for Taking the Math N % Course You re Enrolled In You really enjoy math You like to be challenged You had no choice, it is a school requirement The school counsellor suggested you take it Your parent(s) encouraged you to take it A teacher encouraged you to take it There were no other math courses offered You will need it to get into college You will need it to succeed in college You will need it for your career It was assigned to you Some other reason You don t know why you are taking this course Total =

6 MESA Evaluation 6 Science Course Selection and Performance More participants selected other science course than any other course option, with 32.5% of participants indicating this category as their science class. No other course option was selected by more than 13.0% of participants. Outside of this course selection, other science courses participants enrolled in included: Earth Science (13.0% of participants) Environmental Science (9.1% of participants) Physical Science (7.8% of participants) Biology I (7.8% of participants) Participants were asked to indicate the reasons for enrolling in science courses. As is seen in Table 2, the response You really like science was the option selected most frequently (29.9%), followed by You had no choice, it is a school requirement (20.8%), You like to be challenged (14.3%), and It was assigned to you (13.0%). When related response options are combined, 44.2% of participants used either You really like science, or You like to be challenged as their primary criteria, and 33.8% of participants indicated that the course was a school requirement or assigned to them as their reason for enrolling. Additionally, 10.4% of participants indicated that they selected the course because it would either help them get into or succeed in their college or career; and 5.2% of participants indicated that encouragement from a parent, teacher, or guidance counsellor was a primary factor. When asked, What have most your grades been in science this year? 64.2% of participants reported receiving mostly A grades; 90.6% of participants reported receiving mostly B grades or higher, and no participant reported receiving mostly failing grades.

7 MESA Evaluation 7 Table 2 Participant Reasons for Science Course Selection Reason for Taking the Science N % Course You re Enrolled In You really enjoy science You like to be challenged You had no choice, it is a school requirement The school counsellor suggested you take it Your parent(s) encouraged you to take it A teacher encouraged you to take it There were no other science courses offered You will need it to get into college You will need it to succeed in college You will need it for your career It was assigned to you Some other reason You don t know why you are taking this course Total = Computer Science Course Selection and Performance In total, 10 out of the 77 survey participants (13.0%) indicated that they were taking a computer science class during the school year. Six participants indicated that they were taking computer science courses described as either general, basic, or introductory. The remaining participants reported taking AP Computer Science A, Computer Programming, Other, Computer Programming 1, or Computer Programming 2. Participants reported the following as their top three reasons for their course selection: You really enjoy computer science (40.0%) You like to be challenged (20.0%) It was assigned to you (20.0%) Unlike participants math and science course selections, none of the participants indicated that the course was a school requirement. Furthermore, no participants selected options related to parent, teacher, or counsellor encouragement as their reason for enrolling. Lastly, no participants selected options related to the courses utility in helping them succeed in college or a career.

8 MESA Evaluation 8 Of the 10 participants enrolled in computer science courses, 8 participants indicated that they received mostly A grades in the computer science course, 9 participants indicated they received mostly B grades or higher, and only one participant indicated that most of his/her grades were lower than a B (this participant received mostly C grades). Table 3 Participant Reasons for Computer Science Course Selection Reason for Taking the N % Computer Science Course You re Enrolled In You really enjoy computer science You like to be challenged You had no choice, it is a school requirement The school counsellor suggested you take it Your parent(s) encouraged you to take it A teacher encouraged you to take it There were no other computer science courses offered You will need it to get into college You will need it to succeed in college You will need it for your career It was assigned to you Some other reason You don t know why you are taking this course Note. N = 10 response rate for each item Participant Perceptions on STEM Topics Participants were asked to respond to survey items pertaining to how interested, confident, and competent they felt with math, science, and computer science, respectively. Ten of the 77 total participants (13%) responded to this section. As indicated in Table 4, participant Likert responses showed a degree of interest, confidence, and competence with each of the STEM subjects being addressed. On average, the respondents reported positive opinions on all the STEM statements covered; furthermore, participant means exceeded 3.50 for all 16 of these survey items, and exceeded 3.75 for 14 out of the 16 items. Participants, on average, indicated agreement with the statement: Someday I plan to have a career in a math, science, or computer-science related field (M = 4.40, SD = 1.26). The participant response mean for this question (4.40) represented the highest response mean for this entire section of survey items. Additional

9 MESA Evaluation 9 aggregated findings pertaining to this section are listed below (within each bullet point, subjects are listed in order of their response means): Participants, on average, found each STEM subject to be interesting: computer science (M = 4.10, SD = 1.29), math (M = 4.00, SD = 1.15), and science (M = 3.90, SD = 1.20). Participants, on average, agreed that they understood science (M = 3.90, SD = 1.10), math (M = 3.80, SD = 1.03), and computer science (M = 3.80, SD = 1.23). On average, participants felt confident doing science (M = 4.10, SD = 1.20), computer science (M = 3.80, SD=1.23), and math (M = 3.70, SD = 1.06). Participants, on average, indicated that they liked learning computer science (M = 3.90, SD = 1.29), science (M = 3.80, SD = 1.23), and math (M = 3.70, SD = 1.34) in school. Participants, on average, indicated that they planned to keep taking classes in science (M = 4.10, SD = 1.20), computer science (M = 4.00, SD = 1.25), and math (M = 3.90, SD = 1.20) in the future. As outlined in Table 5, experiments and field trips were the activities most frequently cited as participant favorites in math and science classes. In addition, competitions and design activities were cited by more than 50% of participants as being favorite activities.

10 MESA Evaluation 10 Table 4 Participant Perceptions of STEM Topics Survey Item Strongly disagree Disagree Neither agree or disagree Agree Strongly agree I think math is interesting I understand math I feel confident doing math I like learning math in school I plan to keep taking math classes in the future I think science is interesting I understand science I feel confident doing science I like learning science in school I plan to keep taking science classes in the future I think computer science is interesting I understand computer science I feel confident doing computer science I like learning computer science in school I plan to keep taking computer science classes in the future Someday I plan to have a career in a math, science, or computer science-related field Note. N = 10 response rate M SD

11 MESA Evaluation 11 Table 5 Participant Preferences in Math and Science Class Activities My favorite part(s) of math and N % science classes are... (check all that apply) Coursework Competitions Field trips Design activities Learning about math/science careers Experiments Other Participant Future Plans A total of 12 participants responded to survey items concerning college and career plans (only high school students were eligible to respond to this section). As shown in Table 6, 91.7% of participants indicated that they planned on applying to/attending a 4-year college, with several indicating that they would like to apply to/attend other types of schools as well. As indicated in Table 7, these participants used/were using a variety of different college information-gathering activities as they applied to college. Participants demonstrated unanimous interest in taking STEM-related classes in their postsecondary future, with 100% of participants indicating that they planned on taking math, science, or computer science classes postsecondary. Similarly, 100% of participants indicated that they planned to obtain an advanced degree such as an M.S., Ph.D., or M.D. Lastly; all but two participants indicated that they planned to eventually pursue a STEM career (83.3% of participants). Table 6 Participant College Plans Did you apply to (or are you N % making plans to attend) any of the following types of colleges/schools? (Check all that apply) A 4-year college A 2-year college A vocational, technical or trade school A military school I haven t thought about applying and/or attending college yet Note. N = 12 response rate

12 MESA Evaluation 12 Table 7 Participant College Preparation Activities If you DID apply or make plans N % to attend college/school, have you done any of the following activities? (Check all that apply) Visit any of the colleges you were interested in? Talk with professors or other staff at any of these colleges? Talk with students who attended these colleges about how they liked it there? Get information about specific colleges (for example, on the computer or in the mail)? Attend a test preparation class? Attend a mock interview (practice for college interviews)? Attend a shadow day at a college? Talk with a school counsellor or other school staff members about your college plans? I didn t do any of these things Note. N = 12 response rate Participant Perceptions of MESA Participants were asked to rate the overall influence of the MESA program on a variety of their personal outcomes in STEM, including their excitement toward, and interest and achievement in math, science, and computer science classes. The Likert mean for all 10 survey items in this section exceeded 3.00 (the Likert scale s neutral rating), and exceeded 3.50 for 9 of 10 items. Notably, the item block s final response ( It has made me more interested in having a job in a math, science, or computer science related career ) had the highest response mean (M = 3.95, SD = 1.23) of any item in this section. Additional findings pertaining to this section are listed below: Participants, on average, agreed that MESA made them more excited about science (M = 3.87, SD = 1.15), math (M = 3.62, SD = 1.12), and computer science (M = 3.52, SD = 1.26). Participants, on average, agreed that MESA made them more interested in taking classes in science (M = 3.77, SD=1.16), math (M = 3.64, SD = 1.11), and computer science (M = 3.51, SD = 1.23).

13 MESA Evaluation 13 Participants, on average, agreed that MESA helped them improve their performance in science classes (M = 3.57, SD = 1.28), math classes (M = 3.56, SD = 1.19), and computer science classes (M = 3.27, SD = 1.17). Although the response statistics between survey items in these areas are relatively uniform, two additional trends are worth noting. First, the survey items pertaining to competence (e.g., It has helped me do better in my classes ) have Likert means lower than the survey items pertaining to interest and excitement for all three subjects. Secondly, Likert means for questions pertaining to math and science are higher than computer science in all three key areas (i.e., the program s influence on participants excitement, interest, and performance.)

14 MESA Evaluation 14 Table 8 Participant Perceptions of MESA Survey Item Strongly disagree It has made me more excited about math. It has made me more interested in taking math classes. It has helped me to do better in my math classes. It has made me more excited about science. It has made me more interested in taking science classes. It has helped me to do better in my science classes It has made me more excited about computer science. It has made me more interested in taking computer science classes. It has helped me to do better in my computer science classes. It has made me more interested in having a job in a math, science, or computer science-related career. Disagree Neither agree nor disagree Agree Strongly agree M SD

15 MESA Evaluation 15 Site Leader Interview Findings Two evaluators from CRRE conducted in-person interviews with three MESA site leaders (representing one elementary, one middle, and one high school participating in the program) during May and June The site leaders had been in their positions at their respective schools for 8 12 years. MESA program membership ranged from students at each site. The site leader interview content was developed by the CRRE research team to provide an administrator s view of the program and its use in schools. All participants agreed to participate in the interviews, which lasted approximately minutes. Questions focused on three areas: the school s use of the program, the site leader s attitude toward the MESA program, and his/her perception of student response to the program. Schools Use of the MESA Program MESA provided the school program s base funding, which allowed site leaders to purchase equipment and materials, such as robotics kits. Site leaders provided building materials for the students to use on some of their projects, including items such as scrap wood, plastic bottles, and cardboard. One leader made a point of saying that funding was not an issue for his/her club whereas another spoke of decreased funding resulting in decreased field trip opportunities for his/her students. Site Leaders Attitudes toward MESA When asked about the degree to which degree they believed the MESA program benefitted their schools, two of the leaders stated that they felt the program brought a measure of prestige to their school. One stated that the program had lapsed briefly at his/her school; parents and students alike had been disappointed, with parents calling and ing the principal pleading for it to be restarted. The elementary school site leader said that he/she thought that students who participated in the club were better prepared for middle school. I do have to say for many of the students and for me, it s the highlight of the week... I can t wait... I get to see that glimmer in the eye where they re (the students) like Oh yeah! (SL2) All of the site leaders indicated that they would strongly recommend the MESA program to other educators, with one saying that he/she actually had done so. Two of the three site leaders mentioned that they wished the program was offered at all schools within their county. Administrator and Parental Involvement School administrators did not appear to play a direct role in the program s functioning at the schools that were visited. Administrators uniformly provided help to their MESA clubs by including their news in school announcements, providing funding from the schools PTAs, and occasionally attending team meetings or events. One site leader noted that at one time in the past, his/her assistant principal had helped to decide which students would participate in the program. The site leaders who were interviewed for this evaluation seem to have great autonomy in the way they chose to manage their clubs. One leader ran the club solely with

16 MESA Evaluation 16 the help of parents; two received additional help in the form of supplies, assistance, and work space from their Tech Ed. teachers; and one of the three had an assistant provided by the school. Parental involvement varied from school to school, with the most parental involvement evident at the elementary school. At that school, the site leader stated that the program would not be successful if the students were not able to do a lot of work on their MESA projects after school and on the weekends. Parents provided meeting places (in their homes) and supervision at these times. Some of the parents at the elementary site had served as MESA team leaders for various projects conducted during the year. At both the middle and high schools, the site leaders commented that their students were less inclined to want to have their parents around at school: We ll have them (parents) come in occasionally... I see a tendency for the parents to kind of fall back in middle school. (SL2) All of the site leaders spoke of parents willingness to volunteer, to take time off from work to chaperone at MESA competitions and field trips, and to turn out to take pictures and cheer their students on. How MESA Differs from other STEM Programs Each of the site leaders voiced differing opinions on this topic. One felt that other STEM programs, as compared to the one at his/her school, focused on writing and reading rather than doing. Another believed that his/her school s location within the state gave him/her a possible advantage over other STEM clubs when it came to the support he/she received. A third site leader noted that his/her program was less costly than other STEM programs, as the investment was in the students ideas and academics rather than in equipment. Meeting Student Needs and Interests Overall, the site leaders agreed that the program was meeting the needs of their students. An interesting point raised by one site leader was that in his/her school, the program appealed to students with a variety of interests not just those with an interest in math and the sciences. I think that it (MESA) does a pretty good job in allowing the kids to choose their difficulty of the project that they do. Sometimes it s hard to teach a lot of physics and math to (the youngest students in the club). (SL2) (Students are) learning good engineering, and learning to work together as teams. (SL2) They re exposed to a lot of different things that they might not get in class. (SL3) I think it really opens the door for them because some of them are taking more engineering classes, rethinking what they want to do. (SL3)

17 MESA Evaluation 17 The site leaders unanimously agreed that students enjoyed participating in the MESA program at their schools. Two of the three schools could only accommodate approximately half of the students who would like to participate in their programs due to staffing, funding, and space. I know they look forward to it... and I know that they enjoy their time here. I mean shucks, if they didn t, they wouldn t be here on a Friday afternoon! (SL2) Reported Impact on Student Achievement One site leader had successfully used participation in MESA as a motivation for students to keep their grades up in math. Another felt that the real impact on achievement would be seen in several years time, when students in the program continued to take science classes. One stated that he/she had heard from parents that their children do better in school when they are members of the team. All three site leaders mentioned that they knew of students from their program who had gone on to be MESA club members, or had continued in the sciences at their subsequent schools.... then they come back and they talk about it and you see what they ve accomplished later on. (SL3)... greatest benefit is that kids learn about themselves and their limits, while demonstrating what they are capable of doing if given the opportunity. (SL1) Implementation of MESA The MESA program was in place when all of the site leaders interviewed for this evaluation had taken their positions at their respective schools. One of the present site leaders was currently trying to find someone to assume his/her role with the program but had not been successful in recruiting another teacher from the school. One of the three site leaders began his/her association with MESA as a parent volunteer. He/she went on to become the club leader at the school where he/she was working when no one else was showing interest in the position. On deciding to take on the job and continuing the club, he/she commented: I think it s very important that students learn to problem solve. They need to learn how to read directions, see the objectives, it s just very important. (SL3) Positive Aspects about MESA Site leaders liked the variety of projects that their students had the opportunity to work on, including robotics and steamboats. They also really appreciated seeing students solving problems together and sharing ideas with one another sometimes with students of very different backgrounds or ideas from their own. Another highlight for the site leaders was seeing students make the connection between something they had learned in one of their classes and what they were doing in MESA, and vice-versa. I really like it when the kids get to discover science hands-on without a textbook telling them... (SL2)

18 MESA Evaluation 18 One leader discussed the many unseen benefits of MESA, such as the skills that students acquire by presenting their projects before professionals, thereby gaining a sense of ease and confidence in that type of situation.... (T)hey don t realize it until they get to college, and they go You know, I can do this! (SL3) Suggestions for Improvement Suggestions for improvement varied between site leaders. Two of the three stated that while judging and general organization of the MESA team competitions had gotten better in recent years, there was still room for improvement. Two of the three site leaders also stated that the Effective Communication competition (one of the MESA Day competition events) was not popular with their students, and that they would like to see something done to make that component of MESA more engaging for students. I would like to see them maybe change some of the different projects a little bit more. (SL2) Another issue raised was a desire for more (and varied) hands-on field trips and activities for students, such as trips to museums and venues where they could interact with professionals in STEM-based industry. One site leader said that this component of the MESA program seemed to have diminished in scope during the past few years and that he/she hoped that this pattern would change. He/she viewed the times when students got to meet with STEM mentors as very valuable.

19 MESA Evaluation 19 Conclusions Site leader and student participant reactions to MESA participation demonstrated consistently strong support for the program. Specifically, site observations, survey response statistics, and interview responses corroborated that students found the program interesting and beneficial to their STEM learning. Furthermore, site leaders felt that the program served the needs of the students it reached, but regretted that it could not accommodate all of the students who expressed an interest in participating. Further conclusions concerning the data gathered are summarized below. According to site leader interviews, Maryland MESA provided a framework, funding, and general support for the individual MESA clubs. School administrators did not appear to play a direct role in the MESA clubs functioning but did provide an element of support in facilitating the site leader s autonomy in running the school s MESA program. Participant responses indicated substantial parent support for the program and a degree of direct parental involvement during the elementary school years (visiting the clubs, chaperoning field trips, etc.). Although parental support for the program seemed to continue in middle and high school, direct parental involvement generally became more limited as students progressed through school. Site leaders consistently exhibited highly positive attitudes toward the program s purpose and outcomes. Site leaders consistently indicated that parents and students enjoyed and valued the program and suggested that their (the site leader s) participation in leading the program brought them fulfillment. All interviewed site leaders indicated that they would recommend the MESA program to other educators and that the program was meeting the needs of their students. Site leader interviews as well as participant survey responses indicated a strong level of student interest in the MESA program. Site leaders consistently indicated that students thoroughly enjoyed participating in the program, and student survey responses indicated the program s positive influence in increasing their interest in and excitement about math, science, and computer science. Furthermore, over 80% of the MESA high school students surveyed indicated that they planned to pursue a career in STEM after college. Although not as robust as the support demonstrated for the program s influence on student STEM engagement, site leader interviews and participant survey data provided support suggesting the program s potentially positive influence on student achievement. Site leaders shared multiple anecdotes of student STEM learning, and student survey responses indicated that they frequently felt the program helped them perform better in math, science, and computer science classes.