STEM Academy Workshops Evaluation

OFFICE OF INSTITUTIONAL RESEARCH RESEARCH BRIEF #882 August 2015 STEM Academy Workshops Evaluation By Daniel Berumen, MPA Introduction The current report summarizes the results of the research activities to evaluate the STEM Academy Workshops that were offered during the spring 2015 term at Santa Monica College (SMC). These workshops were held over five consecutive Saturdays and included sessions focused on material related to the college s two specially designated CHEM 10 (Introductory to General Chemistry) sections. Specifically, the workshops included core sessions focusing on chemistry, and additional sessions covering math and counseling topics. SMC designates specific course sections for students who are part of their Adelante, Black Collegians and STEM programs. Of the 64 students enrolled in these sections, 40 participated in the workshops. An additional nine current and prospective SMC students participated. This group included two students who had an official enrollment in non-designated CHEM 10 sections. This brief includes a summary of the results of a survey that was administered to students at the end of the five week workshops and an analysis of the CHEM 10 course outcomes for students in the designated CHEM 10 sections. Survey Results Thirty-six participants completed a 17 item survey at the conclusion of the STEM Academy Workshops. The following tables focus on items that specifically relate to the content provided in the chemistry, math and counseling workshops. Questions related to students demographic background and the structure and delivery of the workshops are included in Appendix A. The first sets of questions require students to indicate the degree to which the workshops they attended increased their understanding of each of the three topics (chemistry, math and counseling). By design, only students who attended at least three sessions for a particular topic provided a response for said topic. Santa Monica College Page 1

Question 1. How much did the Math Workshops increase your understanding of what is required of you to be successful in math coursework? No Increase 1 3.8% Slight Increase 3 11.5% Moderate Increase 14 53.8% Substantial Increase 8 30.8% Total 26 100% Question 2. How much did the Counseling Workshops increase your understanding of what is required of you to be successful in college? No Increase 1 3.7% Slight Increase 4 14.8% Moderate Increase 9 33.3% Substantial Increase 13 48.1% Total 27 100% Question 3. How much did the Chemistry Workshops increase your understanding of what is required of you to be successful in chemistry coursework? No Increase 2 6.9% Slight Increase 2 6.9% Moderate Increase 5 17.2% Substantial Increase 20 69.0% Total 29 100% The majority of students who responded to each prompt noted that the workshops either moderately increased or substantially increased their understanding of each topic. Only 30.8% of students who responded noted that the math workshops substantially increased their understanding of math coursework. This was lower than the rate for counseling and chemistry (48.1% and 69.0%, respectively). The second set of questions asked students to identify how useful, or not useful, they found the different workshops. Students were allowed to provide comments that explained their answers. Tables are included that summarize the general topics students addressed in their comments. Not all students provided comments, and some students included multiple topics in their comments. Santa Monica College Page 2

Question 4. Overall, how useful or not useful did you find the Math Workshops? Please explain: Not at All Useful 1 3.6% Slightly Useful 3 10.7% Somewhat Useful 9 32.1% Very Useful 15 53.6% Grand Total 28 100.0% Summary of Topics Addressed Count Provided a better understanding of math concepts 7 Good refresher 6 Did not cover material at my math level 5 Learned new methods 1 Needed more tutor time 1 Instructor "not into it" 1 Total 21 The majority of students found the math workshops to be somewhat useful or very useful (85.7%). In their comments, students noted that the workshops provided them with a better understanding of different math concepts, with six instances of students noting that these sessions provided a good refresher. There were five instances of students noting that the workshops did not cover material at their math level, indicating that they would have liked to have more advanced math topics be covered. Question 5. Overall, how useful or not useful did you find the Counseling Workshops? Please explain: Not at All Useful 1 3.6% Slightly Useful 2 7.1% Somewhat Useful 8 28.6% Very Useful 17 60.7% Grand Total 28 100.0% Summary of Topics Addressed Provided "insight or "important information" on personal growth/development 10 Workshops provided information on specific resources available 7 Review/had previously received information 3 Need "more time management tools" 1 Total 21 Santa Monica College Page 3

Once again the majority of students found the workshops to be somewhat useful or very useful (89.3%). A higher percentage of students found the counseling workshops to be very useful compared to the responses for the math workshops (60.7% and 53.6%, respectively). Most of the comments noted that the workshops provided students with insightful or important information about what it takes to be a successful student. There were multiple responses noting that students learned about important services that are offered at the college. Question 6. Overall, how useful or not useful did you find the Chemistry Workshops? Please explain: Somewhat Useful 1 3.6% Very Useful 27 96.4% Grand Total 28 100.0% Summary of Topics Addressed Count Provided additional/supplemental information 13 Covered important class topics 6 Found assessments useful 2 Want less group work 1 Want more group work 1 Inspirational 1 Total 24 About 96% of respondents found the chemistry workshops to be very useful, and only one student responded that it was somewhat useful. The responses indicate that students found the chemistry workshops to be more useful than either the counseling or math workshops. In their comments, students noted that the workshops provided additional or supplemental information for their CHEM 10 class. Additionally, students noted that covering important topics from class was also useful. CHEM 10 Outcomes The following section contains the CHEM 10 outcomes for students who participated in the STEM Academy Workshops. Success and GPA by Cohort Success rates are calculated by taking the number of A, B, C and P grades and dividing that number by the total enrollment. Grade point average (GPA) is calculated by assigning a numerical value to A, B, C, D and F grades from four to zero, respectively. SMC designates specific courses for their Adelante, Black Collegians and STEM programs. In the following tables the Attended cohort will include students who were officially enrolled in these sections and attended at least one workshop. For comparison, the tables include students in the two Santa Monica College Page 4

designated CHEM 10 sections that did not participate in any sessions of the workshops and all other spring 2015 CHEM 10 students. Additionally, students enrolled in the same Adelante, Black Collegians and STEM designated CHEM 10 sections during the spring 2014 term are included. The two participating students who enrolled in a different CHEM 10 section are excluded entirely from the tables. Table 1. CHEM 10 Success Rates and GPA Cohort Count Success Rate GPA Attended 40 67.5% 2.50 Did not attend 24 41.7% 2.07 All other Spring 2015 CHEM 10 Students 795 55.5% 2.50 Spring 2014 designated sections 65 43.1% 1.76 Students who participated in the STEM Academy Workshops had a higher success rate in CHEM 10 than students in the designated sections that did not attend any sessions. An independent samples t-test found a statistically significant difference in the success rates of students who attended the workshops and those that did not 1. While the students who attended had a higher GPA than those that did not, there it was not a statistically significant difference. Students who attended the STEM Academy Workshops had a higher success rate and GPA than students in the spring 2014 CHEM 10 courses designated for Adelante, Black Collegians and STEM students. Independent samples t-tests found statistically significant differences between their success rates 2 and GPA 3. The Attended cohort had higher success rates than all other spring 2015 CHEM 10 students, and a similar GPA. An independent-samples t-test was conducted and did not indicate a significant difference between their success rates and GPA. Table 2. CHEM 10 Success Rates and GPA by Gender Did Not All CHEM Spring 2014 Attended Gender Attend 10 Sections Count of Females 25 8 410 35 Success Rate 64.0% 37.5% 49.8% 34.3% GPA 2.24 2.00 2.50 1.57 Count of Males 15 16 385 30 Success Rate 73.3% 43.8% 61.6% 53.3% GPA 2.92 2.10 2.50 1.92 Total 40 24 795 65 Success Rate 67.5% 41.7% 55.5% 43.1% GPA 2.50 2.07 2.50 1.76 1 t(64)=2.06, p=.043, d=.44 2 t(105)=2.51, p=.014, d=.44 3 t(80)=2.85,p=.006, d=.90 Santa Monica College Page 5

The gap between the female and male success rates were lower for students who attended the workshops (9.3%) compared to all other CHEM 10 students and students in the spring 2014 designated sections (11.8% and 19%, respectively) 4. Ethnicity Table 3. CHEM 10 Success Rates and GPA by Ethnicity Did Not Attended Attend All CHEM 10 Spring 2014 Sections Count of Asian/PI Students 3 3 153 7 Success Rate 100.0% 100.0% 69.3% 57.1% GPA 3.33 3.00 2.71 2.00 Count of African-American/Black 8 4 49 Students 11 Success Rate 62.5% 25.0% 40.8% 36.4% GPA 2.50 1.50 2.14 1.22 Count of Hispanic/Latino Students 20 9 325 38 Success Rate 60.0% 11.1% 38.5% 34.2% GPA 2.21 1.00 2.04 1.63 Count of White Students 7 5 206 8 Success Rate 85.7% 80.0% 70.9% 75.0% GPA 3.00 2.75 2.86 2.71 Total* 40 24 795 65 Success Rate 67.5% 41.7% 55.5% 43.1% GPA 2.50 2.07 2.50 1.76 *Includes students reported as Native American, Two or more and Unknown African-American/Black and Hispanic/Latino students who participated in the workshops had higher success rates and GPAs than students in the other three cohorts. It should be noted that the number of students in the Attended cohorts are small (all less than 25). The following table contains students success rates and GPA by the number of Chemistry sessions that they attended. Students had the opportunity to attend up to 5 sessions. Table 4. CHEM 10 Success Rates and GPA by Chemistry Sessions Number of Success Count Sessions Rate GPA 1 3 33.3% 3.00 2 6 66.7% 3.00 3 6 83.3% 2.60 4 9 66.7% 2.29 5 18 66.7% 2.35 Total 42 66.7% 2.49 4 Equity Ratios for gender and ethnicity by CHEM 10 outcomes are available in Appendix B. Santa Monica College Page 6

An increase in the number of sessions that students attended did not have a significant relationship with an increase in their CHEM 10 success rates and GPA. CHEM 10 In-Class Assessments The following tables contain scores students enrolled in the designated CHEM 10 sections received for their in-class assessments. These assessments included seven in-class quiz scores, and three in-class test scores. These assessments make up about 30% of the final course grade 5. Students who withdrew from the class, and therefore did not complete all assessments, are excluded from the tables 6. Table 5. Cumulative Quiz and Test Scores by Cohort Average Average Average Count Cumulative Cumulative Cumulative Cohort Quiz Score Tests Score Total Attended 35 44.3 217.2 261.5 Did Not Attend 15 41.3 214.5 255.8 Total 50 43.4 216.4 259.8 Students who attended the workshops had slightly higher cumulative quiz and test scores than students who did not attend any workshops. An independent samples t-test did not find any significant difference between the two groups. Table 6. Cumulative Quiz and Test Scores by Cohort and Gender Gender Attended Did Not Attend Count of Females 22 5 Average Cumulative Quiz Score 44.3 45.1 Average Cumulative Test Score 211.5 228.1 Average Cumulative Total 255.8 273.2 Count of Males 13 10 Average Cumulative Quiz Score 44.3 39.4 Average Cumulative Test Score 226.9 207.6 Average Cumulative Total 271.2 247.0 Total 35 15 Average Cumulative Quiz Score 44.3 41.3 Average Cumulative Test Score 217.2 214.5 Average Cumulative Total 261.5 255.8 Female students who attended the workshops had a lower cumulative total score than female students who did not attend. It should be noted that the count of students who did not attend the workshop is only five. 5 A course syllabus is available at http://homepage.smc.edu/walker_muriel/syllabus%20chem%2010%20win14.htm 6 Appendix C contains data for all students with available quiz and test data. Santa Monica College Page 7

Table 7. Cumulative Quiz and Test Scores by Cohort and Ethnicity Ethnicity Attended Did Not Attend Count of Asian/PI Students 3 3 Average Cumulative Quiz Score 51.3 47.9 Average Cumulative Test Score 218.9 250.6 Average Cumulative Total 270.2 298.5 Count of African-American/Black Students 6 2 Average Cumulative Quiz Score 44.8 40.9 Average Cumulative Test Score 210.2 193.5 Average Cumulative Total 255.0 234.4 Count of Hispanic/Latino Students 19 4 Average Cumulative Quiz Score 40.8 43.4 Average Cumulative Test Score 210.0 190.8 Average Cumulative Total 250.8 234.2 Count of White Students 6 4 Average Cumulative Quiz Score 51.2 45.6 Average Cumulative Test Score 242.8 235.2 Average Cumulative Total 294.0 280.8 Total* 35 15 Average Cumulative Quiz Score 44.3 41.3 Average Cumulative Test Score 217.2 214.5 Average Cumulative Total 261.5 255.8 *Includes two students reported as Unknown African-American/Black and Hispanic/Latino students who attended workshops had higher cumulative test and quiz scores than students who did not attend any workshops. The count of the Did Not Attend cohorts are small (<5). Santa Monica College Page 8

Summary In total, 49 current and prospective SMC students participated in at least one STEM Academy Workshop. Of those, 40 were enrolled in two specially designated CHEM 10 courses. 69% of survey respondents who attended at least 3 chemistry sessions felt that the workshops "substantially increased their understanding of what is required to be successful in chemistry coursework" 96.4% of survey respondents found the chemistry workshops to be "very useful". This percentage is higher than the responses to the math and counseling workshops (53.6% and 60.7%, respectively) Students who attended the workshops had significantly higher success rates than those students in the same sections that did not attend. Students who attended the workshops had significantly higher success rates and course GPA than students who enrolled in the same sections the previous spring term. A relationship was not found between an increase in the number of sessions attended and an increase in CHEM 10 success rates and GPA. Students who attended the workshops had slightly higher cumulative quiz and test scores than students who did not attend. Santa Monica College Page 9

Appendix A: Survey Results for Student Demographic and Workshop Structure Questions 1. Please indicate your gender: 2. Please indicate your ethnicity: Male 14 38.9% Female 22 61.1% Total 36 100.0% Caucasian/White 8 22.2% American Indian 3 8.3% Asian/Pacific Islander 9 25.0% African American/Black 14 38.9% Latino/Hispanic 2 5.6% Total 36 100.0% 3. Please indicate the highest level of education your parents received: At least one of my parents graduated from a four-year college. 11 30.6% One or both of my parents attended some college or trade school. 12 33.3% Neither of my parents attended any college or trade school. 13 36.1% Total 36 100.0% 4. When do you plan to transfer? Fall 2015/Spring 2016 8 22.9% Fall 2016/Spring 2017 9 25.7% Fall 2017/Spring 2018 18 51.4% Total 35 100.0% Santa Monica College Page 10

5. Please select the Workshop(s) which you participated in at least 3 times (You may check more than one box): Math 25 69.4% Counseling 25 69.4% Chemistry 30 83.3% Total (Unduplicated) 36 100% 6a. Should the duration of breakfast or lunch be lengthened, shortened or stay the same? Breakfast Lunch Lengthened 6 19.4% 14 42.4% Shortened 1 3.2% 0 0.0% Stay the Same 24 77.4% 19 57.6% Total 31 100.0% 33 100% 6b.Should the duration of each program be lengthened, shortened or stay the same? Math Workshops Counseling Chemistry Workshops Workshops Lengthened 7 23.3% 4 12.9% 14 46.7% Shortened 14 46.7% 7 22.6% 1 3.3% Stay the Same 9 30.0% 20 64.5% 15 50.0% Total 30 100.0% 31 100.0% 30 100.0% 7. Should the length of weeks of the STEM Academy be lengthened, shortened, or stay the same? Lengthened 21 65.6% Shortened 1 3.1% Stay the Same 10 31.3% Grand Total 32 100.0% Santa Monica College Page 11

8. Please provide the most valuable aspects of the STEM Academy Workshops. Summary of Topics Addressed Count Tutors and teachers 14 Chemistry help 9 Content 7 Counseling 4 Time and attentiveness 4 Group work/social aspect 4 Resources 1 Math 1 Total 44 9. Please describe aspects of the STEM Academy Workshops that could improve the experience for future students. Summary of Topics Addressed Count More advanced math topics 7 Shorter math sessions 3 More tutor time 3 More group work 3 More advanced chemistry topics 2 Longer sessions/extra days 2 Longer counseling 1 More structure to math 1 Total 22 10. If there is anything not covered by the Workshops that you would have liked to see or experience, please explain. Summary of Topics Addressed Count More advanced math topics 4 More chemistry 3 Longer sessions/extra Days 1 More group work 1 Keep same partners 1 More focused math 1 More activities 1 More counseling info 1 Total 13 Santa Monica College Page 12

11. Why did you decide to attend the STEM Academy? Summary of Topics Addressed Count Extra help 32 Meet others/social aspect 2 SMC experience 2 Friends 1 Professor s suggestion 1 Try new things 1 4-Year college experience 1 Total 40 Santa Monica College Page 13

Appendix B: Equity Ratios by CHEM 10 Completion An equity ratio is calculated by dividing the percentage of a subgroup (ex: female students, white students) that successfully completed an outcome by the percentage of the same subgroup in the initial cohort (outcome percentage/cohort percentage). A ratio of 1.0 indicates that a subgroup is equally represented in the outcome variable when compared with the subgroup s representation in the cohort In this instance, the outcome is CHEM 10 completion, and the cohort is the total number of students enrolled in CHEM 10. Equity ratios are calculated for gender and ethnicity only; the designated CHEM 10 sections did not include veteran students, foster youths, and less than four students with a primary disability. More details about calculating equity ratios can be found on the Institutional Research website 7. 1. CHEM 10 Equity Ratios by Gender Attended Did Not Attend All CHEM 10 Spring 2014 Sections Gender Female 0.95 0.90 0.90 0.80 Male 1.09 1.05 1.11 1.24 Total 1.00 1.00 1.00 1.00 Count 40 24 795 65 2. CHEM 10 Equity Ratios by Ethnicity Did Not Attended All CHEM 10 Ethnicity Attend Asian/PI 1.48 2.40 1.25 1.33 African- American/Black Spring 2014 Sections 0.93 0.60 0.74 0.84 Hispanic/Latino 0.89 0.27 0.69 0.79 White 1.27 1.92 1.28 1.74 Total* 1.00 1.00 1.00 1.00 Count 40 24 795 65 *Includes students reported as Native American, Two or more and Unknown 7 http://www.smc.edu/enrollmentdevelopment/institutionalresearch/documents/reports/student%20equity/smcstudentequitydata2014.pdf Santa Monica College Page 14

Appendix C: CHEM 10 Quiz and Test Scores (All available students) 1. Cumulative CHEM 10 Test and Quiz Scores by Cohort Count Total Quiz Score Total Tests Score Cohort* Average Average Total Points Average Attended 38 42.3 207.3 249.6 Did Not Attend 19 35.7 185.4 221.0 Total 57 40.1 200.0 240.1 *Includes students that received a W. 2. Cumulative CHEM 10 Test and Quiz Scores by Gender Gender* Attended Did Not Attend Count of Females 24 6 Average Cumulative Quiz Score 42.7 39.8 Average Cumulative Test Score 202.3 201.5 Average Cumulative Total 245.1 241.3 Count of Males 14 13 Average Cumulative Quiz Score 41.5 33.7 Average Cumulative Test Score 215.9 177.9 Average Cumulative Total 257.4 211.7 *Includes students that received a W. 3. Cumulative CHEM 10 Test and Quiz Scores by Ethnicity Ethnicity* Attended Did Not Attend Count of Asian/PI Students 3 3 Average Cumulative Quiz Score 51.3 47.9 Average Cumulative Test Score 218.9 250.6 Average Cumulative Total 270.2 298.5 Count of African-American/Black Students 7 3 Average Cumulative Quiz Score 39.5 33.1 Average Cumulative Test Score 185.5 150.5 Average Cumulative Total 225.1 183.6 Count of Hispanic/Latino Students 19 6 Average Cumulative Quiz Score 40.8 34.8 Average Cumulative Test Score 210.0 159.6 Average Cumulative Total 250.8 194.4 Count of White Students 7 4 Average Cumulative Quiz Score 44.6 45.6 Average Cumulative Test Score 218.5 235.2 Average Cumulative Total 263.1 280.8 *Includes students that received a W. Santa Monica College Page 15