The hidden meaning of TIMSS: The relationship between school ethical culture and science achievement in TIMSS 2015 Orly Shapira Lishchinsky Associate professor Department of Educational Administration, Leadership and Policy School of Education Bar-Ilan University Israel shapiro@biu.ac.il The 14th Education Research Symposium MARIE SKŁODOWSKA-CURIE -Individual Fellowships, Horizon 2020, European Commission
The study goals To explore whether a shared concept 'school ethical culture (SEC) came about using cross-national approach. To investigate SEC effect through predicting science achievements among countries participating in TIMSS 2015.
A shared meaning in the concept SEC may be supported by international assessment in education such as TIMSS, focusing on common ethical aspects like equity and quality (Mullis, Martin, Foy, & Hooper, 2016).
TIMSS 2015 continues 20-year trend measures of international assessments in math and science conducted by IEA (the International Association for the Evaluation of Educational Achievement). This study focuses on teachers' TIMSS 2015 questionnaires in relation to their 8th-grade student achievements in science among 45 participating countries.
We explore whether we can elicit additional meaning from teachers questionnaire SEC The original questionnaire s emphases: problems with school conditions and resources, safe and orderly school, job satisfaction, challenges facing teachers and teaching limited by student needs.
Two main approaches in comparative studies that focus on both ethics and culture: Different ethical perceptions that are rooted in country culture and norms (e.g., Rausch, Lindquist, & Steckel, 2014) Universalism through similarities in moral attitudes and behavior across countries (e.g., Donnelly, 2013).
Ethical culture and ethical climate We found confusion in the literature regarding the concepts ethical culture and ethical climate, using them interchangeably (e.g., Denison, 1996).
Our study, based on previous studies, indicated that ethical culture and ethical climate are distinctly contrasted. Kaptein (2011) distinguished between ethical culture and ethical climate, explaining that: Ethical culture presents the actual conditions for ethical behaviors. Ethical climate is defined as the expectation about what constitutes ethical behavior in the organization.
This study is pioneering in its approach, trying to find the meaning of ethical culture in the context of educational systems.
Method Data was based on: (a) TIMSS s questionnaires completed by 8th-grade science teachers. (b) Their students' achievements based on questionnaires focusing on science curriculum.
Data Analysis First step: Exploring whether a shared 'school ethical culture' came about Experts judgment: We asked for 10 voluntary experts in school ethical research, to independently rank relevant items (survey questions) according to their potential SEC meaning (about 86 items). We worked according to ranking procedure (Meyer & Booker, 2001), which recognizes high relevancy (4-5) to SEC. We have chosen 38 out of 86 items initially deemed as SEC.
Exploratory factor analysis Exploratory factor analysis was run on 1/3 of the total teachers' data (n=2,629 teachers). Cronbach's Alpha shows high internal consistency among the factor items (alpha>.80). Table 3 provides the final factor loadings for four dimensions: Teachers' profession Caring for learning Interaction with colleagues Respecting rules.
Our elicited dimensions were supported by several of the ethical culture dimensions developed by Kaptein (2011): Teachers' profession' - standing in professional standards such as: doing science experiences, assessing students, similar to 'Clarity' of EC (standards that are expected in the organization). Caring for learning' - school leadership support for teachers' professional development, similar to 'Supportability' of EC (strengthening the employees by the organizational leadership). Interaction with colleagues' - discussing how to teach and share teaching experiences similar to 'Discussability' of EC (sharing knowledge in order to learn from each other). Respecting rules' includes aspects of clear rules about student conduct similar to 'Sanctionability' of EC (aspects of enforcement through punishment and rewards).
Teacher's profession BTBS17D SCI\CONFIDENT\ENGAGE STUDENTS INTEREST 0.70 0.12-0.06-0.08 BTBS17E SCI\CONFIDENT\APPRECIATE SCIENCE 0.70 0.10-0.02-0.08 BTBS17A SCI\CONFIDENT\INSPIRE STUDENTS 0.65 0.10-0.08-0.03 BTBS17F SCI\CONFIDENT\ASSESS COMPREHENSION 0.64 0.08-0.06 0.04 BTBS17C SCI\CONFIDENT\CHALLENGING TASKS 0.64 0.03 0.03-0.05 BTBS17G SCI\CONFIDENT\IMPROVE UNDERSTANDING 0.62 0.09-0.03 0.01 BTBS17B SCI\CONFIDENT\SCIENCE EXPERIMENTS 0.55 0.12-0.05 0.00 BTBG14G GEN\HOW OFTEN\EXPRESS IDEAS 0.54-0.10 0.04 0.03 BTBG14D GEN\HOW OFTEN\CLASSROOM DISCUSSIONS 0.53-0.12 0.16 0.04 BTBG14C GEN\HOW OFTEN\CHALLENGING EXS 0.51-0.16 0.11 0.07 BTBG14F GEN\HOW OFTEN\PROBLEM SOLVING PRCDS 0.49-0.11 0.15 0.01 BTBG14B GEN\HOW OFTEN\EXPLAIN ANSWERS 0.47-0.18 0.03 0.08 Caring for learning BTBG06P GEN\CHARACTERIZE\AMOUNT OF INSTR SUPPORT -0.10 0.86 0.06-0.04 BTBG06O GEN\CHARACTERIZE\COLLABORATION TO PLAN -0.05 0.84 0.08-0.09 BTBG06Q GEN\CHARACTERIZE\SUPPORT FOP PROF DEVELOPM -0.11 0.77 0.08-0.02 BTBG06N GEN\CHARACTERIZE\CLARITY OF OBJECTIVES 0.06 0.70-0.08 0.07 BTBG06A GEN\CHARACTERIZE\TCHS UNDERSTANDING 0.13 0.52-0.06 0.04 BTBG06D GEN\CHARACTERIZE\TCHRS WORKING TOGETHER 0.05 0.47 0.19 0.02 BTBG06C GEN\CHARACTERIZE\TCHS EXPECTATIONS 0.15 0.36-0.14 0.18 Interaction with colleagues BTBG09E GEN\INTERACTIONS TEACHERS\WORK TOGETHER 0.05 0.01 0.82-0.04 BTBG09C GEN\INTERACTIONS TEACHERS\SHARE LEARNING 0.01-0.03 0.75 0.00 BTBG09F GEN\INTERACTIONS\WORK AS A GROUP 0.05 0.05 0.73 0.03 BTBG09A GEN\INTERACTIONS TEACHERS\DISCUSS TOPIC 0.05-0.05 0.72 0.04 BTBG09G GEN\INTERACTIONS\CONTINUITY IN LEARNING -0.01 0.07 0.71 0.05 BTBG09D GEN\INTERACTIONS TEACHERS\VISITS 0.00 0.09 0.63-0.09 Respecting rules BTBG07D GEN\THINKING ABT CURR SCH\STUD BEHAVE 0.01-0.09-0.01 0.90 BTBG07E GEN\THINKING ABT CURR SCH\STUD RESPECT 0.04-0.06 0.00 0.88 BTBG07F GEN\THINKING ABT CURR SCH\RESPECT PROPERTY 0.00-0.02 0.01 0.81 BTBG07G GEN\THINKING ABT CURR SCH\CLEAR RULES -0.06 0.31 0.00 0.47 BTBG07H GEN\THINKING ABT CURR SCH\RULES ENFORCEMENT -0.10 0.36 0.07 0.46 BTBG06M GEN\CHARACTERIZE\RESPECT FOR CLASSMATES 0.05 0.27-0.03 0.39 Mean Score STD Table 3: Exploratory factor analysis, factor arrangement and loadings (n=2,629 teachers). F1 F2 F3 F4 Reliability Alpha Cronbach.87.86.89.87
Multilevel confirmatory analysis The confirmatory runs were done on 2/3 of the sample (n=5,261). We confirmed the country level. structure of each latent factor on school and In order to find whether there is a universal approach (common factors on school and country level), we compared the model fit of the unconstrained model to the model fit of a constrained model in which loadings are constrained equally across the levels (Table 4). The equal loading constraint does not cause a severe reduction in the goodness-of-fit model ( CFI<.02) The conclusion is that the factor structure as observed within each country remains between countries.
Table 4: The multilevel confirmatory model results factor loadings and invariance test (n=5,261 teachers) Factor Teacher's profession Within Level Between Level ICC Coeff. SE Coeff. SE BTBS17D 0.74*** 0.01 0.97*** 0.02.11 BTBS17E 0.73*** 0.01 0.93*** 0.04.14 BTBS17A 0.67*** 0.02 0.83*** 0.07.18 BTBS17F 0.66*** 0.01 0.84*** 0.07.09 BTBS17C 0.66*** 0.02 0.83*** 0.05.10 BTBS17G 0.63*** 0.01 0.76*** 0.09.10 BTBS17B 0.63*** 0.01 0.59*** 0.11.12 BTBG14G 0.40*** 0.02 0.73*** 0.07.15 BTBG14D 0.37*** 0.02 0.72*** 0.09.21 BTBG14C 0.36*** 0.02 0.64*** 0.10.17 BTBG14F 0.40*** 0.02 0.63*** 0.08.18 BTBG14B 0.30*** 0.02 0.63*** 0.12.13 Unconstrained Model fit: CFI=.956, TLI=.940, RMSEA=.035, Chi Square=722.43, df=97, p<.001, AIC=100691.91, BIC=101158.24 Constrained Model fit: CFI=.952, TLI=.942, RMSEA=.035, Chi Square=792.85, df=109, p<.001, AIC=100730.67, BIC=101118.19
Caring for learning Within Level Between Level Coeff. SE Coeff. SE ICC BTBG06P 0.83*** 0.01 0.97*** 0.02.14 BTBG06O 0.86*** 0.01 0.94*** 0.03.02 BTBG06Q 0.73*** 0.01 0.92*** 0.03.13 BTBG06N 0.70*** 0.01 0.79*** 0.07.09 BTBG06A 0.52*** 0.02 0.42** 0.13.09 BTBG06D 0.58*** 0.01 0.68*** 0.09.11 Unconstrained Model fit: CFI=.949, TLI=.910, RMSEA=.068, Chi Square=424.91, df=17, p<.001, AIC=53359.45, BIC=53563.06 Constrained Model fit: CFI=.948, TLI=.932, RMSEA=.059, Chi Square=438.91, df=23, p<.001, AIC=53394.42, BIC=53558.62
Within Level Between Level Interaction with colleagues Coeff. SE Coeff. SE ICC BTBG09E 0.84*** 0.01 0.98**** 0.01.14 BTBG09C 0.72*** 0.01 0.92*** 0.03.11 BTBG09F 0.79*** 0.01 0.89*** 0.03.18 BTBG09A 0.67*** 0.01 0.88*** 0.05.10 BTBG09G 0.73*** 0.01 0.91*** 0.03.16 BTBG09D 0.63*** 0.01 0.73*** 0.08.29 Unconstrained Model fit: CFI=.982, TLI=.968, RMSEA=.047, Chi Square=212.58, df=17, p<.001, AIC=55935.67, BIC=56139.28 Constrained Model fit: CFI=.980, TLI=.972, RMSEA=.043, Chi Square=236.65, df=22, p<.001, AIC=55936.32, BIC=56107.09
Within Level Between Level Respecting rules Coeff. SE Coeff. SE ICC BTBG07D 0.82*** 0.01 0.88*** 0.05.09 BTBG07E 0.84*** 0.01 0.99*** 0.02.08 BTBG07F 0.84*** 0.01 0.91*** 0.05.09 BTBG07G 0.60*** 0.02 0.58*** 0.13.05 BTBG07H 0.63*** 0.02 0.63*** 0.11.07 BTBG06M 0.56*** 0.02 0.78*** 0.09.09 Unconstrained Model fit: CFI=.998, TLI=.997, RMSEA=.014, Chi Square=29.47, df=15, p<.001, AIC=49703.95, BIC=49920.70 Constrained Model fit: CFI=.994, TLI=.991, RMSEA=.023, Chi Square=78.51, df=21, p<.001, AIC=49754.16.54, BIC=49931.50
Multiple group confirmatory analysis The configural step is the unconstrained model for estimating all countries independently. The comparison is between the unconstrained model as a basis for fit quality and the constrained models. Regarding the configural and the metric model, we found acceptance level of above 0.90 for CFI and the TLI fit indices, while the Metric model may be considered as performing at a relatively similar fit quality as the configural model (Table 5).
The CFI differences between the metric and the configural models for the study dimensions show that the metric model performs as well as the configural model. We can conclude that countries share a very similar factor structure in terms of the suggested 3 dimensions (caring for learning, interaction with colleagues, and respecting rules), whereas for the teachers' profession factor, similarity between countries is less.
Table 5: multiple group analysis for the school ethic factors (n=5,261 teachers) Configural Model Metric Model Scalar Model Metric versus Configural Scalar versus Configural Scalar versu Metric Teacher's profession CFI.945.914.624 0.031 0.29 0.593 TLI.921.902.642 0.019 0.26 0.623 RMSEA.067.075.142-0.008-0.067 0.15 Chi-Square 2627.36 3574.25 8784.80 953.57 6222.58 5227.80 df 1724 2164 2604 P <.001 <.001 <.001 SRMR.066.134.301 # of parameters 1741 1301 861 AIC 89690.47 89826.48 94624.99 BIC 101125.49 98371.54 100280.11
Confirmatory factor analysis To confirm the full factor structure model we ran an integrative measurement model on 2/3 of the sample, that includes all four factors (Table 6). Our results indicate a high level of reliabilities (composite reliability >.70) for all four factors. Model fit is above acceptance level, e.g., CFI=.93, TLI=.92.
Table 6: Confirmatory factor analysis (measurement model), standardized coefficients (n=5,261 teachers) Factor Coefficient Standard Error Factor 1: Teacher's profession; CR=.84 BTBS17D.76*** 0.01 BTBS17A.69*** 0.01 BTBS17F.67*** 0.01 BTBS17C.70*** 0.01 BTBS17G.65*** 0.01 BTBS17B.64*** 0.01 BTBG14G.43*** 0.01 BTBG14D.45*** 0.01 BTBG14C.43*** 0.02 BTBG14F.44*** 0.02 BTBG14B.35*** 0.02 Factor 2: Caring for learning; CR=.87 BTBG06P 0.83*** 0.01 BTBG06O 0.86*** 0.01 BTBG06Q 0.75*** 0.01 BTBG06N 0.72*** 0.01 BTBG06A 0.53*** 0.01 BTBG06D 0.62*** 0.01 Factor 3: Interaction with colleagues; CR=.89 BTBG09E 0.85*** 0.01 BTBG09C 0.74*** 0.01 BTBG09F 0.81*** 0.01 BTBG09A 0.69*** 0.01 BTBG09G 0.77*** 0.01 BTBG09D 0.64*** 0.01 CR.89 Factor 4: Respecting rules; CR=.87 BTBG07D 0.81*** 0.01 BTBG07E 0.84*** 0.01 BTBG07F 0.84*** 0.01 BTBG07G 0.62*** 0.01 BTBG07H 0.65*** 0.01 BTBG06M 0.59*** 0.01 CR.87 Goodness-of-Fit: CFI=.934, TLI=.924; RMSEA=.044, SRMR=.050; CR=Composite Reliability.
Second step: The relationship between SEC and science achievements We used the MIMIC (Multiple Indicators Multiple Causes) model in Mplus v.8.0. To run this analysis we used the multilevel approach for schools at level one and countries at level two (Heck and Ried, 2017). On school level, teachers' SEC dimensions affect positively the mean score on science exams (Table 7). On country level, the higher the teachers' profession (B=-.47, p<.001) and the interaction with colleagues (B=-.40, p<.01), the lower the country mean score.
Table 7: Relationship between ethic factors and science grade (n=7,890 teachers) Unstandard Standard Factor Coefficient Standard Error Coefficient Standard Error Factor 1: Teacher's profession Within countries 8.62*** 0.97.16***.02 R 2.02***.005 Between Countries -26.66*** 6.81 -.47***.11 R 2.22***.10 CFI=.958, TLI=.944; RMSEA=.030, SRMR=.028, Chi-Square=788.50, df=98, p<.001 Factor 2: Caring for learning Within countries 7.38*** 1.37.13***.02 R 2.02**.006 Between Countries -10.51 8.44 -.19.15 R 2.03.06 CFI=.944, TLI=.913; RMSEA=.056, SRMR=.143, Chi-Square=687.41, df=27, p<.001 Factor 3: Interaction with colleagues Within countries 3.89*** 1.24.07*.02 R 2.005.003 Between Countries -22.82** 8.08 -.40**.14 R 2.16.11 CFI=.979, TLI=.968; RMSEA=.037, SRMR=.046, Chi-Square=319.44, df=27, p<.001 Factor 4: Respecting rules Within countries 16.47*** 1.23.30***.02 R 2.09***.01 Between Countries 7.92 13.92.14.24 R 2.02.07 CFI=.987, TLI=.978; RMSEA=.029, SRMR=.108, Chi-Square=196.52, df=25, p<.001 *p<.05. **p<.01. ***p<.001.
Figure 2. The negative relationship between teacher's profession and science achievements (country level)
Figure 3. The negative relationship between interaction with colleagues and science achievements (country lefvel)
CODE 36 Australia 1 48 Bahrain 2 72 Botswana 3 124 Canada 4 152 Chile 5 158 Chinese Taipei 6 268 Georgia 7 344 Hong Kong, SAR 8 348 Hungary 9 364 Iran, Islamic Republic of 10 372 Ireland 11 376 Israel 12 380 Italy 13 392 Japan 14 398 Kazakhstan 15 400 Jordan 16 410 Korea, Republic of 17 414 Kuwait 18 422 Lebanon 19 440 Lithuania 20 458 Malaysia 21 470 Malta 22 504 Morocco 23 CODE 512 Oman 24 554 New Zealand 25 578 Norway 26 634 Qatar 27 643 Russian Federation 28 682 Saudi Arabia 29 702 Singapore 30 705 Slovenia 31 710 South Africa 32 752 Sweden 33 764 Thailand 34 784 United Arab Emirates 35 792 Turkey 36 818 Egypt 37 840 United States 38 926 England 39 5788 Norway (8) 40 7841 United Arab Emirates (Dubai) 41 7842 United Arab Emirates (Abu Dhabi) 42 9132 Canada (Ontario) 43 9133 Canada (Quebec) 44 32001 Argentina, Buenos Aires 45
Discussion Our analysis evokes a deeper cross-national meaning for SEC, by exposing four dimensions of SEC: teachers' profession, caring for learning, interaction with colleagues and respecting the rules. We found some similarity between our findings and part of the Kaptein s (2008a) dimensions of EC, which may give some conceptual validity to our generated SEC dimensional model. Our findings on the country level, indicate that as teachers' profession and interaction with colleagues increased, the country s students' achievement reduction may be explained by country policy, investing more in low achievement countries compared to high achievement countries.
'Marginal addition - In high achievement countries, teachers may perceive the effect of 'teacher's profession' and 'interaction with colleagues' as relatively lower than other countries based on the fact that they are already on a high achievement level, thus these factors do not contribute significantly to their point of view. However, in low achievement countries, teachers may perceive the effect of these dimensions as more significant, because they may have the potential to increase students' achievements.
Conclusions Our findings support a universal perspective to SEC elicited from TIMSS questionnaires, providing additional meaning to TIMSS questionnaires, and contributing to defining EC in schools by its four dimensions, an effort that has never been undertaken before. Our findings also support the national culture by the dimension of teachers' profession which was found to be perceived differently among the participating countries.