The role of TIMSS, PIRLS & PISA in Providing a True Picture of Maltese Students Attainment

The role of TIMSS, PIRLS & PISA in Providing a True Picture of Maltese Students Attainment Dr. Vincent Marmarà Ph.D.(Stir.), M.Sc.(Sheff.) Statistics, B.Sc.(Hons.)(Melit.), FRSS for the Malta Union of Teachers November 2018

Contents 1) Main objectives 2) Methodology 3) PISA, TIMSS & PIRLS 4) One-to-one interviews 5) The quantitative study: Methodology & Demographics Results Familiarity with PISA,TIMSS and PIRLS Results Performance of the three assessments Results Involvement in the three assessments Results a. Your workplace and the three assessments level of importance b. Your workplace and the three assessments level of preparation (academic staff) c. Your workplace and the three assessments level of preparation (students) Results Opinions about the three assessments Results The level of students knowledge in relation to these assessments Results The three assessments and the Maltese syllabus Results The three assessments and the language impact Results Design and implementation of the assessments Conclusion of the quantitative study 2

Main Objectives 3

Main Objectives Based on the information provided by the Malta Union of Teachers (MUT), the scope of work covered the following: To review and understand PISA, TIMSS and PIRLS by analysing the documentation related to these assessments and the perceptions of these tests amongst educators and key personnel within the education sector 4

Methodology 5

Methodology The Methodology for this project included: 1. Setting the necessary meetings with the Malta Union of Teachers (MUT) to understand their contribution in the whole process; 2. Setting the necessary meetings with the Maltese educational authorities to understand their contribution in the whole process. These meetings helped to understand the assessments being carried out in further detail as well as the curriculum and the process of data collection; 3. Setting the necessary meetings to understand the educational system and the perceptions of key stakeholders in relation to these tests. The latter meetings were held with the following members: a. Representatives from the Education department and officials from the Department for Curriculum, Lifelong Learning and Employability; b. Officials from the Faculty of Education, University of Malta; c. Independent Schools representatives; d. Church Schools representatives; 4. Obtained the required data and documentation in relation to the scope of this project to carry 6 out theanalysis;

Methodology 5. Furthermore, a quantitative survey study was carried out amongst Learning Support Educators (LSEs), Kindergarten Educators (KGEs) Teachers, Heads of Departments (HoDs), Assistant Heads (AHs), Heads of Schools (HoSs) and Education Officers (EOs) to: Study and analyse the role of international studies (TIMSS, PIRLS and PISA) to provide a true picture of Maltese Students Attainment. Throughout this study we analysed questions such as: a. What do you consider your level of knowledge to be about these assessments? b. Do you think that your workplace is giving enough importance to these assessments? c. To what extent is your workplace preparing the students for these assessments? Disclaimer: This data collection was carried out during the month of October when some educators were being given training related to the TIMSS assessment. Hence, some educators may have been more knowledgeable about this particular assessment. 7

PISA, TIMSS and PIRLS 8

PISA What is PISA? The Programme for International Student Assessment (PISA) organised by the Organisation for Economic Co-operation and Development (OECD) is a triennial international survey which aims to evaluate education systems worldwide by testing the skills and knowledge of 15-year-old students. Moreover, the PISA programme investigates and compares the performance of schools and education systems worldwide. PISA focuses primarily on the assessment of student performance in reading, mathematics and science as they are foundational to a student s continuing education. However, it also takes into account student attitudes, motivations, and collaborative problem-solving. Reference: oecd.org 9

Key findings for PISA 2015 Based on the current literature, no specific issues or flaws with respect to the actual research methodology exists (data collection and analysis); PISA 2015: 38 countries had a mean Science score higher than Malta (465), while 32 countries had a lower mean score. Malta s mean science score was found to be lower than the international average (493); When comparing European countries, Malta scored higher in Science than in Greece (455), Bulgaria (446), Romania (435), Cyprus (433), Moldova (428), Albania (427), Turkey (425), Montenegro (411), Macedonia (384) and Kosovo (378); Malta scored significantly lower than Estonia (534), Finland (531), Slovenia (513), Netherlands (509), Germany (509), United Kingdom (509), Switzerland (506), Ireland (503), Denmark (502), Belgium (502), Poland (501), Portugal (501), Norway (498), Austria (495), France (495), Sweden (493), Czech Republic (493), Spain (493), Latvia (490), Luxembourg (483), Hungary (477), Croatia (475), Italy (481), Lithuania (475) and Iceland (473); 10

Key findings for PISA 2015 The mean Science score of Maltese students in the PISA 2015 cycle (465) was 4 points higher than the PISA 2009 cycle (461). Compared to PISA 2009, the mean Science scores in PISA 2015 increased by 2 points in State and Church schools and by 20 points in Independent schools; Mathematics was a minor domain in the PISA 2015 study. 33 countries had a mean Mathematics score higher than Malta (479), while 34 countries had a lower mean score. Malta s mean mathematics score was lower than the international average (490); Reading was a minor domain in the PISA 2015 study. 41 countries had a mean reading score higher than Malta, while 28 countries had a lower mean score. The reading average score for Maltese students (447) was lower than the international average (493); Independent and Church schools are scoring higher than the State schools. Student attainment in Science differs significantly between school types. Male and female students attending church (509) and independent schools (540) are scoring significantly higher in Science, compared to the international average (493). Conversely, male and female students attending state schools (427) are scoring significantly lower. 11

TIMSS What is TIMSS? The Trends in International Mathematics and Science Study (TIMSS) is a comparative international study of mathematics and science achievement organised by the International Association for the Evaluation of Educational Achievement (IEA). IEA is an independent international cooperative of national research institutions and government agencies that has been conducting studies of cross-national achievement since 1959. TIMSS has the goal of helping countries make informed decisions about how to improve teaching and learning in mathematics and science. Reference: timssandpirls.bc.edu 12

Key findings for TIMSS 2015 TIMSS 2015: Malta s mean Science score (481) was lower than the international average (500) and was ranked 22nd place out of the 39 participating countries; Science attainment of Maltese students was comparable to students from United Arab Emirates (477). It was found to be higher than 16 other countries including Malaysia (471), Bahrain (466), Qatar (457), Iran (456), Thailand (456), Oman (455), Chile (454), Georgia (443), Jordan (426), Kuwait (411), Lebanon (398), Saudi Arabia (396), Morocco (393), Botswana (392), Egypt (371) and South Africa (358); The 21 countries that scored higher than Malta in Science included Singapore (597), Japan (571), Chinese Taipei (569), Republic of Korea (556), Slovenia (551), Hong Kong (546) Russian Federation (544), England (537),Kazakhstan (533),Ireland (530),United States (530), Hungary (527), Canada (526), Sweden (522), Lithuania (519), New Zealand (513), Australia (512), Norway (509), Israel (507), Italy (499) and Turkey (493); 13

Key findings for TIMSS 2015 On average, Independent school students (550.3 females, 556.4 males) scored higher in Science than Church school students (519.3 females, 517.3 males) who in turn scored higher than State school students (452.9 females, 434.3 males).; Malta s mean Mathematics score (494) was 6 scale points lower than the international average (500) and was ranked 20th out of the 39 participating countries; On average, Independent school students (541.3 females, 553.0 males) scored higher in Mathematics than Church school students (518.3 females, 524.3 males) who in turn scored higher than State school students (473.0 females, 458.1 males); In the TIMSS 2015 cycle, Maltese students scored higher than the 2007 cycle in all content and cognitive domains. This improvement was achieved by both female and male students. 14

PIRLS What is PIRLS? The Progress in International Reading Literacy Study (PIRLS) is a comparative international study of the reading attainment of ten-year-olds (Year 5 students). PIRLS is organised by the International Association for the Evaluation of Educational Achievement (IEA). Furthermore, it investigates reading literacy and the factors involved in acquiring this skill. Reference: timssandpirls.bc.edu 15

Key findings for PIRLS 2016 PIRLS 2016: Malta s mean reading score (452) was significantly lower than the international average (500) and was ranked 40th of 50 participating countries; Reading attainment of Maltese students was comparable to students from United Arab Emirates (450) but was higher than nine countries including Bahrain (446), Qatar (442), Saudi Arabia (430), Iran (428), Oman (418), Kuwait (393),Morocco (358),Egypt (330) and South Africa (320); In all countries, female students scored higher in reading literacy than males. In Malta, female (463) students scored 21 scale points higher than males (442); 16

Key findings for PIRLS 2016 On average, Church school students (470.6) scored higher in reading than State school students (447.5) who in turn scored higher than Independent school students (411.8) (*). Mean reading scores of female students exceed those of males by 26.2 scale points in State schools, 15.4 scale points in Church schools and 5.2 scale points in Independent schools; (*) There is societal bilingualism in Malta and this is reflected in the different school sectors in Malta. The first language of most students in State schools is Maltese. The language situation in Church schools is more varied. The first language of most students in Independent schools is English. This is made clear by the present PIRLS results where Maltese was the test language. There was a significant decrease in the performance for students from Church schools, but especially for those from Independent schools. It is clear that the language of the test was a huge factor which determined the general underperformance of Maltese students on the PIRLS assessment. Source: curriculum.gov.mt (PIRLS 2016, Malta Report) 17

One-to-one interviews 18

One-to-one interviews Several one-to-one interviews were carried out with key players within our educational system. The following are the salient points: 1. There are socio-economic differences between state schools, church schools and independent schools; 2. Our system is very much exam oriented. If students fail the exams, they will most likely be lost in the system. Hence, our students are not trained to focus on other tests which are not their exams; 3. The items found in these tests (PISA, PIRLS, TIMSS) are not in sync with the knowledge provided to students; 4. Family background (incl. economic status) is still a dependent factor on the type of school (state, church and independent) and this affects test scores; 5. In most foreign countries, the above tests are given more importance, hence greater awareness; 6. These tests are mainly based on reasoning and thinking skills. This is an issue for Malta as our system is still exam oriented; 19

One-to-one interviews 7. Our educational system is not based on evidence and research; 8. In order for our ratings to improve we need to put less emphasis on coaching and more emphasis on reasoning and thinking skills. Decisions need to be taken based on evidence; 9. The vision of the different categories of schools towards these tests is different. 10. Perceptions about PISA, PRILS and TIMSS: Teaching in the Independent schools incorporates the same mind set of these assessments. 11. Science teaching: For the independent schools science is a core subject involving around 4 lessons per week. This is completely different when compared to other schools (a lower number of science lessons); 12. More focus needs to be put on primary schooling if we want to improve Malta s ranking. Teachers are not confident enough when teaching science at a primary level; 13. More focus needs to be placed on the science subjects, especially at primary level. 20

The quantitative study 21

Methodology 22

Methodology and Demographics The survey was carried out online. Educators (Learning Support Educators (LSEs), Kindergarten Educators (KGEs) Teachers, Head of Departments (HoDs), Assistant Heads (AHs), Heads of Schools (HoSs) and Education Officers (EOs)) were approached several times through e-mail by inviting them to participate in this research study; A sample size of 475individuals was collected amongst the educators; Level of confidence: 95%; Confidence interval: +/- 4.3%; The data was collected between the 12 th October and the 26 th October 2018. 23

Results Familiarity with PISA,PIRLS and TIMSS 24

Results Familiarity with PISA,PIRLS and TIMSS Most of the respondents heard about these assessments. PISA is the most familiar assessment amongst the educators (86.6%) while PIRLS is the least familiar (73.6%) Have you heard about PISA? Have you heard about PIRLS? Have you heard about TIMSS? 13.4% 26.4% 15.9% Yes No 86.6% 73.6% 84.1% 25

Results Familiarity with PISA,PIRLS and TIMSS Result 3.1 by school type Have you heard about PISA? Have you heard about PIRLS? 100% 80% 60% 40% 20% 85.5% 87.9% 85.7% 14.5% 12.1% 14.3% 100% 80% 60% 40% 20% 78.0% 22.0% 64.8% 35.2% 53.6% 46.4% 0% State Church Independent 0% State Church Independent Have you heard about TIMSS? 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 85.3% 83.7% 69.0% 31.0% 14.7% 16.3% State Church Independent Yes No 26

Results Familiarity with PISA,PIRLS and TIMSS Result 3.1 by occupation Have you heard about PISA? Head of Department (HoD) Head of School (HoS) Assistant Head (AH) Education Officer (EO) 3.1% 0.0% 0.0% 0.0% 96.9% 100.0% 100.0% 100.0% Teacher 10.1% 89.9% Kindergarten Educator (KGE) Learning Support Educator (LSE) 45.0% 55.0% 50.0% 50.0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Ye s 27

Results Familiarity with PISA,PIRLS and TIMSS Result 3.1 by occupation Have you heard about PIRLS? Head of Department (HoD) 18.2% 81.8% Head of School (HoS) Assistant Head (AH) Education Officer (EO) 0.0% 2.3% 0.0% 100.0% 97.7% 100.0% Teacher 28.8% 71.2% Yes Kindergarten Educator (KGE) 42.1% 57.9% No Learning Support Educator (LSE) 38.6% 61.4% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 28

Results Familiarity with PISA,PIRLS and TIMSS Result 3.1 by occupation Have you heard about TIMSS? Head of Department (HoD) 9.1% 90.9% Head of School (HoS) Assistant Head (AH) Education Officer (EO) 0.0% 0.0% 0.0% 100.0% 100.0% 100.0% Teacher Kindergarten Educator (KGE) 12.6% 44.4% 55.6% 87.4% Yes No Learning Support Educator (LSE) 43.5% 56.5% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 29

Results Familiarity with PISA,PIRLS and TIMSS When asked about the level of knowledge regarding these assessments, 26.9% have some knowledge about PISA, while 33.3% are not knowledgeable at all about PIRLS. Only 27.6% are knowledgeable or very knowledgeable about PISA and only 22.1% are knowledge or very knowledgeable about PIRLS. Level of knowledge (PISA) Level of knowledge (PIRLS) Not knowledgeable at all 20.4% Not knowledgeable at all 33.3% Slightly knowledgeable 25.2% Slightly knowledgeable 19.3% Some knowledge 26.9% Some knowledge 25.4% Knowledgeable 20.4% Knowledgeable 17.1% Very knowledgeable 7.2% Very knowledgeable 5.0% 30 0% 5% 10% 15% 20% 25% 30% 0% 5% 10% 15% 20% 25% 30% 35%

Results Familiarity with PISA,PIRLS and TIMSS 25.2% of the respondents are knowledgeable about TIMSS; however 67% are either Not knowledgeable at all or Slightly knowledgeable or have Some knowledge. The mean and median of the level of knowledge of these assessments is also illustrated in the table below. Level of knowledge (TIMSS) Not knowledgeable at all 21.5% Slightly knowledgeable Some knowledge 20.7% 24.8% PISA PIRLS TIMSS Mean/Median Some knowledge Slightly knowledgeable Some knowledge Knowledgeable 25.2% Very knowledgeable 7.8% 0% 5% 10% 15% 20% 25% 30% 31

Results Familiarity with PISA,PIRLS and TIMSS The mean and median of the level of knowledge of these assessments by school type and occupation are illustrated in the tables below. State Church Independent Mean Median Mean Median Mean Median PISA 3 3 3 3 3 3 PIRLS 2 3 2 2 2 1 TIMSS 3 3 3 3 3 3 PISA HoD HoS AH EO Teacher KGE LSE Mean 3 3 3 4 3 2 2 Median 3 3 4 4 3 1 2 1: Not knowledgeable at all 2: Slightly knowledgeable 3: Some knowledge 4: Knowledgeable 5: Very knowledgeable PIRLS HoD HoS AH EO Teacher KGE LSE Mean 3 4 3 4 2 2 2 Median 3 4 3 4 2 1 1 TIMSS HoD HoS AH EO Teacher KGE LSE Mean 3 4 4 4 3 2 2 Median 3 4 4 4 3 1 1 32

Results Performance of the three assessments 33

Results Performance of the three assessments More than half of the interviewed educators think that the Maltese students performed below average in all three assessments when compared to other countries. Performance of Maltese students (PISA) 2.6% Performance of Maltese students (PIRLS) 2.4% Performance of Maltese students (TIMSS) 3.0% Below average 41.4% 56.0% 40.6% 57.1% 37.7% 59.3% Average Above average 34

Results Performance of the three assessments Result 3.2 by school type. Respondents working in independent schools were more optimistic and think that the Maltese students performance was of an average performance. 70% 60% 50% 40% 30% 20% 10% 0% Performance of Maltese students (PISA) 58.1% 54.4% 39.7% 41.2% 2.2% 4.4% 48.1% 51.9% 0.0% State Church Independent 70% 60% 50% 40% 30% 20% 70% 60% 50% 40% 30% 20% 10% 0% 59.9% 37.6% Performance of Maltese students (TIMSS) 61.8% 35.3% 56.6% 38.9% 51.9% 48.1% Performance of Maltese students (PIRLS) 53.8% 43.4% 2.5% 2.8% 44.4% 55.6% 0.0% State Church Independent Below average Average Above average 10% 0% 2.8% 4.4% 0.0% State Church Independent 35

Results Performance of the three assessments Result 3.2 by occupation Performance of Maltese students (PISA) Head of Department (HoD) 0.0% 33.3% 66.7% Head of School (HoS) 0.0% 30.8% 69.2% Assistant Head (AH) Education Officer (EO) 2.6% 0.0% 7.1% 33.3% 64.1% 92.9% Above average Average Teacher 3.2% 39.9% 56.9% Below average Kindergarten Educator (KGE) 0.0% 20.0% 80.0% Learning Support Educator (LSE) 5.4% 24.3% 70.3% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 36

Results Performance of the three assessments Result 3.2 by occupation Performance of Maltese students (PIRLS) Head of Department (HoD) 0.0% 35.5% 64.5% Head of School (HoS) 0.0% 23.1% 76.9% Assistant Head (AH) Education Officer (EO) 0.0% 0.0% 14.3% 32.5% 67.5% 85.7% Above average Average Teacher 3.2% 40.0% 56.8% Below average Kindergarten Educator (KGE) 7.7% 15.4% 76.9% Learning Support Educator (LSE) 2.7% 32.4% 64.9% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 37

Results Performance of the three assessments Result 3.2 by occupation Performance of Maltese students (TIMSS) Head of Department (HoD) 0.0% 39.4% 60.6% Head of School (HoS) 0.0% 19.2% 80.8% Assistant Head (AH) Education Officer (EO) 0.0% 0.0% 7.1% 35.0% 65.0% 92.9% Above average Average Teacher 3.5% 35.5% 60.9% Below average Kindergarten Educator (KGE) 0.0% 15.4% 84.6% Learning Support Educator (LSE) 10.0% 32.5% 57.5% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 38

Results Involvement in the three assessments 39

Results Involvement in the three assessments Very few of the respondents were involved in these three assessments. 16.8% of the educators were involved in PISA while only 8.8% were involved in PIRLS. 18% 16% Involvment in these assessments 16.8% 15.8% 25% Involvment in these assessments (by school type) 14% 12% 10% 8.8% 20% 15% 16.6% 15.3% 18.5% 17.7% 20.7% 17.2% PISA 8% 6% 10% 9.1% 8.9% 6.9% PIRLS TIMSS 4% 5% 2% 0% PISA PIRLS TIMSS 0% State Church Independent 40

Results Your workplace and the three assessments level of importance 41

Results Your workplace and the three assessments level of importance Only 4.8% (PISA), 2.7% (PIRLS) and 3.8% (TIMSS) of the educators said that a high level of importance is given by their workplace to these three assessments respectively. The majority said that their workplace gives a Neutral level of importance (38.3%, 43.5% and 37.6% respectively.) Level of importance (PISA) Level of importance (PIRLS) No importance at all 16.3% No importance at all 19.7% Low importance 19.8% Low importance 16.6% Neutral 38.3% Neutral 43.5% Good level of importance 20.7% Good level of importance 17.5% Very high level of importance 4.8% Very high level of importance 2.7% 0% 10% 20% 30% 40% 50% 0% 10% 20% 30% 40% 50% 42

Results Your workplace and the three assessments level of importance The mean and median of the Likert scale is at Neutral level of importance for all the three assessments. When analysing the Likert scale by school type, all the mean and medians for all the assessments are Neutral ; however educators from the independent schools rated the level of importance by their workplace at a higher level (E.g. PISA: Independent (Mean=3.1), Church (Mean=2.9) and State (Mean=2.7)). Level of importance (TIMSS) No importance at all 16.2% Low importance Neutral 17.1% 37.6% PISA PIRLS TIMSS Mean/Median Neutral Neutral Neutral Good level of importance 25.2% Very high level of importance 3.8% 43 0% 10% 20% 30% 40%

Results Your workplace and the three assessments level of preparation (academic staff) 44

Results Your workplace and the three assessments level of preparation (academic staff) Around a third (33.6%, 36.8% and 32.9%) of the respondents said that their workplace level of preparation for the academic staff for such assessments is Neutral. On the other hand, a considerable amount (31.0%, 30.0% and 27.3%) of the respondents said that there is no preparation at all for these assessments. Level of preperation for academic staff (PISA) Level of preperation for academic staff (PIRLS) No preparation at all 31.0% No preparation at all 30.0% Low preparation 20.4% Low preparation 19.7% Neutral 33.6% Neutral 36.8% Good level of preparation 14.2% Good level of preparation 12.1% Very high level of preparation 0.9% Very high level of preparation 1.3% 0% 5% 10% 15% 20% 25% 30% 35% 40% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45

Results Your workplace and the three assessments level of preparation (academic staff) When calculating the mean for the latter question, this resulted in a low preparation rating for all assessments while the median for PISA resulted in a Low preparation rating and for PIRLS and TIMSS the result was Neutral. Level of preperation for academic staff (TIMSS) No preparation at all 27.3% Level of preparation for academic staff Mean Median Low preparation 20.3% PISA Low preparation Low preparation Neutral 32.9% PIRLS Low preparation Neutral TIMSS Low preparation Neutral Good level of preparation 16.5% Very high level of preparation 3.0% 0% 5% 10% 15% 20% 25% 30% 35% 46

Results Your workplace and the three assessments level of preparation (academic staff) The mean and median of the Likert scale by school type is illustrated below. It is evident that the educators at the independent schools claimed that their workplace prepare more their academic staff for these assessments. PISA Level of preparation for academic staff State Church Independent Mean Median Mean Median Mean Median Low preparation Low preparation Low preparation Low preparation Neutral Neutral PIRLS Low preparation Neutral Low preparation Low preparation Neutral Neutral TIMSS Neutral Neutral Low preparation Low preparation Neutral Neutral 47

Results Your workplace and the three assessments level of preparation (students) 48

Results Your workplace and the three assessments level of preparation (students) 38.0%, 36.5% and 34.3% of the respondents rated their workplace level of preparation for the students for such assessments as Neutral. On the other hand, a considerable amount of the respondents (30.8%, 31.5% and 30.0%) said that there is no preparation at all for these assessments. Level of preperation for students (PISA) Level of preperation for academic staff (PIRLS) No preparation at all 30.8% No preparation at all 31.5% Low preparation 20.4% Low preparation 20.5% Neutral 38.0% Neutral 36.5% Good level of preparation 10.0% Good level of preparation 10.0% Very high level of preparation 0.9% Very high level of preparation 1.4% 0% 5% 10% 15% 20% 25% 30% 35% 40% 49 0% 5% 10% 15% 20% 25% 30% 35% 40%

Results Your workplace and the three assessments level of preparation (students) When calculating the mean for the latter question, this resulted in a Low preparation for all assessments while the median for PISA and PIRLS resulted in a Low preparation rating and a Neutral rating for TIMSS. Level of preperation for academic staff (TIMSS) No preparation at all 30.0% Level of preparation for students Mean Median Low preparation 20.0% PISA Low preparation Low preparation Neutral 34.3% PIRLS Low preparation Low preparation TIMSS Low preparation Neutral Good level of preparation 13.5% Very high level of preparation 2.2% 0% 5% 10% 15% 20% 25% 30% 35% 40% 50

Results Your workplace and the three assessments level of preparation (students) The mean and median of the Likert scale by school type is illustrated below. Similar to the prior findings, it is evident that the educators at the independent schools claimed that their workplace prepares their students more for these assessments. PISA Level of preparation for students State Church Independent Mean Median Mean Median Mean Median Low preparation Low preparation Low preparation Low preparation Neutral Neutral PIRLS Low preparation Low preparation Low preparation Low preparation Neutral Neutral TIMSS Low preparation Neutral Low preparation Low preparation Neutral Neutral 51

Results Opinions about the three assessments 52

Results Opinions about the three assessments The majority of the respondents have a neutral opinion or said it is just a test for all the three assessments when asked What is your opinion about these assessments? Opinion about PISA Opinion about PIRLS 40% 35% 35.3% 35.3% 40% 35% 37.5% 36.2% 30% 30% 25% 25% 20% 20% 15% 15.3% 14.0% 15% 12.9% 13.4% 10% 10% 5% 5% 0% Positive Neutral It is just a test Negative 0% Positive Neutral It is just a test Negative 53

Results Opinions about the three assessments 40% 35% Opinion about TIMSS 34.9% 34.9% 30% 25% 20% 15% 16.0% 14.3% 10% 5% 0% Positive Neutral It is just a test Negative 54

Results Opinions about the three assessments Result 3.5 by school type. Opinion about PISA (by school type) Opinion about PIRLS (by school type) 40% 37.3% 37.3% 60% 35% 30% 25% 20% 15% 10% 5% 13.7% 11.8% 34.0% 32.1% 17.0% 17.0% 28.6% 28.6% 28.6% 14.3% 50% 40% 30% 20% 10% 50.0% 39.4% 37.5% 37.3% 31.4% 21.4% 21.4% 15.7% 15.7% 12.5% 10.6% 7.1% Positive Neutral It is just a test Negative 0% State Church Independent 0% State Church Independent 55

Results Opinions about the three assessments Result 3.5 by school type 45.0% 40.0% 35.0% Opinion about TIMSS (by school type) 42.9% 37.0% 35.8% 35.8% 30.0% 25.0% 20.0% 15.0% 14.5% 28.3% 21.4% 20.8% 21.4% 15.1% 14.3% 12.7% Positive Neutral It is just a test Negative 10.0% 5.0% 0.0% State Church Independent 56

Results The level of students knowledge in relation to these assessments 57

Results Do you think these assessments capture what students know about the topics and provide a true picture? Only a fraction of the respondents (6.0%, 5.7% and 5.9%) said that these assessments provide a true picture of the students knowledge. Many of the respondents (33.6%, 30.4% and 35.9%) said that only a small part of the content is relevant. There was a significant amount of respondents (25.0%, 33.9% and 25.7%) who were unsure about the latter question. Does PISA capture what students know about the subject and provide a true picture? Does PIRLS capture what students know about the subject and provide a true picture? 40% 40% 35% 30% 25% 20% 33.6% 22.4% 25.0% 35% 30% 25% 20% 30.4% 20.0% 33.9% 15% 10% 5% 6.0% 12.9% 15% 10% 5% 5.7% 10.0% 0% Yes Most of the content Small parts of the content No I don't know 0% Yes Most of the content Small parts of the content No I don't know 58

Results Do you think these assessments capture what students know about the topics and provide a true picture? 40% 35% Does TIMSS capture what students know about the subject and provide a true picture? 35.9% 30% 25% 20% 21.1% 25.7% 15% 11.4% 10% 5% 5.9% 0% Yes Most of the content Small parts of the content No I don't know 59

Results Do you think these assessments capture what students know about the topics and provide a true picture? Result 3.6 by school type. Educators at the Independent schools selected the option Most of the content more often for the PISA assessment. 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Does PISA capture what students know about the subject and provide a true picture? (by school type) 5.6% 11.9% 31.9% 23.1% 27.5% 41.2% 21.6% 15.7% 13.7% 7.8% 7.1% 28.6% 28.6% 21.4% State Church Independent 14.3% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Does PIRLS capture what students know about the subject and provide a true picture? (by school type) 31.3% 12.5% 31.3% 18.8% 32.7% 6.3% 6.1% 4.1% 40.8% 16.3% 0.0% 21.4% 0.0% 42.9% 35.7% State Church Independent Yes Most of the content Small parts of the content No I don't know 60

Results Do you think these assessments capture what students know about the topics and provide a true picture? Result 3.6 by school type. 45% 40% 35% Does TIMSS capture what students know about the subject and provide a true picture? (by school type) 36.4% 40.4% 42.9% Yes 30% 25% 20% 15% 10% 5% 0% 28.6% 24.2% 25.0% 20.6% 21.4% 19.2% 12.1% 9.6% 6.7% 5.8% 7.1% 0.0% State Church Independent Most of the content Small parts of the content No I don't know 61

Results The three assessments and the Maltese syllabus 62

Results The three assessments and the Maltese syllabus 44.9% (PISA), 48.2% (PIRLS) and 38.9% (TIMSS) of the respondents do not know if the content of these assessments is in line with the Maltese syllabi. Very few of the respondents said that it is in line with the Maltese syllabi (2.2%, 1.3% and 0.9%). The content is in line with the syllabi? (PISA) The content is in line with the syllabi? (PIRLS) 50% 45% 40% 44.9% 60% 50% 48.2% 35% 30% 40% 25% 20% 15% 10% 5% 0% 2.2% Yes 14.1% Most of the content 20.3% Small parts of the content 18.5% No I don't know 30% 20% 10% 0% 1.3% Yes 13.4% Most of the content 21.0% Small parts of the content 16.1% No I don't know 63

Results The three assessments and the Maltese syllabus Throughout this survey, it is evident that the number of responses I don t know is high. This shows the lack of knowledge that exists amongst the Maltese educators regarding these three assessments. 45% 40% The content is in line with the syllabi? (TIMSS) 38.9% 35% 30% 25% 24.8% 20% 17.9% 17.5% 15% 10% 5% 0% 0.9% Yes Most of the content Small parts of the content No I don't know 64

Results The three assessments and the Maltese syllabus Result 3.7 by school type. The content is in line with the syllabi? (PISA) by school type The content is in line with the syllabi? (PIRLS) by school type 50% 46.5% 60% 57.1% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 13.2% 17.0% 21.4% 1.9% 2.0% 19.6% 27.5% 9.8% 41.2% 7.1% 35.7% 35.7% 7.1% 14.3% State Church Independent 50% 40% 30% 20% 10% 0% 16.5% 13.9% 14.3% 12.2% 1.9% 46.2% 21.5% 22.4% 51.0% 0.0% 0.0% 14.3% 7.1% 21.4% State Church Independent Yes Most of the content Small parts of the content No I don't know 65

Results The three assessments and the Maltese syllabus Result 3.7 by school type. 60% The content is in line with the syllabi? (TIMSS) by school type 57.1% 50% 40% 30% 20% 38.8% 24.2% 18.8% 17.0% 32.7% 28.8% 25.0% 21.4% 13.5% 14.3% Yes Most of the content Small parts of the content No I don't know 10% 7.1% 0% 1.2% 0.0% 0.0% State Church Independent 66

Results The three assessments and the language impact 67

Results The three assessments and the language impact A considerable amount of the respondents (61.4%) said that the language of the assessments has an impact on the Maltese results. Only 5.7% think that the language is not affecting the result. Those working in an independent school (78.6%) are the most in favour that the language is affecting the results. Does the language of the assessment impact on the results? 90% Does the language of the assessment impact on the results? (by school type) Yes 61.4% 80% 78.6% 70% No 5.7% 60% 50% 60.2% 59.3% Yes No Maybe 23.6% 40% 30% 25.1% 24.1% Maybe I don't know I don't know 9.3% 20% 10% 5.3% 7.1% 9.4% 7.4% 9.3% 7.1% 7.1% 0% 10% 20% 30% 40% 50% 60% 70% 0% State Church Independent 68

Results Design and implementation of the assessments and improvement of the Maltese results 69

Results Design and implementation of the assessments and improvement of the Maltese results Below are the opinions of the respondents when they were asked In what ways can these assessments be better designed and implemented? 23.5% of the respondents think that such assessments need to be more in line with the Maltese syllabi. Better ways for design and implementation of such assessments To be more related to our syllabi 23.5% Choice of the assessment's language Do not know Others 15.2% 14.4% 16.7% The assessments are very long More training to the educators More student friendly Incentives and motivation for the students 6.8% 6.8% 6.8% 5.3% Better timing to conduct these assessmnents 4.5% 0% 5% 10% 15% 20% 25% 70

Results Design and implementation of the assessments and improvement of the Maltese results There were different suggestions on how the Maltese results can be improved. 15.9% of the respondents think that the Maltese students have to be better prepared for such assessments. Less routine learning for students was another valid suggestion on how to improve the results. How to improve Maltese results Others 33.3% Better preparation for students for these assessments 15.9% Do not know Less rote learning 11.6% 10.9% To be more related to our syllabi More training to the educators Choice of the assessment's language 8.0% 8.0% 7.2% More importance to science subjects Try to be more inline with these assessment 2.9% 2.2% 0% 5% 10% 15% 20% 25% 30% 35% 71

Conclusion of the quantitative study 72

Conclusion The majority of the respondents heard about PISA, PIRLS and TIMSS. Very few of the respondents are very knowledgeable about these assessments (7.2% (PISA), 5.0% (PIRLS) and 7.8% (TIMSS)). Most of the interviewees think that the performance of the Maltese students is below average in these assessments. 16.8% of the educators were involved in PISA, 15.8% in TIMSS while only 8.8% were involved in PIRLS. A Neutral level of importance was given for the three assessments by the educators. A considerable amount of the respondents (30.8% (PISA), 31.5% (PIRLS) and 30.0% (TIMSS)) said that there is no preparation at all for these assessments. 73

Conclusion Many of the respondents have a neutral opinion or said it is just a test for all the three assessments. Only a fraction of the respondents (6.0% (PISA), 5.7% (PIRLS) and 5.9% (TIMSS)) said that these assessments provide a true picture of the students knowledge. Very few of the respondents said that that these assessments are in line with the Maltese syllabi (2.2% (PISA), 1.3% (PIRLS) and 0.9% (TIMSS)). Only 5.7% of the respondents think that the language is not affecting the result. The independent schools are more positive about the fact that their workplace is giving enough importance to these assessments, that their workplace is preparing the academic staff for these assessments and that their workplace is preparing the students for these assessments. 74

The End vincentmarmara@gmail.com