The Second Year of SEC Vocational Subjects MATSEC Support Unit April 216 University of Malta
1 Table of Contents Executive Summary... Introduction... 8 General Information... 8 Methodology... 8 Survey Findings... 1 Perception and Students Choice... 1 Student Ability and Motivation... 11 Displacement from Other Subjects... 16 The Assignments... 16 Syllabi... 22 Paperwork... 2 Assignment Feedback... 2 Professional Support... 28 Resources... 33 Continuous Assessment in other Subjects... 33 Candidate Results Findings... 3 Registrations... 3 Differences in Achievement 21-18 Cohort... 38 Task Comparison 21-18 Cohort... 39 Year Comparison 214-17 Cohort... 47 The Synoptic Assessment... 47 Pass Rates... 48 Conclusion... 49 MATSEC Support Unit, University of Malta +36 234 2814//6, matsec@um.edu.mt
2 List of Tables Table 1: Students, schools and external verifiers for SEC vocational subjects by year... 8 Table 2: Number of Vocational Subjects offered by Schools... 8 Table 3: Reported numbers of very low ability students per class... 13 Table 4: Subject Choice by Gender 214-17 Cohort... 37 Table : Subject Choice by Gender 21-18 Cohort... 38 Table 6: Subject Choice by School Sector 21-18 Cohort... 38 Table 7: Candidate Attainment by Gender 21-18 Cohort... 38 Table 8: Candidate Attainment by School Sector 218 Cohort... 39 Table 9: Average, Skewness and Kurtosis for the Distribution of Marks in Tasks 1, 2 and 3 21-18 Cohort... 46 Table 1: Average, Skewness and Kurtosis for the Distribution of Marks in Tasks 1, 2 and 3 217 Cohort... 47 Table 11: Eligibility, Registrations and Attainment in the Synoptic Assessments... 48 Table 12: Pass Rates for SEC Vocational Subjects... 48 List of Figures Figure 1: Questionnaire Respondents... 9 Figure 2: Response to item 'Vocational subjects are perceived as having a lower status than more traditional SEC subjects.' (N=86)...1 Figure 3: Response to item 'Candidates are enthusiastic about vocational subjects.' (N=87)...11 Figure 4: Response to item 'SEC vocational subjects were marketed as options for low ability students' (N=86)...12 Figure : Response to item Level 1 and CCP students were not allowed to choose SEC vocational subjects (N=8)...13 Figure 6: Response to item 'Level 1 and CCP students should not be allowed to choose SEC vocational subjects' (N=84)...14 Figure 7: Response to item 'SEC vocational subjects should be marketed as options for low ability students' (N=86)...1 Figure 8: Response to item 'Vocational subjects are turning into dumping grounds for low ability and disengaged students.' (N=8)...1 Figure 9: Response to item 'Students studying vocational subjects feel that SEC vocational subjects are less difficult than other SEC subjects.' (N=8)...16 Figure 1: Response to item 'Assignments 1 and 2 are worked out in class' (N=84)...17 Figure 11: Response to item 'When work is given to candidates to be done at home, they can get unfair help from family members / private tutors' (N=87)...18 Figure 12: Response to item 'All work given to students is part of Assessment 1 or Assessment 2' (N=8)...18 Figure 13: Response to item 'Assignments are divided into tasks so that each assessment criteria is a single task' (N=8)...19 Figure 14: Response to item 'Teaching is done according to the grading criteria: the grading criteria being tackled is made clear to students so that they can fulfil the part/s of the assignment related to it' (N=86)...19 Figure 1: Response to item 'Marking schemes are modified/updated during the marking process' (N=8)'...2 Figure 16: Responses to item Teachers need training to aid them in designing marking schemes.' (N=87)...21
3 Figure 17: Response to item 'All assessments (including the controlled assessment) should be designed by the teachers' (N=87)...22 Figure 18: Response to item 'All assessments should be designed by the MATSEC Support Unit' (N=87)..22 Figure 19: Response to item 'The Syllabus is clear in stating the content expected.'...23 Figure 2: Response to item Students should be given the opportunity to study vocational subjects for their Matriculation Certificate at Advanced Level' (N=87)...24 Figure 21: Response to item Students should be given the opportunity to study vocational subjects for their Matriculation Certificate at Intermediate Level' (N=87)...24 Figure 22: Response to item 'SEC vocational subjects involve more work than traditional SEC subjects.'..2 Figure 23: Response to item 'How is feedback provided to candidates AFTER the assignment has been corrected?...26 Figure 24: Responses to item 'The Masterfile is being updated and maintained by a responsible member of the SMT.'...27 Figure 2: Response to item 'The Masterfile should be the responsibility of the teacher' (N=87)...27 Figure 26: Responses to item 'External verification was professional and helpful.' (N=8)...28 Figure 27: Response to item 'Meetings between teachers, internal verifiers and an SMT member are held on a regular basis' (N=86)...29 Figure 28: Responses to item 'SMT members provide adequate support to SEC vocational subject teachers and internal verifiers.'...3 Figure 29: Responses to item 'Internal verification is in place and beneficial to the teacher and students.' (N=87)...31 Figure 3: Response to item 'The training provided to teachers to teach SEC vocational subjects was sufficient.' (N=88)...32 Figure 31: Response to item 'I feel comfortable contacting the MATSEC Support Unit to discuss any issues.' (N=87)...33 Figure 32: Response to item 'Continuous assessment should be adopted in all other SEC subjects' (N=86)...33 Figure 33: Response to item 'External verification is an additional check that can be done without in all schools' (N=8)...34 Figure 34: Descriptive Statistics 214-17 Cohort...36 Figure 3: Descriptive Statistics 21-18 Cohort...37 Figure 36: Distribution of Marks: All Subjects Task 1 21-18 Cohort...39 Figure 37: Distribution of Marks: All Subjects Task 2 21-18 Cohort...4 Figure 38: Distribution of Marks: All Subjects Task 3 21-18 Cohort...4 Figure 39: Distribution of Marks: Hospitality Task 1 21-18 Cohort...41 Figure 4: Distribution of Marks: Hospitality Task 2 21-18 Cohort...41 Figure 41: Distribution of Marks: Hospitality Task 3 21-18 Cohort...42 Figure 42: Distribution of Marks: Health and Social Care Task 1 21-18 Cohort...42 Figure 43: Distribution of Marks: Health and Social Care Task 2 21-18 Cohort...43 Figure 44: Distribution of Marks: Health and Social Care Task 3 21-18 Cohort...43 Figure 4: Distribution of Marks: Engineering Technology Task 1 21-18 Cohort...44 Figure 46: Distribution of Marks: Engineering Technology Task 2 21-18 Cohort...44 Figure 47: Distribution of Marks: Engineering Technology Task 3 21-18 Cohort...4 Figure 48: Distribution of Marks: Information Technology Task 1 21-18 Cohort...4 Figure 49: Distribution of Marks: Information Technology Task 2 21-18 Cohort...46 Figure : Distribution of Marks: Information Technology Task 3 21-18 Cohort...46 Figure 1: Total mark obtained in Year 1 VS Total mark obtained in Year 9-217 Cohort...47
4 AM Advanced Matriculation List of Abbreviations IM Intermediate Matriculation MATSEC Matriculation and Secondary Education Certificate Examinations Board (MATSEC Board) PSAO Principal Subject Area Officer SBA School Based Assessment SEC Secondary Education Certificate SMT School Management Team UoM University of Malta V-AGRI SEC Vocational Subject Agribusiness V-ENTE SEC Vocational Subject Engineering Technology V-INTE SEC Vocational Subject Information Technology V-HESC SEC Vocational Subject Health and Social Care V-HOSP SEC Vocational Subject Hospitality
Executive Summary SEC vocational subjects have been rolled out in the scholastic year 214/1 such that, at the time of collecting evidence for this report, there are two cohorts following such subjects in Maltese secondary schools. This report aims to shed light on the practices, strengths, and limitations of the project by analysing data obtained from external verification visits and reports, unstructured focus groups and/or interviews with teachers, verifiers, SMT members and candidates, and candidate registrations and achievement. The number of schools offering SEC vocational subjects has increased from eight to 3 and half of these are offering more than one SEC vocational subject. The number of candidates following such subjects has increased six-fold and, as a result, the number of external verifiers has also increased from five to 19. When compared with registrations in the previous year, 21/16 registrations were less male dominated and better spread across the three school sectors. SEC Agribusiness and SEC Engineering Technology, however, remain exclusively male subjects. Nevertheless, female candidates outperform male candidates in SEC vocational subjects. 18 of the 32 candidates following SEC vocational subjects have examination access arrangements, a rate of 2.3% which is roughly double that for other SEC subjects. Candidates of the 21-18 cohort, on average, performed best in Task 2 and worst in Task 3 (controlled assessment) of Unit 1. The mark distributions for Task 3 are usually less negatively skewed, showing a more normal distribution of attainment. Differences between subjects, however, cannot be ignored. Candidates of the 214-17 cohort performed similarly in Year 9 and Year 1 with a medium to high strength correlation between achievements in the two years. The majority of candidates who fail to obtain a mean mark higher than % through the three tasks also fail the synoptic test. Several candidates did not apply to sit for the synoptic test even if they were eligible to do so, but they are eligible to sit for the synoptic test next year. Most respondents do not feel that the perception that SEC vocational subjects are easier than other SEC subjects exists, although a sizeable number of participants do so (4.6%). Similarly, while some candidates did cite reasons of more accessible assessment ( no examinations ) and generalised knowledge content as reasons for opting for vocational subjects, the majority of students cited reasons related to particular career paths and knowledge which interests them. In some schools and subjects, candidates family background seemed to play an important role in shaping their choice as candidates guardians had jobs related to the vocational subject of their children s choice. While in most schools the candidate population seems heterogeneous in terms of ability, in a few schools it seems that only low ability students opted for the subject. Similarly, 27.9% of survey respondents feel that SEC vocational subjects were marketed for low ability students by citing subjects high percentage pass rates and aim in minimizing early school leaving. This situation seems limited to a small number of schools. However, most questionnaire respondents (4.8%), excluding teachers (2.%), do feel that low ability students should be allowed to follow vocational subjects. Teachers and students speak positively of each other as they believe the other part to be motivated in teaching/learning the subject. A number of teachers complained of existing paperwork in vocational subjects, however none of the paperwork was labelled as unnecessary. Most survey respondents (89.%) believe that teaching SEC vocational subjects involves more work than other SEC subjects. It seems that Masterfiles, which should be the responsibility of an SMT member responsible for vocational subjects, are being constructed and maintained by teachers in most instances. In addition, meetings between SMT members and teachers regarding vocational subjects are not scheduled, although most respondents feel that the support provided by SMT members is adequate
6 (76.2%) and that the ad hoc meetings held are regular enough (73.3%). In a few cases, essential actions pointed out in previous external verification reports were not acted upon. This suggests that the level of involvement by SMT members varies between schools but has been taken too lightly in some. Differences between schools are considerable to an extent that they might be unfair on candidates. For instance, MATSEC protocols regulating submission of assignments and deadlines seem to have been disregarded in a particular school while in another school these regulations were strictly adhered to as means of promoting certain values (ownership of one s education and eroding a perceived laissez-faire attitude). This transpired through teacher and student interviews. While most respondents (81.%) claimed that assignments are not done entirely at school, others argued that if such practice is not adopted some candidates will not do their work. While this can be seen as discouraging student impendence and ownership of education, the uneven playing field (help from parents and/or private tutors) created when candidates do their work at home cannot be ignored. Irregularities observed in last year s report concerning feedback given to candidates before submission of the assignment seem to have been addressed. Teachers and students claim that only verbal feedback is being given before actual submission. Most teachers claimed to use more than a single form of feedback and students are generally content with the detailed feedback received from their teachers. However, some teachers complained about the amount of work involved in providing feedback and some external verifiers noted vague feedback on assignment front sheets, so much so that in some instances identical feedback was given to different candidates. While in other SEC subjects it is usual practice for parts of the syllabus to be assessed numerous times before the final assessment, in some instances of SEC vocational education, the only work that candidates are assigned is for one s assignment. This practice is not only unfair considering the work done by candidates studying the same subject in different schools, but undermines mastery of the concepts. It is important that teachers are guided into adopting practices which ensure an assessment that is really conducive for learning. Most respondents (79.3%) agree that the centrally set controlled assessment does serve as an additional measure to enhance validity. However, 46.% of respondents, believe that all assignments should be centrally set, with teachers being split exactly half with regard to this. Some argued for centrally set assignments also as a means of further detailing the expected level to be attained by candidates. In fact, a sizeable minority of respondents (3.6%) believe that MATSEC syllabi are not clear enough in stating the subject content expected. Some problems with the construction and use of marking schemes persist as 44.7% of survey respondents claim that marking schemes are not updated during the marking process while 8.% agree that teachers need further training on constructing marking schemes. Moreover, a number of participants reported problems in teachers preparation in the content knowledge required to teach vocational subjects, with 42.% believing that insufficient training was offered. The process of external verification is a fruitful one. Only two teachers, of the 8 participants who answered this question, disagreed that external verification was professional and helpful. On the other hand, while 93.1% of survey respondents agree that internal verification is in place and functioning properly, a number of appalling shortcomings were found. Interviewees could identify the MATSEC official responsible for SEC vocational subjects and 82.8% of survey participants claimed to be comfortable in contacting the MATSEC Support Unit to discuss any difficulties. Mentors appointed by the DQSE were also deemed beneficial by most interviewees except
7 for a couple of severe shortcomings. At the time of interviewing, teachers were unaware of any education officer in charge of vocational subjects within the DQSE. Most teachers and students view continuous assessment as having a positive impact on education. However, some teachers pointed at increased workloads while some students pointed at enhanced stress-levels which would be caused by continuous high-stakes assessments. Professionals in schools are worried about the outcomes of the Learning Outcomes Framework project, fearing that a barrage of change will be unleashed upon them. 9.4% of survey respondents believe that students should be able to follow vocational subjects as part of their Matriculation Certificate.
8 Introduction The aim of this report is to present how the introduction of SEC vocational subjects in schools are developing and to identify and reflect upon some of the strengths and weaknesses of the implementation process. For this, it condenses several stances taken by teachers, SMT members, students, internal verifiers and external verifiers as collected from interviews during external verification visits and external verification reports along with information about candidates attainment. General Information This is the second year of SEC vocational subjects in Maltese secondary schools. Following national roll out, the number of schools offering these subjects has increased from eight to 3. Consequently, the number of candidates opting for such subjects has increased nearly six-fold, from 92 to 32, of which 11 candidates are studying two SEC vocational subjects. As a result, the number of external verifiers has also increased, as shown in the table below. Table 1: Students, schools and external verifiers for SEC vocational subjects by year Year 214/1 Year 21/16 Percentage Increase Students 92 32 478 % Schools 8 3 27 % External Verifiers 18 26 % The rate of increase of schools offering vocational subjects is smaller than the rate of increase of students following such subjects. This is because half of the 3 schools have opted to offer more than one vocational subject, as shown in the table below. Table 2: Number of Vocational Subjects offered by Schools Number of Vocational Subjects Offered Number of Schools 1 1 2 12 3 1 4 2 Of these candidates, 18 candidates (2.3%) have examination access arrangements. This is roughly double the rate of candidates being granted examination access arrangements in other SEC subjects. Methodology This report follows on the report published last year by the MATSEC Support Unit titled Vocational Subjects at SEC Level: Introduction, Implementation, and Results. Such report is available on the MATSEC website along with other reports published by the MATSEC Support Unit. Thus, the same research question is used: What are the advantages and disadvantages in the current design and practice of SEC vocational subjects, what can be improved, and what are the implications of this project on various parts of the Maltese educational system? This study is important to ensure that students are attended to in the best possible way, while respecting the professionalism of educators and the existing standards of SEC qualifications.
9 Although the numbers of individuals involved in the project have substantially increased, a quasiidentical research methodology was adopted. Data was collected through an analysis of the following: 1. Observation of external verification sessions for a number of SEC vocational subjects which included a. Relatively unstructured focus groups with teacher/s, internal verifier/s, external verifier, and candidates; b. Relatively unstructured interviews with Head of Schools and/or SMT members responsible for the subjects; 2. All external verification reports; 3. Questionnaires, based on the data obtained from 1 and 2 above, distributed to all teachers, internal verifiers, external verifiers and heads of schools; and 4. Candidate achievement in the different assessments. Questionnaires were sent to Head of Schools in order to be passed on the responsible SMT members, teachers and internal verifiers and to external verifiers. A total of 88 individuals submitted the questionnaire, mostly teachers (4%). External Verifier, 1, 11% SMT Member, 23, 26% Teacher, 47, 4% Internal Verifier, 8, 9% Figure 1: Questionnaire Respondents While the exact number of teachers and internal verifiers in the project is unknown by the MATSEC Support Unit (these fall under the responsibility of the DQSE), there were 3 schools/colleges involved in this project. Assuming one SMT member per school responsible for SEC Vocational Subjects, and that only this responsible SMT member from each school filled in the questionnaire, the response rate seems to be quite high at 76.7%. There were 18 external verifiers nominated by the MATSEC Examinations Board of which 1 (.%) filled in the questionnaire. The data collection and analysis and compiling of this report has been carried out by Mr. Gilbert John Zahra (PSAO Assessment Research and Development). All this was done in constant liaison with other members of the MATSEC Support Unit, mainly Mr. Malcolm Micallef (PSAO Vocational Subjects), Mr. Dario Pirotta (Director) and Prof. Frank Ventura (Chairperson of the MATSEC Examinations Board).
Number of Respondents 1 Survey Findings Perception and Students Choice Vocational subjects, many respondents claimed, involve a big change from other SEC subjects: They are more applied and practical. Some teachers claimed that these subjects offer an alternative to students who might not be academically oriented, thus encouraging them to pursue formal education after the end of compulsory school. Both the students studying vocational subjects and their teachers noted that the perception that these subjects are easy persists amongst peers and parents. A considerable number of questionnaire respondents (4.6%) believe that SEC vocational subjects are perceived as having a lower status than other SEC subjects. SMT members usually disagreed with this statement while external verifiers were more likely to agree. 2 18 16 14 12 1 8 6 4 2 Agree Agree Teacher 6 1 18 6 Internal Verifier 1 2 4 1 SMT Member 4 1 4 External Verifier 3 4 2 1 [Vocational subjects are perceived as having a lower status than more traditional SEC subjects.] Figure 2: Response to item 'Vocational subjects are perceived as having a lower status than more traditional SEC subjects.' (N=86) Students reasons to opt for vocational subjects varied per school. In one Engineering Technology class, it was remarkable how all learners had a role model (father, uncle, grandfather) working in the field of Engineering, usually mechanical. This was not the case in other schools though, and students expressed that they chose the subject because of particular career paths or simply out of interest in the subject. In addition, some candidates claimed to have chosen the subject because of its mode of assessment. From these, some claimed that they can learn more this way, while others were happy to have no examinations (SEC vocational candidates have a controlled assessment at the end of each scholastic year) with one student emphasising that the vocational subject he chose is less demanding than Biology (the candidate s teacher teaches biology within the same school, thus, the candidate s remarks could be either due to comparison with the Biology class or a reflection of the teacher s own conviction). In the same school, students remarked how the vocational subject studied (Health and Social Care) offers a wide range of content rather than delving deeply into one area (e.g. science).
Number of Respondents 11 Student Ability and Motivation Most teachers claimed that cohorts are formed of a mixture of students: a few high achievers opted for vocational subjects. A few very low achievers also opted for vocational subjects. Most students are hard-working and exhibit high levels of interest. They spoke very positively about the subject being studied and, more so, about their teachers. Some teachers questioned whether these high levels of motivation will be maintained with students as they proceed throughout the years, but a teacher teaching such class claimed that students attitude, whether positive or negative, persisted in Year 1 (Form 4). The vast majority of questionnaire respondents believe that students are very interested in the subject. 3 2 2 1 1 Agree Agree Teacher 17 27 2 Internal Verifier 1 7 SMT Member 7 16 External Verifier 6 4 [Candidates are enthusiastic about vocational subjects.] Figure 3: Response to item 'Candidates are enthusiastic about vocational subjects.' (N=87) An official letter by the DQSE suggested that students be selected according to marks in related subjects. However, this practice was not carried out in visited schools. One SMT member in a school explained that since less than fifteen students opted for the subject, every student was allowed to pursue his/her studies in the subject. In the same letter, schools were also informed that vocational subjects are not suitable for Level 1 and Core Competence Programme (CCP) students. However, an SMT member in one school perceived this as a contradictory message as s/he claimed the subjects were publicised to have a very high pass-rate and as interesting to those who would otherwise become early school leavers. In total, 27.9% of respondents feel that SEC vocational subjects were marketed for low ability students.
Number of Respondents 12 2 2 1 1 Agree Agree Teacher 3 11 22 1 Internal Verifier 1 2 SMT Member 3 1 External Verifier 1 3 [SEC vocational subjects were marketed as options for low ability students.] Figure 4: Response to item 'SEC vocational subjects were marketed as options for low ability students' (N=86) As a result (or not), some students chose the subject thinking that it is easy, even if teachers advised them to do otherwise based on, for instance, their inability to communicate in written English. In a few schools, it seemed that only low ability students choose vocational subjects. In one particular school, some students complained about the subject being chosen by low-ability students while some teachers are looking into ways of increasing access to candidates with learning difficulties. In one school, SMT members and teachers claimed that more academically oriented students do not go for vocational subjects not because these are easier or undemanding, but because they are interested in an academic stream. Survey data seems to confirm that in most cases very low ability students were allowed to choose SEC vocational subjects.
Number of Respondents 13 2 18 16 14 12 1 8 6 4 2 Agree Agree Teacher 7 18 14 Internal Verifier 1 2 3 2 SMT Member 4 6 9 3 External Verifier 4 2 [Level 1 and CCP students were not allowed to choose SEC vocational subjects.] Figure : Response to item Level 1 and CCP students were not allowed to choose SEC vocational subjects (N=8) An analysis of questionnaire responses suggests that the issue of very low ability students opting for SEC vocational subjects is not widespread. When asked How many, if any, Track 1 or CCP students (low ability students who are not expected to obtain traditional SEC qualifications) are there in your class?, the majority of teachers responded that there were none at all while an analysis of responses suggested there is an average of 1. very low ability students per teacher. Nevertheless, there were two reported cases where the majority of students (/9 and 8/11) were very low ability students. Some respondents did not provide the number of students in their class and, thus, it was impossible to calculate whether these constitute a majority or not. Table 3: Reported numbers of very low ability students per class Number of low ability students per class No Reply 7 I'm not sure. 1 1 2 8 3 2 4 2 1 6 1 7 8 1 Class size varies considerably, ranging from 2 to 17 with a mode of 9 and a mean of 9.9 students per class. Teachers deal with the heterogeneous groups of students in class mainly through the application of differentiated activities and explanations that target different learning styles. However,
Number of Respondents 14 as one teacher pointed out, although we try very hard to push the students to their possible limits ( ), there is so much we can do and so much the student can achieve. Thus, a good number of teachers mentioned the importance of learning support assistants in their class. A good number of teachers also find time for individual explanation and one-to-one tutorials. From questionnaire respondents, teachers were the only group to feel that Level 1 and CCP students should not be allowed to opt for SEC vocational subjects. Nevertheless, nearly all questionnaire respondents (91.9%) agreed that these should not be marketed as options for low ability students. 16 14 12 1 8 6 4 2 Agree Agree Teacher 14 12 13 Internal Verifier 1 3 3 1 SMT Member 2 6 12 3 External Verifier 4 [Level 1 and CCP students should not be allowed to choose SEC vocational subjects.] Figure 6: Response to item 'Level 1 and CCP students should not be allowed to choose SEC vocational subjects' (N=84)
Number of Respondents Number of Respondents 1 3 3 2 2 1 1 Agree Agree Teacher 3 11 31 Internal Verifier 3 SMT Member 2 13 8 External Verifier 2 4 4 [SEC vocational subjects should be marketed as options for low ability students.] Figure 7: Response to item 'SEC vocational subjects should be marketed as options for low ability students' (N=86) Questionnaire respondents, in general (76.%), do not feel that SEC vocational subjects are turning into dumping grounds for low ability and disengaged students. However, the 23.% who think they are, should not be ignored. 2 2 1 1 Agree Agree Teacher 7 22 11 Internal Verifier 2 4 2 SMT Member 2 12 8 External Verifier 1 3 1 [Vocational subjects are turning into dumping grounds for low ability and disengaged students.] Figure 8: Response to item 'Vocational subjects are turning into dumping grounds for low ability and disengaged students.' (N=8)
Number of Respondents 16 Displacement from Other Subjects In some schools, the introduction of some vocational subjects caused a displacement of students from other areas. For example, the introduction of Engineering Technology in one school resulted in half the usual number of students opting for Design and Technology while the introduction of Information Technology in another school resulted in half the usual number of students opting for Computing. In another school, when two vocational subjects (Hospitality and IT) were introduced, the number of learners opting for SEC Physical Education dropped to zero. The school management related this to a displacement of those looking for the easiest possible SEC subject. However, questionnaire respondents generally disagreed that students find SEC vocational subjects easier than other SEC subjects. 3 2 2 1 1 Agree Agree Teacher 4 11 28 2 Internal Verifier 1 3 3 1 SMT Member 1 6 13 2 External Verifier 3 6 1 [Students studying vocational subjects feel that SEC vocational subjects are less difficult than other SEC subjects.] Figure 9: Response to item 'Students studying vocational subjects feel that SEC vocational subjects are less difficult than other SEC subjects.' (N=8) The Assignments In a few instances, teachers praised the good practice of maintaining strict deadlines so that students do not slacken their commitment. In one particular school, the Quality Assurance Nominee and teachers complained about a generation of students expecting everything to be ready-made for them. They claimed that strict deadlines and other measures in vocational subjects give candidates ownership and responsibility for their learning. Through interviews with teachers and students, however, it seemed that in another school candidates were given extensions without proper documentation. In some schools, the majority of work is done in class with the teachers presence. Teachers in these schools held that students would not do well if the work had to be done on their own at home, either because they simply would not do it or because they have difficulties expressing themselves in English. In an independent school, teachers complained that students might be getting extra help from home or from private tutors. Although the majority of respondents (81.%) claimed that work is not done
Number of Respondents Number of Respondents 17 entirely at school, most respondents (6.%) also agreed that doing work at home might result in an unfair advantage to some candidates. Teachers participating in the survey (84.4%) disagree that Assignment 1 and Assignment 2 be done at school, while SMT members and external verifiers are split in two groups about this. 3 3 2 2 1 1 Agree Agree Teacher 1 3 3 11 Internal Verifier 1 6 1 SMT Member 2 7 1 2 External Verifier 2 7 1 [Assignments 1 and 2 are worked out in class.] Figure 1: Response to item 'Assignments 1 and 2 are worked out in class' (N=84) 3 2 2 1 1 Agree Agree Teacher 7 2 14 Internal Verifier 6 2 SMT Member 3 1 9 1 External Verifier 1 4 [When work is given to candidates to be done at home, they can get unfair help from family members / private tutors.]
Number of Respondents 18 Figure 11: Response to item 'When work is given to candidates to be done at home, they can get unfair help from family members / private tutors' (N=87) Questionnaire responses suggest that many times all work given to students is part of Assessment 1 or Assessment 2. This might contradict practice in other SEC subjects where teachers usually assess students on the same skill several times to try and ensure mastery before a final assessment. 18 16 14 12 1 8 6 4 2 Agree Agree Teacher 8 17 16 Internal Verifier 3 SMT Member 6 7 7 1 External Verifier [All work given to students is part of Assessment 1 or Assessment 2.] Figure 12: Response to item 'All work given to students is part of Assessment 1 or Assessment 2' (N=8) Most respondents agreed that assignments are divided into tasks so that each assessment criteria is a single task (69.4%) and that teaching is done according to the grading criteria: the grading criteria being tackled is made clear to students so that they can fulfil the part/s of the assignment related to it (67.4%).
Number of Respondents Number of Respondents 19 18 16 14 12 1 8 6 4 2 Agree Agree Teacher 11 16 14 Internal Verifier 1 4 2 1 SMT Member 13 2 1 External Verifier 2 7 1 [Assignments are divided into tasks so that each assessment criteria is a single task.] Figure 13: Response to item 'Assignments are divided into tasks so that each assessment criteria is a single task' (N=8) 16 14 12 1 8 6 4 2 Agree Agree Teacher 13 1 1 3 Internal Verifier 2 3 2 1 SMT Member 7 11 4 External Verifier 7 1 2 [Teaching is done according to the grading criteria: the grading criteria being tackled is made clear to students so that they can fulfill the part/s of the assignment related to it.] Figure 14: Response to item 'Teaching is done according to the grading criteria: the grading criteria being tackled is made clear to students so that they can fulfil the part/s of the assignment related to it' (N=86)
Number of Respondents 2 Along with the assignments, teachers must also design marking schemes and update them throughout the marking process. However, updating of marking schemes is absent according to 44.7% of questionnaire respondents. 2 2 1 1 Agree Agree Teacher 7 21 13 Internal Verifier 3 SMT Member 3 6 9 3 External Verifier 2 3 4 1 ['Marking schemes are modified/updated during the marking process.] Figure 1: Response to item 'Marking schemes are modified/updated during the marking process' (N=8)' Through school visits, it seemed that some problems with marking schemes have persisted. This also led to marking being unclear in some schools. It seems that a good number of professionals require help in understanding what a marking scheme should be and how to construct one. Survey responses confirm this as 8.% of respondents agree that teachers need further training on how to construct marking schemes.
Number of Respondents Number of Respondents 21 2 2 1 1 Agree Agree Teacher 13 23 9 1 Internal Verifier 2 6 SMT Member 7 11 External Verifier 4 4 2 [Teachers need training to aid them in designing marking schemes.] Figure 16: Responses to item Teachers need training to aid them in designing marking schemes.' (N=87) As observed at the beginning of the SEC vocational subjects pilot project and after the 214/1 scholastic year analysis, the majority of questioned individuals (79.3%) are of the idea that the controlled assessment be designed by the MATSEC Support Unit. This ensures an additional check for validity. However, a little less than half of participants (46.%) believe that all assignments should be designed by the MATSEC Support Unit. 2 2 1 1 Agree Agree Teacher 3 6 21 16 Internal Verifier 7 1 SMT Member 3 11 4 External Verifier 1 8 1 [All assessments (including the controlled assessment) should be designed by the teachers.]
Number of Respondents 22 Figure 17: Response to item 'All assessments (including the controlled assessment) should be designed by the teachers' (N=87) 2 2 1 1 Agree Agree Teacher 1 8 2 3 Internal Verifier 1 3 4 SMT Member 3 12 3 External Verifier 2 3 [All assessments should be designed by the MATSEC Support Unit.] Figure 18: Response to item 'All assessments should be designed by the MATSEC Support Unit' (N=87) Syllabi A number of teachers claimed that syllabi lack detail and examples in specifying the expected depth. After the MATSEC Examinations Board approved new syllabi that were to address, among other facets, issues of inconsistency between schools, some teachers find it very difficult to create an assignment based purely on the syllabus as it is too limited and there is no flexibility. Some teachers have asked for the possibility of sample assignments while one teacher questioned whether assignments should be set centrally by MATSEC. Others argued that some grading criteria are overlapping. From the questionnaire respondents, 3.6% believe that MATSEC syllabi are not clear in stating the content expected.
Number of Respondents 23 2 2 1 1 Agree Agree Teacher 4 22 14 6 Internal Verifier 4 3 1 SMT Member 14 3 1 External Verifier 1 6 3 [The Syllabus is clear in stating the content expected.] Figure 19: Response to item 'The Syllabus is clear in stating the content expected.' Since local syllabi lack a textbook designed for them, students and teachers as also evident through focus group interviews use textbooks designed for BTEC syllabi. Although other resources are used, mostly that obtainable freely over the internet, the current practice involved in the teaching of SEC vocational subjects is, to a certain extent, not completely divorced from BTEC curricula. When prompted through the questionnaire, nearly all respondents agreed that students should be able to study vocational subjects for their Matriculation Certificate at either advanced or intermediate level.
Number of Respondents Number of Respondents 24 3 2 2 1 1 Agree Agree Teacher 27 16 3 Internal Verifier 6 2 SMT Member 11 12 External Verifier 6 3 1 [Students should be given the opportunity to study vocational subjects for their Matriculation Certificate at Advanced Level.] Figure 2: Response to item Students should be given the opportunity to study vocational subjects for their Matriculation Certificate at Advanced Level' (N=87) 3 2 2 1 1 Agree Agree Teacher 27 1 4 Internal Verifier 7 1 SMT Member 13 1 External Verifier 6 4 [Students should be given the opportunity to study vocational subjects for their Matriculation Certificate at Intermediate Level.] Figure 21: Response to item Students should be given the opportunity to study vocational subjects for their Matriculation Certificate at Intermediate Level' (N=87)
Number of Respondents 2 Paperwork Although the majority of teachers noted the large amount of paperwork that they are expected to do, they were unable, when asked, to identify paperwork that is unnecessary. In only one instance where teachers were able to suggest one such reduction, pointing that the assessment decision record of internal verification could be substituted by a signature of the internal verifier on the assessment brief front sheet. Some teachers claimed that each assignment is an examination and involves a lot of work to mark and prepare. In a handful of schools, team teaching is being adopted to make teachers load more manageable. In another school, however, teachers claimed that although at the beginning paperwork seemed overwhelming, it turned out to be easily manageable. Besides, they noted, it aids the existing good practice of maintaining good records. The figure below shows that 89.% of questionnaire participants believe that SEC vocational subjects involve more work than other SEC subjects. 3 3 2 2 1 1 Agree Agree Teacher 29 11 6 Internal Verifier 2 1 SMT Member 12 8 2 External Verifier 4 6 [SEC vocational subjects involve more work than traditional SEC subjects.] Figure 22: Response to item 'SEC vocational subjects involve more work than traditional SEC subjects.' Assignment Feedback Teachers claimed that only verbal feedback is given to candidates before the submission of assignments. Some said that they review assignments with students who ask for such help. Teachers claimed that they are not taking students assignments home before actual submission and a teacher said that when e-mails are sent s/he replies to them verbally at school. Interviewed candidates confirmed this, saying that although the teacher was available to help them, s/he never gave them too much information: they offered guidance and support, but not the answer itself. This seemed to be the case in all visited schools. Written feedback should then be provided once the assignment has been corrected. However, five respondents claimed not to provide feedback on the assignment front sheet. Some teachers lamented about the amount of work involved in writing feedback for each candidate. Students, on the other
Number of Respondents 26 hand, were usually content with this level of feedback claiming, in a particular school, that this allows them to further their attainment. In some schools, however, external verifiers lamented that the level of feedback provided to candidates lacked detail, so much so that in one particular school the same feedback was given to candidates irrespective of their performance and attainment. Teachers rarely limit themselves to written feedback on the assignment front sheet as the only means of providing feedback to learners. Of the respondents, 1 teachers adopt only one method of providing feedback to candidates while 3 adopt three or four different methods. Responses are summarised in the figure below. 4 4 3 3 2 2 1 1 41 Written feedback on the assignment frontsheet 36 24 Verbal feedback to General feedback to individual students the class Feedback After the Assigment has been Corrected 14 Whole class discussion about general strengths/difficulties Figure 23: Response to item 'How is feedback provided to candidates AFTER the assignment has been corrected? Masterfiles, in addition, are being constructed and maintained by teachers. These were generally of a very high standard, although in some instances these had missing documentation. Two of the most common missing pieces of documentation were entries for Health and Safety and Conflict of Interest, even though the latter is present in the policy document issued by the MATSEC Support Unit. Slightly more than half questionnaire respondents who were teachers or internal verifiers (1.9%), pointed out that Masterfiles are not being maintained by SMT members. Most SMT members (91.3%) held a divergent opinion.
Number of Respondents Number of Respondents 27 14 12 1 8 6 4 2 Agree Agree Teacher 12 11 1 13 Internal Verifier 2 1 SMT Member 9 12 1 1 External Verifier 3 3 3 1 [The Masterfile is being updated and maintained by a responsible member of the SMT.] Figure 24: Responses to item 'The Masterfile is being updated and maintained by a responsible member of the SMT.' When asked whether Masterfiles should be responsibility of teachers, most participants agreed (8.6%). 16 14 12 1 8 6 4 2 Agree Agree Teacher 12 14 11 9 Internal Verifier 1 2 SMT Member 4 12 7 External Verifier 6 2 2 [The Masterfile should be the responsibility of the teacher.] Figure 2: Response to item 'The Masterfile should be the responsibility of the teacher' (N=87)
Number of Respondents 28 Professional Support The vast majority of respondents believe that the process of external verification is a fruitful one. Only two teachers, of the 8 participants who answered this question, disagreed that external verification was professional and helpful. It would be ideal if such individuals could have been interviewed to seek reasons for the issue, however, this was not possible due to the anonymity of the survey. 3 2 2 1 1 Agree Agree Teacher 2 24 1 1 Internal Verifier 3 SMT Member 8 14 External Verifier 4 [External verification was professional and helpful.] Figure 26: Responses to item 'External verification was professional and helpful.' (N=8) In most cases, teachers do not hold regular, timed meetings with SMT members regarding vocational subjects. Rather, as happens in other subjects, they seek help on an ad hoc basis. Nevertheless, most questionnaire respondents felt that the meetings are regular, with 26.7% of participants disagreeing with this.
Number of Respondents 29 2 18 16 14 12 1 8 6 4 2 Agree Agree Teacher 12 19 6 8 Internal Verifier 2 4 2 SMT Member 14 4 External Verifier 7 3 [Meetings between teachers, internal verifiers and an SMT member are held on a regular basis.] Figure 27: Response to item 'Meetings between teachers, internal verifiers and an SMT member are held on a regular basis' (N=86) The level of involvement by SMT members varies between schools. In a few cases, essential actions pointed in previous external verification reports were not acted upon. Thus, in a number of schools, the position of Quality Assurance Nominee seems to have been taken quite lightly. Questionnaire responses, however, suggest that 76.2% of respondents feel that SMT members do offer sufficient support to teachers. Nevertheless, the 23.8% that disagree with such statement should not be taken lightly. The level of SMT support to teachers should be increased.
Number of Respondents 3 18 16 14 12 1 8 6 4 2 Agree Agree Teacher 16 16 8 Internal Verifier 1 2 SMT Member 9 1 2 External Verifier 2 3 [SMT members provide adequate support to SEC vocational subject teachers and internal verifiers.] Figure 28: Responses to item 'SMT members provide adequate support to SEC vocational subject teachers and internal verifiers.' In one particular school, the first assignment brief was only released by the internal verifier after being changed a total of five times by the teacher. This suggested that internal verification is in place, at least in this school. In this school the internal verifier is external to the school, acting also as a mentor. External verifiers reported problems with internal verification in a good number of schools. In some cases, internal verifiers failed to note obvious defects, like missing deadline dates and other details that should be stated. An external verifier linked the problems with internal verification to the friendship between teachers since they are close colleagues within the same school. On the other hand, it was noted by another external verifier that internal verification occurs less smoothly when it is not in-house. In some schools, no internal verification had been carried out by the date of external verification. In others, the first assignment had not yet been given to students. Questionnaire responses, however, suggest that the practice of internal verification is mostly in place with only one external verifier and five teachers arguing that it is not.
Number of Respondents 31 2 2 1 1 Agree Agree Teacher 21 2 4 1 Internal Verifier 2 6 SMT Member 13 1 External Verifier 3 6 1 [Internal verification is in place and beneficial to the teacher and students.] Figure 29: Responses to item 'Internal verification is in place and beneficial to the teacher and students.' (N=87) All teachers noted that help from mentors appointed by the DQSE was generally beneficial. Most praised the help and support they receive from these individuals. A particular subject s teachers in one school reported that the mentor of another vocational subject analysed their file and suggested things contradicting their own mentor s, specifically that assignment front sheets be put in a separate section than the assignments themselves. In yet another school, the external verifier pointed at defects in the assignment brief which the teacher claimed were suggested by the mentor. Other teachers complained about the lack of an Education Officer for state schools and Head of Department for non-state schools. Although an EO for vocational subjects was present at the time of external verification, no teacher or SMT member made reference to this. A good number of external verifiers noted a need to invest in teachers professional development, especially in subject content and technical training: 2 half days in workshop training is definitely not enough. A number of teachers also felt insufficiently prepared to teach the subjects. They claimed to rely mostly on personal research and have to deal with uncertainty. However, most teachers replying to the questionnaire held that they are sufficiently prepared to teach SEC vocational subjects. While SMT members mostly agreed with this view, verifiers, both internal and external do not. Most external and internal verifiers highlighted that teachers need more training especially regarding the application criteria. Overall, 8.% of respondents agreed or strongly agreed that teachers require more training with regards to the setting of marking schemes.
Number of Respondents Number of Respondents 32 18 16 14 12 1 8 6 4 2 Agree Agree Teacher 1 16 13 7 Internal Verifier 3 2 4 SMT Member 1 17 4 1 External Verifier 4 1 [The training provided to teachers to teach SEC vocational subjects was sufficient.] Figure 3: Response to item 'The training provided to teachers to teach SEC vocational subjects was sufficient.' (N=88) Nearly all teachers knew who to contact from the MATSEC Support Unit regarding SEC vocational subjects and perceived the MATSEC Support Unit to be approachable. Teachers and SMT members who contacted the MATSEC Support Unit were satisfied with the help they received. This was confirmed through the questionnaire, as shown in the figure below. All external verifiers and SMT members are comfortable contacting the MATSC Support Unit to discuss any issues. 3 2 2 1 1 Agree Agree Teacher 9 26 8 3 Internal Verifier 1 3 4 SMT Member 6 17 External Verifier 8 2 [I feel comfortable contacting the MATSEC Support Unit to discuss any issues.]