Training & Reliability Procedures for MELE MELQO experts and observer-trainees will work together to complete the reliability procedures. Timeline and steps to reliability include: PRE-TRAINING ACTIVITIES I. Confirm training strategy and timeline for pre-training and training activities. A certified MELQO expert will train country level observers (including supervisors, etc.) directly in country. This should be in agreement with the implementation strategy and activities. II. Based on the training strategy, ensure training period allows adequate time for training and standardization of observer-trainees (likely to be 5-12 days). This includes consideration of: A. Total sample size, and geographic area. That is, how training can accommodate the entire team of trainees who will need to get standardized on the tools, to carry out the work across the sample area. B. Number of trainers available 1. It is recommended that in-country training include 1 certified trainer per 10 trainees, at a minimum. C. Number of trainees/enumerators needed for completing project (including those that may be dismissed if they do not qualify for the job). 1. This will vary according to sample size, geography, budget, local capacity, etc. 2. It is recommended that the project hire 10%-20% more trainees than needed, and only select those who are reliable at end of training for actual MELE data collection); D. Time needed for actual training of observers: 1. Minimum of 5 days for training enumerators (recommended, but could be more), including: a. Classroom training i Overview ii Review of videos, explanation of codes iii Practice with videos, for coding iv Quizzes, discussion, etc. b. Live observation experience (1-2 mornings) i Observing in classrooms using MELE, videotaped, as possible c. Review of all quizzes, experiences, and reconciliation for moving forward E. Time between training and study start should not be more than 10 days. Longer lapses will require refresher training II. Obtain all English or other translated tools and materials from MELQO expert needed for completing the pretraining processes below III. Translate and back-translate MELE tool and materials (to be done before or during adaptation workshop; see MELQO toolkit for more information, THIS COULD TAKE 2-6 WEEKS TO COMPLETE): A. As possible, two independent translators familiar with ECD in the local context should translate the entire MELE tool B. As possible, two different translators should independently back-translate the entire MELE tool to English C. MELQO expert members will review back-translation and request improved translation, if necessary. D. Re-worded translations will then be back-translated to English, and submitted to the MELQO expert member(s) for review. This may be an iterative process. E. Translation will be finalized by master trainers with MELQO expert input, following decisions made about the MELE tool translations and back-translations IV. Master trainers complete activities provided by MELQO expert 1
A. Pass with 100% score on the Scavenger Hunt. Use of MELE manual is allowed. B. Pass with score 100% on written quiz (based on unadapted core MELE). Master trainers will have 2 chances to pass the quiz. Use of MELE manual is allowed. C. As possible: evaluate master trainers scoring of 2 local or other videos (minimum 10 minutes) across a variety of MELE items. Agreement among master trainers should be >=80%. This will need to be conducted and scored prior to training. V. Create a minimum of five 30-60 minute videos for training (NOTE: as possible, make extra videos for backup, in case those made have technological difficulties or do not cover content sufficiently) A. Select schools and classrooms: a. Contact 3-5 school directors from a variety of programs (private, public, community, etc.) to explain the project and ask permission to film at the school for training purposes. Consent for filming should be obtained b. The class schedules should be discussed with the directors and teachers to ensure that the videos made cover a variety of activities and lessons c. Videos should be conducted in classrooms that teach younger (for example, 3-4 years of age) and older (5-6 years of age) children, or as relevant to the MELQO project within country. 2. Procure appropriate equipment and personnel required for creating videos: a. A digital video recorder with tripod is preferred. Phones and tablets do not always provide high quality video and audio form a distance more than a few feet away from the main subject. b. Batteries and memory cards should be sufficient to complete a minimum of 1 hour of filming c. Headphones (ear buds) for listening to video playback in the field d. An experienced videographer is preferred. If it is not possible to hire an experienced videographer, the individual filming should do the following: i Conduct 2-3 short (5 minute) videos to become familiar with the camera and tripod and various types of lighting (natural light, overhead, etc.) ii These do not have to be videos of classrooms, but should be shot during the daytime, indoors and outdoors, and include people. The camera should be 10 or more feet (3+ meters) from the people being filmed. iii These videos should then be checked for: visual and audio clarity; lighting; optimal camera distance from people being filmed; capacity to move around as necessary to capture activities and interactions, voices, etc. 3. Follow procedures for classroom filming: a. Videographer and study representative should arrive at school 30 minutes before the start of class. If consent was not gotten previously, this should be obtained. During this time, the videographer may film around the room to help gather information on the materials available in the room. b. Camera should be set up in back or to the side of classroom in a place that does not interfere with teacher or children c. Teacher should introduce videographer and remind children that the videographer is visiting, but will not be participating in any activities. Videographer does not interact with teacher of children. d. Videographer conducts a short practice video and plays it back on camera (using headphones): i The angle of the camera must be able to capture the teacher s voice and facial expressions, activities, and interactions with children. ii The teacher s voice must be clearly audible e. Videos should be 30-60 minutes in length f. Videos should be continuous (i.e., not starting and stopping). Transition times between activities are also important for scoring. g. Videographer may have to move in accordance with classroom activities and lessons. 4. Prepare 3 or more videos for teaching about individual items and practicing coding: a. Training videos should cover as many lessons, activities, interactions as possible. 2
b. Videos should include the presentation of short clips for training on specific items or activities as well as some longer segments that include multiple items i Create multiple short (3-5 minute) clips for training on specific items or activities (such as teaching of math concepts; fine motor activity; book reading, etc.) ii Create or demarcate at least 2 continuous clips 15-20 minutes long for individual practice during training. 5. Prepare videos for standardization/ reliability quizzes: a. There should be at least 2 videos, 30-60 minutes in length b. These should be different from those used for training c. The videos should be continuous clips of classroom activities rather than several short clips edited together. These longer clips mimic real life observation, and likely also include periods of transition from one activity to another, which we also want to capture in codings. d. It is suggested to use videos from 2 different types of schools (private and public; rural and urban, etc.) and classroom types (younger and older) as possible 6. Create master codes for video practice and video quizzes a. Two or more master trainers will code the selected videos or video clips independently b. Consensus for each code will be determined c. Master trainers will discuss any codes with <75% consensus and come to an agreement on the master code d. Overall agreement for videos will be >=85% agreement among master trainers e. NOTE: AT LEAST 2 OF THESE VIDEOS WILL BE NEEDED FOR EVALUATING STANDARDIZATION/ RELIABILITY OF TRAINEES VI. Identify schools and contact directors for conducting live observations during training. These can include the same schools used to create videos. VII. Compile all needed materials for training A. Adapted MELE tool and manual in the appropriate language(s) for training. B. Each observer-trainee must have adequate copies of the MELE tool for completing the various practice observations (videos, live, quizzes, etc.) C. Adapted training materials (powerpoint slides, training activities, written quizzes, etc.) D. Videos for discussing specific items; practicing scoring; delivering video quiz(zes). E. Create PICTURES from videos, to use in training on how to observe TRAINING ACTIVITIES VIII. View pictures of classrooms, and gain experience in describing what is seen. The idea is not to code, but just to look and describe: A. What is teacher doing (is she teaching, helping students, supervising, preparing next lesson, etc.) B. What are children doing (participating by group-individual-etc., attentive, etc.) C. What else is observed in the classroom (no use of materials; etc.) IX. Identify open- versus closed-ended questions A. Quiz on identifying each type of question X. Ensure training includes overview of importance of reliability testing for trainees. This includes: A. Discussion of reliability and standardization concepts B. Describing process for gaining reliability: practice with video and live observation; fidelity exercises (e.g., observers carry-out procedures as they should); scavenger and knowledge quizzes; overall evaluation 3
C. Describing process for selecting workers: if trainees will be evaluated (and assigned to jobs based on performance in training), this should be communicated clearly before training begins. XI. Provide practice for scoring items using videos A. Be certain training venue includes capacity to view videos using a projector B. Master scores on practice videos should be completed by master trainers prior to training and available for discussions C. Be certain trainers have videos, and all have scores for all videos available for discussion D. As possible, review 5-10 short (2-4 minute) videos with examples of individual items 1. Encourage discussion on what was seen what teachers were doing, what children were doing, how various items might be scored E. Provide 1-2 15-20 minute videos for practice scoring across many items, followed with discussion 1. Put trainees into groups of 3-5. 2. Show selected video clips that may include several activities that are easy to observe and score (e.g., math lesson, child participation, teacher s individualized instruction). ALL should be scored by master trainers before viewing in training. 3. Allow time for all to score after viewing (5-7 minutes or so) 4. Discuss what was seen. Ask each group to explain the code assigned, and why. 5. Address any ambiguities in item definitions or scoring, and provide examples as necessary to help clarify item and responses 6. Reconcile any disputes in coding XII. Live observations A. All observer-trainees should complete one or more 30-minute observations in classroom. B. No more than 5 observer-trainees plus 2 trainers/ supervisors per classroom (this depends on classroom size, crowding, etc., and must be agreed upon prior to arrival for observation) C. Trainers and supervisors complete fidelity checklists for each observer D. Following observation, debrief with whole group of trainees before dividing into groups comprised of trainees who observed in the same classroom. Trainees should discuss agreement on codes among themselves. Master trainers can participate in these group discussions or circulate among groups to address questions. E. As possible, one classroom should be videotaped, so that the entire group together can review it F. OPTIONAL: Ideally, observer-trainees will conduct one 60-minute observation while a certified trainer scores at the same time 1. Maximum of 4 observer-trainees plus one master trainer per classroom (this depends on classroom size, crowding, etc., and should be agreed upon prior to arrival for observation) 2. Certified trainer and trainees are in the same space at the same time, and can see the same activities 3. Successful implementation of this is dependent on the ratio of master trainers to trainees G. Master trainers can communicate any concerns, unresolved issues to MELQO expert XIII. Activities and quizzes (can be administered by paper, tablet or via web-based application) A. Observer-trainees must pass the following (score >=85%) 1. Open-ended questions quiz a. Recommended: done in whole group or small group; should be discussed. 2. Scavenger Hunt a. Recommended: open book, can be small group or individual. Should be discussed 3. Quiz a. Administration is individual (open book; no group discussion, sharing of answers, etc.) XIV. Reliability exercises (video quizzes) 4
A. Observer-trainees must pass one or more video quizzes (overall agreement >=80%) as provided by master trainers 1. Use videos of classroom obtained previously. These should be a minimum of 20 minutes in length. 2. Administration is open-book (i.e., can use manual during coding). Observer-trainees can review videos multiple times to make codings. 3. All trainees must work on their own to complete coding (no discussion or sharing allowed) 4. Data entry reliability spreadsheets (in Excel) will be provided. These will include formulas for calculating percent agreement and kappa. 5. Data capture using tablets to compute reliability. The MELQO expert will work with country team to develop a process for computing agreement and kappa using a tablet program XV. Discussion between the master trainers and the MELQO team members A. We encourage open discussion and follow up with master trainers before, during, and after training. We can schedule regular correspondence throughout the process, or arrange for intensive contact during specified periods of time. B. These correspondences can also include (as possible) video-shares for coding or input; live participation in training, etc. POST-TRAINING: Monitoring during data collection XVI. This should include: A. Enumerators having regular check-ins with supervisors on how the completion of observations (schedule to be decided, but minimum 2X per week). Topics to cover include: 1. Difficulties in assigning codes in sections 3 and 4 2. Ambiguities in assigning codes in sections 1, 2, 5, 6 3. Administrative issues (timing, how to complete specific items, etc.) B. Weekly reports by supervisors on data collection, to understand any issues with incomplete data or observers failing to complete information consistently. MELQO expert will provide monitoring forms. C. Random spot-checks by certified: trainers, supervisors or other qualified personnel periodically visiting study sites unannounced to observe data gathering techniques in various locations. These would be people certified on the tool, and capable of completing the fidelity checklist D. A refresher training (e.g., discussion of: items and scoring; procedures for administering the MELE; challenges faced in administering the MELE as recommended for this project; suggested improvements in administration of the MELE) with input from supervisors on any patterns observed form reviewing the daily forms collected. E. Completing a brief video reliability check. F. If research teams travel to areas without direct supervision, supervisors and/ or trainers should text observers daily to: check on how data collection is progressing; address any issues/ concerns; keep high quality data collection on track XVII. OPTIONAL: reliability/irr during data collection A. Inter-rater reliability checks may take place during the data collection period by having 10% of administrations be double-scored (live or by videotape) or repeating the reliability checklist (live or by videotape). 5