NATIONAL REPORT ON PILOT TESTING a. INTERACTIVE COURSE FOR MIGRANT WOMEN / WOMEN FROM ETHNIC GROUPS b. ONLINE SELF-ASSESSMENT TOOL Compiled by Staff Consult S.r.l., Italy 1 Pilot Testing Report by Staff Consult S.r.l.
Introduction (what/who/when) The pilot testing of Interactive course and online self-assessment tool within the ALMA-DC project (510658-LLP-1-2010-1-GR-GRUNDTVIG- GMP) took place in Rome in November 2012. The group of evaluators were represented by 15 women (Interactive Course) and 2 external experts (Self-assessment test). Objective of the pilot test The purpose of the pilot testing of the Interactive Course was to obtain feedback from the TGs regarding the content of the course in terms of: comprehension, user-friendly exercises, function, usefulness, appearance, effectiveness, relevance. The group of evaluators was mainly composed by east-european women (Romania, Poland, Slovakia) temporarily outside the labour market and not recently attending any lifelong learning trainings. The purpose of the pilot testing of the self-assessment tool, which contains 50 comprehension questions to be answered by the beneficiaries, was tested by experts in terms of: Clarity, function, usefulness, availability, effectiveness, user-friendliness, appearance. 2 Pilot Testing Report by Staff Consult S.r.l.
The specific objectives of the pilot test were: To check learners' engagement with the material To detect points where material may be too easy/too difficult To confirm that learners understand the instructions for activities and exercises To evaluate the flow and balance of the course To validate the course assessment tools To locate points where the course should be revised Methodology of administration of the pilot testing a. Interactive Course The first most important step was to involve our target group in pilot testing phase. We used several dissemination tools to attract our target group: 1. Personal meetings local NGOs working in the field of social inclusion 2. Invitations for pilot testing were posted on ALMA-DC Italian facebook page 3. ALMA-DC leaflets were distributed to potential participants, and were displayed in the notice board of the social services office in the Municipality of Ostia (Rome) 4. Press release published in local newspaper The presential part of pilot testing took place within January 2013, in a room equipped with 8 computers. 15 women attended at the pilot testing of Interactive Course and Self Assessment Tool, and some of them already participated at the previous focus workshop. The programme of pilot testing was divided in two different days because it was very difficult to collect compiled questionnaires from all participants in the same day. At the beginning of pilot testing every 3 Pilot Testing Report by Staff Consult S.r.l.
participant received detailed information about pilot testing and instructions which helped them during the pilot process. The first part of the programme was divided in two main sessions: 1. Introductory part- presentation of ALMA-DC project and its progress, sharing of experience regarding FW 2. The main aim and the importance of pilot testing, functionality of e-learning platform and useful instructions related to evaluation of pilot testing The second part of the testing day devoted attention to pilot testing itself. The participants had possibility to study the specific modules and to carry out the pilot testing by their own, which had added value for them because they could test themselves and their skills which they achieved during training. Staff Consult trainers gave to participants a very basic training about how to use PC and work with the Interactive Course platform (most of them don t have PC at home and they have either very low level of ICT skills). Evaluation Tool At the end of the pilot testing, the participants completed the evaluation questionnaires. The questionnaire was prepared by ALMA-DC partnership and translated by Staff Consult team in Italian. The questionnaire included 22 questions which devoted much attention to demographic data, educational background, employment status, quality and content of the developed Modules and Self-assessment Tool. b. Self-assessment tool The Self-Assessment Tool was assessed by 2 external experts. 2 External experts were contacted by phone and email. Before the evaluation we organised a personal meeting with them to explain the ALMA-DC project in general, the objectives of the evaluation, and the self-assessment tool itself. They were evaluating self-assessment tool from their own premises and we provided them with all necessary information for their work. Both experts provided us very valuable and useful feedback which helps us in the next steps. 4 Pilot Testing Report by Staff Consult S.r.l.
INDICATORS OF PILOT TESTING IN ITALY OUTCOMES THAT WERE PILOT TESTED WHAT WHO METHOD INTERACTIVE COURSE CONTENT SELF-ASSESSMENT ONLINE TOOL FOCUS GROUP OF MIGRANT WOMEN/WOMEN FROM ETHNIC GROUPS Online PLANNED INDICATORS REALISED INDICATORS RESULT (number) (number) Course: 1 1 Fully accomplished Modules: 5 5 Fully accomplished Interactive comprehension questions: 50 EXPERTS FROM CEDA Online Interactive software tool:1 1 50 Fully accomplished Fully accomplished TOOL EVALUATION QUESTIONNAIRE FOCUS GROUP OF MIGRANT WOMEN/WOMEN FROM ETHNIC GROUPS Paper version Questionnaire containing 22 questions 22 Fully accomplished ADMINISTRATION OF THE PILOT TESTING TESTING GROUP FOR THE INTERACTIVE COURSE TESTING GROUP FOR THE ONLINE SELF-ASSESSMENT TOOL WHO HUMAN RESOURCES INVOLVED PLANNED INDICATORS (number) REALISED INDICATORS (number) Staff Consult 1 partner country 1 RESULT Fully accomplished FOCUS GROUP OF MIGRANT Fully accomplished 1 group 1 WOMEN/WOMEN FROM ETHNIC GROUPS 25 women 15 Partially accomplished Fully accomplished EXTERNAL EXPERTS 2 2 5 Pilot Testing Report by Staff Consult S.r.l.
FEEDBACK FROM THE WOMEN INTERACTIVE COURSE Below are shown the results and feedback that we received from interviewed women: 1. Indicate the age group you belong to: 3. Indicate your current employment status: 6 Pilot Testing Report by Staff Consult S.r.l.
3. Indicate your educational background: 4. Regarding Vocational Training Seminars: 7 Pilot Testing Report by Staff Consult S.r.l.
5. Did Module 1 contain useful information for you? 6. Before studying Module 2 about The Labour Market in Italy 8 Pilot Testing Report by Staff Consult S.r.l.
7. After studying Module 3 Social Integration of Immigrants in Italy... 8. Before accessing Module 4 Basic IT skills... 9 Pilot Testing Report by Staff Consult S.r.l.
9. After studying Module 4 about Basic IT Skills (WORD & INTERNET) 10. After studying Module 5 about Adult Education 10 Pilot Testing Report by Staff Consult S.r.l.
11. Which Module did you find most useful? 12. The exercises are 11 Pilot Testing Report by Staff Consult S.r.l.
13. The idea to use the self-assessment tool is 14. The overall content of the Interactive Course is 12 Pilot Testing Report by Staff Consult S.r.l.
15. Did you find the links to various sites useful for you and your family? 16. What do you think about the language used in the modules? 13 Pilot Testing Report by Staff Consult S.r.l.
17. What do you think about the graphics included in the Course? 18. What do you think of the structure of the Modules? 14 Pilot Testing Report by Staff Consult S.r.l.
19. What do you think of the exercises in the self-assessment tool? 20. What do you think of the online self-assessment tool? 15 Pilot Testing Report by Staff Consult S.r.l.
21. What other information would you like this course to contain? Please write your suggestion: techniques for conducting a job interview information about the documents to register a VAT number information to make better use of social networks practical information about special health-care services 22. Would you recommend our site www.alma-dc.eu to other migrant women / women from ethnic groups? Please explain your answer below: Yes, it is a special tool to understand better Italy Yes, expecially the link included are very useful Yes, I would. It is very helpful to practice the language There are technical useful informations and a lot of thing to study Yes, useful Yes I will do 16 Pilot Testing Report by Staff Consult S.r.l.
CONCLUSIONS Women participating in the Pilot Testing have for the most part between 25 and 45 years. There are also 3 girls aged between 18 and 24 years, but only 1 woman over 46 and one over 55. Almost all of the women have a fixed-term employment or a part-time job. Only a woman has a permanent job, and only a woman has a full-time job. More than half of women have completed a course of primary education, and almost 40% had a secondary course of study. Although only a woman has attended a course in technical studies, over 80% of participants attended or would like to participate in VET courses. The 2nd module of the interactive course ALMA-DC has been one of the most appreciated. All participants in the PT found in the second module at least some information that they did not have before, and that they felt useful. 20% of participants have found some information that he had, and almost 60% of women had a partial or any knowledge of employment procedures used in Italy before reading the second module. Regarding module 4 (IT skills), women who claim to know how to use your PC independently remain 32% both before and after the study of the course materials. Before studying the fourth module, 68% said they do not know how to use your PC. But after studying the fourth module, a significant proportion of participants (44%) stated that they could use the PC with the instructions contained in the text. 17 Pilot Testing Report by Staff Consult S.r.l.
A large proportion of participants considered the course interesting and especially useful because it allows to deepen the knowledge of the Italian language. A large proportion of participants (49%) believe the course interesting and especially useful because it allows to deepen the knowledge of the Italian language. This finding is confirmed by the 51% of women participants, who appreciated the self-assessment tool because they can practice as many times as they want. Almost all participants found the site links contained in the course, and have requested more information on how to find ways of sharing information among peers, especially on the major social networks. Most of the participants stated that the language of the course is simple and clear, and that the structure of the modules is well made and tailored to their learning needs. 12 out of 15 participants stated that they would recommend the interactive course to other migrant women / women from ethnic groups. 18 Pilot Testing Report by Staff Consult S.r.l.
ASSESSMENT OF THE ONLINE SELF-ASSESSMENT TOOL 1 st EXPERT ASSESSMENT OF SELF-ASSESSMENT TOOL Clarity 10 9 8 7 6 5 4 3 2 1 0 Function 10 9 8 7 6 5 4 3 2 1 0 Usefulness 10 9 8 7 6 5 4 3 2 1 0 Availability 10 9 8 7 6 5 4 3 2 1 0 Effectiveness 10 9 8 7 6 5 4 3 2 1 0 User-friendly 10 9 8 7 6 5 4 3 2 1 0 Appearance 10 9 8 7 6 5 4 3 2 1 0 Comments: The strength of the interactive course ALMA-DC in the Italian language is the simplicity and clarity of language. The course modules lead the reader easily to a better understanding of Italian society, its structures and its social and cultural services for immigrants. The module on IT skills is very helpful for those women who have no knowledge of computers and their use. It would be interesting to investigate further what specific skills are useful for the target audience of women of recent immigration. 19 Pilot Testing Report by Staff Consult S.r.l.
The only weak point of the interactive course ALMA-DC is the interface of the website, which is not the most appealing. 2nd EXPERT ASSESSMENT OF SELF-ASSESSMENT TOOL Clarity 10 9 8 7 6 5 4 3 2 1 0 Function 10 9 8 7 6 5 4 3 2 1 0 Usefulness 10 9 8 7 6 5 4 3 2 1 0 Availability 10 9 8 7 6 5 4 3 2 1 0 Effectiveness 10 9 8 7 6 5 4 3 2 1 0 User-friendly 10 9 8 7 6 5 4 3 2 1 0 Appearance 10 9 8 7 6 5 4 3 2 1 0 Comments: The contents of the interactive course ALMA-DC are certainly interesting and useful for the target group to whom they are addressed. The texts are linear and consistent, and information contained are so well structured that should be posted in the offices of public services for migrants. This approach to the preparation of information materials could be a guide to support the work of the operators of help desk for migrants. The website is well built though without great effort of eye-catching graphics. Nevertheless, probably the contents may reach more people if published even in social networks where migrant women are present and active. 20 Pilot Testing Report by Staff Consult S.r.l.
CONCLUSIONS OVERALL ASSESSMENT by 2 EXPERTS - rate from 10 [highest] to 0 FEATURES 1 st EXPERT 2 nd EXPERT AVERAGE Clarity 8 9 8,5 Function 9 9 9,0 Usefulness 8 10 9,0 Availability 8 8 8,0 Effectiveness 8 8 8,0 User-friendly 7 7 7,0 Appearance 8 7 7,5 The experts share the opinion that the contents of the course are of great use and interest. Similarly, they appreciated the clarity and appropriateness of the language. For both experts the course is excellent, and its use could be promoted in other contexts, such as social services. On the other hand, both experts highlighted that the use of social networks and other web-based tools would make the interactive course even more functional and easily accessible. 21 Pilot Testing Report by Staff Consult S.r.l.