UNESCO Current Policies, Research and programmes on Assessment

Studies and Working Documents No.5 UNESCO Current Policies, Research and programmes on Assessment by Vinayagum Chinapah Project Director & Qutub Uddin Khan UNESCO Consultant Division of Basic Education UNESCO July 1993 -------------------------------------------------------------- The views and opinions expressed in this working document are those of the authors and do not necessarily reflect those of UNESCO

PREFACE UNESCO has, over the years, supported and facilitated research and development (R & D) in the field of educational evaluation, research and assessment at national and international levels. The report UNESCO: Current Policy, Research and Programmeson Assessment (no. 5) is prepared in this series of Studies and Working Documents for the Joint UNESCO-UNICEF International Project Education for All Goals: Monitoring Learning Achievement. The overriding aim of this report is to take stock of UNESCO s existing policies, research and programmes on assessment with a view to situating this International Monitoring Project. The report consists of four sections. Section I introduces UNESCO s current policy, major programmes and projects in Basic Education. In Section II, UNESCO s follow-up of the World Declaration on Education for All (Jomtien, 1990) in educational evaluation, assessment and monitoring as well as the joint UNESCO-UNICEF International Monitoring Project on Learning Achievement are presented. Section III deals with a review of research and studies in the field of learning achievement and presents the most important findings and trends. Finally, in Section IV, a general plan of action is given in order to further improve the existing situation and lacunae. It is impossible to summarize the work done in this field by UNESCO, by national and international educational research institutions, and by the international research community at large. Limited efforts are made, however, to facilitate and promote programmes, projects and activities leading to the development of a sustainable monitoring system that dwells upon different modalities for national capacity-building. This report highlights some practical ways of national capacity-building that are the driving-force of the International Monitoring Project. Vinayagum Chinapah Director, Joint UNESCO-UNICEF Monitoring Project UNESCO July 1993

CONTENTS Introduction I. Policy, Major programmes and Projects in Basic Education 2 II. UNESCO Jomtien Follow-up: Monitoring System of Learning Achievement 5 III. Research and Studies 22 IV. A General Plan of Action 27 References 35

Current Policy, Research and programmes on Assessment UNESCO Introduction Since the World Conference on Education for All (WCEFA), Jomtien-Thailand, March 1990, UNESCO has carried out a number of important follow-up programmes, projects and activities to assist Member States in the implementation and monitoring of Educational-For-All (EFA) Goals. In the follow-up to the Jomtien Conference, monitoring education for all goals has been identified as a key for international cooperation. After his examination of the three kinds of monitoring data available to the world community, namely: (i) official statistics; (ii) survey and research data and (iii) information and ideas exchanged among national educational leaders at international and regional fora, the Assistant Director General for Education at Unesco, Colin N. Power (1991:3) concludes that the basis of all monitoring is the establishment of efficient approaches to the collection, interpretation and use of all the various kinds of data. Thus, countries need to develop their own capacities for information-based management of education. It is the task of the various agencies to assist in this enterprise through direct technical assistance and by building inter-agency cooperation and dissemination networks. Against this background the WCEFAcalled for quality improvement in basic education: making the learning environment better so that children actually Iearn how to improve their lives. But how do education authorities monitor what pupiis actually Iearn? As little is done to assist Member States in this field, a joint UNESCO-UNICEF internationai project Monitoring Educational-For-All Goals was initiated in 1992 in order to develop and/or to strengthen national capacities for monitoring learning assessment. A presentation and discussion of the implementation strategies and modalities of this project will be made later in this paper. The Mandate UNESCO has a special mandate to strengthen educational research capacities of Member States at national and regional levels but the contributions have so far been marginal. The World Declaration on Education for All, Jomtien 1990, gave high priority to the further development and strengthening of national capacities in evaluation and monitoring of basic education. The UNESCO-UNICEF Monitoring Education-For-All Goals project is one major follow-up activity of this Declaration (Article 4: Focusing on Learning Acquisition).

UNESCO: Current Policy, Research and Programmes on Assessment Whether or not expanded educational opportunities will translate into meaningful development -- for an individual or for society -- depends u l opportunities... The focus of basic education must, therefore be on actual learning acquistion and outcome, rather than exclusively upon enrolment, continued participation in organized programmes and completion of certification requirements... It is, therefore, necessary to define acceptable levels of learning acquisition for educational programmes and to improve and apply systems of assessing learning achievement. World Declaration on Education for All Article 4 - Focusing on Learning Acquisition I. Policy, Major programmes and Projects in Basic Education The current approved biennium programme of UNESCO (1992-1993) lays strong emphasis on universal quality primary education. The main objectives of the programme are: (1) to promote the access to primary education or non-formal and non-tradicionai education aiternatives, as a complement to Classroom teaching, for aii children, with special emphasis on girls and those more difficuit to reach; and (2) to contribute to the overall improvement of the quality of primary education with a view to increasing pupils level of lea rning achievement and to reducing drop-outs and repetitions. The programme aims at achieving these two objectives through: (1) new methods and pilot projects for overcoming obstacles to schooling of children in remote, rural, slum and poverty areas; (2) support components (policy frameworks, audio-visual materials, teacher training curricula, etc. ) for integrating into schools the children with special needs; (3) two methodologicai guidebooks and two guides on improving learning achievement; renewed curricula, low-cost teaching materials for those with special needs; and (4) training and re-training of educaticn personnel. For improving quality and learning achievement, the programme, in particular, aims at launching, in collaboration with donor agencies, Member States and nonimprovement, governmental organizations (NGOs), several projects in quality as 2

identified by the International Symposium on Strategies on Ensuring Success in Primary Schooling (Lisbon, 1991 ); technical support for national projects, especially in Africa and Latin America and the Caribbean regions on success and retention programmes; dissemination of information on practical experiences and successes; improvement of school success for children belonging to cultural minorities; and assistance in revising primary school curricula to meet local learning needs. The programme also aims at improving in-service restraining of school personnel through the development and dissemination of methodological guides for headteachers on management and improving school quality in co-operation with Member States, ACCT, ISESCO, the Commonwealth Secretariat and other interested inter-governmental organizations (IGOs) and NGOs; dissemination of information on low-cost teaching materials and the use of productive work in schools; and restraining of teachers of pupils with special needs. The above mentioned activities are supported by projects funded from extrabudgetary resources and initiated with funds from the regular budget to provide advisory and technical services to Member states. The total financial resources earmarked for the above activities amounts approximately to US dollars 3.2 millions. UNESCO s draft biennium programme (1994-1995) continues to emphasize the support to its Plan of Action for the Eradication of illiteracy by the year 2000 and the Jomtien Framework for Action to Meet Basic Learning Needs. The objectives and activities are conceived in a new and more concentrated manner with a view to having direct impact on: (i) (ii) increasing the provision of adequate and sustained basic education for those who do not enjoy this human right, especially girls and women; and improving the quality and relevance of basic education as a means of curbing wastage and reinforcing the learning process. Regional and sub-regional projects will be designed to revise, among others, teaching methods and assessment tools, emphasizing the child s learning process and support provided by the family and the community. They will also see to it that formal and non-formal programmes meet basic learning needs of selected target groups. 3

Figure 1 E D U C A T I O N F O R A L L U N E S C O ' s B a s i c S t r a t e g y I I I I I I EXPANDING ACCESS TARGETING COUNTRIES The big nine Sahelian countries LDCs TARGETING GROUPS Girls and women The young child Street/Working children Refugees Those with special needs Remote population STRENGTHENING EFA ALLlANCE Media and public in interagency collaboration monitoring EFA progress IMPROVING QUALITY AND RELEVANCE TEACHERS AND OTHER EFA WORKERS Pre-service and in-service Distance education Literacy, Pre-school personnel CONTENT AND PROCESS OF EDUCATION The first 4,000 hours Achievement and Assessment Learning materials development INFORMATION BASE AND CONDITION OF LEARNING Health, nutrition and sanitation Learning spaces and furniture Pre-school readiness Parent-teacher interaction: community support

The UNESCO-UNICEF project on Monitoring Education-For-All Goals, which includes a study of capacity-building and of the key variables affecting achievement, will be continued. The original sample of countries to be covered by the study will be extended. Expected outputs are 12 national and sub-regional projects on school assessment and the condition of learning. In co-operation with the Jomtien partners, pilot projects and training activities will be supported in the framework of the regional and sub-regional programmes in order to improve the information available to those responsible for basic education in Member States and in the donor community. These activities will focus on the development of regional and national indicator systems for monitoring and improving various factors affecting learning and the impact of these factors on participation, school wastage and learning achievement. UNESCO s basic strategy is targetted towards programmes of expanding access and improving quality and relevance in education (Figure l). II. UNESCO Jomtien Follow-up: Monitoring System of Learning Assessment Since the WCEFA, UNESCO has initiated a number of programmes and projects to promote the development of national monitoring system of learning assessment. A better climate for evaluation, monitoring and follow-up of educational policies, plans and reforms was demonstrated immediately after the WCEFA at the Roundtabie on Educational Evaluation of International Congress on Planning and Management of Educational Development in Mexico (March, 1990). For more information on this Roundtable, reference of the International Congress (Chinapah, presented here. can be made to UNESCO s Documents 1991 ). A few of such initiatives are The UNESCO-UNICEF International on Learning Achievement Monitoring Project In line WCEFA s appeal for an expanded vision and a renewed commitment to education for all, this project aims at strengthening partnership at both international (inter-agency cooperation) and national (task force) levels. The modus operandi created for partnership all along the implementation phases of the project is presented in the organizational flowchart in Appendix 1. Through inter-agency cooperation, this project seeks to assist member states in the process of building a sustainable monitoring system linked to education-for-all goals with a particular emphasis on the assessment of children s learning acquisitions (literacy, numeracy and life skills). The project forms part of the cooperation 5

agreement signed by the Executive Director of UNICEF and the Director General of UNESCO in October 1991. An outline of the project was developed immediately after the Jomtien Conference and was critically examined by a panel of experts in July, 1990. A central professional team has been established at UNESCO headquarters since 1992 to implement the project with the financial assistance of UNICEF. During the period September 1992--February 1993, five baseline-studies (Chinapah, 1992; Pawar et al. 1992; Audouin et al, 1993; Schnüttgen, 1993 and Chinapah and Khan, 1993), were carried out to map the experiences so far accrued in the field of educational evaluation, research and monitoring of both national and crossnational nature and to prepare a battery of instruments (questionnaires and tests). The battery of instruments have been sent to the national task force in the five countries for critical examination. An international intensive training workshop in survey methodology was organized at UNESCO (February, 1993) to discuss and to finalize the overall design for the project with due consideration of each country s specific contexts, needs and priorities. The results from the workshop were used in fine-tuning a workable survey methodology taking into consideration sampling procedures, the appropriateness of background questionnaires and tests, main features of data collection, data transfer, data analysis and reporting modalities. They were also used to finalize a set of core domains of simple tests of literacy, numeracy and life skills (Figure II). The outcomes from the inter-agency consultations, from the identification missions in the first five countries (China, Jordan, Mali, Mauritius, and Morocco) and from the one-week intensive training workshop in survey methodology (UNESCO, Paris, February 22-26, 1993) have been used to finalize the project application in these five countries. Objectives and Outputs The aim of this joint UNESCO-UNICEF project is to develop the national capacities in monitoring basic education goals and to set up, train and give appropriate support to national monitoring teams. Over an initial 3-year period (1992-1994), the project will: develop a simple and workable monitoring methodology that can be readily applied/adapted to additional countries: core basic education indicators; sample frame for school surveys; model questionnaire; and a simple test of students achievement in basic literacy, numeracy and life skills; 6

Strategy put at the disposal of decision-makers at national and lnternational level basic education indicators for 20 developing countries; assist in setting up, training, and supporting about 20 developing countries with a view to goals monitoring on a permanent basis. national monitoring teams in carrying out education-for-all The project focuses on primary education as the cutting edge of any education-for-all strategy. Working links are being maintained with other, complementary projects which may focus on the monitoring of non-formal basic education and literacy programmes. The strategic goal of this project is to ensure the appropriate linkages for actions at different levels of policy-making and implementation of educationfor-all goals. The key approach is one of school-based sample surveys conducted in each co-operating country on a yearly or two-yearly basis. A simple, flexible, minimalist and sustainable survey methodology -- its nature and scope are to be determined by participating countries themselves -- is proposed in order to provide policy-makers and front-line implementors with a monitoring information-base for the quality improvement of basic education programmes and projects. The mechanism and modalities for promoting capacity building activities in the project are built upon a holistic approach in order to ensure wider national participation all throughout the project and national ownership. Development of Indicators The indicators developed are first and foremost related to basic learning achievement (literacy, numeracy and life skills) of students. But they will also reflect factors found to influence learning achievement. A third group of indicators will relate to education access and equity. Thus, the indicators will cover essentially 3 domains: (1 ) learning achievement; (2) factors influencing learning achievement -- these are organized into three conceptual blocks of variables: personal characteristics, home environment and school environment -- and (3) access and equity. In addition, the project may provide, as a national option, a core set of simple items of literacy, numeracy and life-skills to meet the needs and interests of participating countries in the conduct of comparative analyses.

Figure : 2 1 COMMON CORE OF BASIC LEARNING COMPETENCIES (BLC) I L ITERACY I [ DOMAINS 1 t 1 NUMERACY I i 1 VOCABULARY READING COMPREHENSION WRITING COUNTING WRITING COMPARING UNDERSTANDING....................................... ---... --.-.:<-s-...s,. -<-.,.-,..=, WORD RECOGNITION SYNONYMS / ANTONYMS READ & COMPREHEND SHORT WRITTEN TEXTS, IMAGES... WRITE & COPY SHORT SENTENCES HEALTH/HYGIENE NUTRITION ENVIRONMENT PRODUCTION / PRE- VOCATIONAL CIVIC LIFE

A) Learning achievement Basic learning achievement of students (literacy, numeracy and basic life skills tested on a sample of primary school pupils). Depending upon the profile of the primary school system in each country the survey will cover: government schools, religious schools, private and community organized schools etc. B) Factors influencing learning achievement Student characteristics and selected homebackground factors. The school setting and selected teaching-learning factors. The community environment and selected school-community-related factors. C) Access and equity Girls enrolment - admission and participation rates - dropouts and repeaters, attendance rates, educational disparities, children out of school, enrolment of disabled children etc. The National Task Forces The success of this project depends to a great extent upon the commitment and active participation of the country (government and relevant national institutions) in the process of developing and/or strengthening national capacities for monitoring of basic education programmes. The establishment of a national task force to participate in the design and implementation of the project is, therefore, an important first step. Depending upon the skills and competencies available in the country and the opportunities provided by the project through training activities, the national task force is the focal point for maintaining and performing monitoring activities on a continuous basis. A major objective of the identifications missions was to arrange with governments the establishment of a national task force for the project. The national task forces for the five countries are presented in Table 1. In all five countries the national task forces have been operational in previous evaluation projects. 9

Table 1 : The project National Task Force in the Five Countries China Jordan Mali Mauritius Morocco Project Coordination Department of Basic Education, State Education Commission Task Force Members from National Center for Educational Development & Education Management Information Centre China National Institute for Educational Research Provincial and County Level Bureaus and Units for Educational Research Project Coordination National Centre of Educational Research and Development Task Force Members from Directorate of Curriculum Planning, Research and Statistics Regional Directors Project Coordination Secretariat d Etat chargé de l Education de base Task Force Members Institute Pédagogique de L Education Directions régionales de I Education Inspections d Enseignement Fundamental Project Coordination Mauritius Examination Syndicate Task Force Members from Master Plan Nine-Year Schooling Sub-Task Force Master Plan EMIS Sub-Task Force Master Plan Rodrigues and Agalega Sub-Task Force Project Coordination Direction de la Planification de L Enseignement Fondamentale r Task Force Members from Division des Etudes et Objectifs Division des Statistiques Division de L Information et de L Orientation Bureaux Regional et Provincial 10

Project Activities The objectives set for capacity building programmes can only be realized within the framework of human resources development (training of a group of national trainers) and infrastructure building (development of a national monitoring environment) of a permanent nature. In some countries, skilled human resources are present but not always adequately identified and assisted. Infrastructure building has often been one-shot external inputs with no adequate national involvement and support for its sustainability. The present project envisages capacity building as a continuous activity initially supported through external resources. It is expected that the national task forces will become self-sustaining after 3 years, though this period may vary somewhat with national conditions. The project focuses upon the training of a selected group of people (national team) to carry out the project during its first phase. This group in turn will serve to train a wider clientele serving to monitor basic education programmes on a continuous basis. Snüttgen (1993) prepared within the framework of this project a working document to capture some conceptual and methodological issues for designing instruments to assess pupils learning achievement. The analysis presented in this document is based on the deliberations and discussions that took place in the International Intensive Training Workshop on Survey Methodology (UNESCO, Paris: February,1993). Snüttgen provides answers to several issues concerning tests designs and their contents for assessing pupils learning achievement together with ideas expressed by the participants in the workshop. Modalities for the preparation of questionnaires based on logical and scientific criterion are also considered. The author suggests that tests should be designed in such a way that they become, on the one hand, instrumental to collect reliable, valid and feasible information on pupils linguistic, psychological, and socio-economic environment and, on the other hand, to demonstrate how a particular group of pupils performs as compared with a reference population. The document provides a list of a core of minimum competencies in literacy, numeracy and life-skills included and tested by the joint UNESCO/UNICEF Monitoring Project initially in the five country case studies to determine how far they are relevant and effective as instruments for measuring appropriately pupils learning achievement after certain years of formal schooling. While designing the questionnaires, the author suggests that cultural sensitivity where pupils come from should always be kept in mind when addressing the respondents and also at the time when deciding issues such as from whom to ask what. A set of questionnaires for pupils, parents, and headteachers and teachers encompassing a variety of characteristics, for example, pupils home and social. 11

environment, school conditions, teachers-parents perceptions and so on should form the basis for questionnaire preparation. Other aspects, such as content areas, item presentation and sequence, and so on are also discussed. UNESCO-IIEP/UNDP Inter-Regional Project on Monitoring the Quality of Basic Education The UNESCO Institute for Educational Planning (IIEP) with the financial assistance from UNDP embarked on this inter-regional project to strengthen national capacities of Member States for planning and monitoring quality improvements in basic education services with a view to increasing their efficiency and facilitating their extension to all segments of the society (Carron, 1991 ). An indepth analysis of the teaching/learning conditions of primary schools in different local contexts, the extent to which these schools equip learners with basic education, and the main factors that explain differences in school results were, studied in four countries since 1989 (Guinea r India, China, and Mexico). The findings from these studies were examined in a research review workshop at IIEP (January, 1993) bringing together the representatives from these four countries, senior researchers and central-level educational administrators from different regions of the world and representatives of various donor agencies. In summarizing the major results from these studies and from the review workshop, Govinda (1993:6) argued that the focus of monitoring the quality of basic education has to shift from the macro-system level to the school level. For example, macro-level comparison of test outcomes could be a strategically powerful tool for planners to bargain with decision-makers, but countries have to develop their own Management Information Systems that link examination data to other information on the system at the local level. UNESCO-UIE/DSE Monitoring and Evaluation of Nonformal Basic Level Education The UNESCO institute for Education (UIE), Hamburg is implementing a project concerned with developing an innovative methodology for monitoring and evaluation of non-formal basic level education (NFBE) for out-of-school children, young people and adults as a WCEFA follow-up activity. In association with the German Foundation for International Development (DSE), UIE has developed an effective and appropriate mechanism to support a number of countries in monitoring and evaluation of NFBE programmes. A series of inter-regional intensive orientation seminars and corresponding training workshops are organized in order to design monitoring systems and appropriate evaluation techniques and procedures 12

for NFBE targeted to national level decision-makers and evaluation specialists, on the one hand, and to middle-level technicians and field evaluators, on the other. [n order to monitor, improve and legitimize NFBE programmes, continuous formative evaluation, internal evaluation, qualitative and descriptive techniques combined with Management Information System and the enhancement of an information culture are needed. A paradigm dialogue is built in the programme for the seminars to arrive at a new and expanded vision of evaluation. Regional and Sub-Regional Monitoring Projects Several projects have been launched by UNESCO institutes (IIEP, IBE, UIE) as well as by its regional and sub-regional offices to monitor EFA goals. These projects lay a strong emphasis on developing and/or strengthening national evaluation and monitoring systems of learning assessment. Some examples are given here. Africa Under its programme on Basic Indicators of Monitoring, the IIEP and the Ministry of Education and Culture of Zimbabwe conducted a collaborative research study (1 991) in order to test a mechanism that was designed to illustrate how educational planners could develop indicators of the quality of basic education and then use these for important educational planning issues. At the request of the Zimbabwean Ministry of Education, IIEP in co-operation with the Planning Department prepared a set of basic indicators for monitoring pupils learning achievement. The lessons drawn from these studies have been adequately diffused to encourage other countries both in Africa and other regions to establish similar indicator system. The results of the study are presented in three volumes. Volume I describes the mechanism and some valuable experiences gained, results obtained and suggestions for action and policy implications. Methodological aspects, appendices and results of the statistical analysis are the subject matter of Volumes II and Ill. The study does not claim to be an evaluation study. Its underlying objectives, however, are: (i) to develop and test a mechanism for the accurate and quick collection and analysis of data and rapid reporting of the results to the MEC; (ii) to address a short list of questions in order to demonstrate the potential for the mechanism to yield information as a basis for informed decision making; and (iii) to familiarize the technicians with the sequence of steps to be undertaken in educational survey work. 13

The mechanism employed by the study is based primarily on the scientific principles of survey research. These principles considered by the study are the recognition of the importance of appropriate sample design, the development of instrumentation focusing on policy questions posed by the MEC, the execution of field work operations centred around a well-designed training programme, and the use of the most recent innovations in software for powerful micro-computers. The study uses reading comprehension as the key variable for measuring learning outcomes. Reading comprehension has been considered as one of the key subjects because mastery of this skill is essential if children were to be able to learn effectively. As regards reading achievement, two approaches have been employed to identify the major indicators. [n the first approach, correlation matrices have been estimated between input and output indicators of reading achievement. However, in the second approach, attempts have been made to identify effective and ineffective schools. Finally, 15 most effective schools have been compared with the 15 least effective schools in terms of the provision of inputs to the schools. Baseline indicators of inputs to schools considered in the study include school buildings, toilet facilities, number of pupils per school, number of teachers per school, pupil-teacher ratio, teacher absenteeism, teacher qualifications, libraries, teacher housing, school furniture, distribution of school supplies, and teacher perceptions. As indicators of learning outcomes, the study selects 33 items from an initial list of 59 and indicates that mastery of these 33 items is essential if pupils were to succeed in Grade Vll in Zimbabwe. Thus, the study measures minimum and desirable performance levels for learning outcomes. Finally, the study presents in details the results on inputs to schools. This is followed by data on indicators of learning outcomes and linkages between inputs and learning outcomes. As a follow-up to the collaborative research conducted in Zimbabwe (1991), IIEP organised a sub-regional seminar on Data Building and Data Management in Harare in 1992 attended by 40 participants from Zimbabwe, Botswana, Lesotho, Malawi, Mauritius, Swaziland, Tanzania, and Zambia. The main focus of the seminar was to provide to the participants the necessary materials and training to undertake a national survey similar to the. Zimbabwe study. It provided an opportunity to the participants to gain practical experience in the specification of detailed research plans derived from policy priorities, test construction, preparation of field work manuals, interactive data entry and cleaning, 14

file building and file management, computer-based sample design, data analysis, and report preparation. Within the framework of the project on evaluation of NFBE programmes, UIE organized two orientation seminars, one for English-speaking and one for French- Speaking countries of the African region (Botswana, Ethiopia, Ghana, Kenya, Malawi, Tanzania, Zimbabwe, Benin, Burkina Faso, Congo, Guinea, Mali, Niger and Senegal). These seminars and workshops have altogether improved the understanding of the importance of monitoring and evaluation of educational programmes in general, and of NFBE programmes in particular. The paradigm dialogue in the new and expanded vision of evaluation challenges the validity and reliability. of conventional information-base for educational policy-making. UIE programmes in this field are closely associated with the activities on learning assessment of the Monitoring Project. The design, instrument construction and data analysis components of both projects are jointly discussed and revised in such seminars. The objectives of the Orientation Seminars are: (1) (2) (3) (4) (5) (6) (7) to analyze. existing explicit and implicit formal and nonformal evaluation practices in use in NFBE programmes in the participating countries; to increase awareness of implicit or explicit consequences of evaluation practices on the quality of NFBE; to increase awareness of the need for appropriate evaluation procedures and techniques pertaining to NFBE programmes; to make participants question existing monitoring and evaluation practices and motivate them to improve their quality; to encourage the adoption of an expanded vision of evaluation to include Management Information System, Naturalistic Evaluation and Rationalistic Evaluation as appropriate; to assist participants in preparing plans of action to introduce or review monitoring systems and evaluation procedures suitable for NFBE programmes; and to assist participants in the preparation of their staff members for the, following training workshop with evaluation specialists. 15

Arab States The nature and scope of educational evaluation and research in the Arab States seems to be quite limited. Morsi (1991) attempted to develop a methodology and instruments of evaluation and research that can be used in the Arab States to monitor the implementation of their educational reforms. The model presented is a decision-tree for selecting evaluation models and consists of various important decisions in the design and implementation of educational evaluation and research endeavors. However, UNESCO s Regional Programme for Education in the Arab States (UNEDBAS) emphasizes to the development of Arab educational systems at the level of literacy and primary education so as to ensure the right to education for all. One of the specific objectives of the Programme is to improve the quality of both forma! and non-formal education and raise the level of efficiency of educational systems through their interaction with development programmes, as well as through reform, renewal and integration between both formal and non-formal sectors of education. During its implementation phase, the programme assigns a special priority for renewing planning, management, research and evaluation infra-structures. The programme offers a practical framework for data collection and classification and dissemination of information that would help in decision-making concerning the formulation of strategies, plans and programmes for primary and Iiteracy and adult education. UNEDBAS, in collaboration with UNESCO Secretariat would be responsible for the coordination of work under the Regional Programme as well as providing support for conductings surveys, evaluation and research studies and organizing workshops, seminars, training courses field visits, and so on. Asia and the Pacific In the Asian and Pacific Region, UNESCO has promoted several Jomtien follow-up activities to monitor EFA goals. The meeting of the expert group on the development of EFA monitoring and follow-up information systems (September, 1992), the joint UNESCO/UNDP/AIDAB/DANIDA programme on basic education and life skills (1 992-1 996) and the criterion-referenced study of the Pacific Islands literacy levels are cases in point. Highlights of these activities are presented here. The main objectives of the expert group meeting were: (1) to review and present the state-of-the art in the field of monitoring information systems for EFA planning and management in countries of Asia and the Pacific Region; (2) to determine the relevant quantitative indicators and qualitative aspects that intervene in the EFA

scenario and that are essential to obtain a comprehensive picture of EFA progress in the region; (3) to appraise the feasibility of micro-computer based applications in Educational Management Information Systems of EFA follow-up and monitoring; and (4) to share regional and international expertise and experience in the above fields. Senior representatives from Bangladesh, China, India, Indonesia, Laos PDR, Myanmar, Nepal, Pakistan, Philippines, Republic of Korea, Sri Lanka, Thailand, and Vietnam participated and presented case studies on the formulation and implementation of EFA monitoring systems. They also presented views in regard to the follow-up plans and programmes in their respective countries. However r the only study which had some hints on monitoring learning achievement was the Indian case study. India has developed a Computerized Planning of Education Project (COPE) which apart from the quantitative aspects places a special emphasis on qualitative aspects, i.e., the achievement levels based on minimum learning competencies in various subjects as well as functional literacy. Based on the country case studies, conceptual and methodological presentation and discussions as well as deliberations on national follow-up plans and programmes, the participants made the following suggestion and recommendation on the theme of monitoring of learning achievement: national level training activities should be organised with a view to promoting/strengthening the monitoring systems as well as enhancing the orientation/competencies of the various functionaries in the adoption/use of the system of indicators. Within the framework of the Pacific Education Development Programme (1992-1996) the UNESCO/UNDP/UNICEF/AIDAB/DANIDA proposed programme is aimed to achieve eight. major objectives-of basic education and life skills in the Pacific Island countries. One of the major components of the overall project is monitoring programme implementation and assessment of achievements. Within this area, the programme intends to cover two distinct aspects, the first being the establishment of a system which will provide data on the progressive implementation of each component; and the second being to provide a system of measuring the effectiveness of the interventions in terms of children s learning. Procedures for the collection of both the quantitative and qualitative data (based on the success rates as determined from the pre-set performance indicators for each course) have already been established. This programme also emphasises the need for collection of data highlighting the gender disparities in learning achievement., A special workshop was convened in Suva, Fiji (September 1992) to prepare an inventory of bench-marks. The programme plans to measure the impact of the 17

learning achievement of school pupils over a much longer period than the duration of the project. For each element of the project, modalities have been determined and several instruments for data collection finalised. Periodic reviews will be undertaken during the monitoring and appraisal of programme implementation in order to identify successful pilot strategies and methods which could be adapted by other participating countries to stimulate efficiency and policy level support. Among others, the programme anticipates to yield the following outcomes: (i) (ii) a system of early intervention based on reliable cumulative assessment data; and a mechanism for decision-making in educational management at the national level based on reliable data; Reading and primary education specialists from nine Pacific States were convened by UNESCO s Apia Office in the first regional meeting to discuss common problems of illiteracy and sub-standard reading. One of the gathering s most significant conclusions was that, contrary to the rosy picture suggested by high school enrolment rates in many countries, all countries in the region were in fact afflicted with the problems of poor reading skills in a significant portion of their populations. Worse, all believed that the problem actually extended to the in- school population. The meeting acknowledged that the generally weak data base enormously complicated the task of mounting remedial measures. If there was absolute illiteracy among enrolled school-age children, how extensive was the problem? What was the distribution by gender, school type (Government or Mission) and location (urban, rural, outer island)? Confessing to-having no idea-, the-participants constructed a simple test and administered it to a small random survey. By interviewing a sample of out-of-school, working age people in the greater Apia area, they were able to find at least tentative confirmation of their views. No claims were made about the extent of reliability of the test or the validity of the findings obtained. A shortage of resources and competing priorities made it impossible to act on the meeting s recommendations for nearly two years. Fortunately, however, WCEFA refocused attention on the problem. The heads of the UNESCO office in Apia and the South Pacific Board for Educational Assessment in Suva commenced a dialogue on the quality of literacy education in the Pacific Island States and particularly on the ways in which sub-standard reading skills resulting from a poor grounding in primary school impacted with disastrous consequences on achievement scores at Forms 5 and 6 (classes 11 and 12). They accordingly decided that it was time to 18

put an end to speculation, however well informed, and actually chart the true dimensions of the problem. A Criterion-Referenced Study was prepared under contract with the South Pacific Board for Educational Assessment by Graeme Withers of the Australian Council for Educational Research to review the performance of students in all nine countries constituting the sample population in Cook Islands, Fiji, Kiribati, Solomon Island, Tokelaus, Tongo, Tuvalu, Vanuatu, and Western Samoa. First, it was decided to limit the study to obtaining a straight yes/no answer to a straight question: does absolute illiteracy exist at a significant level in each of the nine countries participating in the study? Second, the definition of literacy would be derived from UNESCO s own. It would include for the present study, the ability to read and write in both the vernacular of the country concerned as well as English as a compulsory subject for all nine countries, and numeracy. The sampled study consisted of 2,529 Class VI students in 94 schools in nine countries. A set of three tests to be completed in 40 minutes was given. The test was made up of (i) a simple writing task in each of English and, where appropriate, a national or vernacular language; (ii) two sets of six reading comprehension questions, one based on a short passage in English and, where appropriate, also a national or vernacular language; and (iii) twenty-five number questions testing the four basic arithmetical processes, The study was designed to test the widely-held impression that there is much hidden illiteracy in Pacific Islands schools and that after six years of formal education, there are significant numbers of students who still cannot read, write or compute given simple tasks to perform. The study was conceptualised. as a snapshot of students achievement during 1990. Due to the availability of limited funds, it was not possible to conduct a huge sampling of student achievement, and the study was designed accordingly. For this reason, only one level of schooling was selected and a sample of students in each country was selected to do the tests. Class VI was selected as the target population. This allowed all participating students to have had a significant amount of literacy education and experience prior to the test. The working definition of the word literacy adopted to underpin the rationale for this study was as follows: Literacy is the confident, appropriate and accurate use of spoken and written languages for the wide variety of personal, public and creative uses demanded by the society in which the user lives. 19

The following shows the literacy levels and their in the Pacific Island Literacy levels: descriptors, as used by assessors LITERACY LEVELS AND THEIR DESCRIPTORS 5. Production and comprehension assured, If any errors occur they are minor. If any omissions occur, they are few, and likely to be mere slips. 4. Some real ability both to produce and comprehend lacking assured control over one or other process. displayed, but 3. 2. 1. Evidence of some capacity to produce or comprehend (perhaps both), but flawed by omissions or mistakes. Minimal success at production and/or comprehension, but some attempt made to complete the test. Little or no response in writing: little or no comprehension of basic printed material displayed. This level includes blank scripts. In achievement terms and in completing the literacy tasks of the Pacific Island Literacy Levels, students who achieved, in either English or a vernacular language, the top three levels (3-5) above were considered to be literate. Similarly, the working definition of the word numeracy for this study was as follows: Numeracy is the confident, appropriate. and accurate use of number and the language of arithmetics for the wide variety of personal and public uses demanded by the society in which the user lives. The following shows numeracy levels and their descriptors, as used by the assessors in this study: NUMERACY LEVELS AND THEIR DESCRIPTORS 4. Assured control over all four. basic processes even if a few errors are made. 3. Assured control over some basic. processes, but marred by major weaknesses or carelessness. 20

UNESCO: Current Policy, Research and programmes on Assessment 2. Some control over one or more omissions or errors to be certain. basic processes, but too many 1. Little or no response to the questions asked. This Ievel includes blank scrip t. In achievement terms, in completing the numeracy tasks of the Pacific Island Literacy Levels, students who achieved the top two levels (3-4) as defined above were considered to be numerate. The study offered only one general conclusion. if there is in fact a greater degree of hidden illiteracy amongst children of Class 6 age, on the scale suggested by other evidence, then it is still hidden. The strong suggestion is that it would be found amongst those students who are not in school, either because they have left school, or they have never been enrolled, or they were absent on the day of this particular testing. The non-appearance of the expected large number of blank papers in this testing, in most countries, support this view. A second general conclusion might be made more tentatively. The aggregate data suggest that girls have out-performed boys on the English and numeracy tests at all levels of competence established for the study. Latin America and the Caribbean The UNESCO Regional Office for Latin America and the Caribbean (OREALC) is providing technical assistance to several countries in the region to further strengthen their national assessment systems for measuring achievement in education. As a Jomtien follow-up activity to assist Member States in developing their national capacities for educational assessment, a joint effort was made in this field by OREALC, the World Bank (EDIHR) and the International Association for Educational Assessment to bring together in a regional seminar key officials from nine of the countries in the region (Argentina, Brazil, Colombia, Costa Rica, Chile, Ecuador, Mexico, Dominican Republic, and Venezuela) which have developed or are developing assessment systems (OREALC, August 31- September 4, 1992). The objectives of this seminar were to: (a) discuss the role of the national assessment systems in the improvement of the quality of education; (b) present, compare and analyze the existing assessment systems currently in operation in the region; (c) analyze in detail, the main aspects associated with the organization, administration and management of such systems; and (d) determine appropriate alternative responses to the different needs of the countries. In his mission report from this regional seminar, Cassasus (1992) describes the progress made in strengthening national capacities for monitoring learning achievement in the region since Jomtien (1990). Although the seminar itself was 21

a first important activity in this field, it still mirrors the important progress made for the creation of an evaluation culture. With the exception of two, all countries in Latin America have developed or are establishing assessment systems. Institutional reform is being conducted in a twofold process of decentralization and the establishment of information systems. Accordingly, agencies for international cooperation have important roles to play in strengthening - and not replacing - national capacities. Several areas for cooperation were identified, namely: (1) to develop human resources for assessment, through the establishment of educational facilities for specialization in this area, and /or the provision of fellowships for study and attachments abroad; (2) to improve and modernize existing technical level through hardware and statistical software; and (3) to foster TCDC through the establishment of a regional item bank (an activity already initiated at OREALC) and through exchange of experiences among countries of the region. As a follow-up to the seminar, two more regional seminars are planned: one at a lower level of sophistication for countries that have not participated in the 1992 seminar and which are starting assessment systems, and another, much more technical, for more advanced countries. III. Research and Studies Immediately after WCEFA in 1990, UNESCO in collaboration with other Jomtien partners moved ahead in developing appropriate follow-up mechanisms to implement WCEFA framework for action. [n the field of educational assessment, UNESCO and its partners have commissioned and/or have conducted a number of research and studies directly or indirectly related to learning achievements. A short overview of these research and studies is presented here. Monitoring Progress Towards Education-For-All Goals After the publication of the World Education Report (1 991), UNESCO organized several consultations on the development of education indicators as an important activity for monitoring the progress towards the EFA goals. The meeting of experts on education indicators and the International Standard Classification of Education (UNESCO, June 1992) emphasized the practical usefulness of educational indicators for policy-makers at the national level and the importance of making comparisons with neighboring countries and/or countries at similar levels of development. Within the field of learning assessments, further emphasis was given to the development of indicators on the educational process and learning outcomes. It was also suggested that the World Education Report of 1993 could present some indicators relating to the educational process from the special survey of primary schooling carried out in 1991. UNESCO organized a similar exercise in April 1993 to reflect upon the progress made and the challenges ahead. For 22