The EFSA Journal (2007) 526, 1-15 Internal and External Review Scientific Advice by the Scientific Committee (Question N EFSA-Q-2007-060) Adopted by written procedure on 3 August 2007 Proposal for a Review System for EFSA s Scientific Activities SCIENTIFIC COMMITTEE MEMBERS Sue Barlow, Andrew Chesson, John D. Collins, Erik Dybing, Albert Flynn, Claudia Fruijtier- Pölloth, Anthony Hardy, Ada Knaap, Harry Kuiper, Pierre Le Neindre, Jan Schans, Josef Schlatter, Vittorio Silano, Staffan Skerfving, Philippe Vannier. SUMMARY The European Food Safety Authority asked its Scientific Committee to develop a proposal for a review system to assess the quality of EFSA s scientific work. The proposal developed by the Scientific Committee is comprised of the following four components: 1. Self-review: During the development of each opinion or other scientific document the compliance with best scientific practice should be checked; 2. Internal scientific review: Before adoption by the relevant Scientific Panel(s) or Committee a sample of EFSA draft opinions or other scientific documents should be reviewed by senior scientific staff not involved in the relevant Scientific Panel or Committee and/or members of a Scientific Panel or Committee, not involved in the preparation or adoption of the opinion; 3. External scientific review: A number of EFSA adopted opinions, or other scientific documents should be reviewed by independent scientists; 4. The appreciation of EFSA s scientific work by the intended users should be assessed. This proposal offers a strategy that should provide EFSA with the means to comprehensively review the quality of its scientific activities, both by internal review and by independent, external review processes, together with additional consultations with institutional and noninstitutional stakeholders. The strategy is designed to give continuing feedback to the Authority about the quality of its work. The strategy should be updated in due course, taking into account the experiences gained with these proposals. Contacts should also be made with other European and international organisations to draw on any experience they may have with similar reviews of their scientific activities. Key words: EFSA, Scientific Committee, Scientific Panels, scientific review, quality of science European Food Safety Authority, 2007
TABLE OF CONTENTS Scientific Committee members... 1 Summary... 1 Table Of Contents... 2 Terms of Reference... 3 Acknowledgment... 3 Introduction...4 Proposed Strategy... 5 1. Self-review... 5 2. Internal scientific review... 6 2.1 Introduction 6 2.2. Aim 6 2.3. Selection of opinions and other scientific documents to be reviewed 6 2.4. Performance and expected outcome of the review 6 2.5. Testing of the above procedure 7 3. External scientific review... 7 3.1. Introduction 7 3.2. Aim 7 3.3. Structure and timing of the external review 7 3.4. Selection of opinions and other scientific documents to be reviewed 8 3.5. Performance of the external review 8 3.6. Expected outcome of the review 9 3.7. Improvements of the procedure 9 4. Appreciation of EFSA s scientific work by the intended users... 9 Conclusion... 10 References... 10 Annex : Self-Review Template... 11 Abbreviations: IRG: Internal review Group ERG: External review Group The EFSA Journal (2007) 526, 2-15
TERMS OF REFERENCE The Scientific Committee is asked to develop a proposal for a review system to assess the quality of EFSA s scientific work. Such a proposal should meet present day standards of audit and peer review and the necessity for openness and transparency. It should consider not only the processes that are followed during the development, completion and delivery of EFSA s scientific outputs, but also the appreciation of the output by the intended users. The approach should include both internal and external reviews and be based on principles of best scientific practice. ACKNOWLEDGMENT EFSA wishes to thank the working group members, Sue Barlow, Pierre Le Neindre, Vittorio Silano, Staffan Skerfving for their contributions. The EFSA Journal (2007) 526, 3-15
INTRODUCTION The EFSA Founding Regulation (EC 178/2002) states that EFSA shall carry out its tasks in conditions which enable it to serve as a point of reference by virtue of its independence, the scientific and technical quality of the opinions it issues and the information it disseminates. Since its establishment, high quality science has been at the core of EFSA s efforts. However to more objectively assess that this goal is fulfilled, there is a need to develop a system to review the quality, objectivity and utility of scientific information provided by EFSA. Many national and international science-based organisations and advisory bodies consider it appropriate and necessary to open the processes by which scientific advice and information is generated to both internal quality assurance checks and external scientific review. Such scrutiny, conducted in an open and transparent way, helps to ensure that best scientific practices are being followed in order to underpin public confidence that the best scientific advice is being obtained. EFSA asked its Scientific Committee for help in developing and implementing such a quality review system applicable both to its own staff and to members of the Scientific Committee, Panels and their working groups. Such a quality review system is meant to be understood as an evaluation of the conformity of EFSA s scientific work with best risk assessment practices and should not result in a new evaluation of the data. A number of governments and international bodies around the world have published reviews of the operation and effectiveness of scientific advisory committees, guidance documents on scientific analysis, and codes of practice for scientific advisory committees.1 A number of these publications were consulted during the preparation of the present guidance document and consideration was given to approaches that were considered potentially useful for the review of EFSA s scientific work. This document focuses on the internal and external reviews of the work processes and outputs of the Scientific Committee and the nine Scientific Panels, e.g. scientific opinions, reports, guidance documents and statements. It also proposes a possible mechanism for ascertaining the appreciation of EFSA s output by the intended users. The latter should be considered complementary to the independent external evaluation of EFSA s achievements, which has to be commissioned every six years, according to the EFSA founding Regulation (EC, 2002), and which should assess the working practices and the impact of EFSA. 1 Examples include: FAO/WHO Framework for the Provision of Scientific Advice on Food Safety and Nutrition. Final Draft for public comments, FAO/WHO, Rome/Geneva, 2006. Food Standards Agency. Review of the Scientific Committees. Food Standards Agency, London, March 2002, FSA/0567/0402. Food Standards Agency. The Governance of Science, Annex 4 Science Checklist. FSA 06/02/07. Paper for the FSA Board 9 February 2006. Food Standards Agency, London. Food Standards Australia New Zealand. Science Strategy 2006-09. Available at: www.foodstandards.gov.au. Australian Government, Department of Agriculture, Fisheries and Forestry Annual Report 2003-2004. Report on Performance; Scientific Advice. Available at: www.daffa.gov.au/about/annualreport/03-03/report-performance/output7. A Framework for Science and Technology Advice: Principles and Guidelines for the effective use of Science and Technology Advice in Government Decision making. Government of Canada, 2000. Available at:http://strategis.gc.ca Federal Research: Peer Review Practices at Federal Science Agencies Vary. Report to Congressional Requesters, GAO/RCED-99-99, United States General Accounting Office, Washington DC, 1999. Final Information Quality Bulletin for Peer Review. Executive Office of the President, Office of Management and Budget, Washington DC, December 16, 2004. The EFSA Journal (2007) 526, 4-15
For the review of other EFSA scientific activities in the Scientific Cooperation and Assistance Department, e.g. process and output of the Pesticide Risk Assessment Peer Review (PRAPeR), Zoonoses monitoring, data collection and other units, a modified review system might be needed as their modus operandi differ from those of the Scientific Panels and Committee, but in principle, the same approach as proposed below should be applied for the review of EFSA s other scientific activities. This proposal should be read in conjunction with the EFSA document Transparency in risk assessment carried out by EFSA: guidance document on procedural aspects (EFSA, 2006). The Scientific Committee would like to have feedback in due course from the Panels on the utility of the provisional template and the process, once they have had an opportunity use it over some months. The Scientific Committee will then finalise the template in the light of any comments received. PROPOSED STRATEGY The proposed strategy comprises four components which, taken as a whole, would provide a comprehensive overview and regular feedback to EFSA on the quality of its science. 1. Self-review: During the development of each draft opinion, or other scientific documents, compliance with best scientific practice should be checked; 2. Internal scientific review: Before adoption by the relevant Scientific Panel(s) or Committee a sample of EFSA draft opinions or other scientific documents should be reviewed by senior scientific staff not involved in the relevant Scientific Panel or Committee and/or members of a Scientific Panel or Committee, not involved in the preparation or adoption of the opinion; 3. External scientific review: A number of adopted EFSA opinions or other scientific documents should be reviewed by independent scientists; 4. The appreciation of EFSA s scientific work by the intended users should be assessed. The proposed mechanisms for the above reviews are discussed in more detail below. 1. Self-review The Scientific Committee and each Panel should carry out a self-review during the course of the development of the work related to each question or mandate. The purpose of these selfreviews would be to provide a quality assurance check that proper procedures had been followed and to encourage best scientific practice. Such self-reviews would contribute to improving the quality of EFSA s scientific advice and the consistency of EFSA s scientific advice within and between the Scientific Panels and Committee and over time. It is suggested that self-reviews be carried out by the respective secretariats of the Scientific Panels and Committee under the responsibility of the chairs of the Scientific Committee or Panel or their working groups. The mechanism for this process should be decided within each Scientific Panel and Committee. Throughout the process of developing opinions and other scientific documents, the Scientific Panels and Committee should check for proper compliance with best assessment practices as outlined in the Annex. The template in the Annex provides a means to conduct this check and to identify any significant issues. The Scientific Panels and Committee should keep an internal record of the completed self-reviews. The EFSA Journal (2007) 526, 5-15
2. Internal scientific review 2.1 Introduction In order to evaluate the conformity of EFSA s scientific work with best risk assessment practices on a regular basis, in addition to self-review, there is a need for a continuous internal scientific review process within the Authority. This review of draft opinions or other scientific documents should be carried out by EFSA senior scientific staff not involved in the relevant Scientific Panel or Committee and/or members of a Scientific Panel or Committee, not involved in the preparation or adoption of the opinion. EFSA should establish an internal review group (IRG) and draft opinions or other scientific documents should be reviewed by one or more persons from within the pool of IRG members. This section sets out the aim of the internal review and provides proposals for the timing/frequency of the review, the selection of opinions and other scientific documents to be reviewed, the procedure to conduct such a review, its expected outcome, and measures to improve it over time. 2.2. Aim The aim of the internal scientific review is to evaluate the conformity of EFSA s scientific work with best risk assessment practices, especially the opinions produced by the Scientific Panels and Committee and other scientific documents. Such a review is intended to contribute to improving the quality, clarity and consistency of the scientific advice provided by EFSA in its opinions and other scientific documents and ensuring that best scientific practice is followed. Summary reports on the main outcomes of the internal review process should be provided periodically to the Executive Director and be made available to the Scientific Panels and Committee so that any useful lessons can be discussed and shared. 2.3. Selection of opinions and other scientific documents to be reviewed The IRG should review a balanced set of draft opinions and other scientific documents produced by EFSA s Scientific Panels and Committee (the aim would be at least 10% of opinions / and other scientific documents produced annually). The selection of opinions/reports to be reviewed should focus on, but not be limited to, complex issues and high-profile public-health aspects and should reflect a balance across the Scientific Panels and Committee. 2.4. Performance and expected outcome of the review To ensure that the internal review process takes account of the practicalities around adoption of opinions and other scientific documents to deadlines and preserves the primacy of the Scientific Panels and Committee in deciding the final advice in their opinions and other scientific documents, the selected draft opinions and other scientific documents should be reviewed in a timely manner. This should be before their adoption by the relevant Scientific Panel(s) or Committee members, so that in their final deliberations they may consider any comments arising from the internal review process. It is recommended that similar elements, as applicable, from the items listed in the template for self review are used for internal-review (see Annex) The EFSA Journal (2007) 526, 6-15
The internal review process would result in written IRG reports. The IRG report should identify high quality elements in the draft opinion or other scientific document, alongside any problem areas and deviations from best practice. It should suggest future actions, if any, that might be taken to ensure best scientific practice is followed and provide any suggestions for improvements. The IRG report should be made available for consideration by the relevant Scientific Panel or Committee as soon as possible before the scheduled meeting at which the draft opinion and other scientific documents is due to be adopted. Normally, the Scientific Panel or Committee would be unaware until then that a particular draft opinion or and other scientific document had been selected for internal review. 2.5. Testing of the above procedure The approach proposed above should be tested during the remainder of the current 3 year cycle of the Scientific Panels and Committee (2006-2009) and improvements suggested by practical experience should be carefully considered before a new cycle of internal review is undertaken. Contacts should be made with other European and International Organizations to draw on any experience they may have with similar internal reviews of their scientific activities. 3. External scientific review 3.1. Introduction To ensure that the review of the quality of EFSA s scientific work is robust and credible, and also is seen to be so by the external world, it needs to include the participation of leading scientists independent of EFSA, drawn from the worldwide scientific community. An external scientific review, on whether EFSA s advice was consistent with the best available scientific information, would offer a powerful tool to improve the scientific quality of EFSA s work and to enhance its reliability for decision-making by risk managers. This section provides proposals for the aim of the external review, the selection of opinions and other scientific documents to be reviewed, the procedure to conduct such a review, the timing/frequency of the review and its expected outcome. 3.2. Aim The external scientific review would cover already adopted and published EFSA opinions, and other scientific documents and would result in a written report focussed on whether: best scientific practice was followed with respect to assembling and describing scientific evidence; conclusions and recommendations were adequately supported by the evidence with due attention paid to any uncertainties; and the terms of reference were adequately addressed. 3.3. Structure and timing of the external review Based on the three years appointment of EFSA s Scientific Panels and Committee, a comprehensive cycle of external review of the scientific outputs of the Scientific Panels and Committee should take place every three years. The EFSA Journal (2007) 526, 7-15
An external review cycle should ensure a balanced consideration of EFSA s activities in all the different sectors. To this end, the establishment of External Review Groups (ERGs) for each Panel and the Scientific Committee is recommended each 3 years. Each ERG should consist of a small number of independent, highly-reputed scientists with backgrounds relevant to EFSA s work The ERGs would be asked to focus on criteria such as: (i) quality of the scientific evidence description, discussion and interpretation, (ii) treatment of uncertainties, (iii) completeness of data gathering, (iv) appropriateness of the underlying assumptions, and (v) whether the conclusions were adequately supported by the quoted data and/or references. It would not be the purpose of the ERGs to reach their own conclusions on the same set of data. When performing their tasks, ERGs should have access to all available information, including scientific comments, if any, by other parties relating to the published opinions and other scientific documents undergoing review. 3.4. Selection of opinions and other scientific documents to be reviewed Each ERG should be asked to review several opinions and other scientific documents in depth. To avoid possible bias on the part of EFSA, each ERG should select the opinions or other scientific documents to be examined from EFSA s entire portfolio. In practice, EFSA could offer for information of the ERG a list of the opinions and other scientific documents, from which each ERG would select several for detailed scrutiny. Such a list would include, but would not be limited to, opinions and other scientific documents that have addressed complex questions, high profile public health issues and controversial topics where the science is still evolving. 3.5. Performance of the external review The external review should be conducted by independent scientists from academia and public institutions with an international reputation. Scientific bodies at non-european, European or national levels could be approached for advice on the identification of such experts. The database of national experts currently being compiled by EFSA may also be useful for identifying external reviewers. EFSA s Scientific Committee should also be asked to propose names for external reviewers, along with the basis for their proposal. The experts should be selected and formally appointed by EFSA. In order to facilitate the involvement of external experts who have limited time available, it is proposed that the review would start with a one-day initiation meeting of each ERG to choose the opinions or other scientific documents to be reviewed and to agree the work plan. The work would continue by remote working and contacts. Another one-day meeting of each ERG would take place at the end of the review to adopt a consensus report. It can be anticipated that, in total, the work would require about 5 full days, spread over a couple of months. Each ERG would be assisted by a Secretariat provided by EFSA. The EFSA Journal (2007) 526, 8-15
The ERG should be provided with: - Remit of the ERG. - A summary description of EFSA s scientific aims and objectives and the way in which it works to deliver those objectives (1-2 pages). - A bibliography of all relevant EFSA s opinions and other scientific documents sorted by Panel. - Reports of internal reviews carried out by EFSA (if available for the selected opinions). - Information on any feedback on EFSA s opinions and other scientific documents from risk managers in the Commission or from risk assessors and risk managers in national food agencies (if available for the selected opinions). - Information on any comments and criticisms on EFSA s scientific opinions and other scientific documents from outside scientific bodies and how EFSA dealt with them (if available for selected opinions). - Access to documents on which the opinion is based (if requested by the ERG). 3.6. Expected outcome of the review The expected outcome would be a written report to the Executive Director giving an overview of the quality of EFSA s science and suggesting future benchmarks that EFSA might use to ensure best scientific practice, identifying high quality aspects of the scientific output, alongside any problem areas and deviations from best practice, and providing any suggestions for improvements. ERGs should also be invited to comment on the review process itself. 3.7. Improvements of the procedure The approach proposed above should be tested during the remainder of the current 3-year cycle of the Scientific Panels and Committee (2006-2009) and improvements suggested by practical experience should be carefully considered before a new cycle of external review is undertaken. In the meantime, contacts should be made with other European and International Organizations to draw on any experience they may have with similar reviews of their scientific activities. 4. Appreciation of EFSA s scientific work by the intended users It should be noted that, under Article 61 of EFSA s founding Regulation (EC, 2002), there is already a requirement for commissioning an independent external evaluation of EFSA s achievements every 6 years. In addition it would also be useful to consider additional ways for structured consultations of both institutional and non-institutional stakeholders to assess their appreciation of EFSA s scientific work. The more general consultation with non-institutional stakeholders could be conducted via the EFSA Stakeholder Consultative Platform. The EFSA Journal (2007) 526, 9-15
CONCLUSION This proposal offers a strategy that should provide EFSA with the means to comprehensively review the quality of its scientific activities, both by internal review and by independent, external review processes, together with additional consultations with institutional and noninstitutional stakeholders. The strategy is designed to give continuing feedback to the Authority about the quality of its work. The strategy should be updated in due course, taking into account the experiences gained with these proposals. Contacts should also be made with other European and international organisations to draw on any experience they may have with similar reviews of their scientific activities. REFERENCES EFSA (European Food Safety Authority), 2006. Transparency in risk assessment carried out by EFSA: Guidance document on procedural aspects. Endorsed by the Scientific Committee in April 2006. The EFSA Journal 353: 1-16. European Communities (EC), (2002). Regulation (EC) No 178/2002 of the European Parliament and of the Council of 28 January 2002 laying down the general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures in matters of food safety. Official Journal of the European Communities, OJ L 31, 1.2.2002, 1-24 The EFSA Journal (2007) 526, 10-15
ANNEX : SELF-REVIEW TEMPLATE Template for self-review of the work process and output of the Scientific Panels and Committee to be completed during the course of the work process 1 Preparatory steps Checked (date) Significant issues identified Is there a common understanding between the experts and the originator of the request of what is needed and expected (clarity of terms of reference)? Has a realistic time frame been discussed with the originator of the request with respect to the priority and complexity of the question? Have, when necessary, appropriate adhoc experts been involved for addressing the particular question (e.g. composition of working groups)? Have the Declarations of Interests of Scientific Panel and Committee members, working groups or other ad hoc experts been obtained and checked by the secretariat for potential conflicts of interest before their involvement in the Scientific Panel or Committee and/or its working groups? Where potential conflicts of interest have been identified, have correct procedures for dealing with them, as set out in EFSA guidance, been correctly followed? Where data have been provided by outside parties, have the dossiers been checked for completeness and compliance with any relevant guidelines for data submission (e.g. adequacy for addressing the question, compliance with GLP, up-to-date, inclusion of unpublished data and comprehensive literature search)? 1 Depending on the Panel, items listed under the respective headings could be more relevant under another heading The EFSA Journal (2007) 526, 11-15
Where primary and/or additional data have been collected by the SC/P or their WGs (e.g. literature searches, consultation of external experts or national authorities, identification of relevant unpublished data, identification of any previous risk assessments on the same or similar topic by EFSA or other expert bodies) how was it ensured that the dataset was comprehensive enough and up-to-date? Have any cross-cutting issues been identified and, if so, have other relevant Panels or internal and external expert groups been appropriately involved? Has any need for external consultation been considered prior to the adoption of the document (especially for guidance documents)? Where necessary, have risk communicators been alerted by the secretariat sufficiently early in the process to ensure appropriate and timely communication? Content of the scientific document: description of the data Have the quality and completeness of the data to be used been considered? Have the data been properly described in the document? Are the sources of information used properly described in the document? Has a check been made that all scientific data have been properly referenced (within the text and in the reference list)? Checked (date) Significant issues identified The EFSA Journal (2007) 526, 12-15
Content of the scientific document: quality of the discussion Where relevant, have key studies for the assessment been properly identified and reasons given for their selection as key studies? Have the implications of potentially contradictory/conflicting data been discussed? Have opinions of other expert bodies been discussed and, if relevant, the reasons for any diverging views been explained? Have significant data gaps been identified and discussed? Have the underlying assumptions, limitations and uncertainties in the data been discussed? Have existing relevant EFSA guidance documents on risk assessment practices/methodologies been followed? Is the approach followed broadly in accordance with internationally agreed guidance on risk assessment procedures? Checked (date) Significant issues identified The EFSA Journal (2007) 526, 13-15
Content of the scientific document: quality of the conclusion Does the assessment adequately address the terms of reference? Are the conclusions coherent with the main content of the opinion? Are any assumptions, limitations and uncertainties adequately addressed? In the case of any minority opinion, has it been clearly expressed and reported? In the case of any provisional/interim conclusion, is there a clear indication of what further research or information may be needed and a timeline for follow-up? Does the assessment respect the boundaries between risk assessment and risk management? Checked (date) Significant issues identified The EFSA Journal (2007) 526, 14-15
Content of the scientific document: quality of the summary Is the summary adequately informative and clear for the target audience? Does the summary adequately reflect the context of the question and the content of the opinion, including the conclusions and any recommendations? Checked (date) Significant issues identified Delivery and follow-up Checked (date) Significant issues identified Has the document been delivered on time? If not, please specify. What means are in place for follow-up of the documents, if relevant? Date Signature by the responsible support unit secretariat If any significant issues have emerged during this best practice check, they should be drawn to the attention of the Scientific Committee/Panel The EFSA Journal (2007) 526, 15-15