Evaluation Systems in Development Co operation REviEw

Pages 138
Views 5
of 138
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Description
Evaluation Systems in Development Co operation 2016 REviEw EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW This work is published under the responsibility of the Secretary-General of the OECD.
Transcript
Evaluation Systems in Development Co operation 2016 REviEw EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW This work is published under the responsibility of the Secretary-General of the OECD. The opinions expressed and arguments employed herein do not necessarily reflect the official views of OECD member countries. This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Please cite this publication as: OECD (2016), Evaluation Systems in Development Co-operation: 2016 Review, OECD Publishing, Paris. ISBN (print) ISBN (PDF) Photo credits: Cover Cover artwork adapted from Freepik.com Corrigenda to OECD publications may be found on line at: OECD 2016 You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgement of OECD as source and copyright owner is given. All requests for public or commercial use and translation rights should be submitted to Requests for permission to photocopy portions of this material for public or commercial use shall be addressed directly to the Copyright Clearance Center (CCC) at or the Centre français d exploitation du droit de copie (CFC) at Foreword Evaluation of development co-operation helps to meet public demands for accountability, and supports transparency by providing evidence about the effectiveness and impact of development policies and programmes. Evaluation also contributes to institutional learning and global knowledge, bringing to light key success factors and obstacles to effective development. High quality evaluation processes and products are important, but strong evaluation systems are needed to encourage the use of findings, ensure the integrity of the evaluation function, and make evaluative evidence readily available for learning. Changes in evaluation systems in recent years reflect evolutions in the development cooperation landscape, including the merger of ministries of international development, foreign affairs, and trade; and the establishment of new evaluation and development oversight bodies. Demand for evaluative evidence is likely to increase with the implementation of the Sustainable Development Goals; in this context growing attention must be given to supporting evaluation capacity development in line with SDG follow-up and review processes. There is a wide variety and diversity of institutional evaluation arrangements across the membership of the OECD Development Assistance Committee s (DAC) Network on Development Evaluation. This report describes how evaluation functions are structured and organised and contains individual profiles on the evaluation systems of members of the DAC Network on Development Evaluation. The report outlines key changes in DAC members and multilateral development banks evaluation systems since 2010 and offers an overview of the current state of evaluation systems in development co-operation ministries and departments. All members of the DAC Network on Development Evaluation contributed to this effort. We hope that the wealth of information contained in this report will be of use for other organisations looking to establish or make improvements to evaluation systems. The study offers useful information on current practices across the DAC Network on Development Evaluation membership and as a follow up to the 2010 report can be used to track changes in development evaluation systems over time. Finally the study highlights areas where further attention is needed, such as increasing support for evaluation capacity development. Efforts to improve evaluation policy and practice are ongoing endeavours; evaluation systems must continuously adapt to changes in development co-operation, finding new ways to support learning and accountability. Evaluation, as an essential component of good governance, encourages periodic reflection and adaptation, and has an important role to play in strategic orientations and evidence based policy and programme decision-making. Penny Hawkins Chair of the DAC Network on Development Evaluation EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW OECD Acknowledgements This report was made possible thanks to the contributions of the members of the Development Assistance Committee s (DAC) Network on Development Evaluation who provided inputs and feedback throughout the process. The high level of participation and engagement demonstrates the interest and support for this updated review of evaluation systems. The Management Group for the review was comprised of Penny Hawkins (Chair of EvalNet, DFID), Per Øyvind Bastøe/Siv Lillestøl (Norad), and Hans Lundgren (OECD Secretariat). Financial contributions were provided by Australia, Denmark, EBRD, Luxembourg, Norway and ICAI (UK) with Norad hosting the pooled funding. The Danish Management Group was contracted for the review and was responsible for the compilation of data, analysis, and drafting of the report. Additional support and editorial assistance was provided by Susanna Morrison-Métois (OECD Secretariat). Publication assistance was provided by Angèle N zinga, Elizabeth Del Bourgo and Angela Stuart (OECD Secretariat). Stephanie Coic (OECD Secretariat) formatted the cover. 4 Table of contents Acronyms and Abbreviations...9 Executive Summary...11 Part I: Evaluation systems in development co-operation...15 Chapter 1: Overview of the 2016 Review of Evaluation Systems in Development Co-operation...17 Introduction...18 Background...18 Research methods...19 Scope of the study...20 Overall trends...21 Conclusions...24 Notes...27 References...27 Chapter 2: Evaluation systems and governance...29 Institutional setup of evaluation units...30 Recent organisational changes in evaluation systems...31 Financial and human resources related to the evaluation system...35 Evaluation policies...40 Accountability and learning...43 Independence of evaluation system...44 Notes...48 References...49 Chapter 3: Evaluation processes Programming...52 Ensuring quality...55 Supplementary evaluation tools...59 Addressing capacity constraints...62 Participation and donor co-ordination...65 Dissemination...67 Evaluation use and demand...69 Notes References EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW OECD TABLE OF CONTENTS Part II: Profiles of members of the DAC Network on Development Evaluation...75 Introduction and key for the member profiles...76 African Development Bank Group (AfDB)...77 Asian Development Bank (ADB)...81 Australia...85 Austria...89 Belgium...93 Canada...97 Czech Republic Denmark European Bank for Reconstruction and Development (EBRD) European Commission (EC) European Investment Bank (EIB) Finland France Germany Iceland Inter-American Development Bank (IADB) International Monetary Fund (IMF) Ireland Italy Japan Korea Luxembourg Netherlands New Zealand Norway Poland Portugal Slovakia Slovenia Spain Sweden Switzerland United Kingdom United Nations Development Programme (UNDP) United States World Bank Group (WBG) Annex A: Budget and staffing - centralised units Annex B: Further reading Annex C: Evaluation resources by member Annex D: Interviews conducted as a part of the review exercise TABLE OF CONTENTS Figures 2.1. Typical types of evaluations that are centralised and decentralised Types of evaluation Volume of evaluation Gender disaggregation of employees working in centralised evaluation units Content of evaluation policies Important components of the evaluation process Work plan duration and organisational synchronisation Percentage of organisations that include management, policy makers, and operational units in the formulation of the evaluation work plan Alignment of evaluation work plan to demand from governing authorities Respondents perception of external expert competence Adequacy of knowledge and skills for evaluating crosscutting issues - gender, environment, and human rights Extent of participation of partners and stakeholders in evaluation activities Number of joint evaluations conducted over the last five years Dissemination of evaluation reports Findings and recommendations publicly available on organisation websites Frequency of responses of ways in which evaluation use could be improved Tables 2.1. Summary of changes over the past five years in budget and staff in multilateral organisations and bilateral agencies Summary of evaluation resources Demand for evaluation to inform decision making Effectiveness of knowledge management systems in being actively used by evaluation/operations and decision-making staff...73 A.1: Budget and staffing for centralised units EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW OECD Acronyms and abbreviations ADA AusAID AFD AfDB CIDA CODE CzDA DAC Danida DED DEval DG-DEVCO DFAT DTATD DFID EBA EBRD EDCF EIB EU EvalNet EvD GAC GIZ ICAI ICEIDA IDB IEAP IEC IEG IFC IFI IOB JICA Austrian Development Agency Australian Agency for International Development Agence Française de Développement African Development Bank Canadian International Development Agency Committee on Development Effectiveness (World Bank Group) Czech Development Agency Development Assistance Committee Danish Development Assistance Development Evaluation Division (Canada) German Institute for Development Evaluation (Deutsches Evaluierungsinstitut der Entwicklungszusammenarbeit) Directorate-General for International Cooperation and Development Department for Foreign Affairs and Trade (Australia) Department of Foreign Affairs, Trade and Development (Canada) Department for International Development (UK) Expert Group for Aid Studies (Sweden) European Bank for Reconstruction and Development Korea Economic Development Co-operation Fund European Investment Bank The European Union OECD DAC Network on Development Evaluation Evaluation Department Global Affairs Canada Deutsche Gesellschaft für Internationale Zusammenarbeit (Germany) Independent Commission for Aid Impact (UK) Icelandic International Development Agency Inter-American Development Bank International Evaluation Advisory Panel (UNDP) Independent Evaluation Committee (Australia) Independent Evaluation Group (World Bank) International Finance Corporation International Financing Institutions Policy and Operations Evaluation Department (Netherlands) Japan International Cooperation Agency EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW OECD LIST OF ACRONYMS Acronyms and abbreviations (cont.) KfW KOICA M&E MAR Kreditanstalt für Wiederaufbau Development Bank Germany Korea International Cooperation Agency Monitoring and Evaluation Management Action Record MAEC Ministry of Foreign Affairs and Cooperation (Spain) MCC MDB MFA-DCD MFAT MIGA MO NAO ODA ODE OECD PDCA RMES SDGs SEO SEQAS SIDA UNDP US USAID WBG Millennium Challenge Corporation Multilateral Development Banks Ministry of the Foreign Affairs of the Czech Republic - Development Cooperation Department Ministry of Foreign Affairs and Trade (New Zealand) Multilateral Investment Guarantee Agency Multilateral Organisations National Audit Office (UK) Official Development Assistance Office of Development Effectiveness (Australia) Organisation for Economic Co-operation and Development Plan, Do, Check, and Action Results Measurement and Evaluation Stream (World Bank) Sustainable Development Goals Special Evaluation Office Specialist Evaluation and Quality Assurance Services Swedish International Development Cooperation Agency United Nations Development Programme United States United States Agency for International Development World Bank Group 10 Executive summary The purpose of this review is to provide information to assist development organisations in building effective evaluation systems and processes to deliver high quality, credible and useful evaluations. It provides information about evaluation systems in development agencies and multilateral organisations, and analyses experiences from managing evaluation systems across organisations. The study covers the evaluation systems of the 37 members of the DAC Network on Development Evaluation (EvalNet) and 9 multilateral organisations (six development banks, the European Commission, the IMF and the UNDP). The study analyses experiences from managing evaluation systems across organisations. It identifies features of evaluation systems, and highlights some of the advantages and limitations of these in the context of various organisational structures and evaluation systems. In addition, the review identifies overall trends in evaluation systems, relating them to the review of evaluation systems performed in 2010 (OECD, 2010). The evolution of evaluation systems In recent years a number of countries have undertaken reforms of the systems that support the delivery of development co-operation. In terms of the evaluation functions, this has manifested in various ways. Several members established bodies to improve the coordination of evaluation across a complex network of agencies, and to provide independent oversight of evaluation across the development co-operation sphere. The creation of the Independent Commission for Aid Impact (United Kingdom) (ICAI), the German Institute for Development Evaluation (DEval) and the Expert Group for Aid Studies (Sweden) (EBA) as oversight or scrutiny bodies, as well as the introduction of the EU Better Regulation package indicate the emphasis on the improvement of performance and greater effectiveness. The merging of foreign policy, trade and development agendas has led to the integration of the semi-autonomous unit of New Zealand Aid into the Ministry of Foreign Affairs and Trade (MFAT), the Australian Agency for International Aid (AusAID) into the Department of Foreign Affairs and Trade (DFAT), and the Canadian International Development Agency (CIDA) merger with the Department of Foreign Affairs, International Trade and Development (DFAIT). Over recent years, systems of evaluation have become increasingly decentralised. The majority (55%) of organisations reported making use of a combination of centralised and decentralised evaluation, with the aim of increasing cost-effectiveness, relevance and ownership of evaluations for operational units, and supporting the development of capacity in partner institutions. However, ensuring the quality, coverage (of sectors, themes or strategic issues) and consistency of decentralised evaluations is challenging, affecting the usability of evaluations for synthesis purposes. Meta-evaluations help in pointing out weaknesses and where support is needed, contributing to the improvement of systems. EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW OECD EXECUTIVE SUMMARY The vast majority of member organisations have a policy document to guide evaluation work, representing an increase since The purpose of the document varies from providing a policy or set of guidelines to outlining the strategy. The emphasis on strategic questions of development is apparent in recently revised evaluation policies which demonstrate an orientation towards thematic work. However, the actual use of policy/ strategy evaluations appears to have become less common (from 92% in 2010 to 60% in 2016). Where 49% of institutions were performing project/activity evaluations in 2010, 76% are now undertaking them. On the whole, agencies responded to changes in resources affecting the evaluation system, both centralised units and decentralised operations, by focussing on learning and making an effort to increase the relevance and strategic impact of the evaluations commissioned. In general, agencies that benefit from increased resources do not increase the number of evaluations, but rather focus on issues likely to bring about new directions, for example meta-evaluation, innovative methodology, and impact at strategic levels. The Paris Declaration (2005) and Busan Partnership for Effective Development Cooperation (2011) have concentrated the efforts of institutions on internalising principles of development effectiveness. They brought greater attention to the importance of ownership and mutual accountability. However, this review finds that less than half of the organisations mention the partner countries role in evaluations, joint evaluations or collaborative partner-donor evaluation in their policies. The balance of emphasis between accountability and learning has continued to shift since the 2010 review of evaluation systems (OECD, 2010). There is an increased emphasis on undertaking evaluation for the purposes of learning; however this is not necessarily at the expense of accountability, as both are recognised to fulfil vital functions and reinforce one another. The recent structural changes mentioned above have to some extent divided the responsibilities for learning and accountability between oversight or scrutiny bodies (accountability) and central evaluation units (learning). The principle of independence as a function of evaluation systems is well rooted in the culture of EvalNet member organisations. The advantages of independent evaluations in terms of contributing to credibility and accountability are widely recognised. Nevertheless, independence needs to be balanced with the potential of isolation from operations, which is considered by many members to reduce acceptance and use of evaluation findings. The majority of members perceive that they are independent for the most part, although collaboration is necessary during the selection of evaluations to improve their relevance and usefulness for operational units. The organisational independence of the evaluation units is clearly described in the policies of the majority of member organisations, showing that this is a priority. Evaluation processes In general, programming of evaluation work is closely linked to operations with evaluations and operational planning becoming increasingly integrated. A significant number of members describe evaluation planning as being synchronised with overall organisational priorities. Decisions on whether evaluations are to be performed for decentralised interventions depend on such criteria as investment thresholds and identified specific needs to provide, develop or improve an evidence-base. Member organisations have set up structures to assess the quality of evaluation outputs, using advisory bodies, steering committees, reference groups, and special advisors to oversee the work and the process. The quality of evaluations is strongly influenced by the evaluators capacity and knowledge, derived from their education, past evaluation 12 EXECUTIVE SUMMARY experience, the subject of the intervention, and most importantly, local knowledge of politics, institutional setups, culture, and traditions. When decentralised evaluations are carried out, it can be challenging to ensure consistency and rigour, but central evaluation units provide a variety of support mechanisms to operations units, from providing input to specific milestones in the evaluation process to engagement throughout the process. Capacity building of operational units to manage evaluations contributes to the improvement of quality, although staff turnover may pose a risk. There is widespread recognition in the development community that stakeholder participation in evaluation processes helps to develop sustainability, evaluation ownership and mutual accountability. Despite this only a small number of agencies confirm that frequent or close stakeholder participation is encouraged during evaluation planning and design phases (12% and 16% respectively). Half of EvalNet members encourage partner participation in reference or steering groups. Joint evaluations are considered to be practical if the specific prog
Advertisements
Related Documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x