ERIE is an approach to conducting retrospective long‐term impact evaluations of completed USAID interventions. These evaluations will leverage and build on existing program data to either assess if the observed short‐term impacts are sustained, or to investigate results which might only be expected to emerge over a long-term horizon. We will use innovative data collection strategies and methods to identify the appropriate counterfactuals, and generate lessons on how to plan for and conduct these long‐term impact evaluations which can be incorporated in USAID bureau and mission planning.

THE CHALLENGE

Many development interventions are believed to achieve long‐term impacts, but evidence of these impacts is often unavailable. Most program evaluations focus on short‐term outcomes because of rigid program cycles, unreliable funding streams for monitoring and evaluation, and changes in programmatic priorities. Without evidence about long‐term impacts, development organizations do not have reliable information to make cost‐effective decisions, particularly in sectors with theories of change that stretch beyond the typical project cycle. The problem is that we rarely measure the results of a policy or program over long periods of time and large geographic scales.

THE INNOVATION

To demonstrate the power of long‐term evaluation, our team will conduct a set of pilot evaluations that assess the impact of a completed aid intervention. These retrospective evaluations might occur either when short‐term impacts have been assessed but not tracked over time, or when interventions are only likely to have impacts over longer periods of time. We will also develop toolkits with guidance to help evaluators and USAID offices assess whether a long‐term evaluation is feasible for a given program, and to integrate plans for long‐term evaluations into future programming. These toolkits will help create low‐cost opportunities for long‐term impact evaluation by providing recommendations on collecting and compiling certain types of program data in systematic ways from the initial stages of measurement.

THE APPROACH

Two recent developments make long‐term evaluation in development programs more feasible now than in the past: First, the recent increase of impact evaluations has created a pool of well‐documented, rigorously studied programs to which we can return for long‐term follow‐up. Second, a rapid improvement in the granularity, coverage, and nature of data— from both aid activities and outcomes—has created new opportunities for rigorous, quasi‐experimental evaluations, including retrospective analyses. For example, advances in mobile data streams and administrative "big data" allow us to construct measures of poverty or economic activity using existing data. Geocoding, data extraction tools, and satellite imagery allow us to define what activities took place where, when, and by whom. When merged, these new kinds of program and outcome data create powerful opportunities for evaluation of development interventions that currently lack evidence of long‐term impacts.

QUICK FACTS

• Tools (illustrative): Tools for data collection and analysis will match the design questions, but may include compilation of administrative or programmatic data, mobile data, geocoding, and satellite imagery analysis
• Funding mechanism: Cooperative Agreement (buy-in option for USAID operating units)
• Partners: Notre Dame Initiative for Global Development (prime), AidData, Center for Effective Global Action (CEGA), Geo-spatial Impact Evaluation (GIE), Mathematica Policy Research
• Period of Performance: 10/01/2016– 9/30/2025

Contact the MERLIN team at merlin@usaid.gov