WLE Evaluation Inception Report - Final

INCEPTION

REPORT
APRIL 2015
Evaluation of the
CGIAR Research Program on
Water, Land and Ecosystems
Charles Batchelor
Sylvie Brouder
Federica Coccia
Elias Fereres Castiel
John Soussan
David Todd (Team Leader)
WLE Evaluation, Inception Report, April 2015
Table of Contents
List of Acronyms ............................................................................................................................... iv
EXECUTIVE SUMMARY .................................................................................................................... 1
1. Introduction ............................................................................................................................... 6
1.1 Origins of This Evaluation .................................................................................... 6
1.2 Evaluation Purpose and Users ............................................................................. 6
1.3 Purpose and Structure of the Inception Report ................................................. 7
2. Background to the Evaluation ............................................................................................... 8
2.1 Context of CGIAR Reform .................................................................................... 8
2.2 Background to the Water, Land and Ecosystems (WLE) Program ................... 10
2.2.1. Context of Research on Water, Land and Ecosystems........................... 10
2.2.2. Institutional and Program Structure and Origins .................................. 17
2.3 The WLE Portfolio .............................................................................................. 21
2.3.1. Focal Regions ........................................................................................... 21
2.3.2. 2015 WLE Portfolio of Activities ............................................................. 22
3. Scope of the Evaluation ........................................................................................................ 27
3.1 Research/Programmatic Performance ............................................................... 27
3.2 Organizational Performance ............................................................................. 29
3.3 Geographical Scope ........................................................................................... 29
3.4 Evaluability Issues and Limitations of the Evaluation ...................................... 30
3.4.1 The WLE Impact Pathways and Theory of Change ................................... 30
3.4.2 Main Limitations of the Evaluation ........................................................... 31
3.4.3 Evaluability .................................................................................................. 32
4. Evaluation Criteria and Questions ...................................................................................... 33
4.1 Introduction ........................................................................................................ 33
4.2 Strategic Questions ............................................................................................ 33
4.3 Criteria for Research/Programmatic Performance............................................ 34
4.3.1 Relevance ..................................................................................................... 34
4.3.2 Quality of Science ....................................................................................... 35
4.3.3 Effectiveness ................................................................................................ 35
4.3.4 Impact .......................................................................................................... 35
4.3.5 Sustainability ............................................................................................... 36
4.4 Criterion for the Evaluation of Organizational Performance ......................... 36
4.4.1 Efficiency ..................................................................................................... 36
iea.cgiar.org
4.5 Performance on cross-cutting issues ................................................................. 37

4.5.1 Partnerships ................................................................................................. 37
4.5.2 Gender, Poverty and Institutions ............................................................... 37
4.5.3 Capacity Strengthening .............................................................................. 37
4.6 Inter-relationships between Evaluation Criteria, Strategic Questions and Cross-
Cutting Issues ................................................................................................................ 37
5. Evaluation Approach and Methods .................................................................................... 39
5.1 Approach ............................................................................................................ 39
5.2 Methodology ...................................................................................................... 41
5.2.1 Collection and Analysis of Existing (Secondary) Sources of Information 41
5.2.2 Methods for Collection and Analysis of Original (Primary) Data ............. 43
5.3 Selection of methods ......................................................................................... 51
5.3.1 Overall Analysis and Triangulation of Findings ........................................ 51
5.3.2 Examples of Analysis Using Triangulation ................................................. 51
5.3.3 Analysis of Governance and Management ............................................... 51
5.3.4 Sequencing and Contribution of Methods ................................................ 52
6. Organization and Timing of the Evaluation .................................................................... 54
6.1 Team Composition ............................................................................................. 54
6.2 Team Roles and Responsibilities ........................................................................ 55
6.3 Evaluation governance/roles and responsibilities ............................................ 56
6.4 Quality Assurance ............................................................................................... 57
6.5 Stakeholder Involvement ................................................................................... 58
6.6 Timeline .............................................................................................................. 59
6.7 Key Deliverables and Dissemination of Findings.............................................. 60
Annex 1 – Evaluation Matrix ........................................................................................................ 62
Annex 2: Documents Consulted .................................................................................................. 64
Annex 3: Persons Consulted During Inception Mission to IMWI ......................................... 67
Annex 4: Persons Met during Inception Mission to Rome .................................................... 68
Annex 5: Outline Work plan ........................................................................................................ 69
Annex 6: Outline Contents of Final Report .............................................................................. 70
Tables
Table 1: 2015 Flagship budgets by funding source and number of activities................ 23

Table 2: Distribution of research projects by CGIAR Center. .......................................... 24
ii
iea.cgiar.org
Table 3: list of projects ...................................................................................................... 26

Table 4: Inter-relationships between Evaluation Criteria, Strategic Questions and Cross-
Cutting Issues ..................................................................................................................... 38
Table 5: areas of expertise of each team member. ......................................................... 54
Table 6: WLE Evaluation Team ......................................................................................... 56
Table 7: Reference Group Members ................................................................................. 57
Table 8: WLE evaluation stakeholders ............................................................................. 58
Table 9: Outline Timeline for WLE Evaluation ................................................................. 60
Figures
Figure 1: Basic WLE Governance and Management structure ........................................ 20

Figure 2: Distribution of WLE Funding by CGIAR Center in 2015 ................................... 24
Figure 3: 1. Figure 2: Gender as a proportion of Flagship Budget in 2015 .................. 25
Figure 4: Relationship between WLE Program Structure and Theory of Change ......... 30
Figure 5: Overview of the Theory of Change and Impact Pathways of WLE ................. 41
Figure 6: Overview of Sequencing and Contribution of Methods ................................. 53
Boxes
Box 1: Major Sources of Funding in the CGIAR System ..................................................... 9
iii
iea.cgiar.org
LIST OF ACRONYMS
AAS CGIAR Research Program on Aquatic Agricultural Systems
ADB Asian Development Bank
AfDB African Development Bank
AWM Agriculture Water Management
BMGF Bill and Melinda Gates Foundation
BRAC Bangladesh Rural Advancement Committee
CCAFS C GIAR Research Program on Climate Change, Agriculture and Food Security
CI Conservation International
CIAT International Center for Tropical Agriculture
CIRAD La recherche agronomique pour le développement
CPWF CGIAR Challenge Program on Water and Food
CRP CGIAR Research Program
DAI Decision Analysis and Information Systems
ES Ecosystem services
ESR Ecosystems Services and Resilience
EAWAG Institute of Aquatic Science and Technology
FAO United Nations Food and Agriculture Organization
FTA CGIAR Research Program on Forests, Trees and Agro‐Forestry
GEF Global Environment Facility
GPI Gender, Poverty and Institutions
GSF Global Soils Forum
GSP Global Soils Partnership of the FAO
GWI Global Water Initiative
GWP Global Water Partnership
GWSP Global Water Systems Project
IAHS The International Association of Hydrological Sciences
ICRAF World Agroforestry Centre
IDO Intermediate development outcome
IEA Independent Evaluation Arrangement
IES Integrating Ecosystem Solutions into policy and investments
IF Innovation fund
IFAD International Fund for Agricultural Development
INVEST Integrated Valuation of Ecosystem Services and Trade‐offs
IPBES Intergovernmental Platform on Biodiversity and Ecosystem Services
ITP IWMI‐Tata Water Policy Research Program
IUCN International Union for Conservation of Nature
iv
iea.cgiar.org
IWA International Water Association
IWMI International Water Management Institute
LWP Land and water productivity
M‐POWER Mekong Program on Water, Environment and Resilience
MRV Managing Resource Variability and Competing Uses
NGOs Non‐governmental organizations
NRM Nature resource management
ODI Overseas Development Institute
PES Payment for ecosystem services
PIM CGIAR Research Program on Policy, Institutions and Markets
RDE Regenerating Degraded Agricultural Ecosystems
RRR Recovering and Reusing Resources in Urbanizing Ecosystems
SDC Swiss Development Cooperation
SDG Sustainable Development Goals
SIDA Swedish International Development Agency
SLM Sustainable Land Management
SRI Systems Rice Intensification
SSA Sub‐Saharan Africa
TNC The Nature Conservancy
UNDP United Nations Development Program
UNEP United Nations Environment Program
UNESCO‐IHE United Nations Educational, Scientific and Cultural Organization – Institute of
Water
USAID United States Agency for International Development
VBA Volta Basin Authority
WB World Bank
WHO World Health Organization
WLE CGIAR Research Program on Water, Land and Ecosystems
WSP Water and Sanitation Program of the World Bank
iea.cgiar.org
EXECUTIVE SUMMARY
1. Since 2010, the CGIAR has undergone radical transformation, driven by a new strategy that
has affected all parts of the CGIAR, including centres, donors, partners, and all levels of leadership
and management. The Centres have re‐aligned their research activities, institutional capacities,
financial resources and partnerships to 15 research programs (CRPs). Two new system‐level pillars
were established, namely the Fund Council and the Consortium. Donors committed to a new funding
strategy and investment in the CGIAR as a whole grew to around $1 billion.
2. This evaluation covers the CRP on Water, Land and Ecosystems (WLE), which is one of ten
CRPs that IEA is evaluating over the period 2013‐15. Its primary purpose is to increase the
contribution that WLE is likely to make toward CGIAR goals, enhancing the productivity and
sustainability of water and land resources in agro ecosystems, as well as the livelihoods of poor
producers and consumers in developing countries.
3. Stakeholders of this evaluation include the management of WLE, all participating Centers,
partners associated to the Program, the CGIAR Fund Council, and the Consortium Board. These
stakeholders will be consulted and engaged throughout the evaluation through structured
interviews, surveys, and site visits.
4. This inception report aims to provide an initial roadmap for the conduct of the evaluation,
which can help guide the evaluation team and inform evaluation stakeholders. In order to do this, its
structure includes the following elements:
 A brief analysis of the context in which the evaluation is taking place, to inform the
evaluation design
 A conceptual framework for evaluation of the key evaluation criteria
 A short list of key evaluation questions that build on the evaluation TOR and preliminary
stakeholder consultation and analysis during the inception phase
 Outline evaluation methodology and tools to be used and analysis, including initial thoughts
on sampling frames, site selection and consultation
 Preliminary work plan, outlining provisional division of duties among the team.
5. The original WLE funding proposal1 states its objective as, “To learn how to intensify farming
activities, expand agricultural areas and restore degraded lands, while using natural resources wisely
and minimizing harmful impacts on supporting ecosystems.”
6. WLE originally consisted of five Strategic Research Portfolios (SRPs). These covered:
 Irrigated systems
 Rainfed systems
 Resource Recovery and Reuse
1
WLE 2011
1
iea.cgiar.org
 River Basins (which incorporated several CPWF projects)
 Information Systems.
7. In 2014, after structural reform to the program for the 2015‐16 Extension Period, the five
original SRPs were replaced by five Flagships. These are as follows:
 Flagship 1: Integrating Ecosystem Solutions into Policy and Investments (IES)
IES pursues policy and investments in ecosystem services through influence and decision
support—with unique, context‐specific impact pathways in four regions where international
rivers occur called Focal Regions—and an Innovation Fund
 Flagship 2: Sustainably Increasing Land and Water Productivity (LWP)

LWP research aims to impact small farmers and large‐scale irrigation; gender equity and
resource access equity; and the sustainability of ecosystem services.
 Flagship 3: Regenerating Degraded Agricultural Ecosystems (RDE)
RDE hypothesizes change through research on degraded landscapes.
 Flagship 4: Recovering and Reusing Resources in Urbanized Ecosystems (RRR)
RRR seeks to reduce negative footprints of urbanization through private sector investments, by
promoting business models and supporting technological innovation.
 Flagship 5: Managing Water Resource Variability and Competing Use (MRV)
MRV research is targeted to reconcile competing claims on variables resources.
8. In both SRPs and Flagships, the program has incorporated crosscutting themes of Gender,
Poverty, and Institutions (GPI) and Ecosystem Services and Resilience (ESR). In early 2014, WLE
issued a comprehensive gender strategy, which aimed to help integrate gender themes into planning
and management processes and into the portfolio. This was followed in late 2014 by an Ecosystem
Services and Resilience framework document. Both of these approaches cut across research areas,
to such an extent that most projects from every Flagship contribute to both, which have been
defined as Core Themes. The previous fifth SRP on Decision Analysis and Information Systems (DAI)
became a third Core Theme, intended to strengthen decision analysis and improve measurement of
an intervention’s impact, thereby helping to build capacity among partners in these areas. As a Core
Theme, it is intended to complement all of the Flagship research areas, although it is budgeted
slightly differently from the ESR and GPI Themes. Furthermore, while most projects will have ESR
and GPI aspects, DAI is currently engaged in relatively few projects across the portfolio.
9. The overall WLE portfolio has a 2015 budget in excess of $53 million. Some 11 CGIAR Centers
currently participate in WLE, with IWMI as Lead Center and disbursing the largest single share of the
overall funding.
10. The evaluation scope encompasses all activities covered by the WLE programme. The
concept of WLE was specified in its initial proposal document and formalised through Program
iea.cgiar.org
Agreements. Much of the first WLE cohort consists of projects, which were initially designed and
implemented as part of existing programs, such as the Challenge Program on Water and Food
(CPWF) and were then incorporated into WLE through a process of mapping Projects, which
commenced before WLE start up in 2012, will be at various stages or progress and some may have
been completed. The extent to which these have produced results at outcome or impact level will be
assessed as part of the summative dimension of the evaluation.
11. The Extension Proposal for CGIAR Research Program on Water, Land and Ecosystems
(Resubmit, September 8, 2014) is expected to lead to some modification in the types of activities
undertaken under WLE, within the guidance offered under the Second Call for Proposals. Although
these activities will be only starting up at the time of this evaluation, their approach (particularly
where detailed through the means of a Theory of Change) will be assessed as part of its formative
dimension.
12. A multidisciplinary team covering a wide range of disciplines, areas of expertise and
experience will undertake the evaluation. The core evaluation team is comprised of five team
members.
13. According to the CGIAR Standards for Independent External Evaluation, there are six key
criteria to be used in its independent CRP evaluations, namely relevance, science quality, efficiency,
effectiveness, impact and sustainability. In line with this requirement, the TOR specify the key
evaluation criteria and related core evaluation questions, which should be used to explore
performance against these criteria. The following set of strategic questions that the evaluation
needs to address has been developed from this process:
 Are the conceptual framework and key hypotheses of WLE coherent, effectively
communicated and appropriate for the wide range of issues and diversity of locations
included in the program?
 Are the impact pathways identified by the program ones that are likely to achieve the
desired development outcomes and are they consistently developed across the different
components of the program?
 Is the quality of science in the program of an appropriate standard, does it reflect the
CGIAR’s comparative advantage in science and is it engaged with and reflective of
international developments in the different fields included in the program?
 Are the partnership strategy and range of partnerships being developed in the program
consistent with the program’s overall goals and the development of impact pathways within
the program?
 Are the management procedures and governance structure of the program appropriate,
efficient and consistently applied and is there clarity on the roles and operational
procedures of different components of the management structure?
 Is the development of the program finding an effective balance between ensuring the
delivery of concrete planned outcomes within the program life whilst at the same time
creating a long‐term development trajectory to ensure the program’s approach and benefits
will continue beyond the presently defined program period?
iea.cgiar.org
14. In order to ensure a consistent approach to both the summative and formative intentions,
the evaluation will adopt a Theory of Change approach. This will assess the achievements and
potential achievements of the WLE programme against an anticipated overarching results chain,
which will lead from individual inputs and outputs towards long term and broad scale effects, at
outcome and impact level. The Theory of Change approach adopted by the team has been derived
from the document CGIAR Research Programme on Water, Land and Ecosystems (WLE) 2014.
Ecosystem Services and Resilience Framework.
15. The evaluation will use primary and secondary data collection and analysis methods as
follows. Secondary data collection and analysis methods, which are likely to be used, are presented
below.
 Portfolio Analysis
 Document Review
 Meta Analysis
 Quality of Science Analysis
 Coherence Analysis
16. Primary data collection and analysis methods, which are likely to be used, are presented
below.
 Semi‐structured interviews
 Group discussions
 Social surveys
 Workshops.
 Case Studies
17. The following Case Studies will be conducted:
 Country Case studies in all WLE Focal Regions
 IMWI – TATA Partnership
 Mekong River Basin
 Field missions.
18. Initial field missions (Laos, Thailand and India) are planned to take place from about 1 to 11
May 2015. The evaluation team will then meet in Sri Lanka for discussions within the team and with
IMWI stakeholders from approximately 12 to 17 May, after which a second set of field missions
(Cambodia, Ethiopia, Kenya and Ghana) will be undertaken in June and July 2015. The exact dates of
missions will be finalised in collaboration with participating stakeholders and taking account of
national constraints.
19. An Independent External Evaluation Team will conduct the Evaluation. The Team Leader has
final responsibility for the evaluation report and all findings and recommendations, subject to
iea.cgiar.org
adherence to CGIAR Evaluation Standards. The IEA is responsible for planning, designing, initiating,
and managing the evaluation. The IEA will also be responsible for the quality assurance of the
evaluation process and outputs, and for the dissemination of the results.
20. WLE management will play a key role in helping provide for the evaluation team’s
informational needs. It is also responsible for giving factual feedback on the Draft Report and for
preparing the Management Response to the Final Report. It will assist in dissemination of the report
and its findings and lessons and it acts on the accepted recommendations. While the evaluation is
coordinated with WLE management, IWMI as the lead Centre is also a key stakeholder in the
evaluation. The Centre and its leadership and board are expected to make themselves available for
consultations during the evaluation process.
21. A Reference Group has been set‐up to work with the IEA Evaluation Manager and Team
Leader to ensure good communication with, learning by, and appropriate accountability to primary
evaluation clients and key stakeholders, while preserving the independence of evaluators.
22. External peer review: The IEA quality assurance of evaluations includes the peer review for
each CRP evaluation by two external peer reviewers at two stages in the evaluation process: the
draft inception report and the draft evaluation report.
iea.cgiar.org
1. INTRODUCTION
1.1 Origins of This Evaluation
23. As noted in the Terms of Reference for this evaluation: “In the CGIAR, the Independent
Evaluation Arrangement (IEA) is responsible for System‐level external evaluations. IEA’s principal
mandate is to lead the implementation of the CGIAR Policy for Independent External Evaluations2
through the conduct of strategic evaluations of CRPs and of other institutional elements of the
CGIAR. IEA is also charged with developing a coordinated, harmonized and cost‐effective evaluation
system in the CGIAR. IEA’s first four‐year Rolling Evaluation Work Plan 2014‐17, approved in
November 2013 by the Fund Council, foresaw the evaluation of 10 CRPs between 2013 and 2015.
24. The CRP on Water, Land and Ecosystems (WLE) is one of the ten CRPs that IEA is evaluating
over the period 2013‐15”.
1.2 Evaluation Purpose and Users
25. As stated in 1.1 above, this evaluation is one of a substantial set, which IEA will manage by
the end of 2015. In common with these3: “The primary purpose of this evaluation is to increase the
contribution that WLE is likely to make toward CGIAR goals, enhancing the productivity and
sustainability of water and land resources in agroecosystems, as well as the livelihoods of poor
producers and consumers in developing countries.
26. As in all CRP evaluations, the purpose of the evaluation of WLE is to provide essential,
evaluative information for decision‐making—by both CRP management and funders — on issues
such as extension, expansion and structure of the program, as well as adjusting some aspects of the
program.
27. In November 2013 the CGIAR Fund Council agreed that all current CRPs should undergo
some form of evaluation before the call for the second round of CRPs and full proposal development
is initiated. In that context, the evaluation of WLE will also provide information for decisions on
program formulation and selection in the 2015 call for second‐cycle CRP funding. Taking into
account the stage of the program and given its nature and timelines for results, the evaluation aims
to provide an overview and critical analysis of the relevance of the program, as well as its
achievements and progress to date.
28. The evaluation provides both accountability and learning. It re‐enforces the principle of
mutual accountability and responsibility among program, donors and partners. And it fosters
institutional learning among the CRP and its stakeholders, for improving program relevance,
efficiency, and the likelihood of sustainable results. Therefore, it will look at the extent to which WLE
is responding within its mandate, to the vision and focus underlying the CGIAR reform—especially
through a delivery orientation, clearer accountability, and synergy through efficient partnerships.
2
http://www.cgiarfund.org/sites/cgiarfund.org/files/Documents/PDF/CGIAR_evaluation_policy_jan2012.pdf
3
Terms of Reference, IEA, 2014. P7.
6
iea.cgiar.org
29. Stakeholders of this evaluation include the management of WLE, all participating Centers,
partners associated to the Program, the CGIAR Fund Council, and the Consortium Board. These
stakeholders will be consulted and engaged throughout the evaluation through structured
interviews, surveys, and site visits. A reference group will be convened to represent WLE
management, governance, partners and stakeholders closely involved in the CRP.
1.3 Purpose and Structure of the Inception Report
30. This inception report builds on the Terms of Reference of the evaluation and aims to provide
an initial roadmap for the conduct of the evaluation, which can help guide the evaluation team and
inform evaluation stakeholders. In order to do this, its structure includes the following elements:
 A brief analysis of the context in which the evaluation is taking place, to inform the
evaluation design
 A conceptual framework for evaluation of the key evaluation criteria
 A list of evaluation questions that build on the evaluation TOR and preliminary stakeholder
consultation and analysis during the inception phase
 Outline evaluation methodology and tools to be used and analysis, including initial thoughts
on sampling frames, site selection and consultation
 Preliminary work plan, outlining provisional division of duties among the team.
 Consideration and explanation of deviations from the evaluation TOR.
iea.cgiar.org
2. BACKGROUND TO THE EVALUATION
2.1 Context of CGIAR Reform
31. Since 2010, the CGIAR has undergone radical transformation, driven by a new strategy that
has affected all parts of the CGIAR, including centers, donors, partners, and all levels of leadership
and management. The Centers have re‐aligned their research activities, institutional capacities,
financial resources and partnerships to 15 research programs (CRPs). Two new system‐level pillars
were established, namely the Fund Council and the Consortium. Donors committed to a new funding
strategy and investment in the CGIAR as a whole grew to around $1 billion.
32. As part of the reform process, CGIAR established a Strategy and Results Framework, which
establishes its overarching goals of less rural poverty, better food security, better nutrition and
health and sustainably managed resources. The CRPs, which are developed and delivered by CGIAR
centers and partners, should be characterized by:
 A focus on development outcomes
 Explicit expectations about the role of partners in shaping strategy and delivering results
 Their larger scale, when compared to the CGIAR’s earlier programmatic initiatives, such as
Challenge Programs
 The complexity of the relationships between CRPs and centers.
33. Although designed as 10 year programs, most CRPs were initially approved for a three‐year
period to run in parallel to the SRF. The SRF is currently being revised and will be submitted by the
Consortium Office for approval by the Fund Council in April 2015. A Mid‐Term Review of CGIAR
reform, commissioned by the CGIAR Fund Council in April 2013 to assess progress and make
recommendations for course correction was recently completed. The Fund Council accepted all its
recommendations except the one related to the governance structure of the CGIAR.
34. At the request of the Consortium, all CRPs have applied for extension funding for 2015‐16
and submitted revised proposals for the extension period that were reviewed by the ISPC, the
Consortium and approved by the Fund Council. The budget allocation for WLE’s extension proposal
for these years was officially approved by the Fund Council in November 2014. The total of
$128,590,000 was supposed to be reached through contributions of $60.69 million through
Windows 1 and 2 and $67.9 million through Window 3 and bilateral funding. The Windows 1 and 2
funding however was recently cut down to $46 million.
35. A second call of CRPs is currently been developed. The process for this second call takes into
account the MTR recommendation that there should be an element of competition in the selection
of future CRPs. It will be a two‐step process, with CRP pre‐proposals expected to be submitted in
mid‐2015. Following an assessment, the Consortium Board and the Fund Council will determine
which pre‐proposals should be developed into full CRP proposals by April 2016.
iea.cgiar.org
36. The new CRP governance and management arrangements4 were designed to build on
existing structures and capacities to provide the CGIAR with the flexibility to adapt its portfolio of
research programs as results are achieved and new opportunities and challenges emerge. Under this
approach, the CRPs function as time‐limited joint ventures, mandated to effectively mobilize
resources and partners. The CGIAR Centers provide an institutional framework for the CRPs and a
critical mass of human and physical research capacity. Within this overall structure, the lead centers
have program agreements with the Consortium to provide it with accountability for CRP
performance. The Centers are able to draw on existing partnerships and collaborations as a starting
point for a more extensive and inclusive partnership strategy. Funding for the CRPs is based on two
essential funding channels: bilateral project funding from individual donors for individual projects
mapped to CRPs and programmatic funding for each CRP as a whole from Windows 1 and 2 of the
CGIAR Fund. Windows 1 and 2 funding is usually referred to as “unrestricted funding” compared to
the more restricted Window 3 and bilateral funding (see Box 1 below for a definition of the sources
of funding in the CGIAR System). The intention is that the CRPs and Centers should be
interdependent for access to and control of resources and results, with functions that overlap, co‐
exist and complement, whilst also in competition for some funding.
The funding sources available to CRPs in the reformed CGIAR are shown in Box 1 below.

Box 1: Major Sources of Funding in the CGIAR System
 To maximize coordination and harmonization of funding, donors to CGIAR are strongly
encouraged to channel their resources through the CGIAR Fund. Donors to the Fund may
designate their contributions to one or more of three funding “windows”:
 Contributions to Window 1 (W1) are the least restricted, leaving to the Fund Council how
these funds are allocated to CGIAR Research Programs, used to pay system costs or otherwise
applied to achieving the CGIAR mission.
 Contributions to Window 2 (W2) are designated by Fund donors to specific CGIAR Research
Programmes.
 Contributions to Window 3 (W3) are allocated by Fund donors to specific CGIAR Centres.
Participating Centres also mobilize financial resources for specific activities directly from donors as
bilateral funding and negotiate agreements with their respective donors for the use of these
resources.
Source: CGIAR website: http://www.cgiar.org/who‐we‐are/cgiar‐fund/
4
Review of CRP Governance and Management, P10
9
iea.cgiar.org
2.2 Background to the Water, Land and Ecosystems (WLE) Program
2.2.1. Context of Research on Water, Land and Ecosystems
37. Research into water, land and ecosystems has seen considerable change and development
over recent years. Disciplinary research into issues such as hydrological modelling, the engineering
of irrigation systems, land and soil erosion and the distribution and species composition of
ecosystems has continued with an emphasis on increasing productivity and sustainability while
minimising environmental trade‐offs. In addition, multidisciplinary efforts that encompass the social
and natural sciences and engineering have expanded rapidly to tackle some of the more complex
WLE issues, making good use of the new ICT tools and methods.
38. Key research topics relating to WLE include:
39. Sustainable land management (SLM): FAO5, CGIAR and many national research
organisations continue to view SLM as an imperative for sustainable development that has the
potential to play a key role in harmonizing the complementary, yet historically conflicting goals of
production and environment. One of the most important aspects of SLM is this critical merger of
agriculture and environment through twin objectives: i) maintaining long term productivity of the
ecosystem functions (land, water, biodiversity) and ii) increasing productivity (quality, quantity and
diversity) of goods and services, and particularly safe and healthy food.
40. Within R4D, Sustainable Intensification (SI) has emerged as the comprehensive framework
for simultaneous pursuit of hunger / poverty eradication and improvement of agriculture’s
environmental performance. SI eschews one‐size‐fits all solutions in favour of data driven,
knowledge intensive management6 and, thus, is inherently challenging to deliver to even the most
sophisticated farmers and land managers. Further, a philosophical divide has emerged between
those who place yield gains as paramount in the context of population growth and the humanitarian
need to alleviate hunger7 and those who advocate a more agro‐ecologically focussed approach that
emphasizes ecology and prioritizes multi‐objective optimisation through harnessing biological
regulation, food webs and more holistic approaches to ecosystem services8. WLE has explicitly posed
the latter approach as a testable hypothesis and has aligned its portfolio accordingly.
41. Integrated approaches to water resource management: A central concept in the
international approach to water has been Integrated Water Resources Management (IWRM); but,
whilst an attractive idea, there are few examples of it working in practice, especially at a river basin
level. In addition, some critics have argued that as a concept IWRM is naïve and idealistic9. Others
5
See http://www.fao.org/nr/land/sustainable‐land‐management/en/
6
Solutions for Sustainable Agriculture and Food Systems, Technical Report for the Post‐2015 Development
Agenda. Prepared by the Thematic Group on Sustainable Agriculture and Food Systems of the Sustainable
Development Solutions Network.
7
e.g. Cassman, K.G. 1999. Ecological intensification of cereal production systems: Yield potential, soil quality,
and precision agriculture. PNAS, 96: 5952‐5959.
8
e.g. Dore, T., Makowski, D., Malezieux, E., Munier‐Jolain, N., Tchamitchian, M., Tittonell, P. 2011. Facing up to
the paradigm of ecological intensification in agronomy: Revisiting methods, concepts and knowledge.
European J. Agron. 34:197‐210.
9
e.g.Biswas, A. K. (2004). Integrated water resources management: A reassessment. Water International, 29, 248–256.
10
iea.cgiar.org
have maintained that IWRM takes insufficient account of the politics that are at the core of most (if
not all) important water‐related decisions1011. Paradoxically, the IWRM concept has also been
criticized for being: 1) Too broad to have any real meaning and 2) So narrow that it focuses mainly
on water and ignores important linkages between land and water management12. Concerns relating
to the way the IWRM concept has been interpreted and implemented led researchers to invest in
development of the concept of light IWRM13. In contrast to prescriptive top‐down IWRM14, light
IWRM aims to be problem‐focused, opportunistic and adaptive/iterative when applying core IWRM
principles, especially at the water‐users level. The intended outcome of applying light IWRM is a
system of managing water resources and water services delivery that has developed incrementally
over many years and, as a result, is better adapted or tailored to the political economy of a given
area.
42. Water/Energy/Power Nexus: During the last 3‐4 years, the Water/Energy/Power nexus
approach has gained traction as an alternative or complementary approach to IWRM15. A key
difference between the two approaches is that IWRM always starts with water resources when
considering inter‐relationships between water, land, food and energy whereas the nexus approach
can start from different perspectives (e.g. water, food or energy)16. Whilst the nexus approach has
significant merit, it has also attracted criticism for being unnecessarily limiting and prescriptive, for
example, by not explicitly highlighting inter‐linkages with climate change, poverty and pro‐poor
development.
43. Agronomy and breeding of cultivars for dry environments: Progress in producing drought
resistant cultivars has been very slow so far, but new approaches are being developed to
complement past efforts. Better understanding of the role of physiology and of the co‐limitation of
water and nitrogen17 should lead to more rapid increases in yield under drought conditions.
Optimizing water productivity18 (Fereres et al., 2014) should also minimize the tradeoffs between
yield and water productivity in the future.
44. Deficit/supplemental irrigation: Water scarcity in irrigated areas is forcing farmers to
explore the profitability of deficit or supplemental irrigation, a technique that uses a fraction of the
10
e.g. Jensen, K. M. (2013). Viewpoint–swimming against the Current: Questioning development policy and practice. Water
Alternatives, 6, 276–283.
11
Molle, F. 2008. Nirvana concepts, narratives and policy models: Insight from the water sector.Water Alternatives 1(1):
131‐156. http://www.water‐alternatives.org/index.php/allabs/20‐a‐1‐1‐8/file
12
Merrey,D.J, Drechsel,P., Penning de VriesF.W.T. and Sally,H. 2005. Integrating ‘‘livelihoods’’ into integrated water
resources management: taking the integration paradigm to its logical next step for developing countries. Regional
Environmental Change (2005) 5: 197–204DOI 10.1007/s10113‐004‐0088‐5
13
Moriarty, P.; Butterworth, J. and Batchelor, C. 2004. Integrated water resources management and the domestic water
and sanitation sub‐sector. Thematic Overview Paper. Delft, the Netherlands: IRC International Water and Sanitation
Centre.
14
Shah, T. and van Koppen, B. 2006. Is India ripe for integrated water resources management IWRM: Fitting water policy to
national development context. Economic and Political Weekly XLI(31): 3413‐3421.
15
e.g. http://www.water‐energy‐food.org/en/news/view__1612/the‐nexus‐approach‐vs‐iwrm‐gaining‐conceptual‐
clarity.html
16
e.g. Hoff, H. (2011). Understanding the Nexus. Background Paper for the Bonn2011 Conference: The Water, Energy and
Food Security Nexus. Stockholm Environment Institute, Stockholm.
17
Sadras and Richards, 2014
18
Fereres et al., 2014
11
iea.cgiar.org
full irrigation requirements in a way that minimizes the negative impacts of water deficits on yield19.
The situation is particularly critical in China and the Middle East20. In horticulture, deficit irrigation
has been found to reduce irrigation water use without negative impact on yield in many cases21.
45. Advances in remote sensing and modelling for performance evaluation of irrigation
networks and for precision irrigation management: Advances in remote sensing and modeling tools
now permit the quantification of irrigation performance and the physical mapping of water stress in
rainfed areas with a degree of precision which was unthinkable only a decade ago22. One of the
applications that would be most useful would be to map yield gaps and water productivity gaps in
irrigated and rainfed systems23. The techniques developed are also paving the way for the
development of precision irrigation management, an approach by which a field that is
homogeneously irrigated, may receive variable water amounts in different parts according to its
precise needs, thus minimizing water waste24:
o Remote sensing of water balance components: Thermal infra‐red based ET monitoring has
reached a stage where it is operationally and economically feasible deliver providing large‐area
ET information at accuracies and spatiotemporal resolutions that are required for many practical
water resource applications25. A similar stage has been reached with estimation: remotely
sensed rainfall26 and remotely‐sensed soil moisture and groundwater conditions27.
 Recycling transpiration: Technically speaking, it is now possible to collect transpirational water
and recycle it, thus reducing the consumptive use of water in enclosed areas (greenhouses) to
negligible amounts. New research activities have been undertaken to make this option
commercially viable for intensive horticultural production under protected cultivation.
 Wastewater reuse: The adoption of re‐cycling of drainage water and wastewater use in
agriculture tends to be positively correlated with water scarcity. Re‐use of drainage water is a
reality in most large irrigation schemes, in particular in the large rice‐based systems of Asia. Of
lesser global significance, but locally important, is the re‐use of urban wastewater (it is estimated
that world‐wide some 20 million hectares of agricultural land is irrigated with wastewater). Efforts
19
Boyer, C. N., Larson, J. A., Roberts, R. K., McClure, A. T., & Tyler, D. D. (2014). The impact of field size and
energy cost on the profitability of supplemental corn irrigation. Agricultural Systems, 127, 61‐69.
20
Oweis, T. Y. (2014). The role of water harvesting and supplemental irrigation in coping with water scarcity
and drought in the dry areas. Drought and Water Crises, 191.).
21
Pérez‐Pérez, J. G., Robles, J. M., & Botía, P. (2014). Effects of deficit irrigation in different fruit growth stages
on ‘Star Ruby’grapefruit trees in semi‐arid conditions. Agricultural Water Management, 133, 44‐54.
22
Ghahroodi, E. M., Noory, H., & Liaghat, A. M. (2015). Performance evaluation study and hydrologic and
productive analysis of irrigation systems at the Qazvin irrigation network (Iran). Agricultural Water
Management, 148, 189‐195.
23
van Noordwijk, M., & Brussaard, L. (2014). Minimizing the ecological footprint of food: closing yield and
efficiency gaps simultaneously?. Current Opinion in Environmental Sustainability, 8, 62‐70.
24
(Gonzalez‐Dugo, V., Goldhamer, D., Zarco‐Tejada, P. J., & Fereres, E. (2015). Improving the precision of
irrigation in a pistachio farm using an unmanned airborne thermal system. Irrigation Science, 33(1), 43‐52)
25
Anderson, M.C, Allen, R.G, Morse, A. and Kustas, P. 2012. Use of Landsat thermal imagery in monitoring
evapotranspiration and managing water resources. Remote Sensing of Environment Volume 122.
26
e.g http://trmm.gsfc.nasa.gov/
27
e.g. http://drought.unl.edu/MonitoringTools/NASAGRACEDataAssimilation.aspx
12
iea.cgiar.org
are needed to better assess re‐use and its potential, and promote safe recycling of wastewater in
agriculture, in particular in water‐scarce areas28.
 Information management technologies & applications: New information management

technologies and cyberinfrastructure29 are emerging that have the potential to transform many
aspects of WLE. These include:
o Virtual observatories can link and integrate online: 1) Global, national and global information
bases (containing both terrestrial and remotely sensed biophysical and societal information); 2)
Networks of environmental sensors; 3) Information collected by users of water services or by
citizen scientists; and, 4) Inconnnected web or cloud‐based services or application30.
o Unmanned aircraft systems (UAVs) offer the opportunity for individual scientists and small
teams to obtain low‐cost, repeat imagery at high resolutions (;1 cm to 1 m) tailored to the
specific areas, products and delivery times of research interest31;
o GPS‐enabled smartphones are having a major impact on the abilibity of users to report
problems or inadequacies in the services they receive using applications such as AKVO Flow32 or
Water Point Mapper33;
o IT applications that have built‐in capabilities for locating and accessing water‐related
information via the internet. An example of this approach is the CUAHSI Hydro‐Desktop34
application that provides access to remote data archives using the CUAHSI WaterOneFlow web
service35 36.
28
http://www.fao.org/docrep/016/i3015e/i3015e.pdf
29
The generic term cyberinfrastructure is used to describe the infrastructure that takes advantage of recent
advances in information technology.
30
Laniak, G.F et al. 2013. Integrated environmental modeling: A vision and roadmap for the future.
Environmental Modelling & Software, 39, 3‐23.
31
Vivoni,E.R, Rango,A, Anderson,C.A, Pierini,N.A, Schreiner‐McGraw,A.P., Saripalli,S., and Laliberte,A.S. 2014.
Ecohydrology with unmanned aerial vehicles. Ecosphere Volume 5(10) v Article 130
32
For more information see: http://www.akvo.org/web/introducing‐akvo‐flow
33
For more information see: http://www.waterpointmapper.org/
34
For more information see: http://hydrodesktop.codeplex.com/
35
Ames, D.P., Horsburgh, J.S. Yang Cao, Kadlec, J., Whiteaker, T., Valentine, D. 2012. HydroDesktop: Web
services‐based software for hydrologic data discovery, download, visualization, and analysis. Environmental
Modelling & Software 37
36
Tarboton, D.G., Horsburgh, J.S., Maidment, D.R., Whiteaker, T., Zaslavsky, I., Piasecki, M., Goodall, J.,
Valentine, D., Whitenack, T., 2009. Development of a community hydrologic information system. 18th World
IMACS Congress and MODSIM09 International Congress on Modelling and Simulation. In: Anderssen, R.S.,
Braddock, R.D., Newham, L.T.H. (Eds.), Modelling and Simulation Society of Australia and New Zealand and
International Association for Mathematics and Computers in Simulation, pp. 988e994.
http://www.mssanz.org.au/modsim09/C4/tarboton_C4.pdf
13
iea.cgiar.org
46. Agriculture is both a cause and a victim of water scarcity37. Inter‐sectoral competition for
water is most obvious in the hinterlands of large urban centres, but water scarcity can arise in all
catchments where the intensification of agriculture in headwater areas reduces water supply
downstream. Unsustainable groundwater use can have long‐term impacts on agricultural production
in areas such as South Asia, where a boom in groundwater‐based irrigation in the 1980s and 1990s
led to a major increase in agricultural production that is now constrained by aquifer depletion. Many
WLE challenges are as much societal as they are biophysical. As a result, much of WLE research
focuses on issues related to governance, institutions, legislation, economics, gender and the wide
political economy of areas of interest. Increasingly WLE‐related research in both funding
programmes and journal submissions and publications focuses on issues that include:
 Governance: the nature of governance of natural resources, including the institutional
arrangements through which governance operates, has become a major issue across all of these
themes. This has emerged from the understanding that effective management is contingent upon
good governance arrangements, whilst management failures and a consequent lack of
sustainability are in part a reflection of governance failures. There are strong and obvious links
between research and policy in this field. One important research theme here is the nature of
formal and informal governance arrangements, themselves often a reflection of different
stakeholder interests in these areas.
 New concepts and ways of working: New (or relatively new) societal concepts relating to
addressing WLE challenges include: “thinking politically, working differently”38, “politically smart,
locally‐led development”39 and “good enough governance”40. New ways of diagnosing issues and
opportunities include: governance assessment41 and political economy analysis42. Finally new
approaches to action research include: problem‐driven iterative adaptation43.
 Valuation: understanding and, where this is possible, placing a quantitative monetary value on
resources and the ecosystems services that they generate is one of the key research issues to
emerge in recent years44. This includes major international initiatives such as the Stern Report on
37
FAO. 2012. Coping with water scarcity: An action framework for agriculture and food security. FAO Water
Reports 38. http://www.fao.org/docrep/016/i3015e/i3015e.pdf.
38
Rocha Menocal, A. 2014. Getting real about politics: from thinking politically to working differently.
http://www.odi.org/sites/odi.org.uk/files/odi‐assets/publications‐opinion‐files/8887.pdf
39
Booth,D. and Unsworth,S. 2014. Politically smart, locally‐led development. ODI London.
http://www.odi.org/sites/odi.org.uk/files/odi‐assets/publications‐opinion‐files/9158.pdf
40
Grindle,M. 2007. Good Enough Governance Revisited. Development Policy Review, 2007, 25 (5): 553‐574
http://onlinelibrary.wiley.com/doi/10.1111/j.1467‐7679.2007.00385.x/abstract
41
UNDP. 2013. User’s Guide on Assessing Water Governance.
http://www.watergovernance.org/documents/WGF/Reports/20058‐UNDP‐Assessing‐water_web.pdf
UNESCO (2009a). IWRM Guidelines at River Basin Level, Parts 1: Principles. Paris: UNESCO.
http://www.hydrology.nl/images/docs/ihp/IWRM_Guidelines/IWRM_Part_1_Principles.pdf
42
Fritz, Verena, Brian Levy, and Rachel Ort. 2014. Problem‐ Driven Political Economy Analysis: The World
Bank’s Experience. Directions in Development. Washington, DC: World Bank. doi:10.1596/978‐1‐4648‐0121‐1.
License: Creative Commons Attribution CC BY 3.0
43
Andrews, Matt. 2013. The limits of institution reform in development. Cambridge Press
44
Schomers, S. & Matzdorf, B. (2013). Payments for ecosystem services: a review and comparison of
developing and industrialized countries. Ecosystem Services. Ranasinghe T & Kallesoe, M. (2006) Valuation,
Rehabilitation and Conservation of Mangroves in Tsunami Affected Areas of Hambantota, Sri Lanka: Economic
14
iea.cgiar.org
climate change, the TEEB initiative in relation to the value of biodiversity and ecosystems and,
more recently, the Economics of Land Degradation Initiative45. Valuation methods and models are
a major research field, with environmental economics increasingly important within overall
economics. However, they are often also the source of debate and controversy, in part because
they require numerous assumptions and judgment calls, but also because of the paucity of reliable
data. There are clear links between valuation research and policy, including fields such as Payment
for Ecosystems Services (PES) and carbon offsetting initiatives46. The ability to demonstrate a
monetary value for ecosystems and ecosystem services that are not traditionally valued through
markets is seen as crucial evidence in policy debates.
 The relationship between natural resources and livelihood systems: although it is a well‐
established principle in many parts of the developing world that livelihood systems have an
intimate relationship with the local natural resource base, this still poses many complex research
questions47. In particular the specific form of this relationship in particular ecosystems and/or
livelihood systems (including how these relate to the governance issues raised above) has formed
the focus of extensive local‐level studies48. The dynamics and direction of change within these
relationships is also a key issue, including how they are conditioned by external factors such as
changing markets, climate change and globalization49. Differential gender roles within natural
resources and livelihoods systems are also an important research issue, as is understanding and
valuing the sustainable management of ecosystems within these livelihood systems50.
 Resilience and sustainability: although sustainability is not a new issue, there is an increasing
research concern to understand what it means in terms of the ability of specific ecosystems and
Valuation of Tsunami Affected Mangroves — The World Conservation Union, Ecosystems and Livelihoods
Group Asia. Costanza, R., de Groot, R., Sutton, P., van der Ploeg, S., Anderson, S.J., Kubiszewski, I., Farber & S.,
Turner, R.K. 2014. Changes in the global value of ecosystem services. Global Environmental Change. 26: 152‐
158.
45
TEEB (2009) The Economics of Ecosystems and Biodiversity for National and International Policy Makers –
Summary: Responding to the Value of Nature. ELD (2012) The economics of land degradation, the global
initiative for sustainable land management ELD Secretariat, Bonn.
46
Greiber, T. (2009) Payments for ecosystem services: legal and institutional frameworks. IUCN
Environmental Policy and Law Paper No.78.
47
Haines‐Young, R. and Potschin, M. 2010. The links between biodiversity, ecosystem services and human well‐
being in Ecosystem Ecology: A New Synthesis, eds. David G. Raffaelli and Christopher L. J. Frid. Cambridge
University Press, Cambridge (Ch 6). Daw, T., K. Brown, S. Rosendo, and R. Pomeroy. 2011. Applying the
ecosystem services concept to poverty alleviation: The need to disaggregate human well‐being. Environmental
Conservation. 38:370–379.
48
See, for example, J. Chong, (2005), Valuing the Role of Wetlands in Livelihoods: Constraints and
Opportunities for Community Fisheries and Wetland Management in Stoeng Treng Ramsar Site, Cambodia.
IUCN Water, Nature and Economics Technical Paper No. 3, IUCN — The World Conservation Union, Ecosystems
and Livelihoods Group Asia, Colombo. Willemen, L., Drakou, E.G., Dunbar, M.B., Mayaux, P. & Egoh, B.N.
(2013) Safeguarding ecosystem services and livelihoods: Understanding the impact of conservation strategies
on benefit flows to society. Ecosystem Services. 4: 95‐103.
49
Ruckelshaus, M., E. McKenzie, H. Tallis, A. Guerry, G. Daily, P. Kareiva, S. Polasky, T. Ricketts, N. Bhagabati, S.
a. Wood, and J. Bernhardt ( 2013) Notes from the field: Lessons learned from using ecosystem service
approaches to inform real‐world decisions. Ecological Economics. doi: 10.1016/j.ecolecon.2013.07.009
50
Plummer, M. L. (2009) Assessing benefit transfer for the valuation of ecosystem services. Frontiers in
Ecology and the Environment 7:38–45.
15
iea.cgiar.org
natural resource systems to withstand the impacts of external factors that change and potentially
degrade them51. These factors include both sudden ‘shocks’, rapid change caused by factors such
as natural disasters or major pollution events, and more incremental change from soil degradation,
the over‐exploitation of resources or persistent pollution. The key research issues relate to the
ability of ecosystems (and land resources and hydrological systems) to survive these degradation
forces, their resilience, as well as the levels of change and of resource exploitation that are
sustainable without threatening the integrity of the natural systems in question. However, to date,
there are no commonly agreed approaches to the measurement of Ecosystem Resilience.
 Carbon stocks, flows and sequestration: climate change is an increasingly pervasive issue across
policy and research programmes and is central to concerns over the dynamics of land and water
systems and the nature of ecological change. One important challenge within this broad agenda is
to understand levels of carbon stocks in different ecosystems (including above and below ground
biomass carbon as well as soil carbon), the nature of carbon flows over time (especially where
there is ecological change) and the actual and potential role of different ecosystems in carbon
sequestration. This issue has catalysed the establishment of policy links to REDD initiatives in
many parts of the developing world.
 Climate change adaptation: adaptation is increasingly recognized as both necessary and inevitable
as the effects of climate change are felt by ecosystems and livelihood systems throughout the
developing world. It is of particular significance in areas, such as coasts and drylands, which are
the most vulnerable to the impacts of climate change. The role of land and water management in
adaptation is recognized as central, as is the ability to respond to changes in ecosystem services
availability that occur as a result of climate‐induced ecological change. Research into sustainable
adaptation systems is increasingly recognised as important and essential if effective adaptation
policy systems are to emerge. It is understood that this item and the one above are being primarily
tackled by the CCAFS CRP and they will not, therefore, be investigated in detail by this evaluation.
The evaluation will however look at the linkages that WLE has with CCAFS in this area.
 Local management and knowledge systems: it is now widely understood that local‐level
livelihoods and resource management systems are based on traditional knowledge systems that
differ widely from mainstream science. Their effectiveness in sustainably managing local resources
and their ability (or otherwise) to respond to externally‐induced change is a key research issue, as
is the nature of the relationship between traditional local and mainstream scientific knowledge.
 Land tenure and entitlements: research into these issues has become increasingly important in
relation to customary rights and usufruct entitlements, with the emergence of large‐scale land
allocations to external (including foreign) interests for production of such commercial crops as oil
palms and rubber. The institutional arrangements that underpin the management and allocation
of land resources under traditional, customary tenure systems are a further focus of research into
51
Biggs, R., Schlüter, M., Biggs, D., Bohensky, E.L., BurnSilver, S., Cundill, G., Dakos, V., Daw, T.M., Evans, L.S.,
Kotschy, K., Leitch, A.M., Meek, C., Quinlan, A., Raudsepp‐Hearne, C., Robards, M.D., Schoon, M.L., Schultz, L.
& West, P.C. (2012) Toward principles for enhancing the resilience of ecosystem services. Annual Review of
Environment and Resources. 37:421‐448.
16
iea.cgiar.org
land issues. It is notable also that, at the international level, FAO has developed voluntary
guidelines for land tenure52 and is currently working on similar guidelines for water tenure.
2.2.2. Institutional and Program Structure and Origins
47. The original WLE funding proposal53 states its objective as, “To learn how to intensify
farming activities, expand agricultural areas and restore degraded lands, while using natural
resources wisely and minimizing harmful impacts on supporting ecosystems.”
48. WLE originally consisted of five Strategic Research Portfolios (SRPs). These covered:
 Irrigated systems
 Rainfed systems
 Resource Recovery and Reuse
 River Basins (which incorporated several CPWF projects)
 Information Systems.
49. In 2014, after structural reform to the program for the 2015‐16 Extension Period requested
by the CGIAR Consortium Office, the five original SRPs were replaced by five Flagships. These are as
follows:
 Flagship 1: Integrating Ecosystem Solutions into Policy and Investments (IES)
IES pursues policy and investments in ecosystem services through influence and
decision support—with unique, context‐specific impact pathways in four
international river basin zones called Focal Regions—and an Innovation Fund
 Flagship 2: Sustainably Increasing Land and Water Productivity (LWP)
LWP research aims to impact small farmers and large‐scale irrigation; gender equity
and resource access equity; and the sustainability of ecosystem services.
 Flagship 3: Regenerating Degraded Agricultural Ecosystems (RDE)
RDE hypothesizes change through research on degraded landscapes.
 Flagship 4: Recovering and Reusing Resources in Urbanized Ecosystems (RRR)
RRR seeks to reduce negative footprints of urbanization through private sector
investments, by promoting business models and supporting technological
innovation.
 Flagship 5: Managing Resource Variability and Competing Use (MRV)
MRV research is targeted to reconcile competing claims on variables resources.
52
http://www.fao.org/nr/tenure/voluntary‐guidelines/en/
53
WLE 2011
17
iea.cgiar.org
50. Research on integrating ecosystem solutions into policies and investments (IES, Flagship 1) is
based on the ecosystems approach described in the Ecosystem Services and Resilience framework,
notably around the concepts of resilience and sustainability. Flagship 2 (LWP) addresses sustainable
land and water productivity enhancement. It focuses on sustainable land management and on
adaptive water management, as discussed in Section 2.2.1 and it conducts research on biophysical
and engineering aspects of land and water management and conservation. Flagship 3 on
regenerating degraded agricultural ecosystems (RDE) combines biophysical and socio‐economic
models and other approaches to quantify the levels of ecosystem degradation and to find new ways
of landscape restoration. The recovery and reuse of resources in urbanized ecosystems is the subject
matter of Flagship 4 (RRR). The focus here is on peri‐urban environments where multiple partners
can develop new business models to bring back water, nutrients, and energy into the production
cycle. Finally, Flagship 5 on managing water resource variability and competing use (MRV) aims to
formulate innovative solutions to deal with the impact of spatial and temporal variability on
agricultural ecosystems at different scales.
51. In both SRPs and Flagships, the program has incorporated crosscutting themes of Gender,
Poverty, and Institutions (GPI) and Ecosystem Services; but by the time of the revised proposal,
Resilience had been added to the latter theme to give its current scope of Ecosystem Services and
Resilience (ESR). In early 2014, WLE issued a comprehensive gender strategy, which aimed to help
integrate gender themes into planning and management processes and into the portfolio. This was
followed in late 2014 by an Ecosystem Services and Resilience framework document. Both of these
approaches cut across research areas, to such an extent that most projects from every Flagship
contribute to both, which have been defined as Core Themes. The previous fifth SRP on Decision
Analysis and Information Systems (DAI) became a third Core Theme, intended to strengthen decision
analysis and improve measurement of an intervention’s impact, thereby helping to build capacity
among partners in these areas. As a Core Theme, it is intended to complement all of the Flagship
research areas, although it is budgeted slightly differently from the ESR and GPI Themes.
Furthermore, while most projects will have ESR and GPI aspects, DAI is currently engaged in
relatively few projects across the portfolio.
WLE Intermediate Development Outcomes (IDOs)
52. Under its new Flagship structure, WLE is intended to contribute towards a set of
Intermediate Development Outcomes (IDOs). These are as follows:
 Improved land, water and energy productivity in rainfed and irrigated agro‐ecosystems
 Increased and more equitable income from agricultural and natural resources management
and ecosystem services in rural and peri‐urban areas
 Women and marginalized groups have decision making power over and increased benefits
derived from agriculture and natural resources
 Increased ability of low income communities to adapt to environmental and economic
variability, demographic shifts, shocks and long term changes
 Increased resilience of communities through enhanced ecosystem services in agricultural
landscapes.
18
iea.cgiar.org
53. Whereas the previous IDOs in WLE’s 2012 Strategic Plan were aligned with the old SRP
structure, but were not specifically measurable, the new IDOs are intended to enable such
monitoring. In particular, results should be trackable across four thematic criteria: Environment;
Gender and Equity; Income and Productivity; and Adaptation.
54. Under the Extension Proposal for WLE, significant changes were made to its structure and
approach54:
“The changes to WLE’s structure draw on lessons from the CGIAR Challenge Program on Water
and Food’s (CPWF) external review and two years of WLE implementation. Changes include:
1. The establishment of a significant region‐based flagship, Integrating Ecosystem Solutions into
Policy and Investments (IES), which is defined by open and competitive calls for ecosystem
service and sustainable intensification research within defined geographical areas.
2. WLE has introduced an Innovation Fund to support impact‐driven research that features its
ecosystems‐based approach. The Fund, which was implemented through an open and
competitive call, encourages integrated, curiosity driven research, cross‐regional and global
development and use of tools, methods and analysis that support equitable ecosystems based
development and investments.
3. WLE’s heritage projects have been largely realigned to demonstrate clear contributions to the
outcomes of the flagships they represent. The realignment is an ongoing process that will be
completed by the end of 2014.
4. Decision Analysis and Information (DAI) has been reclassified as a core theme in order to
provide more cross‐cutting support to all WLE’s flagships.
5. The development of a robust monitoring and evaluation process building from the work of
CCAFS that includes hiring an M&E Coordinator and an online reporting tool for the flagships”.
Challenge Program on Water and Food (CPWF)
55. WLE has inherited some of the work commenced under the previous CGIAR Challenge
Program on Water and Food (CPWF)55. Some of the elements carried over include:
 Focal Regions for WLE research
 Emphasis on context‐specific development impact from scientific research (“participatory

action research” and research for development)
54
Extension Proposal for CGIAR Research Program on Water, Land and Ecosystems. Resubmit, September 8,
2014. P3.
55
This initial impression may be modified by later research, since some stakeholders already interviewed
expressed a stronger distinction between the two programs.
19
iea.cgiar.org
 Approach towards partnership, with 26 of 36 WLE projects in the Focal Regions led by non‐
CGIAR institutions
 Members of governance committees, and research managers in the Focal Regions
56. Thirty‐four CPWF projects were mapped to WLE at the outset in 2012, while 10 remained
active in 2013. CPWF projects carried into 2014 were no cost extensions. There appear to be CPWF‐
heritage aspects of many more research grants and activities, often within projects that received a
new name during their transition from CPWF to WLE. This evaluation will therefore trace to what
extent projects, research ideas and relationships formed under CPWF and under other pre‐existing
programs have been continued without change, modified, or discontinued. Changes made to
heritage projects “mapped” to WLE will be examined; particularly in order to assess to what extent
these processes affected their relevance and coherence.
Governance and Management of WLE
57. WLE is headed by a Steering Committee of experts with international standing in WLE
research areas, with the Director General of IWMI and Director of WLE as ex officio members. The
Steering Committee provides recommendations to the Lead Center (IWMI) Board. It is responsible
for scientific direction, quality of science and oversight of the WLE CRP. It includes a representative
from FAO, the Deputy Director of the Land and Water Division.
Figure 1: Basic WLE Governance and Management structure
58. Reporting by way of the Lead Center DG to the WLE Steering Committee and to the IWMI
Board, management of WLE is split into two bodies, both headed by the WLE Program Director. On
one side, the central Operations Team is responsible for day‐to‐day operations, and includes a
20
iea.cgiar.org
Communications and Knowledge Management unit. On the other, the Management Committee
comprises Flagship and Theme leaders and is responsible for research performance of the Flagships.
59. IWMI and the 11 Partner Centers (and FAO) each have a Science Focal Point for that Center’s
participation in WLE. The Partner Centres are shown in Table 2 below.
60. Program Management and Coordination (PMEC) for WLE was originally listed as a $5.3
million line item in the 2014 budget, to consist of inter alia management (Operations Team),
research coordination, knowledge management, communications (including WLE’s blog), monitoring
and evaluation (and learning), and support to WLE’s cross‐cutting themes —gender, resilience, and
decision support tools — each of which is convened as a WLE working group to integrate with
research strategies. Aside from these items of PMEC, program management has also issued plans for
partnership, capacity‐building, marketing, fundraising, and regional implementation.
61. A Review of overall CGIAR governance56 was completed in 2014. This evaluation will
therefore not cover this topic in detail. However, based on discussions held during the Inception
mission to IWMI, there is little doubt that respondents will raise governance issues and their
relationship to the specific efficiency and effectiveness of WLE, so they will need to be incorporated
into the assessment of these evaluation criteria.
2.3 The WLE Portfolio
The most comprehensive set of budgeted data on the portfolio is contained in the 2014 year‐end
budget and activity plans. The WLE portfolio has been re‐conceived several times since 2011, which
makes it difficult to precisely trace individual research activities from one year to the next. The
section below focuses on the current WLE portfolio, which consists of the 149 projects that are
active in 2015. Although the evaluation will assess progress achieved during previous years of the
program, in particular by identifying research activities that have been ongoing (or that were
completed) since the start of WLE, it is expected that the coherence between the program activities
and its overall objectives, both at Flagship and at CRP level, will focus on the 2015 projects.
2.3.1. Focal Regions
62. CPWF featured six main river basins: the Volta, Nile, Ganges, Mekong, Limpopo and Andes.
WLE expanded upon the CPWF basin concept by incorporating relevant areas surrounding basins
(e.g., from the Mekong Basin to the Greater Mekong Region). With effect from June 2014, the first
four (expanded) basins listed above became Focal Regions (the Limpopo and Andes were originally
supposed to become focal regions but had to be removed due to cuts in funding). The basis for
adopting these regions was that they incorporate both rainfed and irrigated agricultural ecosystems
and have good potential for long‐term engagement with local research partners. They are seen as
representing “critical poverty‐environment hotspots” where WLE will place particular emphasis on
research, partnership, outreach and policy activities. The Limpopo and Andes became regions of
56
Review of CGIAR Research Programs Governance and Management. Final Report. Independent Evaluation
Arrangement, Rome. March 2014.
21
iea.cgiar.org
secondary emphasis in terms of funding allocation. The Amu Dharya/Syr Dharya river basin in
Central Asia and the Tigris/Euphrates river basin in the Middle East were added to this second tier.
63. The WLE Focal Region evolved from the CPWF basin concepts in the following aspects:
 Broadening of the range of operational partners from the original basin approach, which
mainly focussed on CGIAR institutions
 Efforts to encourage integration among projects within each region, rather than pursuit of
disconnected individual activities
 Core approach of Ecosystems Services and Gender to underlie all activities
 20% of budget of each project to focus on gender research
 Movement of budget from CGIAR Centers to national and regional partners (40% of budget
spent at this level)
 Stronger emphasis on open competitive processes for proposals within broad areas
determined by WLE.
64. A major vehicle for the new regional approach is Flagship 1, Integrating Ecosystem Solutions
into Policy and Investments (IES). Including its Innovation Fund, this Flagship has a 2015 budget of
approximately $9 million, almost exclusively from W1‐2 with the exception of some Australian DFAT
bilateral co‐financing of IES research partnerships in the Greater Mekong Sub‐Region WLE Focal
Region.
2.3.2. 2015 WLE Portfolio of Activities
65. In 2015, the WLE portfolio is currently composed of 149 projects, funded through a
combination of bilateral and core funding (CGIAR W1/2).
66. Table 1 below presents the 2015 WLE portfolio according to Flagship budget, with notes on
W1/2 proportions. The LWP Flagship (Sustainably Increasing Land and Water Productivity) has the
largest overall budget. However, LWP also receives the lowest proportion of its funds from WLE
W1/2. Therefore, it could be said that this block of research is heavily driven by bilateral funds. The
IES Flagship, on the other hand, is almost entirely funded through core CGIAR funding.
22
iea.cgiar.org
Table 1: 2015 Flagship budgets by funding source and number of activities (the figures in the table
below are likely to change following budget cuts to the program)
Share of N. of
Flagship Cluster W1/W2 Total Funding
W1/2 Projects
1.1 1,520,936 2,217,919 69% 13
1.2 2,448,410 2,448,410 100% 8
1.3 1,848,021 1,848,021 100% 6
1 ‐ IES
1.4 1,495,159 1,495,159 100% 5
1.5 2,461,334 2,461,334 100% 4
Total F1 9,787,060 10,470,843 93% 36
2.1 3,738,865 16,157,537 23% 29
2 ‐ LWP 2.2 1,001,322 1,262,974 79% 7
Total F2 4,740,187 17,420,511 27% 36
3.1 2,986,475 5,448,230 55% 17
3.2 384,692 2,057,096 19% 6
3 ‐ RDE
3.3 150,000 532,382 28% 2
Total F3 3,521,167 8,037,708 44% 25
4.1 1,159,456 2,622,092 44% 5
4.2 299,244 840,044 36% 2
4 ‐ RRR
4.3 103,493 447,047 23% 2
Total F4 1,562,193 3,909,183 40% 9
5.1 1,718,689 3,099,768 55% 15
5.2 1,192,515 4,086,365 29% 14
5 ‐ MRV
5.3 915,001 1,703,707 54% 7
Total F5 3,826,205 8,889,840 43% 36
DAI 1 424,675 1,026,657 41% 1
DAI DAI 2 1,533,305 3,030,478 51% 5
Total DAI 1,957,980 4,057,135 48% 6
ESR ESR 310,797 478,228 65% 1
Total 25,705,589 53,263,448 48% 149
67. Figure 1 below shows the distribution of overall funding across participating CGIAR Centres.
After the Lead Center IWMI that has 35% of the funding, the CGIAR Centers with more funding
mapped to WLE are IITA, Bioversity and CIAT. The partners with the largest proportion of W1/2
funding are IWMI, CIAT, Bioversity, IFPRI and ICRAF.
23
iea.cgiar.org
Figure 2: Distribution of WLE Funding by CGIAR Center in 2015
ILRI CIP IRRI World Fish

ICARDA 2% 1% 1% 1%
5%
IFPRI
5%
ICRAF
IWMI
7%
35%
ICRISAT
9%
Bioversity
11%
IITA
12%
CIAT
11%
68. Table 2 below shows the distribution of research projects by Flagship and CGIAR Center.
Table 2: Distribution of research projects by CGIAR Center.
CGIAR Center Total Funding W1/W2 Share of W1/W2 N. of activities

IWMI 16,241,701 8,172,293 50% 50
IITA 5,666,866 874,344 15% 5
CIAT 5,388,541 2,113,677 39% 12
Bioversity 5,110,885 3,237,742 63% 11
ICRISAT 4,341,000 945,000 22% 10
ICRAF 3,235,366 1,296,000 40% 2
IFPRI 2,412,191 1,043,159 43% 14
ICARDA 2,182,342 882,000 40% 9
ILRI 1,155,539 546,140 47% 5
CIP 460,400 383,400 83% 2
IRRI 350,112 363,312 104% 1
World Fish 326,000 153,000 47% 2
Total 46,870,943 20,010,067 43% 123
24
iea.cgiar.org
69. While the overall proportion of gender funding across all the Flagships stands at
approximately 20%, the percentage of gender funding in the individual flagships is uneven, ranging
from 25% for IES to 5% in DAI.
Figure 3: 1. Figure 2: Gender as a proportion of Flagship Budget in 2015
25,000,000
20,000,000
15,000,000
10,000,000
5,000,000
‐
IES LWP RDE RRR MRV DAI
Large‐scale projects
70. There are 8 projects with a yearly budget (combined W1/2 and W3/bilateral) greater than $1
million in 2015. These larger projects will require detailed examination, although not necessarily
through field missions. Projects funded through the innovation fund are not included in the table
below because they will be the subject of a separate analysis. Larger projects are not the only
projects that the evaluation team intends to review, a sub‐set of projects containing innovative
approaches will also be identified in collaboration with the WLE program.
25
iea.cgiar.org
Table 3: list of projects
Center Flagship Project Title Funding in

2015
ICRISAT 2 1,176,600
Enhancing Water Use Efficiency through various agricultural
water management interventions in rainfed and irrigated areas
IITA 2 Identification and dissemination of Integrated Soil Fertility 1,866,926
Management recommendations for the sustainable
intensification of root crop‐based systems within farming
landscapes
IITA 2 Identification and dissemination of Integrated Soil Fertility 2,298,731
Management recommendations for the sustainable
intensification of root crop‐based systems within farming
landscapes within farming landscapes, with a specific focus on
striga management
IWMI 2 1,058,690
Water Management Solutions for Flood Recession and Dry
Season Farming in Nigeria
Bioversity 3 Providing ecosystem services and resilience through the use of 1,093,216
agricultural biodiveristy within production systems of in South
Asia (Nepal, Sri Lanka)
CIAT 5 Sustainable development options and land‐use based 1,770,888
alternatives to: Enhance climate change mitigation and
adaptation capacities in the Colombian and Peruvian Amazon,
while enhancing ecosystem services and local livelihoods
ICRAF DAI 1,026,657
DAI 1: Decision Analysis and Risk Assessment ‐ Stochastic
Impact Evaluation
ICRAF DAI 2,208,709
DAI 2: Information Systems for Land, Water & Ecosystems ‐ Soil
Information Systems
26
iea.cgiar.org
3. SCOPE OF THE EVALUATION
71. The evaluation is being undertaken at a time when the CRP has completed its first funding
phase and is entering the 2015‐2016 extension phase. The Program went through various
adjustments and restructuring into Flagship Projects, and within them clusters of activities, defining
program theories of change as well as impact pathways for each Flagship Project. Intermediate
Development Outcomes have been defined with target achievement goals for the medium‐term
(about a 10‐year time span), according to specific agroecologies and beneficiary groups for them,
with measurable indicators for progress and results.
72. The scope is structured around two main and inter‐linked aspects of performance:
Programmatic Performance and Organizational performance.
3.1 Research/Programmatic Performance
73. The evaluation scope encompasses all activities covered by the WLE programme. The
concept of WLE was specified in its initial proposal document and formalised through Program
Agreements. As discussed in Section 2.2.2 above, much of the first WLE cohort consists of projects,
which were initially designed and implemented as part of existing programs such as the Challenge
Program on Water and Food (CPWF) and were then incorporated into WLE through a process of
“mapping.” These (“legacy”) projects commenced before WLE start up in 2012 and are at various
stages of progress, possibly including some completed projects.
74. The Extension Proposal for CGIAR Research Program on Water, Land and Ecosystems
(Resubmit, September 8, 2014) has led to further modification in the types of activities undertaken
under WLE, within the guidance offered under the Second Call for Proposals. Although some of
these activities will be only starting up at the time of this evaluation, their approach (particularly
where detailed through the means of a Theory of Change) will be assessed as part of its formative
dimension.
75. The various timescales of projects now included under WLE, together with the restructuring
of the program and the change in conceptual framework to place ESR as the over‐arching concept all
offer potential challenges to evaluation coherence. However, to a large extent measures have
already been taken by WLE management to bring all current activities under a common frame of
reference. This is evident from the WLE 2014 Activity Plans for Flagships, which require a common
set of planning and management information for each project, regardless of origin or start‐date.
76. With regard to timescale, all projects, regardless of their date of origin, now provide
information on their expected contribution to the ESR and gender , as well as the specific
Intermediate Development Outcomes to which they will contribute. Timeframes are provided for
anticipated outcomes and impact pathways are specified. This means that projects with different
start dates can be assessed according to common criteria, taking into account the level of progress
towards the anticipated outcomes of their Impact Pathways. The evaluation will also pay particular
attention to issues of scaling up or the catalytic role of WLE interventions, in addition to
sustainability.
27
iea.cgiar.org
77. Concerning program restructuring, all activities have been mapped from their original SRPs
to Flagships. Flagship Projects incorporate clusters of activities, which should lead towards impacts
according to their defined Theories of Change (ToC). In the 2014 Activity Plans, the originating SRPs
are not listed and it seems that the Flagships are actually broader in concept than the original SRPs,
so that migration from one to the other has been readily accomplished. Further, one SRP (Resource
Recovery and Reuse) moved from one system to the other without change. The major departure
from the SRP structure has therefore been the institution of Flagship 1, notably through its
Innovation Fund and Open Call, which have a very specific focus on the interpretation and
implementation of the ESR concept. This is interpreted by many stakeholders57 as the major
conceptual innovation of the WLE Program. Accordingly, the evaluation will pay particular attention
to the analysis of progress towards results in Flagship 1. Since this Flagship is very much in its start‐
up phase, a major aspect will be the quality of research project design, notably through the use of
innovative procedures such as Write Shops. An assessment will also be made through the assembly
of expert opinion on the ESR Framework itself, based on its 2014 document. This will cover the
coherence and utility of the concept and the extent to which it builds on previous WLE frameworks
and research.
78. In view of the relatively long timescale for the realization of outcomes, much of the focus of
the evaluation, particularly for more recent research activities, will be on assessing to what extent
the design and early implementation of activities are appropriate to effectively contribute towards
the delivery of targeted benefits at intermediate stages of the results chain. In addition, the extent
to which completed research projects have produced results at outcome or impact level will be
assessed as part of the summative dimension of the evaluation, while the progress of continuing
legacy projects will be addressed within the formative dimension.
79. In addition, other performance‐related aspects to be explored include:
 the comparative advantage of the CGIAR and the WLE centers vis‐à‐vis other suppliers
 the relevance of research clusters and Flagship Programmes to the objectives of the CRP
and the CGIAR more broadly, as well as to global, regional and national priorities and the
needs of the targeted intermediate and final beneficiaries
 the quality of science of original research project concepts
 measures, which have been taken to build sustainability of benefits, will be assessed at a
level appropriate to the current stage of implementation.
80. In addition to its focus on WLE‐specific activities, the evaluation will place these within the
broader context of the total set of CRPs. This assessment will be conducted from both a “top‐down”
and a “bottom‐up” perspective. The former will assess the extent to which systematic “high level”
exchanges of information and approaches have taken place among CRPs, to ensure maximum
consistency and synergy. The latter approach will begin with the “linkage to other CRPs” listed for
each project in its annual Activity Plan.
57
Notably by the Steering Group, which presented the ESR Framework as a hypothesis to be tested by WLE, at
a meeting with the valuation team on 3rd March 2015.
28
iea.cgiar.org
3.2 Organizational Performance
81. As specified in the Terms of Reference, this issue will enable a particular focus on efficiency
of governance, management and (where relevant) administration processes, including with regard
to finance58. However, there may also be areas to explore with regard to any perceived or
demonstrated influences of these processes on effectiveness: for example, the implications on
research management and activities of the unexpected reduction and delayed fund flow in 2014 and
more recently in 2015.
82. An important issue on this dimension is likely to be the issue of continuity as opposed to
change in CGIAR overall direction, CRP structures, objectives and regulations and the extent to which
these have provided sufficient consistency to enable substantive research on complex issues to be
designed and implemented. This issue spans the concepts of formative and summative evaluation,
since it will be necessary to examine both the effects to date of the numerous changes and whether
it appears that the systems now in place will provide an adequate enabling environment for effective
research for development.
83. A robust approach to monitoring and evaluation is invaluable in terms of assessing progress,
ensuring accountability and providing a sound evidence base from which lessons can be learned. The
coverage and outputs from the program’s M&E strategy and results framework will be examined,
including the consideration of whether it is possible to aggregate M&E data across the portfolio in
order to assess cumulative results.
84. Other key organizational aspects, which will be examined, include:
 The efficiency of organizational processes
 Governance and management structure
 Partnership management
 Intellectual property rights and management
 Organizational learning and knowledge management.
3.3 Geographical Scope
85. The WLE program now has a focus on four regions and its activities are gradually being
aligned with these. However, there are also many projects outside of these regions, as well as global
projects. This means that there will need to be focussed fieldwork covering all or most of the four
Focal Regions, while other geographical locations will be accessed through secondary resources and
telephone/internet‐based interviews and discussions.
58
This evaluation will not conduct detailed financial analysis, which will be separately addressed under
auditing procedures, but will consider the interaction between finance and implementation
29
iea.cgiar.org
3.4 Evaluability Issues and Limitations of the Evaluation
3.4.1 The WLE Impact Pathways and Theory of Change
86. As described in the WLE Extension Proposal59, the over‐arching question guiding WLE’s
Theory of Change is: “How can we ensure that agricultural intensification and productivity increases
are accomplished in ways that are sustainable and that make use of the services provided by
ecosystems?” This proposal outlines that the overall WLE program approach incorporates four layers
of integration and synergy, as shown in Figure 3 below.
87. First, flagships are integrated into WLE’s program by having impact in its focal regions.
Second, each flagship operates in alignment with WLE’s Theory of Change. This Theory proposes that
WLE activities will promote change through an approach, which emphasises working through
partnerships to generate knowledge, engage in multiple sectors and help to shift mind‐sets. The
third area of synergy happens through integration with WLE’s three core themes of Gender Poverty
and Institutions (GPI), Ecosystem Services and Resilience (ESR) and Decision Analysis and Information
(DAI). These core themes provide strategic, cross‐cutting research and support to implementing
sustainable intensification within the flagships. The core themes also support use of decision‐making
tools to assess the power and benefits that woman and marginalized groups receive from natural
resource management. Finally, the flagships integrate around WLE’s core concepts of sustainable
intensification: livelihoods, productivity, efficiency and sustainability.
Figure 4: Relationship between WLE Program Structure and Theory of Change
59
Extension Proposal for CGIAR Research Program on Water, Land and Ecosystems Resubmit, Page 2, September 8, 2014.
30
iea.cgiar.org
88. The levels at which WLE can contribute towards impacts was a central topic of discussion
during assessment of the Extension Proposal. This Proposal confirms that WLE focuses its work at a
national and regional scale supplemented with important investment pathways to local and global
scales. Through this national and regional focus, WLE is able to leverage the extensive capacity and
research of its partners. WLE recognizes that a transition to integrated and holistic sustainable
intensification systems requires decision makers at all levels—from local to international—to make
complex choices among competing uses of, and management strategies for, water, land,
ecosystems, energy and other resources.
89. Whilst the Theory of Change presented in the Extension Proposal provides an overview of
how WLE intends to contribute towards its goals and objectives, the document does not suggest
how this theory might be operationalised for monitoring and/or evaluation purposes. This has been
partially addressed in the WLE Annual Report for 2014. This WLE Theory of Change is presented and
discussed in Section 5.1 of this report.
90. The potential articulation of the WLE objectives with those of the broader CGIAR, notably
through its System Level Outcomes is another area, which will be carefully explored during the
evaluation. It is likely that the linkages from WLE‐specific results to those of CGIAR Intermediate
Development Outcomes and still further up the Impact Pathway to System Level Outcomes will be
increasingly difficult to specify on the basis of evidence; since the SLOs are identified at a very high
level and the number and scope of potential contributions towards their realization will be too
complex to enable WLE’s role to be specified.
3.4.2 Main Limitations of the Evaluation
91. The evaluation faces a number of challenges, which have the potential to limit the extent to
which it can draw definitive findings. The first of these concerns the evolving identity of WLE and of
the activities within it. However, as noted in 3.1 above, the 2014 Activity Plans required all
continuing activities, regardless of origin, to be specifically mapped to current WLE objectives, so
that this challenge is not a major one.
92. Secondly, the scale of the WLE CRP is substantial, with a projected 2015 budget of $60.5
allocated to 149 projects. The geographical distribution of project activities is also widespread, with
four focal basins and a substantial volume of activities outside of these. There is also a substantial
range of types of activity, distributed among five Flagships. This overall diversity presents some
limitations in the extent to which evaluation methods and results can be proven to generate
evidence and findings, which are “representative” of WLE. Related issues arise concerning the
appropriate characteristics for aggregation. On the one hand, WLE has adopted an approach
emphasising Focal Regions, which might present a suitable analytical category. On the other, a
substantial body of work is active outside of these regions or on a global basis. Cross‐cutting these
geographical criteria are five “Flagships”. Conceptually, the five Flagships are not mutually exclusive.
For example, an activity categorised under “Sustainably Increasing Land and Water Productivity”
could also contribute towards “Managing Resource Variability and Competing Use;” while activities
funded through the Flagship “Integrating Ecosystem Solutions into Policy and Investments” could cut
across all other Flagships. The complexity outlined above will pose some challenges for the extent
to which the evaluation can derive WLE‐wide findings, conclusions and recommendations, since it
31
iea.cgiar.org
will not be simple to “add‐up” the results from component parts (whether flagships or regions) to
give an aggregate overview of WLE performance.
93. The evaluation will address the challenges posed by the complexity of the programme in a
variety of ways. Firstly, it will undertake detailed assessment of each Flagship separately. Secondly, it
will give particular attention to Flagship 1, since this has the most specific focus on the new
Ecosystem Services and Resilience Framework, which is widely regarded as the substantive new
direction of the WLE program. Thirdly, it will assess the relationship of Flagships 2 to 5 as currently
delivered to the ESR Framework. Fourthly, it will assess to what extent and how the results achieved
or in process relate to the new Framework and what perspective this gives on the degree of overall
coherence of the Programme.
3.4.3 Evaluability
94. According to the OECD DAC Glossary of Key Terms in Evaluation and Results Based
Management, evaluability can be defined as: “the extent to which an activity or a program can be
evaluated in a reliable and credible fashion.” Within this rather broad approach, the Team confirms
that WLE is definitely evaluable.
95. Challenges could occur if, under a stricter definition of evaluability, the evaluation were
required to provide measurable conclusions on progress towards “impact” as specified by the
Intermediate Development Objectives. This is because these objectives are expressed at such a high
level (e.g., “irrigated area in Africa expands”) that it will be difficult to present a convincing
contribution analysis, still less any consideration of attribution. Further, although the overall budget
of WLE is substantial ($50m+ per annum), once it is divided among five Flagships (and other
elements) it is small in comparison with the overall scale of the issues to be addressed. With regard
to results statements (effectiveness), the evaluation will therefore pay particular attention to the
need to make only statements, which can be substantiated by evidence and detailed analysis of the
Theory of Change of specific interventions, of Flagships and of the CRP as a whole. In view of the
relatively recent start‐up of many WLE activities, the evaluation will not attempt to report
measurable progress towards impacts, so that this potential challenge is not considered a major
factor at this time.
32
iea.cgiar.org
4. EVALUATION CRITERIA AND QUESTIONS
4.1 Introduction
96. The evaluation criteria used in CGIAR evaluations include the agreed international (“OECD‐
DAC”) evaluation criteria, plus one additional criterion to reflect the research mandate of the CGIAR.
As a result, there are six key criteria to be used in independent CRP evaluations: relevance, science
quality, efficiency, effectiveness, impact and sustainability.
97. In line with this requirement, the TOR specify the key evaluation criteria60 and related core
evaluation questions, which should be used to explore performance against these criteria. The Team
Leader and the IEA conducted an inception mission to IWMI in December 2014, which led to
refinement of these questions and the specification of some specific areas to be explored as a
platform to answer them. These were further developed during the full team discussion with
different WLE stakeholders in Rome in March 2015.
4.2 Strategic Questions
98. The Inception Phase involved extensive discussions between evaluation team members and
a broad range of WLE stakeholders. During these discussions, various perspectives on the
development and implementation of the WLE programme emerged. These have been assessed by
the team and structured to form the following set of key strategic questions that the evaluation
needs to address:
 Are the conceptual framework and key hypotheses of WLE coherent, effectively
communicated and appropriate for the wide range of issues and diversity of locations
 Are the impact pathways identified by the program ones that are likely to achieve the
desired development outcomes and are they consistently developed across the different
 Is the quality of science in the program of an appropriate standard, does it reflect the
CGIAR’s comparative advantage in science and is it engaged with and reflective of
international developments in the different fields included in the program?
 Are the partnership strategy and range of partnerships being developed in the program
consistent with the program’s overall goals and the development of impact pathways within
the program?
 Are the management procedures and governance structure of the program appropriate,
efficient and consistently applied and is there clarity on the roles and operational
procedures of different components of the management structure?
 Is the development of the program finding an effective balance between ensuring the
delivery of concrete planned outcomes within the program life whilst at the same time
60
The criteria are outlined in the document CGIAR Standards for Independent External Evaluation, Annex 2.
33
iea.cgiar.org
creating a long‐term development trajectory to ensure the program’s approach and
benefits will continue beyond the presently defined program period? This aspect
encompasses such key areas as communications and uptake strategies and approaches.
99. These strategic questions encompass many more specific issues, which are discussed below.
The relationship between the strategic questions and the CGIAR evaluation criteria are outlined in
Table 5 below.
4.3 Criteria for Research/Programmatic Performance
4.3.1 Relevance
100. As explained in the Terms of Reference for this evaluation, relevance will be considered from
two perspectives: demand‐side and supply‐side. These have specific definitions under the CGIAR
Standards, including:
101. “Supply side relevance relates to the alignment of the objectives of research and program
components with the overall objectives of the CRP and its Intermediate Development Outcomes and
the objectives and System‐level Outcomes of the CGIAR.
102. Demand side relevance relates to how well the program research and other activities
correspond with the global, regional and national priorities and the needs of the targeted
intermediate and final beneficiaries, differentiated by social groups”.
103. The Terms of Reference indicate three key areas of relevance. The first of these is
coherence:
 “Coherence relates to the internal alignment and linkages between program components,
and whether activities funded from different sources strategically complement each other”.
104. The evaluation of the coherence of the programme will need to consider whether WLE CRP
is strategically consistent with the main goals and System Level Outcomes presented in the CGIAR’s
Strategy and Results Framework. Since these have been set at a high level, it is considered likely that
a very broad range of activities will be consistent with them. A second issue in this area concerns the
extent to which Flagship priorities have been integrated across focal regions, so that research
lessons learned can be tested in terms of their relative dependence on or independence from
contextual factors. The evaluation will also need to assess whether all research activities are
coherent within the WLE strategy and examine the extent to which some may be isolated
thematically and/or geographically. In this respect the possibility of loss of coherence because of the
need to conform to the priorities of bilateral funders will be explored.
105. The second important aspect of relevance is comparative advantage, focusing on whether
the CGIAR’s mandate of delivering international public goods gives WLE a comparative advantage
relative to other international initiatives/research efforts, including the private sector; to partner
country research institutions and to development agencies. The engagement of WLE with research
partners will be an important aspect of this issue, which will be explored through contacts with both
existing partners and with potential partners, particularly at regional and national level, which have
34
iea.cgiar.org
not participated in WLE activities. Important aspects of comparative advantage to be explored
include: external perceptions of WLE quality of science, of the accessibility of its partnership
processes and of the relevance of its activities both in academic and developmental terms..
106. The third aspect of relevance is that of Program Design. This will consider whether the
program targets an appropriate set of Intermediate Development Outcomes (IDOs), the extent to
which the CRP activities are adequately prioritised in relation to the resources available to contribute
towards the outcomes and whether gender issues and capacity building are sufficiently incorporated
into program design and targeting to ensure that these aspects are well‐addressed.
4.3.2 Quality of Science
107. In evaluation terms, Quality of Science is a criterion specific to CGIAR which reflects its
research mandate. The IEA has developed a comprehensive approach towards it which will form the
basis of the approach taken in this evaluation. As discussed in Section 5.3(e), this CGIAR‐wide
approach will be tailored to WLE, to consider whether the research process reflects state of the art
knowledge and approaches and will ensure novelty in all areas of research. The evaluation will also
assess the quality of research outputs and whether WLE research, which is mostly context‐specific,
will deliver international public goods.
4.3.3 Effectiveness
108. A major focus of this evaluation will be on the effectiveness of WLE. This will be examined
through an assessment of progress along the impact pathways specified for WLE, from outputs
towards Intermediate Development Outcomes (IDOs). The likelihood of achievement of results
relates to a number of factors including: duration of implementation to date and complexity of the
results chain and Theory of Change. A further dimension concerns the extent to which design
assumptions have held true during implementation. During this evaluative process, information on
other (non‐WLE) relevant interventions and processes will also be gathered and assessed to address
issues of “contribution versus attribution” of results, which appear to be related to WLE activities.
109. Findings with regard to effectiveness will also address the issue of the extent to which
supported research has been able to link outputs and outcomes to broader development processes,
in order to generate effects, which are both “scaled up” and sustainable. This will include attention
to the potential catalytic role of research findings and conclusions.
4.3.4 Impact
110. WLE must enable its research outputs to contribute towards development outcomes, which
require both scaling up and continuity through time. The evaluation will assess WLE achievements
on these dimensions, in order to focus on the impact and sustainability of its activities. This will
require attention to the full range of research projects that have contributed, including those, which
began under the CPWF and have been incorporated directly, or in modified form to the reformed
WLE programme. In particular, the evaluation will consider whether the Theory of Change and
defined impact pathways address the challenges associated with scaling up the research outputs
(and especially innovative changes to production or resource management systems) generated by
the program, both within and beyond the program’s designated focal regions.
35
iea.cgiar.org
111. Another important aspect of impact concerns the possible extent of any unanticipated
consequences of WLE activities, whether positive or negative. For example, by focussing on certain
aspects of its research agenda, including its Theory of Change, the programme could be perceived to
reduce attention to other potential approaches that could generate positive results.
4.3.5 Sustainability
112. As suggested in the Terms of Reference, WLE began in 2012 so that sustainability will
primarily be assessed in terms of “the likelihood that actual and anticipated results will be sustained
beyond the lifetime and presence of program intervention”. However, Inception Phase work
suggests that numerous WLE portfolio activities actually began life under the CPWF and may already
have been running for some years, with varying amounts of reorientation to meet the requirements
of WLE. This raises the possibility that some activities may already be assessable in terms of the
actual rather than the likely sustainability of their outcomes.
113. The evaluation of sustainability will be based upon the Theories of Change at different
levels; including individual research projects, programs, basin level and above, up to CGIAR‐level.
This will include the consideration of the extent to which benefits from past research have been
sustained and whether WLE anticipated and was able to mitigate challenges of sustainability by
clearly introducing the concept of continuity in program design and implementation.
4.4 Criterion for the Evaluation of Organizational Performance
4.4.1 Efficiency
114. Only one of the six core CGIAR Evaluation Criteria pertains to Organizational, rather than
Programmatic Performance. This is efficiency, which includes the consideration of whether the
governance and management arrangements and functions conform to the program partnership
requirements of independence, accountability, transparency, legitimacy and fairness. The evaluation
will examine whether WLE’s institutional arrangements, management and governance mechanisms
are efficient and cost‐effective and whether WLE research management provide effective
leadership, culture and ethos for advancing the program’s objectives.
115. The concept of cost‐effectiveness is likely to be difficult to specify for WLE. No concept of
expected cost‐effectiveness is presented in the Extension Proposal and, as noted in the Review of
CRP Governance and Management61 CRPs have different degrees and patterns of partnership,
which can be expected to have varying transaction costs. This being so, even internal
“benchmarking” across the CRPs is not expected to generate definitive results. One area in which it
may be possible to explore to draw some conclusions concerns the Open Call for Proposals under
Flagship 1, where issues of the relative cost‐effectiveness of different research
collaborations may be susceptible to analysis.
61
See RQ 7.7 Transaction Costs, P66.
36
iea.cgiar.org
4.5 Performance on cross-cutting issues
116. The Inception Missions and preliminary documentary analysis raised a number of questions
that cut across the six evaluation criteria in various ways. These are initially listed here to ensure that
they receive attention during the course of the evaluation, so that they can appropriately inform the
final analyses within and possibly in addition to the six evaluation criteria.
4.5.1 Partnerships
117. WLE is a large‐scale CRP, which means that the concept and practice of partnership need
detailed attention. The evaluation will critically examine whether the structure of management of
partnerships is efficient and effective and whether there are adequate systems for communicating
with partners and other stakeholders. The nature and level of participation of different partners will
also be examined.
4.5.2 Gender, Poverty and Institutions
118. Gender, associated with poverty and institutions, is a major emphasis of all of the current
CGIAR CRPs and is a stated crosscutting theme in WLE. Under the 2014 Gender Strategy, projects
should assess the potential gender dimensions of their activities and, as necessary, incorporate
these into their design and Plan of Action. The evaluation will examine the specific ways that gender
has been operationalised within the WLE portfolio and consider whether the approach should be
applied uniformly across all types of project.
4.5.3 Capacity Strengthening
119. As identified in the WLE Extension Proposal62, capacity strengthening is an important aspect
of the potential for scaling up and ensuring sustainability of the results from WLE. The evaluation will
consider the extent to which the design of Flagships, cross‐cutting themes and projects have
incorporated capacity strengthening, as well as the adequacy of resources allocated, results and
potential sustainability.
4.6 Inter-relationships between Evaluation Criteria, Strategic Questions and

Cross-Cutting Issues
120. The inter‐relationships between different issues to be addressed are shown in Table 5
below. It can be seen that the issues, which emerged from inception discussions as important topics
for the evaluation to assess overlap with or complement those already identified through the
standard CGIAR research criteria. It should be noted that Strategic Questions in the area of
governance and management are important for the assessment not only of efficiency, but also
because they contribute towards effectiveness.
62
Page 15, under section WLE Partnerships
37
iea.cgiar.org
Table 4: Inter‐relationships between Evaluation Criteria, Strategic Questions and Cross‐Cutting Issues
Relevance Quality of Effectiveness Impact Sustaina‐ Efficiency63

Science bility
Strategic Questions
Conceptual Framework * *
Impact Pathways * * *
Quality of Science * * *
Partnership Strategy * * *
Management and Governance * *
Balance Outcomes/Impacts * * *
Cross Cutting Issues
Partnerships * * *
Gender, Poverty, Institutions * * *
Capacity Development * * *
63
Headings in italics relate to organizational performance issues, while all others primarily refer to programme
performance.
38
iea.cgiar.org
5. EVALUATION APPROACH AND METHODS
5.1 Approach
121. The evaluation will have both summative and formative approaches, assessing achievements
to date as well as design and processes established for future delivery of results under the reform
agenda. It will therefore have both accountability and lesson learning elements, which will be inter‐
related. In order to ensure a consistent approach to both the summative and formative intentions,
the evaluation will adopt a Theory of Change (ToC) approach. Achievements of completed (“legacy”)
projects will be assessed in terms of their original ToC, whether within CPWF or another programme
and then updated by a rapid comparison with the current WLE Theory.
122. This approach will encompass all levels, from the contribution of WLE to CGIAR System Level
Outcomes to the intentions and achievements of selected individual research activities. Historically,
various versions of a Theory of Change for WLE as a whole have been presented as the conceptual
underpinning of the program has developed. For this Inception Report, the evaluation team has
utilised the summary Theory of Change included as Figure 1 in the draft WLE Annual Report for
2014. The underlying Theory of Change according to this document is presented in Figure 5 below64.
64
Source: CGIAR Research Programme on Water, Land and Ecosystems (WLE) 2014. Ecosystem Services and
Resilience Framework. Figure 6.
39
iea.cgiar.org
Figure 5: Overview Theory of Change for the WLE Research Program
40
iea.cgiar.org
123. It is understood that most recent research projects have been designed to deliver according
to this Theory of Change approach. Earlier activities, which have since been “mapped” to WLE and
are therefore included in the overall portfolio, were not. However, as part of the Annual Work Plan
process, most (if not all) have been “retro‐fitted” to meet the current approach.
124. As part of the process of using the WLE Theory of Change, the evaluation team will assess
the extent to which the projected outcomes will be susceptible to evidence‐based analysis. This
assessment will relate the potential contribution of WLE to those of other stakeholders attempting
to address the same (or overlapping) issues it covers, in order to ensure that the ToC is realistic and
can be verified over time. Other important factors with the potential to affect results, such as risks
identified and key assumptions will also be determined and explored. The evaluation will also
explore the extent to which the current Theory of Change is known, understood and used by a broad
range of stakeholders within the WLE system and whether it has been embraced by external
partners, including in Government.
125. The evidence necessary to assess progress along the intended WLE results chains will be
gathered through a mixed method approach, using quantitative and qualitative evaluation methods
and overall triangulation of findings, as discussed below. This approach will also be used to assess
other criteria and key questions concerning WLE, such as relevance and research efficiency.
5.2 Methodology
126. The overall methodology will draw on a variety of primary and secondary data collection
methods. Some of these, such as portfolio analysis and social surveys, look across the portfolio to
identify overall characteristics and trends. Others, such as case studies and field missions take
specific issues or sets of activities and explore them in depth. The two approaches (across portfolio
and in‐depth) are inter‐related. Portfolio wide trends will signal issues that could be explored
through in depth approaches, while the latter will suggest processes and possible cause and effect
chains for verification across parts of the portfolio (e.g., flagships) or WLE as a whole. Such chains
may provide specificity to the over‐arching ToC contained in the ESR Framework document or may
add new linkages to it.
5.2.1 Collection and Analysis of Existing (Secondary) Sources of Information
127. The main existing data sets for review and analysis are presented below:
Portfolio data
128. There is a broad range of material available within the WLE system, which can be used to
assess the distribution of activities on a variety of dimensions, including:
 WLE phase (e.g. “legacy” from CPWF or other program, WLE , WLE extension proposal)
 Geographic distribution (including regions and river basins)
 Scientific discipline (e.g. soils, irrigated and rainfed agriculture, land and water management)
 Activity Scale (e.g. budget, geographical area covered (where applicable), duration)
 Funding sources (Windows 1, 2 and 3/bilateral)
41
iea.cgiar.org
 Implementing partners.
129. Analysis of this portfolio data will play two roles in the evaluation. On the one hand, it will
supply information necessary to ensure appropriate selection or sampling of activities for more
detailed examination through in depth methods. On the other, it will provide a substantial body of
independent evidence on several dimensions of the distribution of the portfolio.
Document Review
A broad variety of documents will be reviewed, using analysis templates to ensure
consistency of evidence extracted. These will include documents covering:
 CGIAR strategy and other policy documents
 WLE programming
 WLE financial reporting
 Selected project proposals
 Selected project reporting
 Thematic, program and project reviews and evaluations.
130. Documentary review will be utilised both for portfolio‐wide assessment (e.g., through
aggregation of reviews of individual project documents) and as part of in‐depth analysis (e.g.,
documents informing case studies of regional or thematic programs). An assessment will also be
made through the assembly of expert opinion on the ESR Framework, based on its 2014 document.
In order to assess linkages and collaboration with other CRPs, frequency and minuted content of
meetings and extent to which actions have ensued will be considered. The latter approach will begin
with the “linkage to other CRPs” listed for each project in its annual Activity Plan and will explore the
extent to which these indicate actual linking activities or simply potential areas of common interest.
Synthesis of Completed Evaluations
131. In this evaluation, evaluation synthesis refers to an approach, which integrates and
summarizes the findings from the available body of evaluative evidence relevant to WLE65. There
are several completed assessments, reviews and evaluations of aspects of the work of CGIAR, WLE
and IWMI, which may provide valuable inputs and evidence to this evaluation. Those already
identified include:
 2013 IWMI Center Commissioned External Review (CCER) of Science Quality and Relevance
 2013 Independent Evaluation of the IFAD‐funded network for Improved Management of
Agricultural Water in Eastern and Southern Africa (IMAWESA)
 2013 Reviews of EC‐Funded CGIAR Projects, IFAD/EU support to the Challenge Program on
Water and Food, Volta Basin Development Challenge Project (VBDC)
65
The available evidence is not appropriate for the use of statistical meta‐analysis.
42
iea.cgiar.org
 2013‐2014 Forward‐Looking Review of the Challenge Program for Water and Food (CPWF)
 2014 Review of CGIAR Research Programs Governance and Management. Final Report.
Independent Evaluation Arrangement.
 2014 IWMI‐led impact studies on land reform, political drivers, and agricultural and
environmental effects of water pumpset electrification in West Bengal
 2014 CGIAR Independent Science and Partnership Council (ISPC) Standing Panel on Impact
Assessment (SPIA) “Evaluation of CGIAR Centers’ Impact Assessment Work on Irrigation and
Water Management Research.”
132. These documents will be carefully reviewed to draw out key evidence, which may be useful
for the current evaluation. Findings and recommendations may also be used, where detailed
assessment by the evaluation team indicates that they are adequately supported by the evidence
provided.
WLE‐managed Databases
133. The evaluation will draw on information contained in project, research and financial
management databases of WLE and its participating centres. Wherever possible, the team will
obtain direct connection to these databases, in order to allow for timely access and to avoid
burdening the centres’ information system managers with repeated requests.
Coherence analysis of the WLE programme
134. The analysis will consist of a desk review of WLE projects contained in 2015 portfolio. It will
draw on project documentation and project level Theory of Change/impact pathways/log frames (if
available). A template will be developed which will include rating scales as well as short narrative
assessments by team members.
135. It will assess the relevance and coherence of activities with regard to WLE objectives;
identify gaps; examine to what extent bilateral and W3 funded projects’ objectives match Flagship‐
level objectives and WLE overall program objectives; and verify whether cross‐cutting issues have
been appropriately considered in the projects.
5.2.2 Methods for Collection and Analysis of Original (Primary) Data

Semi‐structured interviews66
136. These are conducted on the basis of a set of common issues that must be addressed in each
interview, whilst also allowing respondents to introduce their own perspectives and additional
themes. Analysis will be conducted on the basis of templates to ensure consistency, whilst allowing
opportunity to raise additional issues to those specified in advance.
66
All research methods involving respondents will meet confidentiality requirements established under UNEG
Norms and Standards, Guidelines and Ethical principles.
43
iea.cgiar.org
137. Semi‐structured interviews will be a core method of the evaluation, used with all categories
of stakeholder, either in person or by telephone.
138. The evaluation team will draw on the experience and understandings of WLE researchers
and of key partner researchers to discuss and understand the backgrounds to, and progress in, the
various research activities of the WLE. The team will conduct interviews to obtain views on various
aspects, including the following: the relevance and quality of research, comparative advantage of
WLE in the research area, products of the research, and the likely impacts of research, as well as the
quality of and management of research partnerships. The evaluation team will conduct visits to
selected countries as well as to the participating centre’s offices, with two team members visiting
each site wherever possible. Detailed interviews will be conducted as part of the Centre visits and
field visits. Interviews will also be conducted with other stakeholders, both directly and through
virtual means where appropriate. Interviews will cover representatives from a wide range of
different stakeholder groups. Detailed notes of each interview will be taken.
Group Discussions and Workshops
139. These will be used to explore issues and opinions on aspects of WLE context and
performance, which can best be explored through the interplay of varying perspectives and where
confidentiality is not regarded as essential. Group discussions are likely to be used in such
circumstances as assessment by implementing partner institutions, or project implementing offices
of their experience and achievements under WLE. A specific type of group discussion, which could
make a valuable contribution is the use of mini‐workshops to present emerging issues and findings
to stakeholders and obtain their feedback, including additional data sources, which may provide
verification or alternative perspectives.
Social Surveys
140. Social surveys will be used to explore issues where reaching the maximum range of
respondents is more important than the depth and flexibility of responses. They present the
opportunity for quantification of defined elements of the program. Surveys will be conducted
electronically, to ensure that the broadest range of potential respondents is reached. Two surveys
are envisaged:
 Researcher survey: To get the views of a broad range of internal stakeholders in this
evaluation on issues around management effectiveness and efficiency, quality of science,
gender, capacity development, etc.
 Partner survey: To get Partner institutions’ perspectives on participation in planning
processes and their feedback on the collaboration with the program.
Expert knowledge of team members
141. The areas of expertise of the team members are described section 6.1 below. Drawing on its
own expert knowledge, the evaluation team will map and interview recognized peer researchers and
institutions in all research domains of WLE to tap into expert knowledge outside of the evaluation
team where required. It will be ensured that external expertise covers a broad range of perspectives
(from different types of institutions and different geographic zones).
44
iea.cgiar.org
Evaluation of Quality of Science of Research Activities
Framework
142. The framework for evaluating Quality of Science has four dimensions: (a) processes for
assuring quality; (b) input quality; (c) output quality; and (d) perceptions of quality.
a. Quality Assurance Processes
143. The evaluation will determine whether WLE management has developed science quality
assurance processes, which are used consistently across the different components of the
programme and will suggest where these could be improved.
144. It will look at all internal processes that are explicitly aimed at assuring quality. These
include:
 Internal peer processes (e.g. internal reviews of publications prior to journal
submissions or the internal review of research proposals) and how they function
 Use of external evaluations/reviews as a management tool
 Role of science quality in staff performance assessment (IWMI and to the extent
possible for other participating Centers) and to what extent it is used for enhancing
quality and as a talent management process
 Incentives for assuring and stimulating high quality.
b. Inputs to Science Quality
145. This assessment will be done at Flagship project level. The assessment aims at identifying
variability within the CRP and noting areas of excellence and identifying areas where improvements
could be made. The evaluation will look at quality of CRP team leaders and scientists, research and
resources.
 Team leaders include all Principal Investigators, Flagship and Cluster Leaders, and
Focal Points. For these lead scientists, information about their scientific track record
will be assessed
 The adequacy of research support and resources
 Quality of data management at project, flagship and full programme levels
 Research conducted and specific methodologies used for sampled projects
 ISPC comments on science quality will be taken into account (original proposal and
extension proposal).
c. Output quality
146. The evaluation will look at both the quantity and the quality of outputs from the main
participating Centers. This analysis will include:
45
iea.cgiar.org
 Qualitative assessment of a sample of key publications offered by WLE scientists
 Quantitative assessment (bibliometric analysis) of publications.
147. The nature of WLE research is largely multidisciplinary and its outputs are generally more
diverse than those in strict disciplinary sciences. Thus, the evaluation of research outputs should
consider not only the scientific publications in JCR journals, but publications in other type of journals
as well as books and book chapters and other media such as policy papers and media targeted at
disseminating results to wider stakeholder audiences. Expert opinions are considered of substantial
importance here relative to routine bibliometric analysis.
d. Perceptions of quality
148. The evaluation will draw on perceptions of quality, particularly by thematic areas and by
participating Center, on issues such as overall science quality performance, excellence and ambition,
critical mass and comparison with other organizations.
Scope of Quality of Science analysis
149. The evaluation will assess the quality of science conducted in the WLE CRP at two levels: the
program as a whole and Flagship level. The science and evidence generated by the research,
together with new tools, technologies, and practices developed will be assessed for each of the five
Flagships.
150. The assessment of science quality in the five flagships will be complemented with similar
efforts to evaluate the outputs and outcomes from the three core themes, namely, Ecosystem
Services and Resilience (ESR), Gender, Poverty and Institutions (GPI) and Decision Analysis and
Information Systems (DAI). The WLE CRP has been positioned to test the overarching hypothesis that
sustainable intensification (SI) is better addressed through an Ecosystem Services optimization
approach (which may incorporate technological innovation), than through a technology‐led
intensification approach supported by environmental mitigation (or enhancement). Thus, the
evaluation team will assess to what extent the programme has been appropriately designed and
implemented to allow comparison between these alternatives. Indicators for this assessment are
expected to include novel experimental methods, the development and testing of appropriate
(effective and useful) metrics for early, medium and long‐term indicators of system performance,
and explicit strategies for integrating results from disparate projects within and across flagships.
151. The evaluation of these issues will look at the quality of science in relation to
appropriateness of the methods used for data collection and analysis, including the effectiveness of
participatory approaches where relevant. The methodological frameworks used should show an
understanding of the character of WLE, which is to produce research that generates results that are
not just valid in scientific terms but that are also able to promote concrete changes to policies and
practices to enhance sustainable productivity improvements in the agricultural systems researched.
This aspect will focus on the appropriateness of the research design to feed into policies, rather than
on implementation aspects, which relate to development effectiveness.
46
iea.cgiar.org
Case Studies
152. Case studies combine a range of methods, including desk and field‐based to gather different
types of evidence in order to develop an in‐depth understanding of specific issues. In research terms,
case studies have high internal validity (reliability concerning the specific case studied), but need to
be supported by other methods, such as surveys and broader documentary research, to attain
external validity (applicability beyond the specific case). The WLE evaluation will therefore conduct a
range of case studies at different levels and relate their findings to those from its other, more
aggregate and cross‐cutting methods. These were selected on the basis of documentary analysis and
through discussions with WLE management and staff during inception missions to IWMI and Rome,
during which a broad range of stakeholders gave specific suggestions for the most appropriate
countries to host field missions, which will play an important role in data gathering for the cases.
Country/Regional Case Studies
153. The revised Extension Proposal for WLE has emphasised that the programme is primarily
seeking to contribute towards impacts at two levels; firstly national and secondly, regional. In order
to assess results already achieved and progress towards outcomes and, eventually, impacts the
evaluation will therefore focus on the four WLE regions. Within these, it will select a number of
countries for missions, which will form the basis of case studies at country or regional level. These
studies will assess activities, processes and achievements at country and regional level and will be
internal documents of the evaluation, contributing to its overall body of evidence.
154. Geographically, the case studies will be spread across the four WLE Regions and will cover
the following countries:
 Nile Region (Ethiopia and Kenya)
 Ganges Region (India and Bangladesh)
 Greater Mekong Region (Thailand, Cambodia and Laos)
 Volta Region (Ghana).
155. The Greater Mekong region has received slightly great coverage in terms of number of
countries for two reasons. Firstly, it has the greatest concentration of activities generated by the
Flagship 1 Open Call, which the evaluation team regards, on the basis of its discussions with WLE
stakeholders during the Inception Phase, as one of the major innovations of WLE, therefore
warranting particular attention. Secondly, a specific thematic Case Study will be conducted in this
region on a major research area with important policy implications, which commenced under CPWF
and is continuing strongly under WLE. This concerns the inter‐relationship between water
governance, hydropower and ecosystem services management in the region (see below). In order to
develop a sufficiently detailed understanding of this complex programme of activities, the evaluation
will visit three participating countries, namely Thailand, Cambodia and Laos. It will assess activities in
these countries from two perspectives: firstly, that of national level results (where these are
targeted) and progress towards impacts and secondly, the contribution towards regional progress
concerning water governance, hydropower and ecosystem services management. Key dimensions of
the country case study approaches to Ghana, Ethiopia and Kenya will be specified on the basis of a
review of the first generation of country missions; which will be conducted by the evaluation team in
Sri Lanka in mid‐May 2015.
47
iea.cgiar.org
Use of Country/Regional Level Case Studies
156. The Country/Regional Case Studies will produce concise data sets, which will be used by the
evaluation team as a key source of detailed information on the following areas:
 Progress of WLE activities in country (including regional and global projects with national
level activities)
 Range and effectiveness of partnerships
 Key research results from legacy projects and their follow‐up through WLE
 Strategies, activities and progress to influence national and/or regional policy makers
 Application of the ESR framework and results achieved or anticipated
 Application of the Gender Strategy and results achieved or anticipated
 Awareness and perceptions concerning WLE among other stakeholders and relevant experts
in the country.
157. The selection of respondents will be based on an iterative process, drawing on the following
sources:
 Analysis of key programme and research documents
 Identification of key stakeholders from distribution of WLE staff responsibilities
 Discussions with WLE managers covering various activities in the country concerned
 Follow up of additional contacts emerging from initial discussions and interviews
 Call backs (telephone/Skype) on respondents for clarification or additional detail.
Thematic Case Study: Water Governance, Hydropower and Ecosystem Services Management in
the Mekong Region
158. The inter‐relationship between water governance, hydropower development and ecosystem
services management has been a major area of research and debate within the Mekong Region.
Issues surrounding this relationship were a key component of the CPWF programme in the Mekong
(addressed by 14 of its 19 projects) and continue to be a major focus in many of its current WLE
projects. Currently, 10 out of the 14 new WLE projects in the Mekong region consider this inter‐
relationship in one way or another. The issues raised are of great concern to governments in the
region, with each of the five lower Mekong countries having hydropower as a key focus of their
power development plans and a number of specific policy initiatives in the area. The inter‐
governmental organization, the Mekong River Commission, has sustainable hydropower as a focal
programme and has initiated a series of research projects in this area. International organizations
such as the Asian Development Bank, The World Bank, DFAT (covering areas formerly addressed by
AusAID), and NGOs such as WWF and Nature Conservancy all have activities in this field.
159. There has been a substantial debate in particular on the wisdom or otherwise of developing
dams on the Mekong River mainstream; but there are also concerns over hydropower development
48
iea.cgiar.org
on tributaries and in other river basins. This debate has focused on the most appropriate balance
between meeting rapidly expanding demand for electricity, the maintenance of ecosystems integrity
and the availability of key ecosystems services such as fish production. It is widely recognized that
this debate faces a challenge of incomplete information, limited capacities in key institutions and the
scarcity of widely‐accepted success stories of innovative approaches to water governance. The
attention paid to these issues under WLE will therefore be a potentially important contribution to
knowledge in a key area for sustainable regional development.
160. The case study in this evaluation will consider to what extent the approach being adopted
under WLE is on track to provide this contribution. Some of the major issues it will consider include:
 Whether the specific issues being researched are ones that the wider community considers
to be key knowledge gaps.
 The nature of the involvement of WLE in the wider regional discourse on these issues and
the specific characteristics of the uptake and impact pathways of the projects, especially in
relation to governmental power sector planning systems.
 The nature of the partnerships in the research projects and in the regional approach as a
whole in the Mekong focal region.
 The inherent quality and feasibility of the research projects in the WLE programme,
especially given the limited time available for the existing portfolio of projects, currently
financed up to the end of 2016.
161. The case study will be undertaken through extensive documentary review (there is a large
number of documents on these issues in the region, including ones that specifically identify
knowledge gaps and key areas for institutional and policy development) and through stakeholder
consultations (both with research partners and with uptake target stakeholders, including
governments), particularly through targeted country missions, but also through telephone or Skype
interviews.
Key Stakeholders within Mekong Countries
162. There are numerous key stakeholders for consultation during the case study, including
partners directly involved in research projects as well as institutions likely to be engaged in the
uptake pathway process. Discussions with these stakeholders will address the perceived standard,
relevance and utility of research activities, the views of participating researchers on research
management and views of internal and external stakeholders on the overall WLE approach.
Use of Thematic Case Study
163. The Mekong Thematic Case Study will be used by the evaluation team to provide detailed
information concerning the inter‐relationship between water governance, hydropower development
and ecosystem services management in a region, where this is a major policy area, to which WLE
expects to contribute. It may also supply an Annex of the Final Report of the evaluation.
49
iea.cgiar.org
Thematic Case Study of the National Policy Influence of the WLE IWMI – TATA Partnership
164. The IWMI‐TATA Water Policy Research Program is a collaborative initiative with the Sir Ratan
TATA Trust (SRTT). The partnership emerged from a shared concern regarding the growing water
stress in different parts of India, it has been active across the CPWF – WLE transition, and it is
considered an important activity in WLE. In particular, it is regarded as a strong example of research,
which has been successfully pursued into the field of policy advice, leading to impacts. The case
study will explore the results of the research “legacy” from CPWF and the effects of the new WLE
approach, including the ESR Framework, on the partnership and its activities. Its key focus will be on
the extent to which the research has contributed to the formulation and implementation of policies
intended to reduce water stress and on evidence available to assess its impacts.
165. The Case Study will include:
 Desk Review of the extensive documentation concerning the partnership and its activities
from start‐up to current
 Review of information generated by the M&E and reporting system
 Inclusion of partnership in country mission to India: to include interviews with key
informants in IMWI and TATA and other key stakeholders, based on guidelines prepared
after documentary analysis of the partnership
 Follow up interviews with additional stakeholders identified during the country mission.
Use of the Case Study
166. The IMWI‐TATA Case Study will be used by the evaluation team to provide detailed
information concerning the uptake of research findings by policy makers, which is a major policy
area, to which WLE expects to contribute. It may also supply an Annex of the Final Report of the
evaluation.
Field Missions
167. Field missions will be conducted to selected locations in focal river basins, regions and
possibly other locations where there are concentrations of WLE activities. These missions will
normally be conducted by two‐person teams. They will build on portfolio and documentary analysis
covering the mission location and will use semi‐structured interviews and group discussions to
gather data, explore opinions and perspectives, and verify emerging findings.
168. The following field missions are envisaged:
 IWMI Headquarters. IMWI stakeholders
 Cambodia, Laos, Thailand (Greater Mekong Region)
 India, Nepal (mission cancelled) (Ganges Region)
 Ethiopia, Kenya (Nile and East Africa Region)
 Ghana ‐ (Volta and Niger Region) .
50
iea.cgiar.org
5.3 Selection of methods
169. A preliminary overview of the application and mix of methods is presented in the Evaluation
Matrix supplied as Annex 1. Whilst some specific issues or areas of interest may require only one
method, most will call for several in sequence; such as initial portfolio analysis, documentary review,
field missions involving individual interviews and group discussions, additional documentary analysis,
workshop (or email exchanges, phone calls, etc.) to verify or modify evidence and findings.
5.3.1 Overall Analysis and Triangulation of Findings
170. Overall analysis will involve compiling, comparing and cross‐checking (triangulating) the
findings from the different lines of inquiry with each other (e.g., document review, interviews, group
discussions, surveys and meta‐evaluation) to address the review questions. Those findings regarded
as verified, substantive and important to the evaluation will be analysed within the cause and effect
chains contained in different levels of Theory of Change (e.g. CGIAR, WLE, Regional and Project level
theories), enabling the team to arrive at conclusions and recommendations about the concept,
design and implementation of WLE and how these might be improved. An additional triangulation
approach will compare the evidence provided by stakeholders in different positions within the
overall CGIAR and WLE systems. Here variations between different categories of stakeholder may be
valid, particularly with regard to “opinion” questions; for example concerning the effects of CGIAR
reform processes. In such cases, the purpose of triangulation is not to move towards one verified set
of evidence, but rather to explore perceptual variation, which may have fundamental effects on
implementation and on the interpretation of results.
5.3.2 Examples of Analysis Using Triangulation

Gender in the WLE Programme
171. The incorporation of gender into all activities is one of the underlying principles of WLE,
which is now supported by a Gender Strategy, in‐house expertise and budgetary allocations. This
analysis will explore the processes involved in this incorporation and the extent and effectiveness of
the gender approach as implemented across the portfolio. The analytical approach will include:
 Desk Review of Documents concerning gender and WLE activities from start‐up to current
 Review of information generated on gender by the M&E and reporting system
 Inclusion in Country missions: topic to be addressed in interviews with key informants from
WLE, donors, regional and national institutions and other key stakeholders
 Follow up interviews with additional stakeholders identified by the above methods
 Inclusion in surveys of staff and partners.
5.3.3 Analysis of Governance and Management
172. The program’s governance (e.g. oversight, stakeholder participation) and management
(planning, reporting, finance and human resource management, the quality and co‐operation of
leadership, the interactions between centres) will be reviewed. This will involve interaction with
51
iea.cgiar.org
centre directors and staff, with partners and a range of other stakeholders. Evaluative methods will
include:
 Analysis of current G&M structures and processes (including ToRs, interrelationships,
reporting lines)
 Desk review of documents including Management Committee and Steering Committee
meeting documentations, IWMI Board of Trustees minutes, programme proposals and
annual WLE reports
 Review of M&E outputs
 Visits to Centres for interview with management and researchers and observation of
processes.
5.3.4 Sequencing and Contribution of Methods
173. Figure 5 below provides an overview of the sequencing and contribution of the methods and
approaches described above. It should be noted from the figure that each individual method both
provides direct evidence and contributes towards other methods. Thus there will be direct evidence
and analysis from documentary reviews, but they will also provide inputs into the content of
subsequent field missions and surveys. Further, the notional sequencing of activities and methods
shown in Figure 5 does not indicate discrete chronological phases. For example, the portfolio and
document reviews will be updated as additional information becomes available.
52
iea.cgiar.org
Figure 5: Overview of Sequencing and Contribution of Methods
Portfolio Analysis
Semi‐structured interviews Narrative Analysis
Group discussions
Documentary Review
Social surveys Theory Based Analysis
Case studies
Triangulation
of Evidence
Meta Analysis
Thematic Analysis
Field missions
Inception Mission :
53
iea.cgiar.org
6. ORGANIZATION AND TIMING OF THE EVALUATION
6.1 Team Composition
174. A multidisciplinary team covering a wide range of disciplines, areas of expertise and
experience will undertake the evaluation. The core evaluation team is comprised of five team
members. The Evaluation Team Leader has broad experience in leading complex evaluations at the
international level, and is supported by a team of specialists who will have between them extensive
and proven experience at a variety of different levels, working for international and development
agencies, on issues, programmes and policies related to WLE’s activities.
Table 6 below summarises the areas of expertise of each team member.
Table 5: areas of expertise of each team member.
Name Nationality Area of Expertise Bio
David Todd UK Consultant. Formerly, Senior Evaluation

 Evaluation Methodology and Specialist, Global Environment Facility;
Management Social Development Adviser, DFID; Head of
Social Planning, Environment Division, WS
 Theory‐Based Impact Evaluation
Atkins Consultants, UK; Senior Lecturer,
 Environmental Evaluation School of Environmental Studies, University
of Zambia.
 Poverty and Gender
Sylvie M. USA Professor of Agronomy, Wickersham Chair

Brouder  Soils of Excellence in Agricultural Research,
Department of Agronomy: Crops, Soils and
 Crops Environmental Sciences, Purdue University,
USA.
 Agronomy
 Agroecology
 Ecological Sciences
Charles UK Consultant. Former Principal Scientist,
Batchelor  Hydrology and Water Resources Institute of Hydrology, Wallingford, UK.
Management
 Irrigated Agriculture
 Rainfed Agriculture, Rangeland
Management and Forestry
 Urban Water Management
54
iea.cgiar.org
 Water Supply and Sanitation
 Software Development
 Advisory and Review Studies
Elias Spain Professor of Agronomy in the School of

Fereres  Agricultural Engineering Agricultural and Forestry Engineering,
University of Córdoba, Spain, and
 Irrigation Science Researcher at the Institute of Sustainable
Agriculture, Scientific Research Council of
 Surface Water Management
Spain (IAS‐CSIC)
 Ecology
 Sustainable Agriculture
 Crop Ecology
John UK Consultant. Formerly Professor of
Soussan  Ecosystems Services Assessment Environment, University of York, UK and
Senior Research Fellow, Stockholm
 Payment for Ecosystems Services Environment Institute; and Professor and
Director of the Centre for Water Policy and
 Sustainable Livelihoods
Development, Development, University of Leeds, UK.
 Water Management
 Coastal Zone Development
 Poverty Reduction
 Mainstreaming of Environmental

Issues into National Development
Policies
175. It is possible that some additional expert resources will be added, possibly through the
vehicle of an Expert Panel, as the evaluation develops.
6.2 Team Roles and Responsibilities
176. The first meeting of the Evaluation Team in Rome agreed the following preliminary
distribution of roles and responsibilities among members.
55
iea.cgiar.org
Table 6: WLE Evaluation Team
Team Member Responsibilities Field Missions

David Todd Team Leader. Lead, ESR Framework. Joint‐ Cambodia, Laos, Thailand
Lead, GPI Theme.
Elias Fereres Lead, Flagship 2, LWP. Joint‐Lead, Flagship 5, Cambodia, Laos, Thailand
MRV, Joint‐Lead Case Study of Mekong River
Basin.
Charles Batchelor Lead, Flagship 4, RRR. Lead, DAI Theme. Joint‐ India, Nepal, (mission cancelled)
Lead Case Study of IMWI‐TATA partnership. Ghana
Sylvie Brouder Lead, Flagship 3, RDE. Joint‐Lead, Flagship 5, India, Ethiopia, Kenya
MRV. Joint‐Lead Case Study of IMWI‐TATA
partnership.
John Soussan Lead, Flagship 1, IES. Joint‐Lead, GPI Theme. Cambodia, Laos, Thailand,
Joint‐Lead Case Study of Mekong River Basin. Ethiopia, Kenya
177. The initial field missions (Cambodia, Laos, Thailand and India) are planned to take place
between about 1 and 11 May 2015. The evaluation team will then meet in Sri Lanka for discussions
within the team and with IMWI stakeholders from approximately 12 to 17 May, after which a second
set of field missions (Ethiopia, Kenya, Ghana) will be undertaken in June 2015. The exact dates of
missions will be finalised in collaboration with participating stakeholders.
6.3 Evaluation governance/roles and responsibilities
178. A Team of Independent External Experts will conduct the Evaluation. The Team Leader has
final responsibility for the evaluation report and all findings and recommendations, subject to
adherence to CGIAR Evaluation Standards. The Evaluation Team is responsible for submitting the
deliverables as outlined in more detail below.
179. The IEA is responsible for planning, designing, initiating, and managing the evaluation. The
IEA will also be responsible for the quality assurance of the evaluation process and outputs, and for
the dissemination of the results. The IEA will take an active role in the preparatory phase of the
evaluation by collecting background data and information and by carrying out preliminary analysis
on WLE. An Evaluation Manager, supported by an Evaluation Analyst, will provide support to the
team throughout the evaluation.
180. WLE management plays a key role in helping provide for the evaluation team’s informational
needs. It provides documentation and data, information on all WLE activities, access to staff for
engagement with the evaluators, and information on partners and stakeholders. It facilitates
arrangement of site visits and appointments within the lead Centre and other stakeholders. WLE
management is also responsible for giving factual feedback on the Draft Report and for preparing
the Management Response to the Final Report. It assists in dissemination of the report and its
findings and lessons and it acts on the accepted recommendations. While the evaluation is
coordinated with WLE management, IWMI as the lead Centre is also a key stakeholder in the
evaluation. The Centre and its leadership and board are expected to make themselves available for
consultations during the evaluation process.
56
iea.cgiar.org
181. A Reference Group has been set‐up to work with the IEA Evaluation Manager and Team
Leader to ensure good communication with, learning by, and appropriate accountability to primary
evaluation clients and key stakeholders, while preserving the independence of evaluators. The
Reference Group provides views and inputs at key decision stages in the evaluation design and
implementation process, including for the Terms of Reference, the Inception Report and the Draft
Report. The Reference Group may also play an important role in leading evaluators to key people
and documents. The reference group consists of ten representatives, listed in the Table 8 below.
Table 7: Reference Group Members
NAME POSITION ORG
Johan Rockstrom Chair of WLE Steering Committee Stockholm Resilience

Centre
Charlotte de Fraiture Member of WLE Steering Committee UNESCO‐IHE Institute

for Water Education
John Williams Member of WLE Steering Committee Natural Resources

Commission, Australia.
Letitia Obeng IWMI Board Member
Andrew Noble CRP Director IWMI
Emma Geatrix CRP Program Manager IWMI
Alain Vidal Previous Director of the CPWF Consortium Office
Peter Mc Cornick Deputy Director General Research IWMI
Claudia Ringler Focal Point and co‐leader of Managing IFPRI

Resource Variability and Competing Use
Flagship
Fabrice De Clerck Focal Point and Leader of the Ecosystem Bioversity

Services and Resilience cross‐cutting
Theme
Nicoline de Haan Core Theme Leader Gender, Poverty and IWMI

Institutions
6.4 Quality Assurance
182. In order to ensure evaluation rigor, the following quality assurance will be implemented
during the evaluation exercise.
57
iea.cgiar.org
183. The IEA, as manager of the Evaluation, will play a crucial role in assuring its quality. The IEA
will work closely with the Evaluation Team throughout the evaluation, and will ensure that the tools
and methodologies, as well as the process followed, are in line with the CGIAR Evaluation Policy and
Standards as well as with those used in other ongoing CRP evaluations.
184. External peer review: The IEA quality assurance of evaluations includes the peer review for
each CRP evaluation by two external peer reviewers at two stages in the evaluation process: the
draft inception report and the draft evaluation report. The primary function is not ex‐post quality
control but represents an additional quality review to the IEA evaluation managers. It is timed so
that it can help improving the outputs (whether the inception or the evaluation report) and make
them in line with CGIAR‐IEA standards. IEA has developed Guidance for Inception Reports and
Evaluation Reports, as well as an Outline for External Peer Reviewers.
6.5 Stakeholder Involvement
185. The WLE Evaluation involves a large set of stakeholders. These include the management of
WLE, all participating Centers, partners associated with the Program, the CGIAR Fund Council, and
the Consortium Board, as shown in Table 9 below. These stakeholders will be consulted and engaged
throughout the evaluation through such means as structured interviews, surveys and site visits.
186. An additional method intended to ensure the participation of stakeholders is through small
workshops at which the evaluation team will present emerging findings and issues in order to obtain
verification (or not), access to additional viewpoints and data, and contextualisation of its evidence
base. In view of budget restrictions, these will be held as far as possible in association with other
activities, such as debriefing meetings at national or regional level and with WLE management. The
team is currently working with WLE management to identify an opportunity for such a workshop in
(or around) July 2015, by which time a substantial body of work will have been conducted.
Table 8: WLE evaluation stakeholders
Type of stakeholder Role in WLE Interest in evaluation

CRP level
WLE management Program management Lessons learned to increase performance of
(Operations Team and WLE
Management Committee)
WLE Steering Committee Program oversight and Accountability
strategic advice WLE performance
Lessons learned about effectiveness of
Governance committees
WLE Researchers Carry out research in line Research performance
with WLE IDOs
Center level
IWMI Management Contribution to program Organizational performance
management Comparative advantage
58
iea.cgiar.org
IWMI Board Fiduciary responsibility Organizational performance

Oversight of the WLE Comparative advantage
Boards and management Oversight of WLE activities Organizational performance
of participating centers carried out by its center Comparative advantage
CGIAR level
CGIAR Fund Council Oversight on use of funds Accountability
for WLE WLE performance
Decision making for resource allocation
Donors of bilateral Funding source Accountability
projects WLE performance
Decision making for resource allocation
CGIAR Consortium Signatory to Program Lessons learned to
Implementation increase the effectiveness and relevance of
Agreements with WLE lead‐ the work of the CGIAR;
Center, strategic advisor Lessons learned to increase the efficiency
and oversight body. and accountability of the CGIAR.
Partners
Research partners Participate in the design Research Performance
and conduct of CRP Collaboration mechanisms, Capacity
Research development
Development and Targeted stakeholders for Relevance of WLE and its research, Research
Boundary Partners implementing change Performance, Collaboration mechanisms,
Capacity development
6.6 Timeline
187. The overall timeline for the evaluation has been established by its Terms of Reference.
Currently, this timeline looks realistic. However, this will be kept under review by the Team Leader,
in coordination with IEA in order to take account of any potential delays for logistical reasons. For
example, if some projects or regions cannot be visited until later than planned. The outline timeline
is shown in Table 10 below. This is regarded as preliminary since the precise timing of field missions
has not yet been determined and will depend partly on the availability of project and key
institutional stakeholders, which is not yet known.
59
iea.cgiar.org
Table 9: Outline Timeline for WLE Evaluation
Phase Period Main outputs Responsibility
Preparatory Phase Sep – Dec 2014 Final ToR IEA
Evaluation team recruited
Inception Phase Jan – April 2015 Inception Report Evaluation team
Inquiry phase April – Sep 2015 Working documents and Evaluation team

analysis products as inputs
into preliminary findings
Presentation of preliminary Sep 2015 Presentation of preliminary Evaluation team

findings findings
IEA
Feedback from main
stakeholders
Reporting phase
Drafting of Report Sep – Oct 2015 Draft Evaluation Report Evaluation team
Final Evaluation Report Dec 2015 Final Evaluation Report Evaluation team
Management and CGIAR Feb 2016 Management and CRP Management

Consortium Responses Consortium Responses
Consortium Board
6.7 Key Deliverables and Dissemination of Findings
188. The key deliverables of the evaluation are outlined in the Terms of Reference.
189. The Inception Report builds on the original terms of reference for the evaluation and
outlines the emerging issues as well as the proposed approach to the main phase of the evaluation.
It constitutes the guide for conducting the evaluation by:
 Outlining the scope of the evaluation
 Clarifying the analytical frameworks which will be utilized by the evaluation
 Developing the methodological tools
 Providing a detailed evaluation matrix
 Providing a detailed work plan for the Evaluation.
190. The Evaluation Report is the principal output of this evaluation. It will describe findings,
conclusions, and recommendations, based on the evidence collected in the framework of the
60
iea.cgiar.org
evaluation questions defined in the Inception Report, as refined and developed during the course of
the evaluation. The recommendations will be evidence‐based, relevant, focused, clearly formulated
and actionable. They will be prioritized and addressed to the different stakeholders responsible for
their implementation. The main findings and recommendations will be summarized in an executive
summary.
191. The Team Leader and Team Members will prepare Presentations for disseminating key
elements of the Report to targeted audiences. The exact forms of these presentations will be agreed
upon during the course of the evaluation.
192. Adequate consultations with WLE stakeholders will be ensured throughout the process, with
debriefings on key findings held at various stages of the evaluation. The final report will be
presented to key CGIAR stakeholders. Following this, the IEA will interact with WLE management
during preparation of the Management Response.
193. WLE Management will prepare a Management Response to the evaluation for the
consideration of the Consortium Board. The management response will be specific in its response to
evaluation recommendations, regarding the extent to which it accepts each recommendation and
reasons for partial acceptance or non‐acceptance. For those recommendations which it accepts
partially or in full, management will enumerate the follow‐up action(s) it intends to take, and in what
timeframe. The consolidated response of WLE management and the Consortium Board will be a
public document made available as a package together with the Evaluation Report, for the
consideration of the CGIAR Fund Council.
194. WLE will develop a specific strategy for disseminating evaluation results during the course of
its implementation, responding to emerging findings and the balance of stakeholders expected to
engage with and respond to them.
61
iea.cgiar.org
ANNEX 1: EVALUATION MATRIX
Potential Methods of Evaluation
Evaluation Criteria Portfolio Document Group Semi‐ Surveys Workshops Case Field
Review Review discussions structured Studies missions
(includes interviews
meta
analysis)
Relevance * * * * * * *
Quality of Science * * * * * *
Effectiveness * * * * * * *
Impact * * * * * *
Sustainability * * * * * *
Efficiency * * * * * *
Additional Areas for Questions
Partnerships * * * * * *
Gender * * * * * * * *
Capacity Development * * * * * * * *
Monitoring, Evaluation and Learning * * * * * * *
Over‐Arching Questions
Are the conceptual framework and key * * * * * *
hypotheses of WLE coherent, effectively
communicated and appropriate for the wide
range of issues and diversity of locations
Are the impact pathways identified by the * * * * *
program ones that are likely to achieve the
desired development outcomes and are they
consistently developed across the different
62
iea.cgiar.org
Is the quality of science in the program of an * * * *
appropriate standard, does it reflect the
CGIAR’s comparative advantage in science
and is it engaged with and reflective of
international developments in the different
fields included in the program?
Are the partnership strategy and range * * * * * *
of partnerships being developed in the
program consistent with the program’s
overall goals and the development of impact
pathways within the program?
Are the management procedures and * * * * *
governance structure of the program
appropriate, efficient and consistently
applied and is there clarity on the roles and
operational procedures of different
components of the management structure?
Is the development of the program * * * * *
finding an effective balance between
ensuring the delivery of concrete planned
outcomes within the program life whilst at
the same time creating a long‐term
development trajectory to ensure the
program’s approach and benefits will
continue beyond the presently defined
program period?
63
iea.cgiar.org
ANNEX 2: DOCUMENTS CONSULTED
Biggs, R., Schlüter, M., Biggs, D., Bohensky, E.L., BurnSilver, S., Cundill, G., Dakos, V., Daw, T.M.,
Evans, L.S., Kotschy, K., Leitch, A.M., Meek, C., Quinlan, A., Raudsepp‐Hearne, C., Robards, M.D.,
Schoon, M.L., Schultz, L. & West, P.C. (2012) Toward principles for enhancing the resilience of
ecosystem services. Annual Review of Environment and Resources. 37:421‐448.
CGIAR. Changing Agricultural Research in a Changing World. A Strategy and results framework for
the Reformed CGIAR. 2011. Montpelier.
CGIAR. Agreement establishing the Consortium of International Agricultural Research Centers as an
International Organization.
CGIAR. CGIAR Policy for Independent External Evaluation. 2012.
CGIAR Standards for Independent External Evaluation. 2014.
Chong, J. (2005), Valuing the Role of Wetlands in Livelihoods: Constraints and Opportunities for
Community Fisheries and Wetland Management in Stoeng Treng Ramsar Site, Cambodia. IUCN
Water, Nature and Economics Technical Paper No. 3, IUCN — The World Conservation Union,
Ecosystems and Livelihoods Group Asia, ColomboReview of CRP Governance and Management
Costanza, R., de Groot, R., Sutton, P., van der Ploeg, S., Anderson, S.J., Kubiszewski, I., Farber & S.,
Turner, R.K. 2014. Affected Mangroves — The World Conservation Union, Ecosystems and
Livelihoods Group Asia. Changes in the global value of ecosystem services. Global Environmental
Change. 26: 152‐158.
Daw, T., K. Brown, S. Rosendo, and R. Pomeroy. 2011. Applying the ecosystem services concept to
poverty alleviation: The need to disaggregate human well‐being. Environmental Conservation.
38:370–379.
Estrada‐Carmona, N., Hart, A.K., DeClerck, F.A.J., Harvey, C.A. & Milder, J.C. 2014. Integrated
landscape management for agriculture, rural livelihoods, and ecosystem conservation: An
assessment of experience from Latin America and the Caribbean. Landscape and Urban Planning.
129: 1‐11.
Greiber, T. (2009) Payments for ecosystem services: legal and institutional frameworks. IUCN
Environmental Policy and Law Paper No.78.
Haines‐Young, R. and Potschin, M. 2010. The links between biodiversity, ecosystem services and
human well‐being in Ecosystem Ecology: A New Synthesis, eds. David G. Raffaelli and Christopher L.
J. Frid. Cambridge University Press, Cambridge (Ch 6).
Hall, A, A. Bullock, B. Adolph. Forward Looking Review of the Challenge Programme for Water and
Food (CPWF) 2013 – 2014, IEA, Rome. 2014.
64
iea.cgiar.org
Independent Evaluation Arrangement. IEA Guidance Note G1. Guidance for Managing the
Independent External Evaluation of CGIAR Research Programs (CRPs). 2014.
Independent Evaluation Arrangement. Guidance on Inception Reports. 2014.
Independent Evaluation Arrangement. Review of CGIAR Research Programs Governance and
Management. Final Report, Rome. March 2014.
Independent Evaluation Arrangement. Terms of Reference for the evaluation of the CGIAR Research
Program on Water, Land and Ecosystems (WLE) Rome, 2014
Independent Evaluation Arrangement. Evaluation of the CGIAR Research Program, “Forests, Trees
and Agroforestry” (FTA). 2014.
Matthews, N. 2012. Water grabbing in the Mekong basin ‐ An analysis of the winners and losers of
Thailand's hydropower development in Lao PDR. Water Alternatives. 5(2): 392‐411.
Merrey, D. An Evaluation of CGIAR Centers’ Impact Assessment Work on Irrigation and Water
Management Research. 2014.
OECD/DAC Network on Development Evaluation: “Assessing the Development Effectiveness of

Multilateral Organizations: Guidance note on the Methodological Approach”. (2012).
Plummer, M. L. (2009) Assessing benefit transfer for the valuation of ecosystem services. Frontiers
in Ecology and the Environment 7:38–45.
Poppy, G.M., Jepson, P.C., Pickett, J.A. and Birkett, M.A. 2014. Achieving food and environmental
security: new approaches to close the gap. Phil. Trans. R. Soc. B. 369: 20120272
Ruckelshaus, M., E. McKenzie, H. Tallis, A. Guerry, G. Daily, P. Kareiva, S. Polasky, T. Ricketts, N.
Bhagabati, S. a. Wood, and J. Bernhardt ( 2013) Notes from the field: Lessons learned from using
ecosystem service approaches to inform real‐world decisions. Ecological Economics. doi:
10.1016/j.ecolecon.2013.07.009
Schomers, S. & Matzdorf, B. (2013). Payments for ecosystem services: a review and comparison of
developing and industrialized countries. Ecosystem Services. Ranasinghe T & Kallesoe, M. (2006)
Valuation, Rehabilitation and Conservation of Mangroves in Tsunami Affected Areas of Hambantota,
Sri Lanka: Economic Valuation of Tsunami
TEEB (2009) The Economics of Ecosystems and Biodiversity for National and International Policy
Makers – Summary: Responding to the Value of Nature. ELD (2012) The economics of land
degradation, the global initiative for sustainable land management ELD Secretariat, Bonn.
Willemen, L., Drakou, E.G., Dunbar, M.B., Mayaux, P. & Egoh, B.N. (2013) Safeguarding ecosystem
services and livelihoods: Understanding the impact of conservation strategies on benefit flows to
society. Ecosystem Services. 4: 95‐103.
65
iea.cgiar.org
WLE. Flagship 4: Recovering and Reusing Resources in Urbanized Ecosystems (RRR). 2014 Activity
Plans
WLE. Flagship 3: Regenerating Degraded Agricultural Ecosystems (RDE). 2014 Activity Plans
WLE. Flagship 5: Managing Resource Variability and Competing Use (MRV). 2014 Activity Plans
WLE. Flagship 2: Sustainably Increasing Land and Water Productivity (LWP). 2014 Activity Plans
WLE. Decision Analysis and Information Systems (DAI). 2014 Activity Plans
WLE. Decision Support Tools for Catalysing Change in Water, Land and Ecosystem Management.
Design notes on the Centre Commissioned External Evaluation (CCEE)
WLE. Recovering and Reusing Resources in Urbanising Ecosystems. Design notes on the Centre
Commissioned External Evaluation (CCEE)
WLE. Water, Land and Ecosystems. Improved natural resources management for food security and
livelihoods. CGIAR Research Program 5. September 2011
WLE. Extension Proposal for CGIAR Research Program on Water, Land and Ecosystems. Resubmit,
September 8, 2014. P3.
WLE. Ecosystem Services and Resilience Framework. IMWI. Undated.
WLE. CGIAR Research Program on Water, Land and Ecosystems: Annual Report 2012
WLE. CGIAR Research Program on Water, Land and Ecosystems: Annual Report 2013
Wright, H. L., Lake, I. R., & Dolman, P. M. (2012). Agriculture—a key element for conservation in the
developing world. Conservation Letters, 5(1), 11‐19.
Links to web pages consulted
http://www.cgiarfund.org/sites/cgiarfund.org/files/Documents/PDF/CGIAR_evaluation_policy_jan20
12.pdf
http://www.cgiar.org/who‐we‐are/cgiar‐fund/
66
iea.cgiar.org
ANNEX 3: PERSONS CONSULTED DURING INCEPTION
MISSION TO IMWI
Name Position
Andrew Noble WLE Director
Jeremy Bird IWMI Director General
Emma Greatrix WLE Program Manager
Elizabeth Weight IWMI Global Science Uptake Coordinator
David Rider Smith WLE Monitoring, Evaluation and Learning Manager
Michael Victor WLE Engagement, Communication and Knowledge Management Coordinator
Pay Drechsel WLE RRR Flagship Leader
Vladimir Smakhtin WLE MRV Flagship Leader
Nathanial Matthews WLE IES Flagship Leader
Fabrice DeClerck WLE ESR Core Theme Leader
Nicoline de Haan WLE GPI Core Theme Leader
Donald Blackmore IWMI Board Chair
Barbara Schreiner IWMI Board Member
George Rothschild IWMI Board Member
Isher Ashluwalia IWMI Board Member
Chemutai Murgor IWMI Board Member
Alain Vidal CPWF, CGIAR Consortium Senior Partnerships Advisor
Peter McCornick IWMI Deputy Director of Research
Charlotte de Fraiture WLE Steering Committee Member
John Williams WLE Steering Committee Member
Amol Khisty IWMI Director, Finance and Administration
Olufunke Cofie IWMI Senior Researcher and Head of Office (West Africa)
Arif Anwar IWMI Principal Researcher (Irrigation) and Head of Office (Pakistan)
Nicole Lefore IWMI Senior International Manager, Research for Development
Miriam Otoo IWMI Researcher ‐ Economics
Javier Mateo‐Sagasta IWMI Senior Researcher (Water Quality and Safe Reuse)
Jayne Curnow IWMI Researcher ‐ Social Sciences
Robyn Johnson IWMI Representative (Myanmar)
Fraser Sugden IWMI Researcher ‐ Social Sciences
Tim Williams IWMI Director for Africa
Meredith Giordano IWMI Advisor, Research Strategy and Management
Rinku Dokania IWMI Manager, Financial Planning and Analysis
Nadeesha Rajapaksha IWMI Project Officer (Finance)
Matthew McCartney IWMI Principal Researcher (Hydrology) and Head of Office (Laos)
Simon Langan IWMI Principal Researcher and Head of Office (Nile Basin and East Africa)
Srabani Roy IWMI Director for Asia
67
iea.cgiar.org
ANNEX 4: PERSONS MET DURING INCEPTION MISSION
TO ROME
Name Position
Andrew Noble WLE Director
Emma Greatrix WLE Program Manager
Elizabeth Weight IWMI Global Science Uptake Coordinator
David Rider Smith WLE Monitoring, Evaluation and Learning Manager
Michael Victor WLE Engagement, Communication and Knowledge Management Coordinator
Pay Drechsel WLE RRR Flagship Leader
Vladimir Smakhtin WLE MRV Flagship Leader
Nathanial Matthews WLE IES Flagship Leader
Fabrice DeClerck WLE ESR Core Theme Leader
Nicoline de Haan WLE GPI Core Theme Leader
Donald Blackmore IWMI Board Chair
Meredith Giordano WLE LWP Flagship co‐leader
Theib Oweis WLE LWP Flagship co‐leader
Deborah Bossio WLE RDE Flagship co‐leader
Suhas Wani WLE RDE Flagship co‐leader
Lisa‐Maria Rebelo WLE DAI co‐leader
John Williams WLE Steering Committee Member
Peter Gardiner ISPC Executive Director
Tim Kelley SPIA Senior Officer
WLE Management
Peter McCornick
Committee
Johan Rockstrõm, John Williams, Gretchen Daily Olcay Űnver, Mihir Shah, Lindiwe
WLE Steering Committee Sibanda, Jeremy Bird (ex‐officio)
68
iea.cgiar.org
ANNEX 5: OUTLINE WORK PLAN
Activity J F M A M J J A S O N D J F
Inception
Document Analysis
Portfolio Analysis
Telephone Interviews
Selection of activities for field research
Preparation for field missions
Field missions
Non Field Case Studies
Surveys
Analysis
Presentation of Preliminary Findings *
Feedback from main stakeholders *
Report drafting
Draft Report circulated *
Feedback from main stakeholders *
Final Report Prepared
Final Report Presented *
Management and Consortium Responses
69
iea.cgiar.org
ANNEX 6: OUTLINE CONTENTS OF FINAL REPORT
Outline of the Final Report
Abbreviations
Executive summary
1. Introduction
1.1 Purpose and audience
1.2 The evolving CGIAR context
1.3 Evaluation questions
1.4 Evaluation methodology
1.5 Analysis framework
1.6 Timeline and organization
1.7 Deviation from inception report
1.8 Limitations to the evaluation
1.9 Structure of the report
2. WLE background
2.1 WLE Context
2.2 WLE program
2.2.1 Objectives, structure and activities
2.2.2 WLE funding
2.2.3 Donors
2.2.4 WLE project portfolio
3. Relevance
3.1 Coherence and program design
3.1.1 Research strategies
3.1.2 Consistency with reform principles
3.1.3 Targeting and impact pathways
3.1.4 Use of W1/2 funding
3.1.5 Priority setting
70
iea.cgiar.org
3.2 Comparative advantage
3.3 Conclusions and recommendations
4. Quality of Science
4.1 Introduction
4.2 CRP‐wide assessment of staff and publications output
4.2.1 Research staff in WLE
4.2.2 Research staff perceptions of QoS
4.2.3 Publication quality
4.3 Quality in Flagship 1
4.3.1 FP1 Research staff
4.3.2 Processes in FP1
4.3.3 FP1 Outputs
4.4.2 Processes in FP2
4.4.3 FP2 Outputs
4.5.2 Process in FP3
4.5.3 FP3 Outputs
4.6.3 FP4 Outputs
4.7.3 FP5 Outputs
5. Effectiveness
5.1 Overview
71
iea.cgiar.org
5.2 Progress toward outputs
5.2.1 Flagship 1: Integrating Ecosystem Solutions into Policy and Investments
5.2.2 Flagship 2: Sustainably Increasing Land and Water Productivity
5.2.3 Flagship 3 – Regenerating Degraded Agricultural Ecosystem
5.2.4 Flagship 4 ‐ Recovering and Reusing Resources in Urbanized
5.2.5 Flagship 5 ‐ Managing Resource Variability and Competing Use
5.3 Monitoring, evaluation and enabling for internal processes for enhanced effectiveness
6. Gender, capacity development and partnerships
6.1 Introduction
6.2 Gender
6.3 Capacity development
6.3.1 Training
6.4 Partnerships
6.4.1 Research partners
6.4.2 Boundary partners
7. Impact and sustainability
7.1 Introduction
7.2 Results of studies
7.3 Assessment
8. Overall Conclusions, Way Forward and Recommendations
Conclusions structured around overarching questions
72
iea.cgiar.org

WLE Evaluation Inception Report - Final

Uploaded by

Copyright:

Available Formats

WLE Evaluation Inception Report - Final

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

WLE Evaluation Inception Report - Final

Uploaded by

Copyright:

Available Formats

INCEPTION

4.5 Performance on cross-cutting issues ................................................................. 37

Table 1: 2015 Flagship budgets by funding source and number of activities................ 23

Table 3: list of projects ...................................................................................................... 26

Figure 1: Basic WLE Governance and Management structure ........................................ 20

Box 1: Major Sources of Funding in the CGIAR System ..................................................... 9

 Flagship 2: Sustainably Increasing Land and Water Productivity (LWP)

1.2 Evaluation Purpose and Users

1.3 Purpose and Structure of the Inception Report

2.2 Background to the Water, Land and Ecosystems (WLE) Program

2.2.1. Context of Research on Water, Land and Ecosystems

 Information management technologies & applications: New information management

2.2.2. Institutional and Program Structure and Origins

 Emphasis on context‐specific development impact from scientific research (“participatory

2.3 The WLE Portfolio

2.3.1. Focal Regions

2.3.2. 2015 WLE Portfolio of Activities

ILRI CIP IRRI World Fish

CGIAR Center Total Funding W1/W2 Share of W1/W2 N. of activities

Center Flagship Project Title Funding in

3.1 Research/Programmatic Performance

3.2 Organizational Performance

3.3 Geographical Scope

3.4 Evaluability Issues and Limitations of the Evaluation

3.4.1 The WLE Impact Pathways and Theory of Change

3.4.2 Main Limitations of the Evaluation

4.2 Strategic Questions

4.3 Criteria for Research/Programmatic Performance

4.3.2 Quality of Science

4.4 Criterion for the Evaluation of Organizational Performance

4.5 Performance on cross-cutting issues

4.5.2 Gender, Poverty and Institutions

4.5.3 Capacity Strengthening

4.6 Inter-relationships between Evaluation Criteria, Strategic Questions and

Relevance Quality of Effectiveness Impact Sustaina‐ Efficiency63

5.2.1 Collection and Analysis of Existing (Secondary) Sources of Information

5.2.2 Methods for Collection and Analysis of Original (Primary) Data

5.3 Selection of methods

5.3.1 Overall Analysis and Triangulation of Findings

5.3.2 Examples of Analysis Using Triangulation

5.3.3 Analysis of Governance and Management

5.3.4 Sequencing and Contribution of Methods

Social surveys Theory Based Analysis

Name Nationality Area of Expertise Bio

David Todd UK Consultant. Formerly, Senior Evaluation

Sylvie M. USA Professor of Agronomy, Wickersham Chair

Elias Spain Professor of Agronomy in the School of

 Mainstreaming of Environmental

6.2 Team Roles and Responsibilities

Team Member Responsibilities Field Missions

6.3 Evaluation governance/roles and responsibilities

NAME POSITION ORG

Johan Rockstrom Chair of WLE Steering Committee Stockholm Resilience

Charlotte de Fraiture Member of WLE Steering Committee UNESCO‐IHE Institute

John Williams Member of WLE Steering Committee Natural Resources

Andrew Noble CRP Director IWMI

Emma Geatrix CRP Program Manager IWMI

Alain Vidal Previous Director of the CPWF Consortium Office

Peter Mc Cornick Deputy Director General Research IWMI