Number 14 February 2016

Centre for Development Impact

PRACTICE PAPER
Innovation and learning in impact evaluation

Balancing Inclusiveness, Rigour

and Feasibility: Insights from
Participatory Impact Evaluations in
Ghana and Vietnam
Abstract This paper by Adinda Van Hemelrijck and Irene Guijt explores how impact evaluation can
live up to standards broader than statistical rigour in ways that address challenges of complexity
and enable stakeholders to engage meaningfully. A Participatory Impact Assessment and Learning
Approach (PIALA) was piloted to assess and debate the impacts on rural poverty of two government
programmes in Vietnam and Ghana funded by the International Fund for Agricultural Development
(IFAD). We discuss the trade-offs between rigour, inclusiveness and feasibility encountered in these two
pilots. Trade-offs occur in every impact evaluation aiming for more than reductionist rigour, but the
pilots suggest that they can be reduced by building sufficient research and learning capacity.

1 Introduction complex environments; they do not explain impacts or

The past ten years have seen a surge in interest and enable stakeholders to engage and learn. Rigour is purely
investment in impact evaluation in development. Bulletproof statistical and method-specific, thus inadequate for other
numbers are required to justify programme investments (including participatory) methods (Stern et al. 2012). How
at scale. Credible explanations of observed changes are then to ensure that impact evaluation of complex programmes
needed to influence national policy and local responsibility is rigorous and inclusive, and triggers reflection and learning
for greater impact. Large programmes with big investments, about contributions, while also remaining feasible?
though, are increasingly complex and political (Wild et al.
2015). Interventions are less standardised, stakeholders are Responding to this challenge, the International Fund
more diverse, influences are more dense, problems are more for Agricultural Development (IFAD)1 and the Gates
intertwined and systemic, solutions are less straightforward, Foundation (BMGF) commissioned the development of a
and changes are emergent and less predictable (Burns 2014a; Participatory Impact Assessment and Learning Approach
Woolcock 2013). Additionally, Sustainable Development (PIALA) that could help IFAD and partners rigorously and
Goals (SDGs) are adding demands for greater inclusiveness collaboratively assess, explain and debate their contributions
and sustainability as well as effectiveness, forcing a rethink of to reducing rural poverty (IFAD and BMGF 2013b).2 PIALA
impact evaluation. was designed and piloted around standards of rigour,
inclusiveness and feasibility that are considered vital to create
Mainstream practice is still dominated by the need impact evaluations of value.
for numbers that attribute impact to investments.
Counterfactual-based approaches using statistical and Rigour3 in this approach refers to the quality of thought
(quasi-)experimental methods fit this need (White 2009). put into the methodological design and conduct of every
Generally, these are very costly and difficult to pursue in step in the evaluation – including sampling, triangulation
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 2 PRACTICE PAPER CDI
Figure 1 PIALA design elements
Rigour

Focus and frame the evaluation Systemic

ToC approach

Multi-stage sampling
Collect and link data of open systems
Participatory mixed-methods

Synthesise evidence Participatory sensemaking

Analyse and debate contributions Configuration analysis

Inclusiveness Feasibility
Source: Presentation given by the authors at the IDEAS Global Assembly, 29 October 2015, Bangkok.

of methods, facilitation of processes, data collation, cross- First, clean ‘control’ groups are rarely found and quantitative
validation and causal analysis. This is to ensure consistency estimation of net attributable impacts on ‘treated’ compared
and responsiveness to the purposes and constraints of the to ‘control’ groups is often impossible or inadequate. This is
evaluation, necessary to establish sufficient confidence the case in IFAD programmes where institutional and policy
among stakeholders in its findings and conclusions (Rogers work ‘contaminate’ entire populations, where other donors
2009; Stern et al. 2012). Inclusiveness involves meaningful and influences augment ‘causal density’, where self-targeting
engagement of stakeholders with diverse perspectives, mechanisms make ‘treatments’ highly diverse, and where
which has an intrinsic empowering value while also innovations have emergent results (Woolcock 2013). Hence
enhancing credibility of the evaluation through triangulation the need for different ways of arriving at rigorous causal
and cross-validation of evidence. Feasibility concerns the inference that better fit complex environments (Befani 2012;
budget and capacity needed to meet expectations of rigour Guijt and Roche 2014).
and inclusiveness and to enhance learning (Chambers 2015).
Second, programme managers and even funders can feel
PIALA was first piloted in Vietnam (IFAD and BMGF 2014) threatened by traditional types of evaluation that focus on
and then in Ghana (MOFA/GOG, IFAD and BMGF 2015). performance against pre-set targets, whereas in complex
Both pilots used these three standards for framing the environments there is less control over results (e.g. similar
evaluation: data collection and linking; synthesising findings; processes can lead to different outcomes). This hinders
and analysing and debating programme contributions. solid debate and learning about impact. By seeking to
Insights from the first pilot enabled the second to address understand programme contributions to impact, alongside
some of the challenges encountered. This paper describes key many other influences, and by taking a broader systemic
trade-offs and lessons. First, we present PIALA as a response perspective, fear of failure can partially be sidestepped
to the main challenges of impact evaluation in complex (Eyben et al. 2015).
environments. Then, we discuss our insights about the
possible trade-offs. We conclude by presenting reflections on Third, it is important to analyse and understand
how to balance rigour, inclusiveness and feasibility. development impacts more systemically when seeking
transformational or systemic change that is more inclusive
and sustainable and grounded in rights and democracy
2 PIALA’s response to challenges in (Eyben 2008). The tendency in mainstream evaluation
impact evaluation practice is to slice programmes into measurable parts
Challenges in impact evaluation and then look at intervention and effect for each in
IFAD-funded government programmes are implemented isolation (Befani, Ramalingam and Stern 2015). In Ghana,
in rather complex ways and environments that challenge for instance, in some studies of specific programme
mainstream evaluation. Four challenges are common to mechanisms conducted before the PIALA study, such
many contexts. a reductionist perspective resulted in quite perilous
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 3 PRACTICE PAPER CDI
Table 1 PIALA methods and processes
Methods and processes Purposes

Focusing and framing the evaluation

1 Outlining of design options and budget implications (full scale–full ■ Enable commissioners to decide on scope and scale of evaluation
scope; full scale–limited scope; or limited scale–full scope)

2 Reconstruction and visualisation of programme Theory of Change ■ Identify causal claims and assumptions
(ToC) ■ Formulate evaluation questions
■ Create shared understanding among stakeholders of programme
theory and broader influences

Collecting and linking data

3 Multi-stage sampling with ‘open systems’ as principle sample unit ■ Enable systemic inquiry and comparative analysis of impact on
(e.g. value chain systems) livelihoods and household poverty

4 Selection of methods for data collection, and drafting of ‘how-to’ ■ Enable rigorous use of methods and facilitation of processes
guidance and templates for each method and for quality monitoring ■ Enable systematic data quality monitoring and reflective practice

5 Data collection on changes and causes in household food and ■ Collect and triangulate data on impacts
income through: ■ With intended beneficiaries visually reconstruct and discuss causal
■ Household survey flow of changes in livelihoods affecting household wealth and
wellbeing
■ Generic change analysis in gender-specific groups (social mapping,
timeline, wealth and wellbeing ranking, causal flow mapping)

6 Data collection on livelihood changes and causes through: ■ Collect and triangulate data on effects of livelihood changes on
household food and income
■ Generic change analysis (see above)
■ Visualise and discuss with intended beneficiaries causal flow of
■ Livelihood analysis in gender-specific groups (livelihood change changes and causes in different areas affecting their livelihood
matrix, causal flow mapping, SenseMaker)

7 Data collection on reach and effects of selected programme ■ Collect and triangulate data on effects of programme mechanisms
mechanisms through: on changes and causes in various areas affecting livelihoods

■ Livelihood analysis (see above) ■ With intended beneficiaries discuss and anonymously score reach,
benefits, outcomes of mechanisms
■ Constituent Feedback (CF) in mixed groups (questionnaire for
discussing and anonymous scoring)
■ Semi-structured interviews with service providers and officials
(CF-linked questionnaire)

8 Data linking and quality monitoring using a standard data ■ Enable instant data processing and cross-checking to identify gaps
collection tool and questionnaire for team reflections on quality of and weaknesses
methods, processes and evidence using a standard questionnaire ■ Ensure robust evidence (inclusive, sufficient, consistent, rigorous)

Synthesising evidence and analysing and debating programme contributions

9 Local and national participatory sensemaking using a workshop ■ Probe to fill remaining data gaps
model consisting of design principles and methods for enabling voice ■ Enable stakeholders to understand impact systemically
and facilitating cross-validation and contribution scoring
■ Engage stakeholders in valuing programme contributions and
10 Configurational analysis using standardised data collation and identifying priority investment areas
scoring tools

Source: Drawn from the Root and Tuber Improvement and Marketing Programme (RTIMP) impact evaluation report in the Ghana pilot
(cf. MOFA/GOG, IFAD and BMGF 2015).

CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi

PAGE 4 PRACTICE PAPER CDI
recommendations – e.g. scaling up production in the smallholders’ capacity to commercialise, and linking local
absence of markets. A systemic approach therefore must businesses to markets and industries. While the DBRP
inquire not just ‘what works’ but also ‘why’ and ‘how’ in focused on diversified short value chains, the RTIMP sought
order to understand likely sustainability (Burns 2014a). to develop longer commodity chains linked to national and
export markets and industries. Both programmes were at
Fourth, greater inclusiveness and sustainability of completion.
development outcomes and impacts requires stakeholders
to learn systemically and adapt responsibly in complex ■ The DBRP was implemented from 2008 to 2014 in
environments, thus understanding broader influences and two provinces (Cao Bằng and Bến Tre) with a budget of
individual and collective responsibilities (Bawden 2010). US$51m, including US$36m from IFAD. The evaluation
Mohan and Hickey (2004) plead for critical participation was conducted in 2013 at a cost of US$90,000. It
based on ‘rigorous, debated and contested’ evidence from covered five years of work in Bến Tre province only,
participatory research. Mainstream evaluation largely fails to where the programme was implemented in 50 of 164
meaningfully engage stakeholders (particularly beneficiaries), communes in eight of nine districts.
assuming that asking them directly about attribution is
highly susceptible to bias (Copestake 2013). Mixing different ■ The RTIMP was implemented from 2007 to 2015 in
methods, including participatory methods, to investigate 106 of Ghana’s 216 districts across all ten provinces,
contribution (broader than attribution) is suggested as a with a total budget of US$24m, of which US$19m was
better way forward (Bamberger 2012; Stern et al. 2012). from IFAD. The evaluation took place after programme
completion in 2015 and was conducted countrywide at
PIALA design elements a cost of US$233,000. It covered the period from 2010,
To address these challenges, PIALA is constructed after the mid-term review.
around five key design elements (presented in Figure 1):
a systemic theory of change (ToC); multi-stage sampling
centred on ‘open systems’; participatory mixed-methods; 3 Reflections about trade-offs in practice4
configurational analysis; and participatory sensemaking. Focusing and framing the evaluation
Table 1 shows how these design elements were translated In the first phase of an impact evaluation, clarity and shared
into a practical sequence of methods. understanding must be created among key stakeholders
about what is to be evaluated and for what purposes.
The ToC approach is most critical, as it provides the This involves considering different design options,
structure for the entire evaluation. This involves visualising a identifying causal claims and assumptions, deciding on
programme’s ToC showing the systemic links and feedback focus and questions, and specifying criteria and standards
loops between different programme components and for evaluation (BetterEvaluation 2014). Using a systemic
mechanisms and other influences, and their collective ToC approach makes it possible to do this rigorously and
outcomes and impacts. Reconstruction and visualisation collaboratively with stakeholders, yet requires sufficient
happens through iterative discussions with stakeholders. time and budget.
Data collection methods are chosen related to the ToC
to inquire about cascading causes and effects (expected Ensuring ownership of the ToC (rigour and inclusiveness versus
and unexpected, positive and negative), from household feasibility)
impacts down to programme mechanisms. This process Inclusiveness is critical to ensure that the ToC is rigorously
permits rigorous and systematic assessment of multiple used for framing and focusing the evaluation, creating
interacting causal claims together with stakeholders. In ownership, and facilitating inclusiveness in the analysis
sensemaking workshops, participants construct a causal stage too. Yet this also makes feasibility more elusive.
flow diagram with the evidence that mirrors – and thus
validates or refutes – the ToC. This enables stakeholders In Vietnam, due to budget and time limitations, we
to collectively value programme contributions along two made three choices that compromised inclusiveness and
scales from ‘strong contribution’ to ‘contribution overrun by subsequent rigorous use of the ToC. First, we organised a
other influences’, and from ‘positive’ to ‘negative’ impact brief workshop with the programme steering committee
(IFAD and BMGF 2015; Van Hemelrijck 2013). and managers only to discuss the programme logic and
expectations for evaluation prior to reconstructing the ToC.
About the pilot cases Other stakeholders were not involved. Second, we did
PIALA was piloted in the Developing Business with the not allow sufficient time for the researchers to engage in
Rural Poor (DBRP) Programme in Vietnam and the Root desk review and national stakeholder interviews to inform
and Tuber Improvement and Marketing Programme the ToC process. Instead, we prioritised their involvement
(RTIMP) in Ghana. Both programmes focused on improving in refining field methodology. Third, we assumed that
livelihoods and increasing incomes as part of sustainable the broader evaluation questions outlined in the PIALA
and equitable poverty reduction through enhancing strategy paper (IFAD and BMGF 2013b) would be sufficient
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 5 PRACTICE PAPER CDI
to guide fieldwork and to focus analysis. However, the In Vietnam, this trade-off was not yet well understood.
evaluation scope remained too wide, and researchers and We believed that participatory research in a subsample
stakeholders showed limited ownership of the ToC, making of the already small-scale sample of villages where a
it difficult for them to relate the evidence to the causal representative household survey was conducted would be
narrative. The resulting lack of shared understanding of sufficient to conduct a full scope inquiry of programme
evaluation frame and focus hindered arriving at greater contribution to impact for the entire province. Rigorous
precision of evaluation findings and agreement on strategic causal inference was hindered by our limited confidence in
directions for the next programme. linking not case-specific household-level findings to case-
specific participatory research findings, and our inability to
Armed with these insights, we required the researchers generalise participatory research findings regarding the
in Ghana to conduct the desk review and lead on the ToC effects of programme mechanisms on livelihoods.
reconstruction and visualisation. This helped them thoroughly
understand the programme’s impact and contribution claims. Collecting and linking data
A design workshop convened stakeholders to build shared In this phase, decisions are made about how to obtain and
understanding of the ToC and agree on the mechanisms, collate data. This includes determining the sample frame
assumptions and questions the evaluation should focus and selecting the mix of methods that must ensure data
on. This more robust and collaborative process laid the sufficiency, data linking and quality monitoring while also
foundation for the entire evaluation. enabling meaningful participation (BetterEvaluation 2014).
The pilots experienced multiple trade-offs at this stage.
Value for money of design options (rigour and inclusiveness Those discussed here are: (1) sampling of open systems
versus feasibility) versus well-defined units; (2) prioritising depth or breadth
Purpose and budget strongly influence design options. The of inquiry; and (3) independent field mobilisation versus
more rigorous an evaluation design, the less feasible it is involving programme staff.
for low-capacity, low-resource situations.
Sampling of open systems versus well-defined units (rigour
In Ghana, prior to signing contracts, commissioners were versus feasibility)
offered three design options explaining what value for A systemic perspective for analysing contributions to
money they could expect from each: ‘full scope – full impact requires identifying the system that forms the
scale’, ‘limited scope – full scale’, or ‘full scope – limited main unit for the sample frame. In complex development
scale’. Scale refers to the size of the principal sample of contexts, systems have open boundaries, meaning they
‘systems’ (in the PIALA pilots: value chain systems) from interact with their environment (Burns 2014b; Humphrey
2014). Open systems are difficult to discern and sample.
which households for survey and intended beneficiaries
Time and budget constraints – particularly if there is
for participatory research are subsampled. Scope refers
no shared understanding of the system – may lead to
to coverage of programme components and mechanisms
compromising on rigour and finding proxy sample units.
reflected in the ToC. In Ghana, commissioners chose the
first and most expensive option for reporting and learning In Vietnam, two factors made it ostensibly easy to
purposes (MOFA/GOG, IFAD and BMGF 2015). When determine the sampling frame. First, the programme
operating on a shoestring budget, either scope or scale focused on developing short value chains close to the
may need to be downsized. farmers and involving just a few local actors. Hence we
assumed that the villages formed an adequate proxy as
A ‘full scope – limited scale’ design emphasises learning main sample unit. Second, demographic data from which
about the programme’s contribution to impact in select to sample households were readily available from provincial
cases under specific conditions. Fieldwork and analysis are government agencies. However, because we had not clearly
less resource-intensive, but findings are not generalisable identified the value chains and subsampled the households
(unless the programme itself is case-based). In a ‘limited from these, it was difficult to probe for systemic interactions
scope – full scale’ design, the purpose is to learn about and link data on changes in business environment and local
the effects of selective programme mechanisms on the capacity of service providers to changes in livelihoods within
entire population. A ToC approach is not mandatory, these value chains, thus compromising on analytical rigour.
saving time and money; but this runs the risk of arriving
at wrong conclusions and limiting stakeholders’ learning. Learning from this, much more work was put into the
For example, a cost-effectiveness study of Farmer Field sampling frame in Ghana. The RTIMP developed long
Forums in Ghana recommended scaling up because of commodity chains, consisting of many supply chains around
the high adoption of new technologies (MOFA 2014), the country. The evaluation focused on four commodities,
while the PIALA study showed that in Ghana’s downward so had four different populations of supply chains from
conjuncture, this contributed to market saturation, which which to sample. Supply chains are loose geographic
negatively affected livelihoods. areas where smallholders supply raw products to a small
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 6 PRACTICE PAPER CDI
enterprise or industrial off-taker manufacturing higher- In Vietnam, the researchers were comfortable with survey
value products for bigger markets. The chains interact statistics, but under pressure of time they struggled with
and overlap geographically and administratively. Hence linking and triangulating data from participatory research.
they were hard to discern and, in practice, often differed Their quantitative background and limited understanding
from what was sampled on paper. Ensuring that evidence of the ToC, with methods focusing more on generic
collected on these entities remained comparable entailed changes, meant that qualitative data capture and collation
much creativity and coordination. was not as structured and rigorous as in Ghana. The large
amount of qualitative data was overwhelming and left
Also, there were no population lists available for sampling them with little time during fieldwork for daily reflections
households. Census lists and national data could not be on data quality and interim findings. Methods and tools
matched with the sampled supply chain areas. Budget for data linking and quality monitoring were insufficient to
limits did not permit extra fieldwork to construct the adequately guide them. We also underestimated post-
lists. Instead, we systematically sampled every fifth or collection tasks. Consequently, data collation tables and
tenth household in a straight or zigzag line from a central village sensemaking workshop reports were submitted
point of the main community in each supply chain area. long after fieldwork ended, making verification and data
Separately, focus group participants were sampled from gap-filling impossible – in turn challenging aggregated
beneficiary lists (where available) or from lists created by analysis and causal inference.
applying a snowballing technique.
In Ghana, the evaluation did not sacrifice depth or breadth;
Independent field mobilisation versus programme staff all questions were answered adequately. The researchers
engagement (rigour versus feasibility) were selected based on their experience of mixed-
Field mobilisation is best undertaken independently from methods and participatory research. With methods tightly
the programme to avoid positive bias. If this is impossible, focused on causal links in the ToC, their solid grasp of
strong facilitation skills are needed to minimise undue the ToC helped them facilitate the processes and collate
influence by programme staff. and triangulate data more rigorously. Data collation and
quality monitoring was undertaken daily and systematically.
In Vietnam, the evaluation took place in one province Data collation was structured according to the causal
and communities were easily accessible. Researchers had claims and links in the ToC. Quality monitoring focused
manageable travel distances. Focus group discussions were on assessing the inclusiveness of participatory processes
easy to organise within the villages, so participants did not and the robustness of emerging evidence for each of the
have to travel. Local transportation and mobilisation was causal links. Teams were able to identify data gaps and
organised by local officials and programme staff, which weaknesses in good time, and prepare well for the district
made it more difficult to maintain independence and avoid sensemaking workshops.
interference. On the other hand, staff and officials were
more engaged as the evaluation unfolded. Yet the fatigue of six weeks’ fieldwork undoubtedly affected
their ability to produce persistently high quality. No real
In Ghana, the researchers took responsibility for answer exists to this: more teams in parallel would have
transportation and mobilisation to warrant greater led to greater variation, while extending fieldwork to allow
independence. Staff and officials were only engaged in more breaks would have affected researchers’ levels of
interviews and workshops. However, budget and schedule commitment. The trade-off between breadth and depth
were challenged by the scale of the evaluation (national), is clearly less prominent, however, when working with
very poor infrastructure, remoteness of communities, and highly competent and motivated research team leaders
spread of communities in the supply chain areas. Researchers and a research coordinator who takes pride in high-quality
and participants had to travel far on poor roads to conduct research. In contexts where local research capacity is
the focus group discussions. Finding central locations suitable weaker, feasibility of a ‘full scope – full scale’ design becomes
for these meetings that were known and trusted by all questionable. Options include more investment in training,
participants from all communities proved cumbersome. coaching and supervision and/or less ambitious designs.

Depth versus breadth of inquiry (inclusiveness versus rigour Synthesising evidence, and analysing and debating
and feasibility) contribution claims
PIALA’s mixed-methods approach pursues depth, through In this phase, data are ‘zipped up’ again along the ToC
focused participatory inquiry of ‘open systems’, and breadth, to show what evidence upholds or refutes the assumed
through representative household surveys. To enable data contribution claims. This leads to answering causal questions
linking, households are sampled within the sample of these about what has produced which observed outcomes and
‘systems’. However, there are trade-offs when using this impacts, for whom and why (BetterEvaluation 2014).
approach at scale, as participatory research becomes onerous Participatory sensemaking and configurational analysis
and limited resources can further compromise rigour. (cf. Table 1) form the backbone of this phase.
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 7 PRACTICE PAPER CDI
Figure 2 Part of the RTIMP configurational analysis
Contributions of
Contribution Claim of Contribution Claim of Contribution Claim of
RTIMP Components
RTIMP Component 3 RTIMP Component 2 RTIMP Component 1
1, 2 and 3

↓ ↓ ↓ ↓
Improved
Enhanced Processing (O3) Enhanced Production (O2) Enhanced Market-Linking (O3)
Livelihoods (I2)

MEF GPC FFF M1c+M1b+ DSF

(M3c)+C1a (M3b)+C3c Evidence M2a+M2b+ C2a+C2b Evidence Evidence O1+O2+ Evidence
DSF FFF GPC MEF → O2
O3+O2 C1a+(M1)
+M3b → C3b Strength (M2c) Strength +O1 → C1b → O1 Strength O3 → I2 Strength
→ C3c → O3 → C2a

Tano North
1 1 1 1 3 6 5 5 5 5 4 4 5 5 5
(Apesika) (CZ)

Techiman (CZ) 1 1 1 1 4 5 5 5 5 5 4 4 5 5 5

Gomoa East (SZ) 1 1 1 0 2 5 3 5 5 5 4 4 5 5 6

Assin South (SZ) 1 1 1 1 3 4 4 6 5 4 3 3 4 4 4

Birim Central
1 1 1 1 3 3 4 5 5 4 3 4 4 4 5
(CZ)

Nkwanta South
1 1 1 0 3 4 5 5 4 5 3 3 5 4 5
(NZ)

Upper West
1 1 1 1 2 4 4 5 5 4 3 3 5 4 5
Akim (CZ)

Ashanti
1 1 1 1 3 4 5 5 5 5 3 3 5 4 5
Mampong (CZ)

West Gonja
1 1 1 0 3 4 5 5 4 5 3 3 5 4 5
(Damongo) (NZ)

Abura Asebu
1 1 1 1 3 3 5 5 5 6 3 3 5 4 4
Kwamankese (SZ)

Nanumba North
1 1 N/A N/A 5 5 5 3 3 5 4 5
(NZ)

East Gonja (NZ) 1 1 N/A N/A 4 3 5 3 3 5 4 5

Central Gonja
1 1 N/A 2 3 5 5 4 5 2 2 5 4 5
(NZ)

NZ=Northern Zone
CZ=Central Zone Gari HQCF Yam PCF Other
SZ=Southern Zone

Source: MOFA/GOG, IFAD and BMGF (2015).

Classical counterfactual or configurational counterfactual difficult to align findings with the ToC. These concerns and
(rigour versus feasibility) limitations made us hesitant to generalise certain findings
Mainstream impact evaluation assumes that comparative data regarding programme contribution. Rigour and feasibility
analysis from treated and non-treated sites is both accessible appeared as an ‘either/or’ type of trade-off.
(thus feasible) and necessary (thus rigorous) to reach
generalisable conclusions about impact on rural household In Ghana, this trade-off was solved by choosing a
poverty. However, where this is not the case (which is quite different causal approach combining ‘configurational’ with
common), other forms of rigorous analysis are needed. ‘generative’ perspectives (Punton and Welle 2015; Stern
et al. 2012). We used systemic heterogeneity as the basis
In the Vietnam pilot, concerns about heterogeneity in for identifying and analysing programme contributions.
programme treatment and sample limitations made it Instead of a classic counterfactual inquiry of household-
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 8 PRACTICE PAPER CDI
level impact, we employed a counterfactual approach that A second option involves investing in research partnerships
looked at the effects of different patterns of treatment with in-country research firms, thus strengthening local
that combined presence/non-presence, functional competencies. Integrating PIALA in programme design
conditions and differentiated effects of programme as part of an impact-oriented monitoring and evaluation
mechanisms on livelihoods and households. (M&E) process would increase cost-effectiveness, lay
the foundation for better knowledge for policy and
We developed a configurational analysis method to compare decision-making, and create more democratic space for
the evidence collected for each causal link in each of the stakeholders to influence decisions (Guijt 2014; Peersman
three contribution claims in the ToC across our sample of et al. forthcoming).
supply chains (first column in Figure 2). For each supply chain,
the formal presence of programme mechanisms such as We shifted towards the second option in Ghana,
Farmer Field Forum (FFF) or Micro-Enterprise Fund (MEF) ensuring that those involved in the tendering process fully
was inputted as a binary code (next four columns in Figure 2). understood the requirements and including far more days
We scored the causal link between the contribution claims for in-country supportive supervision of the pilot. The
and the impact claim (columns on ‘livelihood improvements’), research coordinator was strongly involved in the entire
and the evidence for this link, on ‘strength’ and ‘consistency’. evaluation process, leading to stronger ownership and
responsibility than in the case of Vietnam. IFAD’s country
Similarly, we scored the evidence for each causal link programme manager also actively engaged in design and
in which a mechanism operated, and the reach and sensemaking. The Ghana initiative was thus experienced
performance of the mechanism in each contribution claim as more of a joint learning journey – a partnership rather
(columns on ‘enhanced processing’, ‘enhanced production’ than a technical consultancy.
and ‘enhanced market-linking’). The scoring was done
based on detailed explanatory evidence collected for Degree of participation in sensemaking (inclusiveness versus
each of the claims independently (with different sets of rigour and feasibility)
methods and different groups). The analysis then looked Rigorous facilitation of participatory sensemaking – i.e.
at similarities and differences of various configurations of being responsive to local conditions and dynamics, while
clusters of scores across the supply chains supported by the consistently employing the same set of models and tools
evidence (MOFA/GOG, IFAD and BMGF 2015). in every locality – is essential for enhancing credibility and
confidence in evaluation findings (i.e. rigour) as well as
Not all contexts will allow for such a thorough and detailed generating solid debate and systemic learning among key
configurational analysis, as it requires high-quality data and stakeholders (i.e. inclusiveness). Again, this makes feasibility
analytical capacity. While there will always be a tension more elusive.
between rigour and feasibility, the approach can be adapted
to bring rigorous analysis within reach, in contexts where Engaging beneficiaries, service providers and decision-
classical counterfactual approaches cannot be applied. makers in collective sensemaking of emerging evidence
before turning to final analysis and reporting has both
Involving international experts or investing in local capacity instrumental and empowering value (MOFA/GOG, IFAD
(rigour versus feasibility) and BMGF 2015). Doing this in all researched localities
Undertaking a rigorous aggregated multi-causal analysis and at programme level helps to improve and strengthen
demands high-level analytical skills. In contexts where the evidence, overcome bias, and create ownership
local research institutions do not yet have these skills, of evaluation findings among stakeholders. For this to
conducting impact evaluations of complex programmes succeed, it is crucial to design and facilitate the processes
such as those funded by IFAD becomes less feasible. carefully, in ways that enable all participants to critically
engage and express their views in the presence of power-
In both pilots, researchers were not experienced with holders, and adopt a systemic perspective in valuing
rigorous aggregated multi-causal analysis. Because of the programme contributions to impact (cf. Section 2). A
methodological innovation, the authors took responsibility participatory sensemaking workshop model was developed
for the final analytical product. Ideally, however, the national that was first piloted in Vietnam and further expanded and
research coordinator should undertake the aggregated analysis improved on in Ghana.
and final reporting as part of delivery of the evaluation.
Using this model, in Vietnam, we organised six village-
To ensure sufficient analytical and reporting capacity, level workshops with 180 participants and one provincial
we see two options. The first is to work – as we did in workshop with 100 participants, while in Ghana, there
Vietnam – with international impact evaluation specialists were 23 district workshops with 650 participants and
leading on final analysis and reporting. This option is not one national workshop with 100 participants. Participants
optimal for fostering in-country capacity and responsibility were purposively sampled from the research participants.
for conducting rigorous impact evaluations. Beneficiaries comprised more than 70 per cent of those
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 9 PRACTICE PAPER CDI
attending local workshops, and more than 30 per cent collection on poverty characteristics for designing the
at provincial/national workshops. The workshops were household survey. Another example from Vietnam
quite successful in both pilots. Participants gained a more involved the unavoidable presence of programme staff
complete picture of the development processes. There during fieldwork, which was necessary to make it feasible.
were lively debates about programme contribution to Undoubtedly, this must have had some influence on what
impact and priority areas for future investment. Critical villagers chose to share. Yet we were able to mitigate
to this success were the time and resources invested in excesses by rigorous facilitation and triangulation of
organising the workshops, and the capacity to rigorously different processes. These examples show the win-wins
design and facilitate them. When operating on a that can be achieved from putting serious thought into
shoestring, the number of workshops and participants may balancing rigour, inclusiveness and feasibility by carefully
need to be limited. But this undoubtedly has implications reflecting on potential losses in value for money if one were
for rigour and inclusiveness. to be prioritised over the other.

Third, anticipating possible trade-offs while searching

4 Making multiple standards work in for win-wins is worthwhile to help think of the critical
impact evaluation competencies needed to conduct the evaluation. Having
Rigour, inclusiveness and feasibility are crucial for rethinking learned much in Vietnam, we discussed in detail where
impact evaluations to meet the challenges of complexity and how we needed to fare considerably better in Ghana.
and sustainability (described in Section 2) and to enable Competencies to handle participatory processes and mixed
stakeholders to learn and adapt responsibly. But these three methods proved critical. In Vietnam, researchers struggled
standards are not easily compatible. Decisions are influenced with integrating the large amounts of qualitative and
by politics, capacities, and contextual factors; trade-offs seem quantitative data, and were also challenged by the more
inevitable. In this paper, we have merely given a flavour of the open-ended design. In Ghana, researchers were well
range of possible trade-offs. We encountered many others. versed in research involving participatory processes and
large amounts of qualitative and quantitative evidence, and
So, if impact evaluation needs to move beyond being driven were not restrained by the confirmative design to probe
by reductionist rigour, how does one deal with multiple for alternative explanations. Having a statistician on the
standards? We conclude with three recommendations. team gave sufficient confidence.

First, being clear about how to meet each of the standards To conclude, trade-offs are clearly not absolute, and it is
is crucial. Rigour is arguably the standard around which worthwhile exploring win-wins in any context to reduce
most was achieved in Ghana as we learned from what losses and enhance the evaluation’s value for money. One
had happened in Vietnam and became much clearer about does not have to forfeit inclusiveness completely if rigour
what it entailed. Rigour involved being thorough and is deemed a non-negotiable. Nor does rigour have to be
careful methodologically and analytically, as well as being compromised totally when budgets and capacities are
thoughtful about whose voices informed the findings. restricted. Each of the standards can be conceived of as
Inclusiveness was considered instrumental to arrive at a gradient. For example, inclusiveness can be approached
greater rigour, while also being essential for learning and from a minimalist perspective – ensuring enough to cross-
for influencing future policy and practice. In both PIALA validate key findings but perhaps cutting short on the
pilots, considerable space was created for stakeholders to aspiration of more collective learning and empowering
cross-validate and debate emerging evidence. Rigour also forms of inclusiveness. Similarly, operating constraints may
involved sampling thoroughness and appropriate method mean that it is not feasible to pursue more detailed surveys
selection, which affects the ability to conduct a ‘full scale in larger samples to build more airtight statistical rigour,
– full scope’ evaluation in a way that permits rigorous yet still permit building sufficient confidence in findings to
configurational analysis. This was achieved in Ghana, we stand up to scrutiny.
argue, despite the classic counterfactual (using control
groups) not being feasible. Being inclusive and rigorous does, however, make PIALA
more demanding of time, capacity and budget. Evaluations
Second, being clear with commissioners upfront about of smaller programmes with smaller budgets and limited
which standards are essential to serve which purposes scale will not need large samples and are therefore likely
helps decisions to reduce problematic trade-offs and to be cheaper, but may still produce little value without
accept those that remain inevitable. We sought to pursue sufficient capacity. Arguably, impact evaluation is always
all three standards equally as part of the piloting process, difficult, so there are no real shortcuts. But win-wins are
but made clear choices. For example, the importance more likely where quality standards are clearly defined
of inclusiveness for both analytical quality and uptake of and where sufficient guidance is provided for every step
findings in Ghana led us to invest more in participatory in the evaluation process – and above all, where there is a
sensemaking, instead of additional participatory data commitment to building learning and research capacity.
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 10 PRACTICE PAPER CDI
Notes participatory methods evaluations. The premise is that bias cannot
1 IFAD is a specialist UN agency providing loans and support to be avoided by a single method or procedure but can be mitigated
governments for smallholder agricultural development. through triangulation of different methods and perspectives
2 The piloting of PIALA was made possible with financing from (Camfield, Duvendack and Palmer-Jones 2014).
IFAD and BMGF. However, this paper does not represent the 4 The paper builds on methodological reflections conducted with
views of the funders; it presents views of the individual authors. stakeholders and researchers that were documented by one of
3 In dominant evaluation practice, rigour connotes the controlled the authors (Adinda Van Hemelrijck) as part of the IFAD and
avoidance of bias through statistical procedure (Befani, Barnett BMGF innovation project and her doctoral study (cf. IFAD and
and Stern 2014). This narrow definition is inadequate for mixed BMGF 2013a, 2015; Van Hemelrijck 2014).

References Guijt, I. and Roche, C. (2014) ‘Does Impact Evaluation in

Bamberger, M. (2012) Introduction to Mixed Methods in Impact Development Matter? Well, It Depends What It’s For!’,
Evaluation, Guidance Note 3, Washington DC: InterAction European Journal of Development Research 26.1: 46–54
Bawden, R. (2010) ‘Messy Issues, Worldviews and Systemic Humphrey, J. (2014) Market Systems Approaches: A Literature
Competencies’, in C. Blackmore (ed.), Social Learning Review, Beam Exchange, Brighton: IDS
Systems and Communities of Practice, Springer, 89–101 IFAD and BMGF (2015) Improved Learning Initiative for the
Befani, B. (2012) Models of Causality and Causal Inference, Design of a Participatory Impact Assessment and Learning
review prepared as part of the DFID study, Broadening Approach (PIALA): Methodological Reflections following the
the Range of Designs and Methods for Impact Evaluation, Second PIALA Pilot in Ghana, Rome: IFAD and BMGF
London: Department for International Development IFAD and BMGF (2014) Impact Assessment of the ‘Doing
Befani, B.; Barnett, C. and Stern, E. (2014) ‘Introduction Business with the Rural Poor’ Project in Bến Tre, Vietnam
– Rethinking Impact Evaluation for Development’, (2008–2013). Report of the Pilot Application of a
IDS Bulletin 45.6: 1–5 Participatory Impact Assessment and Learning Approach,
Befani, B.; Ramalingam, B. and Stern, E. (2015) Rome: IFAD and BMGF
‘Introduction – Towards Systemic Approaches to IFAD and BMGF (2013a) Improved Learning Initiative for the
Evaluation and Impact’, IDS Bulletin 46.1: 1–6 Design of a Participatory Impact Assessment and Learning
BetterEvaluation (2014) BetterEvaluation Rainbow Framework Approach (PIALA): Insights and Lessons Learned from the
and Planning Tool, http://betterevaluation.org/resources/ Reflections on the PIALA Piloting in Vietnam, Rome: IFAD
download_the_Rainbow_Framework (accessed and BMGF
11 December 2015) IFAD and BMGF (2013b) ‘PIALA Research Strategy. Improved
Burns, D. (2014a) Assessing Impact in Dynamic and Complex Learning Initiative’, Internal Document, International Fund
Environments: Systemic Action Research and Participatory for Agricultural Development, Rome: IFAD
Systemic Inquiry, CDI Practice Paper 8, Brighton: MOFA (2014) Impact Assessment of FFF on Farmer
IDS, www.ids.ac.uk/publication/assessing-impact-in- Beneficiaries of RTIMP. Final Report, Accra: Government
dynamic-and-complex-environments-systemic-action- of Ghana, Ministry of Food and Agriculture
research-and-participatory-systemic-inquiry (accessed MOFA/GOG, IFAD and BMGF (2015) Final Report on
11 December 2015) the Participatory Impact Evaluation of the Root & Tuber
Burns, D. (2014b) ‘Systemic Action Research: Changing Improvement & Marketing Program (RTIMP) conducted by
System Dynamics to Support Sustainable Change’, PDA with support from the MOFA/GoG. Pilot Application of
Action Research 12.1: 3–18 a Participatory Impact Assessment and Learning Approach
Camfield, L.; Duvendack, M. and Palmer-Jones, R. (2014) (PIALA) Developed with Support from IFAD and the BMGF
‘Things You Wanted to Know about Bias in Evaluations Mohan, G. and Hickey, S. (2004) ‘Relocating Participation
but Never Dared to Think’, IDS Bulletin 45.6: 49–64 within a Radical Politics of Development: Critical
Chambers, R. (2015) ‘Inclusive Rigour for Complexity’, Modernism and Citizenship’, in S. Hickey and G. Mohan
Journal of Development Effectiveness 7.3: 327–35 (eds), Participation: from Tyranny to Transformation?
Copestake, J. (2013) Credible Impact Evaluation in Complex Exploring New Approaches to Participation in Development,
Contexts: Confirmatory and Exploratory Approaches, Bath: London: Zed Books, 59–74
Centre for Development Studies, University of Bath Peersman, G.; Rogers, P.; Guijt, I.; Hearn, S.; Pasanen, T.
Eyben, R. (2008) Power, Mutual Accountability and and Buffardi, A. et al. (forthcoming) Guidance on When
Responsibility in the Practice of International Aid: A and How to Develop an Impact-Oriented Monitoring and
Relational Approach, Working Paper 305, Brighton: IDS Evaluation System, London: Overseas Development
Eyben, R.; Guijt, I.; Roche, C. and Shutt, C. (2015) Institute
The Politics of Evidence and Results in International Punton, M. and Welle, K. (2015) Straws-in-the-wind, Hoops
Development: Playing the Game to Change the Rules?, and Smoking Guns: What can Process Tracing Offer to
Rugby: Practical Action Publishing Impact Evaluation?, CDI Practice Paper 10, Brighton: IDS,
Guijt, I. (2014) Participatory Approaches, Methodological www.ids.ac.uk/publication/straws-in-the-wind-hoops-
Briefs, Impact Evaluation 5, Florence: UNICEF Office of and-smoking-guns-what-can-process-tracing-offer-to-
Research – Innocenti impact-evaluation (accessed 11 December 2015)
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi
PAGE 11 PRACTICE PAPER CDI
Rogers, P. (2009) ‘Matching Impact Evaluation Design Settings’, in J. Servaes (ed.), Sustainability, Participation
to the Nature of the Intervention and the Purpose of and Culture in Communication: Theory and Praxis, Bristol:
the Evaluation’, Journal of Development Effectiveness 1.3: Intellect
217–26 White, H. (2009) Some Reflections on Current Debates in
Stern, E.; Stame, N.; Mayne, J.; Forss, K.; Davies, R. and Impact Evaluation, Working Paper 2009-1, International
Befani, B. (2012) Broadening the Range of Designs and Initiative for Impact Evaluation (3ie)
Methods for Impact Evaluations, Working Paper 38, Wild, L.; Booth, D.; Cummings, C.; Foresti, M. and Wales, J.
London: Department for International Development (2015) Adapting Development: Improving Services to the Poor,
Van Hemelrijck, A. (2014) ‘Understanding “Rigor”: London: Overseas Development Institute,
Challenges in Impact Evaluation of Transformational www.odi.org/sites/odi.org.uk/files/odi-assets/publications-
Development’, PhD Research Outline Paper (Revised), opinion-files/9437.pdf (accessed 11 December 2015)
Brighton: IDS Woolcock, M. (2013) ‘Using Case Studies to Explore
Van Hemelrijck, A. (2013) ‘Powerful Beyond Measure? the External Validity of “Complex” Development
Measuring Complex Systemic Change in Collaborative Interventions’, Evaluation 19.3: 229–48

“ … trade-offs clearly are not absolute and win-wins [are] worthwhile exploring… to reduce losses

and enhance the evaluation’s value-for-money. One does not have to forfeit inclusiveness completely if rigour

is deemed a non-negotiable. Nor does rigour have to be compromised totally when budgets and capacities are

”
restricted. Each of the standards can be conceived of as a gradient.

Centre for Development Impact (CDI) This CDI Practice Paper was written by Adinda Van Hemelrijck
The Centre is a collaboration between IDS (www.ids.ac.uk), and Irene Guijt.
Itad (www.itad.com) and the University of East Anglia The opinions expressed are those of the author and do not
(www.uea.ac.uk). necessarily reflect the views of IDS or any of the institutions
The Centre aims to contribute to innovation and excellence involved. Readers are encouraged to quote and reproduce
in the areas of impact assessment, evaluation and learning material from issues of CDI Practice Papers in their own
in development. The Centre’s work is presently focused on: publication. In return, IDS requests due acknowledgement
and quotes to be referenced as above.
(1) Exploring a broader range of evaluation designs and
methods, and approaches to causal inference. © Institute of Development Studies, 2016
(2) Designing appropriate ways to assess the impact of ISSN: 2053-0536
complex interventions in challenging contexts. AG Level 2 Output ID: 323
(3) Better understanding the political dynamics and other
factors in the evaluation process, including the use of
evaluation evidence.

Institute of Development Studies, Brighton BN1 9RE, UK

T +44 (0) 1273 915637 F +44 (0) 1273 621202 E ids@ids.ac.uk W www.ids.ac.uk
CDI PRACTICE PAPER 14 February 2016 www.ids.ac.uk/cdi