Training Guide: Evaluation Methods (Ver 1.

0)
An overview in the context of the UNICEF Yemen work
This guide has been prepared in light of my own experiences and research
Hussain Tawawalla, June 7, 2016
Definition ................................................................................................................................................ 1
Quality of evaluation............................................................................................................................... 1
Evaluation types ...................................................................................................................................... 2
Summative evaluations ....................................................................................................................... 2
Formative (AKA diagnostics or developmental) Evaluations .............................................................. 2
Important concepts and components of evaluations ............................................................................. 3
Theory of Change ................................................................................................................................ 3
The Evaluation Framework (matrix) ................................................................................................... 3
DAC evaluation principles ................................................................................................................... 3
Evaluation of supply chains and SCM ................................................................................................. 3
Disaster logistics.............................................................................................................................. 4
Key definitions to know and understand ................................................................................................ 4

Definition

An evaluation is an assessment of the structure, process, and outcome, of a given intervention and
heavily depends on (a) the stated focus of the appraisal and (b) the applicable standards.
The evaluation should yield an informed opinion on the value, worth, character, and effectiveness of
the intervention; it should also result in an accurate measurement of progress.
The evaluation approach must be context-specific and should take into account the purposes of the
study.
N1 Definition 1.7 Evaluation is therefore about - Are we doing the right
thing? It examples the rationale, the justification of the undertaking, makes a
reality check and looks at the satisfaction of intended beneficiaries.
Evaluation is also about - Are we doing it right? It assesses the effectiveness of
achieving expected results. It examines the efficiency of the use of inputs to
yield results. Finally, evaluation asks - Are there better ways of achieving the
results? Evaluation looks at alternative ways, good practices and lessons
learned. Norms for Evaluation in the UN System

Quality of evaluation

Ideally, at the end of the assessment, the evaluand should feel that the evaluator has
comprehensively and succinctly covered all aspects of the project or programme. Therefore, the
evaluation process should be as consultative and inclusive as possible, without compromising the
independence of the evaluator or the quality of the assessment. The evaluators must be holistic in
their approach.
Page 1 of 6

Training Guide: Evaluation Methods (Ver 1.0)

N8 Quality of Evaluation
8.1 Each evaluation should employ design, planning and implementation
processes that are inherently quality oriented, covering appropriate
methodologies for data-collection, analysis and interpretation.
8.2 Evaluation reports must present in a complete and balanced way the
evidence, findings, conclusions and recommendations. They must be brief and
to the point and easy to understand. They must explain the methodology
followed, highlight the methodological limitations of the evaluation, key
concerns and evidenced-based findings, dissident views and consequent
conclusions, recommendations and lessons. They must have an executive
summary that encapsulates the essence of the information contained in the
report, and facilitate dissemination and distillation of lessons. Norms for
Evaluation in the UN System

Evaluation types

Summative evaluations are usually conducted at the end of an intervention and should
produce objectively verifiable information on the project's / programme's effectiveness. Such an
evaluation involves the use of quantitative methods and looks at the realisation of committed
outcomes or the likelihood of those materialising. Any unintended consequences of the intervention
and lessons learnt are identified and highlighted. The SMART (Specific, Measurable, Attainable,
Relevant, and Timely) approach is used to evaluate the quality of the indicators and
accomplishments against those.
Outcome Assessments, Impact Evaluations and Cost-Benefit Analysis, are types of summative
assessments.

Formative (AKA diagnostics or developmental) Evaluations are inward-looking

appraisals of the processes and are conducted either (a) immediately after the design stage (i.e. the
lead-up stage) of the project or programme, or (b) at a strategic point during implementation. The
objective is to determine if the project's or programme's strategies are implemented as planned and
to arrive at timely corrections in the design and management of implementation (prescriptive). Such
evaluations usually use qualitative methods of inquiry and are interactive in nature. Quantitative
information may also be used to measure the degree of compliance with procedures. In such cases
Formative Evaluations become Formative-Normative.
The list of some of the key questions you should seek answers to before conducting the evaluation is
given below:
1. What is the context of the intervention?
2. Was a Theory of Change developed?
3. What management practices, process benchmarks and standards did the sponsor or
proponents of the project or programme stipulate?
4. What are the different types of processes defined by the sponsors or executing agency?
5. What is the Institutional Framework of the project or programme?
6. Who are the primary drivers or agents of change?
7. What is the prescribed or adopted communication methodology?
8. What type of information is collected or produced by the implementing partners and
agencies?
Page 2 of 6

Training Guide: Evaluation Methods (Ver 1.0)

Important concepts and components of evaluations
Theory of Change

"Theory of change is both a process and a product" ~ Isabel Vogel, April 2012, UK Department of
International Development.
In most cases the proponents of the intervention internally conduct 'formative-clarificative'
appraisals to gauge the theory of change that the design of the project or programme is based.
External evaluators must review the theory of change and the design assumptions and the resulting
implementation strategies. Studying the model against best practices and current opinions is
important. Some important questions to ask in this regard are:
1. Was the Theory of Change (ToC) formed through a consultative process?
2. What are the best practices incorporated into the ToC?
3. Does the ToC relate to, or, consider current opinions?
4. Is the ToC relevant to the context of the intervention?

The Evaluation Framework (matrix)

The evaluation framework will help us establish the focus of the evaluation in terms of questions to
address and the issues to be covered. It structures what must be analysed and what will not be
analysed. The scope defines the parameters of the evaluation that are presented in the Terms of
Reference.
Although we will develop this together, many of the key questions given under various sectionsof
this guide will structure and populate matrix.

DAC evaluation principles

The complete concepts and guidelines of Disaster Assessment and Coordination are presented in the
document United Nations Disaster Assessment and Coordination (UNDAC) Field Handbook, Office
for the Coordination of Humanitarian Affairs (OCHA) - 6th Edition (2013) 1.

Evaluation of supply chains and SCM

The efficiency and effectiveness of supply chains and SCM have a significant impact on beneficiaries.
It is impossible to evaluate the quality of the SCM in instances where performance data (etc.) is
missing or is unreliable. In such situations anecdotal evidence should be collected from the field.
Based on an analysis of such information we can estimate the impact on beneficiaries and indirectly
assess the strength or weakness of the supply chain processes. The idea is to plot the current
situation on the SCM continuum - inability to deliver to beneficiaries to all needs covered on time
and in full. Some of the questions we need to ask are listed hereunder:
1. What data and tools are used to manage the supply chain?
a. Which best practices are adopted?
b. Is the supplier-to-camp delivery process fully documented and informed?
c. How often are SCM transactions recorded and when?
2. Are process-oriented targets defined?
a. Are the processes flexible enough to allow managers to adjust and fine tune
methods and approaches to ground realities?
b. Is staff fully aware of these targets?
1

The document can be accessed at

http://reliefweb.int/sites/reliefweb.int/files/resources/UNDAC%20Handbook%202013_english_final.pdf

Page 3 of 6

Training Guide: Evaluation Methods (Ver 1.0)

3.
4.
5.
6.
7.
8.
9.
10.

c. Are the right people hired for the job?

What are the average lead times for the supply chain processes?
Is the supply chain (pipeline) visible enough?
How often are goods held up in transit?
What is the number of warehouse in the supply chain?
What is the efficiency of warehouse operations?
What are the procedures for transactions at the warehouse? Are these followed?
Are inventory snapshots taken / recorded and if so how often?
Is feedback solicited (and recorded) from all stakeholders, particularly from beneficiaries?

Disaster logistics
Please download and familiarise yourselves with Chapter G. Disaster Logistics of the UNDAC Field
Handbook.

Key definitions to know and understand

(w) Baseline
(w) Benchmarks
(w) Effective
(w) Efficient
(w) Evaluand
(w) Formative
(w) Normative
(w) Summative
Beneficiary
Effectiveness (1)

Effectiveness (2)
Efficiency (1)

Efficiency (2)
Humanitarian
assistance

Quantitative (descriptive statistics) and qualitative (inferential statistics)

information about the situation or condition before the intervention.
Performance data used for comparative purposes; data that is used to
measure the degree of compliance by implementing agents.
Producing a result that is wanted: having an intended effect (MW)
Achieving maximum productivity with minimum wasted effort or expense
The subject of the assessment, typically a project, programme or a system
rather than a person.
(adjective) "used to describe the time when someone or something is growing
or being formed: helping to develop something" (MW)
"based on what is considered to be the usual or correct way of doing
something" (MW)
"additive, cumulative" (MW)
"The individual, group, or organisation, whether targeted or not, that benefit,
directly or indirectly, from the implementation of a programme, project or
output" (UN)
The degree to which objectives are achieved and the extent to which
targeted problems are solved. In contrast to efficiency, effectiveness is
determined without reference to costs and, whereas efficiency means doing
the thing right, effectiveness means doing the right thing. (BD)
The extent to which a project or programme attains its objectives, expected
accomplishments and delivers planned outputs (UN)
The ratio of the useful work performed by a team working towards a goal to
the total level of effort and costs expended (adapted from the technical
definition)
The ability to do something or produce something without wasting materials,
time, or energy : the quality or degree of being efficient (technical definition)
(MW)
A measure of how well inputs (funds, expertise, time, etc.) are converted into
outputs. (UN)
Humanitarian assistance is generally accepted to mean the aid and action
designed to save lives, alleviate suffering and maintain and protect human
Page 4 of 6

Training Guide: Evaluation Methods (Ver 1.0)

Indicator

Institutional
Analysis

Intervention logic
Objective

Objectives, Specific
Outcomes
Outputs
Results
Situation analysis
Stakeholder

dignity during and in the aftermath of man-made crises and natural disasters,
as well as to prevent and strengthen preparedness for the occurrence of such
situations (Source: Good Humanitarian Donorship). What marks it out from
other forms of aid and foreign assistance is that it should be guided by the
principles of:
humanity saving human lives and alleviating suffering wherever it is found;
impartiality acting solely on the basis of need, without discrimination
between or within affected populations;
neutrality acting without favouring any side in an armed conflict or other
dispute where such action is carried out;
independence the autonomy of humanitarian objectives from the political,
economic, military or other objectives that any actor may hold with regard to
areas where humanitarian action is being implemented. (GHA)
A measure, preferably numerical, of a variable that provides a reasonably
simple and reliable basis for assessing achievement, change or performance. A
unit of information measured over time that can help show changes in a
specific condition. (UN)
Institutional Analysis (IA) focuses on the formal and informal institutions in the
public, private and not-for profit sectors. Institutional analysis assesses the
institutions and institutional linkages critical to project success; identifies
constraints to equitable access to benefits and/or services for the poor;
examines the institutional assets existing in poor communities. Institutional
analysis can also assist in the design of incentives which give existing groups in
poor communities a stake in project objectives and outcomes and finally, IA
lends insight into processes of social exclusion in cases where the 'rules of the
game' are different for different actors; and where participation and
entitlement also vary dramatically. (WB)
"Intervention logic shows a logical relationship between the project objectives
and planned activities, and is used to check whether planned activities will
ensure the achievement of planned objectives" (EU)
"Description of an overall desired achievement involving a process of change
and aimed at meeting certain needs of identified end-users within a given
period. A real objective meets the criteria of being impact oriented,
measurable, time-limited, specific and practical. The objective is set at the
next higher level than the expected accomplishments" (UN)
"The development outcome at the end of the project namely the expected
benefits to the target group(s)" (EU)
"The effects that follows from products or services delivered during the
project implementation; the consequence of project outputs" (EU)
"The direct/tangible results (good and services) that the project delivers, and
which are largely under project managements control" (EU)
"The direct/tangible results (good and services) that the project delivers, and
which are largely under project managements control (EU)
The process of understanding the status, condition, trends and key issues
affecting people, ecosystems and institutions in a given geographic context at
any level (local, national, regional, international). (IFAD)
Agencies, organizations, groups or individuals who have a direct or indirect
role and interest in the objectives and implementation of a programme or
project and its evaluation. In participatory evaluation, stakeholders assume an
increased role in the evaluation process as question-makers, evaluation
planners, data gatherers and problem solvers. (UN)
Page 5 of 6

Training Guide: Evaluation Methods (Ver 1.0)

Supply chain
Supply Chain
Management (SCM)
(1)
Supply Chain
Management (SCM)
(2)

Sustainability (1)
Sustainability (2)
Sustainability (3)
(d)
(d) BD
(d) MW
(w)
EU
GHA
IFAD
UN
WB

The sequence of processes involved in the production and distribution of a

commodity
A set of three or more entities (organizations or individuals) directly involved
in the upstream and downstream flows of products, services, finances, and/or
information from a source to a customer. (Mentzer, et al.)
Management of material and information flow in a supply chain to provide
the highest degree of customer satisfaction at the lowest possible cost. Supply
chain management requires the commitment of supply chain partners to work
closely to coordinate order generation, order taking, and order fulfilment.
They thereby create an extended enterprise spreading far beyond the
producer's location. (BD)
"Sustainability is the likelihood of a continuation of the stream of benefits
produced by the project after the period of external support has ended" (EU)
"A form of economic growth that is self-maintaining without exhausting
natural resources."
"The extent to which the impact of the programme or project will last after its
termination; the probability of continued long-term benefits" (UN)

Dictionary
Business Dictionary (www.businessdictionary.com/)
Merriam-Webster: Dictionary and Thesaurus (www.merriam-webster.com/)
Word
European Union
Global Humanitarian Assistance (Organisation)
International Fund for Agricultural Development
United Nations (including all subsidiary organisations, like UNDP, UNICEF, UNHCR, etc.)
World Bank

Page 6 of 6